Messages

I don't have native ipv6 WAN so I use tunnelbroker via gif device. Local address (in gif device) is set to WAN carp vip in both nodes. All works well in master node but once backup node is in charge, I can't get packets flowing through ipv6 gateway (based on gif to opt interface). I tried toggling down/up manually gif device, disabling and enabling of respective opt interface, saving once again ipv6 gateway and nothing helps. I have 100% packets loss...
ipv6 opt interface doesn't have carp vip since itself doesn't have any regular ip address.
Am I doing something wrong?

In the meantime: do not use DTLSv1.1. It's going away.

Thanks! Are you going to introduce DTLS 1.2/1.3?

Without more debug output this won't resolve easily so let's try this when it's not working:

# fetch -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
# curl -o data.pkg -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg

So, with DTLS set to "1" all works:

Code Select Expand

root@bkp-OPNsense:~ # pkg update
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
root@bkp-OPNsense:~ # fetch -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
resolving server address: pkg.opnsense.org:443
SSL options: 82004850
Peer verification enabled
Using OpenSSL default CA cert file and path
Verify hostname
TLSv1.3 connection established using TLS_AES_256_GCM_SHA384
Certificate subject: /CN=pkg.opnsense.org
Certificate issuer: /C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA
requesting https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
remote size / mtime: 322185 / 1784295958
data.pkg                                               314 kB 3342 kBps    00s
root@bkp-OPNsense:~ #  curl -o data.pkg -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
13:31:42.013785 [0-0] * [HTTPS-CONNECT] added
13:31:42.014044 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 0 socks
13:31:42.018466 [0-0] * [HTTPS-CONNECT] connect, init
13:31:42.018557 [0-0] * [HTTPS-CONNECT] 1st attempt uses h2 from wanted versions
13:31:42.018574 [0-0] * [SETUP] happy eyeballing to origin pkg.opnsense.org:443
13:31:42.018622 [0-0] *   Trying [2001:1af8:5300:a010:1::1]:443...
13:31:42.018826 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:31:42.018851 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
13:31:42.019480 [0-0] * Host pkg.opnsense.org:443 was resolved.
13:31:42.019526 [0-0] * IPv6: 2001:1af8:5300:a010:1::1
13:31:42.019549 [0-0] * IPv4: 89.149.222.99
13:31:42.019569 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:31:42.019586 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
13:31:42.221458 [0-0] *   Trying 89.149.222.99:443...
13:31:42.221591 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:31:42.221616 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 2 socks
13:31:42.245167 [0-0] * [SETUP] added SSL filter for origin
13:31:42.247120 [0-0] * ALPN: curl offers h2,http/1.1
13:31:42.247481 [0-0] } [5 bytes data]
13:31:42.247532 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
13:31:42.247546 [0-0] } [1562 bytes data]
13:31:42.247628 [0-0] * SSL Trust Anchors:
13:31:42.247649 [0-0] *   CApath: /etc/ssl/certs
13:31:42.247672 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:31:42.247693 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
13:31:42.274063 [0-0] { [5 bytes data]
13:31:42.274120 [0-0] * TLSv1.3 (IN), TLS handshake, Server hello (2):
13:31:42.274134 [0-0] { [122 bytes data]
13:31:42.274561 [0-0] * TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
13:31:42.274584 [0-0] { [1 bytes data]
13:31:42.274619 [0-0] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
13:31:42.274632 [0-0] { [25 bytes data]
13:31:42.274670 [0-0] * TLSv1.3 (IN), TLS handshake, Certificate (11):
13:31:42.274683 [0-0] { [3550 bytes data]
13:31:42.276081 [0-0] * TLSv1.3 (IN), TLS handshake, CERT verify (15):
13:31:42.276102 [0-0] { [264 bytes data]
13:31:42.276254 [0-0] * TLSv1.3 (IN), TLS handshake, Finished (20):
13:31:42.276283 [0-0] { [52 bytes data]
13:31:42.276354 [0-0] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
13:31:42.276372 [0-0] } [1 bytes data]
13:31:42.276425 [0-0] * TLSv1.3 (OUT), TLS handshake, Finished (20):
13:31:42.276438 [0-0] } [52 bytes data]
13:31:42.276537 [0-0] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
13:31:42.276561 [0-0] * ALPN: server accepted http/1.1
13:31:42.276578 [0-0] * Server certificate:
13:31:42.276598 [0-0] *   subject: CN=pkg.opnsense.org
13:31:42.276615 [0-0] *   start date: Jan 20 00:00:00 2026 GMT
13:31:42.276628 [0-0] *   expire date: Jan 20 23:59:59 2027 GMT
13:31:42.276648 [0-0] *   issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA
13:31:42.276673 [0-0] *   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha384WithRSAEncryption
13:31:42.276688 [0-0] *   Certificate level 1: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
13:31:42.276701 [0-0] *   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha384WithRSAEncryption
13:31:42.276722 [0-0] *   subjectAltName: "pkg.opnsense.org" matches cert's "pkg.opnsense.org"
13:31:42.276736 [0-0] * OpenSSL verify result: 0
13:31:42.276748 [0-0] * SSL certificate verified via OpenSSL.
13:31:42.276763 [0-0] * [SETUP] query ALPN
13:31:42.276776 [0-0] * [HTTPS-CONNECT] connect -> 0, done=1
13:31:42.276801 [0-0] * Established connection to pkg.opnsense.org (89.149.222.99 port 443) from 172.16.99.4 port 56453
13:31:42.276814 [0-0] * [HTTPS-CONNECT] removing connected setup filter
13:31:42.276826 [0-0] * [HTTPS-CONNECT] destroy
13:31:42.276838 [0-0] * [SETUP] removing connected setup filter
13:31:42.276850 [0-0] * [SETUP] destroy
  % Total    % Received % Xferd  Average Speed  Time    Time    Time   Current
                                 Dload  Upload  Total   Spent   Left   Speed
  0      0   0      0   0      0      0      0                              013:31:42.276976 [0-0] * using HTTP/1.x
13:31:42.277008 [0-0] } [5 bytes data]
13:31:42.277056 [0-0] > GET /FreeBSD:15:amd64/snapshots/latest/data.pkg HTTP/1.1
13:31:42.277056 [0-0] > Host: pkg.opnsense.org
13:31:42.277056 [0-0] > User-Agent: curl/8.21.0
13:31:42.277056 [0-0] > Accept: */*
13:31:42.277056 [0-0] >
13:31:42.277116 [0-0] * Request completely sent off
13:31:42.300220 [0-0] { [5 bytes data]
13:31:42.300296 [0-0] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
13:31:42.300310 [0-0] { [297 bytes data]
13:31:42.300416 [0-0] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
13:31:42.300433 [0-0] { [297 bytes data]
13:31:42.301188 [0-0] < HTTP/1.1 200 OK
13:31:42.301216 [0-0] < Date: Sat, 18 Jul 2026 11:36:47 GMT
13:31:42.301230 [0-0] < Server: Apache
13:31:42.301242 [0-0] < Last-Modified: Fri, 17 Jul 2026 13:45:58 GMT
13:31:42.301255 [0-0] < ETag: "4ea89-656cec6c03180"
13:31:42.301280 [0-0] < Accept-Ranges: bytes
13:31:42.301293 [0-0] < Content-Length: 322185
13:31:42.301306 [0-0] < Content-Type: application/vnd.apple.installer+xml
13:31:42.301320 [0-0] <
13:31:42.301333 [0-0] { [7948 bytes data]
100 314.6k 100 314.6k   0      0 825.4k      0                              0
13:31:42.394953 [0-0] * Connection #0 to host pkg.opnsense.org:443 left intact


And after having it changed to "1.1"

Code Select Expand

root@bkp-OPNsense:~ # pkg update
Updating OPNsense repository catalogue...
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
repository OPNsense has no meta file, using default settings
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
Unable to update repository OPNsense
Error updating repositories!
root@bkp-OPNsense:~ # fetch -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
resolving server address: pkg.opnsense.org:443
SSL context creation failed
1050062A474C0000:error:0A000180:SSL routines:SSL_CONF_cmd:bad value:/usr/src/crypto/openssl/ssl/ssl_conf.c:995:cmd=MinProtocol, value=DTLSv1.1
1050062A474C0000:error:0A0001A3:SSL routines:SSL_CTX_new_ex:error in system default config:/usr/src/crypto/openssl/ssl/ssl_lib.c:4274:
fetch: https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg: Authentication error
root@bkp-OPNsense:~ # curl -o data.pkg -vv https://pkg.opnsense.org/FreeBSD:15:amd64/snapshots/latest/data.pkg
13:32:54.264461 [0-0] * [HTTPS-CONNECT] added
13:32:54.264665 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 0 socks
13:32:54.265958 [0-0] * Host pkg.opnsense.org:443 was resolved.
13:32:54.265987 [0-0] * IPv6: 2001:1af8:5300:a010:1::1
13:32:54.266003 [0-0] * IPv4: 89.149.222.99
13:32:54.266015 [0-0] * [HTTPS-CONNECT] connect, init
13:32:54.266026 [0-0] * [HTTPS-CONNECT] 1st attempt uses h2 from wanted versions
13:32:54.266043 [0-0] * [SETUP] happy eyeballing to origin pkg.opnsense.org:443
13:32:54.266068 [0-0] *   Trying [2001:1af8:5300:a010:1::1]:443...
13:32:54.266248 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:32:54.266284 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
13:32:54.266305 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:32:54.266318 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 1 socks
13:32:54.474497 [0-0] *   Trying 89.149.222.99:443...
13:32:54.474649 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
13:32:54.474678 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 0, 2 socks
13:32:54.498280 [0-0] * [SETUP] added SSL filter for origin
13:32:54.500281 [0-0] * SSL: could not create a context: error:0A000180:SSL routines::bad value
13:32:54.500308 [0-0] * [HTTPS-CONNECT] connect, all attempts failed
13:32:54.500321 [0-0] * [HTTPS-CONNECT] connect -> 27, done=0
13:32:54.500346 [0-0] * closing connection #0
curl: (27) SSL: could not create a context: error:0A000180:SSL routines::bad value

I know DTLS applies to UDP but this settings affects openssl (upgraded) package as pkg also didn't work properly:

Code Select Expand

# pkg update
Updating OPNsense repository catalogue...
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
repository OPNsense has no meta file, using default settings
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
pkg: An error occurred while fetching package: No error
Unable to update repository OPNsense
Error updating repositories!
And once I set DTLS to v1 all came back to normal....

After upgrading to 26.7 I lost an access to webgui (https) and pkg stopped working correctly. I think I've found reason for this issue but not a root cause. Anyway, as long as in System - Trust - Settings - MinProtocol (DTLS) is set to "DTLSv1.1" webgui and pkg doesn't work... After changing it (had to switch webgui to http) to None or DTLSv1 all works again!

> fetch: /usr/local/opnsense/changelog/changelog.txz.sig appears to be truncated: 0/1332 bytes

Usually a sign of DNS timeouts.

This tip got me somewhere. I use local DNS (adguard with unbound as upstream) which is run separately from opnsense. On backup node Unbound doesn't work, initially I thought it was because of not available interfaces (ipv6 tunnel) unbound is binded to. But after deselecting them still doesn't work issuing error:

Code Select Expand

Unable to open pipe. This is likely because Unbound isn't running.
In cli:

Code Select Expand

root@OPNsense-bkp:~ # dig opnsense.org
;; communications error to 127.0.0.1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused
;; communications error to 127.0.0.1#53: connection refused
;; communications error to 172.16.1.4#53: timed out
;; communications error to 2001:xxxxxxxxx::4#53: timed out 
; <<>> DiG 9.20.22 <<>> opnsense.org
;; global options: +cmd
;; no servers could be reached
Despite of:

Code Select Expand

root@OPNsense-bkp:~ # ping 172.16.1.4
PING 172.16.1.4 (172.16.1.4): 56 data bytes
64 bytes from 172.16.1.4: icmp_seq=0 ttl=64 time=0.227 ms
64 bytes from 172.16.1.4: icmp_seq=1 ttl=64 time=0.105 ms
--- 172.16.1.4 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.105/0.166/0.227/0.061 ms
root@OPNsense-bkp:~ # nc -vzu 172.16.1.4 53
Connection to 172.16.1.4 53 port [udp/domain] succeeded!
After enabling

Do not use the local DNS service as a nameserver for this system

cli started working fine:

Code Select Expand

root@OPNsense-bkp:~ # dig opnsense.org 
; <<>> DiG 9.20.22 <<>> opnsense.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40326
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;opnsense.org.   IN A 
;; ANSWER SECTION: opnsense.org.  3 IN A 89.149.225.137 
;; Query time: 0 msec
;; SERVER: 172.16.1.4#53(172.16.1.4) (UDP)
;; WHEN: Thu Jun 11 09:57:59 CEST 2026
;; MSG SIZE  rcvd: 57
But update check still fails:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 26.1.9 (amd64) at Thu Jun 11 10:00:41 CEST 2026
Fetching changelog information, please wait... fetch: transfer timed out
fetch: /usr/local/opnsense/changelog/changelog.txz appears to be truncated: 0/217364 bytes
done
Updating OPNsense repository catalogue...
Waiting for another process to update repository OPNsense
All repositories are up to date.
Checking for upgrades (107 candidates): .......... done
Processing candidates (107 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
***DONE***
So, I'm a bit lost....

EDIT:
It seems like that on top of not working unbound, the main culprit was not workable ipv6 interface (tunnel broker). After removing ipv6 entry of local DNS server (System - Settings - General) and enabling preference to use ipv4, finally system was able to fetch changelog. 
Ultimate verification will be during next upgrade :-)

So both nodes have configured an IP with the correct mask in the switch DMZ subnet?

Also ensure, that you have an outbound or source NAT rule for the source subnet 127.0.0.0/8, which uses the WAN IP as translation target on both.

Yes, both nodes WAN match the mask set in DMZ switch interface.
Could you provide more detailed info about setting up inbound/outbound rules for FW itself? I tried with "This firewall" and didn't work...

So, I have 2 instances of Opnsense in 2 VMs (PVE), configured in HA CARP mode. I have only one public IP, therefore I use Edgerouter X as "dumb router" with switch interface (DMZ) as WAN gateway for both Opnsense instances. Failover / maintenance mode works fine.
But it seems like backup instance (and same applies to master node when acts as backup) has an issue with WAN routing (LAN is ok). When I try to upgrade it, I get:

Code Select Expand

***GOT REQUEST TO CHECK FOR UPDATES***
Currently running OPNsense 26.1.9 (amd64) at Tue Jun  9 13:49:57 CEST 2026
Fetching changelog information, please wait... fetch: transfer timed out
fetch: /usr/local/opnsense/changelog/changelog.txz.sig appears to be truncated: 0/1332 bytes
done
Updating OPNsense repository catalogue...
Fetching meta.conf: . done
Fetching data.pkg: ....... done
Processing entries: .......... done
OPNsense repository update completed. 927 packages processed.
All repositories are up to date.
Checking for upgrades (107 candidates): .......... done
Processing candidates (107 candidates): . done
Checking integrity... done (0 conflicting)
Your packages are up to date.
It takes quite long time to get it displayed plus even longer to be presented with:

You cannot view this attachment.

When system was on 26.1.8 there were timeouts too when checking repository for update info but I was able to download upgrade files. Now, download attempt ends up with timeout.
I've tried to set a routing for firewall itself (This firewall) to use physical interface for WAN connections - it didn't help.

How to troubleshoot it further or fix it? Or maybe I have wrong versions of kernel running in opnsense?

Thanks! I will check both export options, as I ended up with disabling netflow - vacuum caused also CARP flapping.

Writing above post triggered better thinking ;-). The culprit was selection of VPNs' interfaces in netflow settings. Once they've been removed, all went back to normal...

So it seems all started with upgrade to 26.1 and at that time (January) issues with Neighbours Discovery. That was sorted out by the next upgrades/fixes but I started having similar issues with netflow/rrd. Once they are enabled, they cause high I/O and CPU demand. I tried, repair and reset netflow/rrd data plus manual removal of content of /various/netflow/ folder (with stopped neighbours discovery and netflow). Nothing helps; currently system is up to date: 26.1.6.
How to fix it?

So, for those who may experience similar issues:

• WG (only in HA CARP mode) requires additional outbound rule for LAN interfaces and source as WG net
• Very unexpected reason of duplicated data was my set up of CARP VIPs in unicast mode. After removing peers (back to multicast CARP), all of sudden there's no more duplicated data flow and pings plus no webgui/ssh flapping... I wonder why?

One more thing to add is when I reach my LAN over VPN (either Wireguard or OpenVPN) I can't communicate with backup instance (its physical interface addresses) at all while FW rules allow them to send requests to any hosts...

With your assistance in previous topics, I got HA in working condition, but...

To describe my setup:

• 2x Opnsense instances in high availability mode with carp vip interfaces on single pve host. I know it's not full HA but I want software HA and also simply to test it.
• VMs are connected through 3 bridges: 1 on WAN side, the other on LAN side (and further trunk physical link to switch) and pfsync bridge.
• IGMP snooping, storm control are disabled in (UniFi) switches.

In order to change above configuration and (trying to) test my issue, I created additional LAN bridge for backup instance and instead of having them (2x opnsense) connected over single linux bridge - within proxmox, I connected them over physical switch.
This of course requires second downlink:

• master/regular LAN bridge would remain connected as it is now
• backup/new LAN bridge is connected to switch via additional downlink

But problem I'm facing is duplicated communication/data flow to and from both VMs; both instances have same looking graphs in proxmox webgui - network flow and also cpu. Despite they don't change their master/backup status (no flapping at carp status) I have something similar to split brain situation, for example if I communicate with opnsense webgui or ssh on carp vip interface, reply comes either from one of those two and toggles every few seconds. If I ping them, reply is duplicated ("DUP!"). Communication to other hosts and WAN is ok.  I have already set Mac filter to "no" in proxmox VM's firewall options (pve firewall is disabled). I tried ovs and Linux bridges with same results.

To me, it is something related to MAC and network switches; is it possible to set it up correctly?

An update: out of blue (almost) ipv6 started working! My guess is that is because of "routes" - I had them configured in my previous setup. I deleted them from RA when preparing HA setup, but maybe my laptop had cached them (?)... Anyway, since couple of hours ago it did start working and continue doing so...