Messages

Hard to tell unless you provide additional information about the tunnels involved  :)

I had a similar situation (tunnel up, no traffic, restarting ipsec service did not help, restarting my gateway did) with tunnels where two conditions were both met:

• the remote gateway had a dynamic IPv4 adress
• on my side I was using manual SPD entries

The reason was, that whenever the IPv4 address of the remote gateway changed, the manual SPD entries didn't get updated in the kernel security database. Deleting them in the webinterface (/ui/ipsec/spd) and restarting the tunnel made the tunnel work again.

This is a somewhat exotic situation and possibly not yours, but maybe it helps.

10.10.1.1/32  zu 192.168.1.1
10.10.1.2/32 zu 192.168.1.2

Ich nehme mal an, das sind wirklich zwei verschiedene Phase2 Einträge.
Schon mal die Option "Tunnel Isolation" in den OPNSense seitigen Phase1 Settings versucht ?
Das hat bei mir schon ähnliche Probleme mit CISCO Gegenstellen beseitigt.

Same here after upgrading from 22.7.4 to 22.7.6.
OpenVPN with empty CRL List does not connect anymore.
Works fine after deleting and recreating the CRL List

The easiest job here would be to report it at least: https://github.com/opnsense/core/issues/new?assignees=&labels=&template=bug_report.md&title=

done, see https://github.com/opnsense/core/issues/5912

Seems to be a bug in OPNSense, not associated with your client cert, but (as the error message says) with your VPN Root CA, which as CA should indeed have a KeyUsage extension of type "critical" with values "Certificate Sign, CRL Sign". I just tested with OPNSense 22.1.10, internal CA's created via Webgui dont have this extension.

The only solution I can see right now, is to create a Root CA having a correct KeyUsage Extension with OpenSSL or any appropriate tool, import this into your OPNSense and reissue the client certificates using this Root CA.

As it turnes out, this was a false alarm and apparently not related to the 22.1.10 update.

Immediately before this update our stacked Netgear M4300-8X8F core switch got an update to firmware release 12.0.17.7. This update turned of both fans on both switch units, the switch overheated and the described CARP behaviour was correct due to link loss. This is a bad update bug, shame on Netgear. Since turning on the fans via CLI again we did not experience any futher problems.

Sorry for the noise :)

Access to internal services via WAN IPv4 ist usually accomplished by setting up a rule under Firewall->NAT->Port Forward.
Within this rule make shure to set "NAT reflection" to "enable", this should allow to access the service from inside the LAN.

To be honest I have 15 OPNsense VM's in production (6 /6 in HA, 3 single)with zero update issues in 2 years.

I can confirm this for another 19 production setups, only minor upgrade issues, none of them a real show stopper.
You might also consider to use the business edition

https://shop.opnsense.com/product/opnsense-business-edition/

with an upgrade path lagging behind the community edition

/var/log/dmesg.today might provide more insights, unfortunately there are no timestamps:

Code Select Expand


dmesg.today:carp: 1@igb1: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 7@lagg0_vlan10: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 11@lagg0_vlan180: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 1@igb1: INIT -> BACKUP (initialization complete)
dmesg.today:carp: demoted by -240 to 1200 (interface up)
dmesg.today:carp: demoted by -240 to 960 (pfsync bulk done)
dmesg.today:carp: 1@igb1: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 1@igb1: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 9@lagg0_vlan11: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 5@lagg0_vlan15: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 5@lagg0_vlan15: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 11@lagg0_vlan180: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 3@lagg0_vlan12: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 7@lagg0_vlan10: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 1@igb1: BACKUP -> MASTER (master timed out)
dmesg.today:carp: demoted by -240 to 720 (send ok on lagg0_vlan15)
dmesg.today:carp: demoted by -240 to 480 (send ok on lagg0_vlan11)
dmesg.today:carp: demoted by -240 to 240 (send ok on lagg0_vlan12)
dmesg.today:carp: demoted by -240 to 0 (send ok on lagg0_vlan180)
dmesg.today:carp: demoted by 240 to 240 (send error 55 on lagg0_vlan11)
dmesg.today:carp: demoted by 240 to 480 (send error 55 on lagg0_vlan180)
dmesg.today:carp: demoted by 240 to 720 (send error 55 on lagg0_vlan15)
dmesg.today:carp: 7@lagg0_vlan10: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 1@igb1: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 3@lagg0_vlan12: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 5@lagg0_vlan15: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 11@lagg0_vlan180: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 9@lagg0_vlan11: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 7@lagg0_vlan10: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 5@lagg0_vlan15: BACKUP -> MASTER (master timed out)
dmesg.today:carp: demoted by -240 to 480 (send ok on lagg0_vlan15)
dmesg.today:carp: 3@lagg0_vlan12: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 7@lagg0_vlan10: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 5@lagg0_vlan15: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 9@lagg0_vlan11: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 3@lagg0_vlan12: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 11@lagg0_vlan180: BACKUP -> MASTER (master timed out)
dmesg.today:carp: 9@lagg0_vlan11: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: 11@lagg0_vlan180: MASTER -> BACKUP (more frequent advertisement received)
dmesg.today:carp: demoted by 240 to 720 (sysctl)
dmesg.today:carp: demoted by -720 to 0 (sysctl)
dmesg.today:carp: 11@lagg0_vlan180: BACKUP -> MASTER (preempting a slower master)
dmesg.today:carp: 9@lagg0_vlan11: BACKUP -> MASTER (preempting a slower master)
dmesg.today:carp: 7@lagg0_vlan10: BACKUP -> MASTER (preempting a slower master)
dmesg.today:carp: 1@igb1: BACKUP -> MASTER (preempting a slower master)
dmesg.today:carp: demoted by -240 to -240 (send ok on lagg0_vlan180)


but I cant still see what's happening

For the time, the last switch happened, System -> Logfiles -> General contains

Code Select Expand


2022-07-27T01:00:53 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual BMCs IP (10.254.11.1).
2022-07-27T01:00:53 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual BMCs IP (10.254.11.1) (9@lagg0_vlan11)" has resumed the state "BACKUP" for vhid 9
2022-07-27T01:00:53 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual BMCs IP (10.254.11.1).
2022-07-27T01:00:53 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual BMCs IP (10.254.11.1) (9@lagg0_vlan11)" has resumed the state "MASTER" for vhid 9
2022-07-27T01:00:52 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual LegacyServer IP (192.168.180.1).
2022-07-27T01:00:52 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual LegacyServer IP (192.168.180.1) (11@lagg0_vlan180)" has resumed the state "BACKUP" for vhid 11
2022-07-27T01:00:52 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual Administration IP (10.254.15.1).
2022-07-27T01:00:52 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual Administration IP (10.254.15.1) (5@lagg0_vlan15)" has resumed the state "BACKUP" for vhid 5
2022-07-27T01:00:51 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual WAN IP (168.119.27.6).
2022-07-27T01:00:51 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual WAN IP (168.119.27.6) (1@igb1)" has resumed the state "BACKUP" for vhid 1
2022-07-27T01:00:51 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual ManagementInterfaces IP (10.254.12.1).
2022-07-27T01:00:51 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual ManagementInterfaces IP (10.254.12.1) (3@lagg0_vlan12)" has resumed the state "BACKUP" for vhid 3
2022-07-27T01:00:50 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual WAN IP (168.119.27.6).
2022-07-27T01:00:50 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual WAN IP (168.119.27.6) (1@igb1)" has resumed the state "MASTER" for vhid 1
2022-07-27T01:00:49 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual LegacyServer IP (192.168.180.1).
2022-07-27T01:00:49 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual LegacyServer IP (192.168.180.1) (11@lagg0_vlan180)" has resumed the state "MASTER" for vhid 11
2022-07-27T01:00:49 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual Administration IP (10.254.15.1).
2022-07-27T01:00:49 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual Administration IP (10.254.15.1) (5@lagg0_vlan15)" has resumed the state "MASTER" for vhid 5
2022-07-27T01:00:48 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual ManagementInterfaces IP (10.254.12.1).
2022-07-27T01:00:48 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual ManagementInterfaces IP (10.254.12.1) (3@lagg0_vlan12)" has resumed the state "MASTER" for vhid 3
2022-07-27T01:00:44 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual ManagementInterfaces IP (10.254.12.1).
2022-07-27T01:00:44 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual ManagementInterfaces IP (10.254.12.1) (3@lagg0_vlan12)" has resumed the state "BACKUP" for vhid 3
2022-07-27T01:00:44 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual ManagementInterfaces IP (10.254.12.1).
2022-07-27T01:00:44 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual ManagementInterfaces IP (10.254.12.1) (3@lagg0_vlan12)" has resumed the state "MASTER" for vhid 3
2022-07-27T01:00:23 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual LegacyServer IP (192.168.180.1).
2022-07-27T01:00:23 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual LegacyServer IP (192.168.180.1) (11@lagg0_vlan180)" has resumed the state "BACKUP" for vhid 11
2022-07-27T01:00:22 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual Administration IP (10.254.15.1).
2022-07-27T01:00:22 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual Administration IP (10.254.15.1) (5@lagg0_vlan15)" has resumed the state "BACKUP" for vhid 5
2022-07-27T01:00:21 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual BMCs IP (10.254.11.1).
2022-07-27T01:00:21 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual BMCs IP (10.254.11.1) (9@lagg0_vlan11)" has resumed the state "BACKUP" for vhid 9
2022-07-27T01:00:21 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual ManagementInterfaces IP (10.254.12.1).
2022-07-27T01:00:21 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual ManagementInterfaces IP (10.254.12.1) (3@lagg0_vlan12)" has resumed the state "BACKUP" for vhid 3
2022-07-27T01:00:20 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface ProductionServer Virtual IP (10.254.10.1).
2022-07-27T01:00:20 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "ProductionServer Virtual IP (10.254.10.1) (7@lagg0_vlan10)" has resumed the state "BACKUP" for vhid 7
2022-07-27T01:00:20 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Resyncing OpenVPN instances for interface Virtual WAN IP (XXX.XXX.XXX.XXX).
2022-07-27T01:00:20 Error opnsense /usr/local/etc/rc.syshook.d/carp/20-openvpn: Carp cluster member "Virtual WAN IP (XXX.XXX.XXX.XXX) (1@igb1)" has resumed the state "BACKUP" for vhid 1


Thats quite a lot of lines which would be significantely less if OpenVPN was only bound to the WAN Interface, as is should. It doesn't give me a clue as to what happened, though.

The last working release was 22.1.8
The last reboot was after the 22.1.10 update on friday.
I can try a reboot of the primary node, next time the switching occurs.

Since upgrading to 22.1.10 last friday, our 2-Node OPNSense CARP Cluster switches the Master node to BACKUP for no apparent reason.
The slave node takes over on all six configured interfaces. The Master shows an error as in the attached screenshot. By klicking on "Enter persistent CARP Maintenance Mode" twice I can switch back as expected.

The cluster has been running stable with different OPNSense releases for months now, there are no signs for hardware errors.

• has anyone else noticed CARP errors with 22.1.10, that where not there before upgrading ?
• Is there a CARP logfile I can have a look at to find out, what's happening ?

[10]

Ich würde mal raten, das ist ein Konfigurationsfehler.

Du scheinst den Screenshots gemäß für beide Endpoints Netze (/24 für IPv4 und /64 für IPv6) zu verwenden und da möglicherweise die gleichen (ist ja ausgegraut :) )
Auf die "Allowed IP's" wird aber von der OPNSense auf den jeweiligen Peer geroutet und routigtechnisch ist (am Beispiel IPv4 erklärt) das Netz 10.x.y.10/24 das gleiche wie 10.x.y.11/24.

Ich würde den Peers unter "Allowed IP's" /32 und /128 Adressen geben, dann sollte das schon klappen.
Oder aber, falls nötig, abweichende Netze.

Ganz kurz gefasst:

Im AD einen NPS (Microsoft Radius) Server aufsetzen und entsprechend konfigurieren (nicht ganz easy).
In der OPNSense diesen als Authentication Backend einrichten und in der Phase1 des Mobile-Users mit Typ "EAP-Radius" eintragen.

Bei richtiger Wahl der Phase1 und Phase2 Settings klappt das dann mit Windows10, Apple und Android (StrongSwan) Clients problemlos.
Wir verwenden

Phase1 AES256, SHA256, DH 2+14, Lifetime 57600
Phase2 AES256, SHA1, PFS off, Lifetime 7200

Die Phase1 Lifetime ist absichtlich doppelt so groß, wie die von Windows10 verwendeten 28800s.
Damit erfolgt das Rekeying immer zuerst vom Client und die Verbindungen laufen auch länger als 8 Stunden :)

If there are many certificates I would do a backup and restore and edit the generated xml file in between.
The certificates, ca's and crl's are easy to find in the xml file.

set it up and use it. no issues here for months... (using go implementation)

Same here for wireguard-kmod.
No issues for >6 months of production use and much faster than the Wireguard-Go implementation :)