Messages

There is another thread with the same issue, where the user bamypamy did track it down to the issue that when the OPNsense systems do not have direct internet and have to use a proxy server, see https://forum.opnsense.org/index.php?msg=238761

I have just upgraded an HA setup from 25.1.5 to 25.1.7 with running multiple OpenVPN servers running. Still in legacy mode one for end users clients, two for Site2Site to some Ubuntu servers running OpenVPN and one where another OPNsense does connect from with the legacy client. Plus also a new instance for end users.
So far all of them are working fine, as they already did when the secondary system had been updated and then was CARP master.

Maybe check in VPN / OpenVPN / Log File for anything indicating what may be the issue.

I have just upgraded an HA setup from 25.1.5 to 25.1.7 with running multiple OpenVPN servers running. Still in legacy mode one for end users clients, two for Site2Site to some Ubuntu servers running OpenVPN and one where another OPNsense does connect from with the legacy client. Plus also a new instance for end users.
So far all of them are working fine, as they already did when the secondary system had been updated and then was CARP master.

Maybe check in VPN / OpenVPN / Log File for anything indicating what may be the issue.

This is the difference between the base system (FreeBSD) installed version and the one installed from Ports (pkg).

Code Select Expand

root@scrambler:~ # /usr/bin/which -a zstd
/usr/bin/zstd
/usr/local/bin/zstd
root@scrambler:~ #

Everything installed from the base FreeBSD system is in /bin /usr/bin and such, and all from Ports is in /usr/local/.

The path shows in which order things are used:

Code Select Expand

root@scrambler:~ # echo $PATH
/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin
root@scrambler:~ #

So to use the Ports zstd you have to use /usr/local/bin/zstd

Code Select Expand

root@scrambler:~ # /usr/local/bin/zstd -v
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
stdin is a console, aborting
root@scrambler:~ #


How would any resolving system (OPNsense itself or a client in the network) if the outbound traffic will go trough the ProtonVPN "uplink" or anything else?

It will be impossible to direct DNS requests through ProtonVPN if you don't know before doing a lookup which destination will be used. Or the bigger question may be, how do you decide where traffic will go out?

There is a standard for dynamic DNS updates defined in RFC 2136, but I guess none of the DNS Providers does support it. I am running my own name servers, so I do not depend on their ugly APIs.

I am not using caddy or even ACME on OPNsense (yet), but I try to give another view and maybe a possibility to work around this issue with caddy and only that one DNS Provider.

I have a setup for ACME with DNS-01 challenge running on a FreeBSD server with lego. I do have a dedicated dynamic acme.example.com subzone (with only one name server running ISC bind 9.x), which then is only used from lego with the rfc2136 DNS Provider [1]. The cool thing is, that also CNAMEs in multiple other domains with _acme-challenge.example.net CNAME to e.g. example_net.acme.example.com can be used. For certificate deployment I have a hand full of custom shell scripts.

[1] https://go-acme.github.io/lego/dns/rfc2136/

Maybe something like this could be build on OPNsense as well, e.g. with the os-bind plugin serving the dynamic acme.example.com subdomain. Lego seems not to available on, but maybe acme.sh (os-acme-client plugin) is also able to use RFC 2136 and _acme-challenge CNAME. Then you only need to add an IN NS for acme in the example.com zone pointing to the IP address of your OPNsense.

Happy hacking! :-)

The ISC DHCPd does support HA setup on its own. As I am not yet using Kea, I just checked the configuration options and I do see the following things:

1) In "Kea DHCPv4" in "Settings" there is the "High Availability" part.
2) In "Kea DHCPv4" there is also "HA Peers"

Without consulting any further documentation, for me this looks like Kea itself should be able to properly do HA on its own without the need to manually start/stop on CARP change.

Or do I see this wrong?

[primary]

I never had to dig into any HA issue so far and the two setups have been done a long time ago and I don't remember the potential issues I may have run into and fixed.

So just some more things to (re-)check:
- Did you use the /24 netmask for the PFSYNC interface?
- Is this /24 subnet not used anywhere else in your internal network, e.g. may have routes elsewhere?

And below my settings for both systems (primary and secondary).

In Systems / High Availability / Settings on the primary I have the following settings (activate "advanced mode"):
General Settings:
- Disable preempt -> not set
- Disconnect dialup interfaces -> not set
- Synchronize all states via -> PFSYNC
- Sync compatibility -> OPNsense 24.7 or above
- Synchronize Peer IP -> 192.168.x.z (the IP assigned to the PFSYNC on the secondary)
Configuration Synchronization Settings (XMLRPC Sync):
- Synchronize Config -> 192.168.x.z (the IP assigned to the PFSYNC on the secondary)
- Verify peer -> not set
- Remote System Username -> root
- Remote System Password -> the password of the root user (should be the same on both systems)
Services to synchronize (XMLRPC Sync):
- Services -> select the services you want to be synced (be careful and think what should not be synced)

In Systems / High Availability / Settings on the secondary I have the following settings (activate "advanced mode"):
- Disable preempt -> activated
- Disconnect dialup interfaces -> not set
- Synchronize all states via -> PFSYNC
- Sync compatibility -> OPNsense 24.7 or above
- Synchronize Peer IP -> 192.168.x.z (the IP assigned to the PFSYNC on the primary)
Configuration Synchronization Settings (XMLRPC Sync):
- Synchronize Config -> not set
- Verify peer -> not set
- Remote System Username -> not set
- Remote System Password -> not set
Services to synchronize (XMLRPC Sync):
- Services -> Nothing selected

Something to probably also check would be the "Sync compatibility -> OPNsense 24.7 or above" settings.
And I also do remember that sync only works properly when the root user is used.

As you already mention to have a pass any to any IPv4+IPv6 firewall rule on the PFSYNC interface, I have also enabled "Quick" for this rule, so no any other rules (e.g. floating) should kick in and may prevent something.

I am not using NUT, as I have APC UPS and so I am using apcupsd. Depending on the used UPS and Firmware on it, it may be possible to send the UPS a shutdown signal (with delay to execute) to turn itself off as well. Make sure that all the other systems are shutdown before with longer run time / higher battery level when they should initialize the shutdown.
When the power returns and the UPS has a certain (configurable) level of battery charge, it will power the outlets again and your system should start again. Some UPS even have groups of outlets which can be powered up with more or less delay.

Not sure, do you see something wrong?

Looks good, mine are the same.

Could it be, that e.g. the link is not up (and has an IP) until ntpd is starting?

Do you have static IP address on the WAN and configured on the interface? Or are they dynamic or assigned from ISP when the WAN gets initialized?

I am thinking along the line that this could be a timing issue during startup and interfaces may not yet have IP addresses assigned.

I may have to clarify my wording, as I suspect that there the issue may be.

In the HA setups I run, one system is clearly the primary (and most of the time the CARP master), and the other one the secondary (the CARP backup).
Only during system updates (when a reboot is needed), I do switch the CARP master to the secondary system for the update of the primary system. After the update I switch back.

Depending on your setup the CARP master may currently be on the secondary system (from the HA point of view), so it would make sense that you are not able to see the status.

Hope this helps.

I also have 2 HA setups, and in both this works just fine as it always did.

I did just re-check and I see that the status is only available on the primary system. But on the backup systems the messages "The backup firewall is not accessible (check user credentials)." is displayed. But this makes sense, as on the backup system in the Settings everything below "Configuration Synchronization Settings (XMLRPC Sync)" is empty.

For me this looks like something is broken in the ntpd.conf. Did you add own configuration in Advanced, which had been changed but not loaded prior to the update?

Maybe login with ssh or trough console to the system and have a look into /var/etc/ntpd.conf file to see for anything wrong.

To me this looks like you are syncing also from the backup to the main system.

On the backup system in "System: High Availability: Settings" only in "General Settings" for "Synchronize Peer IP" the sync ip of the main system should be set. Everything below "Configuration Synchronization Settings (XMLRPC Sync)" should be empty.

Or did you accidentally make the changes on the backup system?

Use another subnet in your LAN, e.g. 192.168.2.0/24.