Messages

I should add that the entire web interface becomes slow and is unusable when making any change, the command line remains ok.
And everything is ok again after a reboot, but this means that if you want to make any changes the second firewall has to be turned off, then switched on again when the changes are made, they will then be replicated on the other firewall and the systems remains responsive.
It is however a pain to do so when the firewall is in a datacenter and you have to go there to switch them back on (I tried to setup the AMT/IPMI access but couldn't do so in a secure way (ssl) so I disabled it).
So when I have just a few changes I'll reboot the second one and make the changes while its rebooting, I usually have to reboot a few times to give enough time to update some rules, with the risk of making both systems unresponsive if the changes take too long and have to reboot both to get a responsive system again.... you see the picture, its a real pain and it would be great to know why this is happening and to fix it!

Another observation, if you turn the second firewall off, make the changes on the main firewall, then start the 2nd firewall it will update all the changes through CARP and the systems remain fast.
However if you make any change while both are up it will take about 5 Min to replicate the change to the other firewall.

The problem remains with 18.1.8, and I have it on all 6 firewalls now, so its a real pain!

I have just setup the CARP VIP's in the NAT outbound rules, but nothing else in the HA setup.
I'm starting to suspect that the upgrade to 18.1.6 has corrupted some of the CARP/HA configuration, as the HA conf was gone on one of the appliances after the upgrade and the outbound rules seem to have been set back to int instead of the vip. And maybe other things I haven't noticed yet as the system is really slow.
The command line isn't slow however, so looking through the logs, but I'll probably try to recover from a backup and hopefully it will help.

I am configuring 2 other OPNSense Appliances, running 18.1.6, and I've followed the documentation on how to setup CARP, and have set it up as I have done on 2 other pair of systems.

This time however as soon as I enabled CARP the 2 systems became very slow. They have a dedicated interface and cable connected between them, they can ping eachother fine, but as soon as I change something on the main system its spinning for about 10 minutes before the change is reflected on the other appliance and before I can do anything else on the system, so its very painful.

I tried to look into the logs, but...it takes about 10min of waiting after I do something, and then the logs seem to have changed since v17x..., so not sure where to find all the infos. That said the CARP communication are shown in Green, its just very slow.
The system reacts normally again if I disable CARP.

Does anyone had a similar experience or any idea what can cause this ?

Thanks for your replies, as you may have noticed I was a bit frustrated by this when I posted about it, and it didnt make much sense to me either..

Anyway to clarify the CARP configuration on the main backup was flushed somehow during the upgrade, only the ip of the master in "Synchronized peer" remained all the rest was unconfigured/unchecked, while it was there before the upgrade.
I don't know why, its just what happened..  I can only speculate there may have been a check that didnt pass and as a result the config was removed, or it wasn't copied over or.. I don't know.

When I check the history there are a number of occurrences as I update the CARP config again, tried to downgrade/upgrade etc..
But if I check between the last change and update I only see this in the diff:
"<description>/usr/local/opnsense/mvc/script/run_migrations.php made changes</description>"

In any case the configuration is done again, but I have 3 vlans where the master and backup are reversed, I'll also have to investigate if the instability has anything to do with the switch they are connected to.

I have two lines of firewalls on opnsense appliances, I have updated all 4 of them from 18.1.2 to 18.1.5, for two of them no problem but CARP was messed up for the other 2.
Apparently the CARP configuration got lost during the 18.1.5 update on the backup firewall, so I configured it again, the system was very slow after that but was ok again after a reboot.
The problem however remains with CARP, it has been stable with 18.1.2, and I haven't touched any hardware during the update, but now both system's 7 VLANs Master/Backup will change randomly after some time or a reboot.
for example I have about 3 VLAN's with Master/Backup as they should be and 4 are Master on the Backup and Backup on the Master, and this can change after some time or a reboot seemingly randomly.
Again this was all stable before the update, and its not the first time CARP gets unstable after an update, last time it went stable again after some time and another update, I just don't dare use this system fully in production yet without a backup connection :/

I have tried to revert to 18.1.4, but that didnt help so I updated again to 18.1.5.

Does anyone have any advice where to look and troubleshoot this  ?

This issue seems to have been due to an outbound NAT rule.
I had a NAT rule configured for a network range that I was using and changed as it was conflicting with a new WAN network range.
This was created automatically and should have been removed/disabled manually, as I changed the outbound NAT to manual afterwards, when the change was made, but it was still active.
I didnt notice it right away as the connexion was working for some time and only recently stopped working, without any changes in the NAT rules. 

In any case the connexion seems to work again after the NAT rule was removed.

I have setup 2 appliances with CARP initially on 17.7.7 (or a version close to that one), and at the time the backup appliance could only connect to the internet when the master was down, at the time I didnt know why, but it wasn't that bad as main purpose for the backup was to be available if the master is down, which worked. It was however annoying for updates as the master had to be disconnected for the backup to update.
However after some updates and up until 17.7.12 the backup was able to connect again, but now after I updated to 18.1.1 it cannot anymore, sometimes a couple of pings go through but most of the time it doesnt.

What seems to happen is that when a package is sent out from the VIP ip, it tries to go back to the master not the backup, so it never returns. Although there could be some logic as the backup should respond only if the master is down, why was it working for 6 months, and it is also working on the other 2 appliances in CARP which are behind the first layer of firewalls ?

Has a anyone an idea what could be wrong ?

working fine for me as well

ok, thanks for the update. I can leave it for now, I'll try again tomorrow morning

I have four opnsense appliances running (2x2 in CARP). One pair updated fine from 17.7.12 to 18.1.1 and 18.1.2, the other pair updated fine to 18.1.1, but then I lost connectivity partially with the slave and the master can connect fine, but when I try to update to 18.1.2 it says that the repo cannot be found on that server or that there is no update available. I tried several mirrors, one of them (Switzerland) did offer the update and it was updating for a long time, I selected update again and then it showed it had updated to 18.1.2 but it doesnt seem that it has, as it is still mentioning that the system is running on 18.1.1.

I tried to reboot a couple of times but that didnt help.

This is the error message I get when I update from the console:
"updating OPNsense repository catalogue...
pkg-static: Repository OPNsense load error: access repo file(/var/db/pkg/repo-OPNsense.sqlite) failed: No such file or directory
pkg-static: http://pkg.opnsense.org/FreeBSD:11:amd64/18.1/latest/meta.txz: Not Found
repository OPNsense has no meta file, using default settings
pkg-static: http://pkg.opnsense.org/FreeBSD:11:amd64/18.1/latest/packagesite.txz: Not Found
Unable to update repository OPNsense
Error updating repositories!
A firmware update is currently in progress"

I have however just rebooted, so there should not be another update in progress ?!

Anyway has anyone an idea what can be done to solve this ?

I did some more tries, so apparently if you enter both the server and user intermediate CA in an Authority its only going to read the first one, so you have to create two Intermediate CA's and thats why it sees them as a mismatch.

The issue here, as mentioned, is that at SwissSign they use the same root CA, but a dedicated Intermediate CA for Servers and one for users, so two different intermediate CA's, and this doesnt seem to work.

Does anyone know a way to make this work ?

I noticed one difference between the self cert and the SwissSign one in the opnsense gui, in System, Trust, Certificates, the self cert mentions: CA:No, Server: Yes and the SwissSign mentions: CA:No, Server No

I tried both ways. Full chain and only intermediate, that didnt seem to make a difference
I mean I imported the CA and then imported the intermediate as well, and in the intermediate I tried to enter only the intermediate CA and both the CA and intermediate.

Using the latest 17.7.8 version of opnsense on opnsense hardware - https://www.applianceshop.eu/security-appliances/19-rack-appliances/opnsense-based/opnsense-quad-core-gen3-10gb-ssd.html

OpenVPN works fine when using a self generated CA and Certificates, the issue however is that we want to use our own CA and certificates, and this doesnt seem to work.
The issue seems to be that at SwissSign the server certificate and the user certificate are made from their respective intermediate CA (the intermediate CA is however made from the same root CA), so opnsense/openvpn seems to think that there is a mismatch.

Does anyone know if there is anything that can be configured to make it work ?

There is an issue on the pfsense forum from someone that has the same issue
https://forum.pfsense.org/index.php?topic=136116.0

And a description of the issue on the openvpn forum
https://forums.openvpn.net/viewtopic.php?f=6&t=25322