Topics

I have a pair of opnsense routers with CARP which haven't been updated for quite a time. They were running 17.1.2 fine with 16 VIFs defined on the WAN interface (since it's still impossible to define IP aliases on a VIF), and some more on the LAN and DMZ sides.

After upgrading both machines to 17.7.6 the backup machine does't receive CARP announcements any more on the WAN interface (other interfaces are ok), so it will switch to master (on WAN only), messing up traffic badly.

Checking on the master, I still see CARP announcements generated on the WAN if, but apparently they are not passed out. As soon as I pfctl -d the firewall, I can see CARP arriving at the secondary as well; pfctl -e and announcements are lost again.

I added explicit rules on the WAN interface, allowing CARP from the firewall, and even any traffic from the firewall, no result.

Any hint how to get CARP working again? I'm non-redundant now, giving me a bad feeling after I had a kernel crash on the master lately.

Regards
Andreas

As far as I understand carp/pfsync, the backup firewall should take over the running sessions from the master more or less interruption free. On an heavy traffic tcp session, maybe some packets might need retransmission since there's a time gap between pfsyncs, but not more.

Apparently, this assumption is wrong. Even on a virtually idle system, connections are broken. I have a OS 17.1.8 pair (virtualized on Xen 4.4.1), machines are connected via 10GBit. The web server in the DMZ accesses the file storage via NFS4 tcp port 2049. When rebooting the master (for whatever reason, e.g. fw upgrade), the nfs mount might report a stale handle with no chance to recover but reboot the web server VM. I tried to demote the master to slave upfront, with persistence, and promote later back again, but this doesn't seem to help too much.

All other connections might break as well, but the nfs connection is really fatal.

Is there any means to make the carp/pfsync stuff work without breakage? Even a fancy solution improving the nfs breakage only would help. I've been observing this problem since pfSense 1.x/2.x/OS16.7 and it's still annoying.

Regards
Andreas

I encountered this problem with pfsense (they say it's not bug...), and checked with opnsense now getting the same results:

The definition of a gateway implicitely creates a route via that gateway, if a monitoring ip is given. I found this unexpected, and would like to have a checkbox to disable this behaviour (it will create icmp-redirects unless disabled by kernel params).

Anyhow, if this gateway entry is disabled, the routing will still persist, which I'd call a bug. IMHO a disabled entry shouldn't have any side effect.

I tried installing on Virtualbox for testing purposes, but the copying would fail at various positions. After trying different NIC and chipset settings, I found that increasing memory from 512MB to 1G did the trick. Apparently, 512MB is not sufficient to install (using 15.7.18-amd64).

Regards,
Andreas