Asymetric routing firewall issue

[Nico]

Hello,

this is more for informational purposes than an actual cry for help since I managed to find a work around. However, I recently found an issue with asymetrical routing and apparently unsynchronized firewall states. The setup is as follows:

• 2x OPNsense in HA configuration with a dedicated HA link
• Both having the FRR routing suite and iBGP setup with some bigger routers and exchanging routes over the WAN interface
• CARP on all LAN interfaces

Firewall A was the primary ingress router since the BGP peering routers learned those routes first and had them active. So all traffic from the WAN side came in through Firewall A. This was working as expected as long as Firewall A was the CARP master. As soon as the master switched to Firewall B, the routing was asymetric and the firewall states obviously not synched anymore leading to traffic not being able to return since there was no corresponding permit rule on WAN ingress.

I solved this by

• Overriding the next-hop on the BGP peering routers to point to the WAN CARP IP instead of the loopback IP of each Firewall
• Enabling preempt for carp thus making it failover all interfaces at once
• Surprisingly /etc/sysctl.conf was ignored for this so I added the tiny command to a newly created file in "/etc/rc.conf.d/preempt" which is processed fine upon reboot

Maybe I oversaw something but I thought this was appropriate to share with you. A question aside from that: I assume the firewall states are synchronised using the configured credentials but this is of course only configured one way: from A to B. Am I supposed to configure this vice versa? Since documentation is not clear about that, we tested that once but ran into issues according to my colleague. So maybe someone can shed some light on this matter.


Thanks!