HA with 2 BGP sessions/transfer networks - asynchronous routing/NAT (configprob)

[arti]

Hi Guys,

short disclaimer: I posted this already in the german sub and decided to additionally try the broader mass here, because our problem seems quite specific.

the last few weeks we searched docs, the www, this forum here and had talks with different people to get into the right direction.

"Firewalls should firewall and routers route." The global shortage right now somehow forced us to also do the routing part...

Prerequisites
- 2 Transfer networks (1.2.10.12/30, 1.2.20.248/30)
- 1 ecternal network (2.3.30.0/24)
- 2 BGP Sessions (1 pro Firewall)
- HA OpnSense Master-Backup
- wokring CARP Failover on all interfaces except EXT
- No influence on provider side of this architecture

Interfaces
EXT - Our BGP Uplink (1 per FW)
PUB - Internal servers and switches and the external network (2.3.30.0/24)
OPT1-3 - different private networks and VLANs

High availability
For PUB and OPT interfaces there is a CARP-IP on both Firewalls which is used as default gateway for all servers.
Both firewalls have their own BGP session (and transfer IP) to announce the external network (2.3.30.0/24). Additionally both firewalls are NATing over their transfer net IP (1.2.10.13 & 1.2.20.249) all traffic (also the external network).

Problem description
(see attached png: BGP-Transfernetze.png)
(see attached png: asynchroner-traffic.png)
If both BGP sessions are running (our preferred scenario) there is a problem with the routing of the traffic. It seems like some packets take the RED route through FW2 in and take the route GREEN out over the default-GW. We don't really know if this form of asynchrounus traffic even is a problem or the cause that the packets are dropped. Additionally the states seem to play some sort of role - to exclude this, we already disabled the states on both EXT interfaces.
How is it possible to get into a state, where both firewalls have a running BGP session and the traffic is always routed to the www - even if one of the Uplinks (BGP sessions) is failing to reach real HA?

BGP-Sessions
Firewall-1

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.10.13
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.10.14
--------------------------
Firewall-2

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.20.249
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.20.250

BGP CARP Failover
Routing -> General -> Enable CARP Failover

Not possible since both EXT-Interfaces in both FWs have different transfer networks. Additionally the BGP failover takes a lot of time to create the new session.

This only works if another interface on which some CARP IP is configured fails. Since the EXT interface doesn't have CARP this is not feasible.


Both BGP Sessions
(see attached png: asynchroner-traffic.png)

VPN connection possible to both firewalls to their transfer-IP (1.2.10.14 / 1.2.20.250).
SSH, ping, curl... to VM only possible, if routed through FW1 (MASTER) (GREEN).

If we disconnect FW1-EXT (connection to BGP peer transfernet), there is no failover and there are no answers to the NAT-packets anymore. VPN to FW2 is of course still possible.

Both FWs are directly connected through a sync interface to enable CARP.

[mimugmail]

To be honest, opnsense is not the best solution for such a design.

Redesign or different vendor.

[arti]

Hey mimugmail,

thanks for your reply. To be honest we kinda hoped for an answer from your side, because we read your name in countless other threads on the seek for help and you seemed to always have an answer. Besides that, we are both from munich
However, your answer is not a very productive one. It completly kills any discussion and gives everyone a nice reason not to interact with our problem, which is (at least in our opinion) not the way an "ask for help" should be answered.
We are aware, that opnsense is not the "one size fits them all" product and completly agree with the part that says "you could do that better/easier with other vendors". Sadly we currently don't really have this choice, since ressources and possibilites are not endless on our end.
So let me rephrase our question a little bit - hopefully this will lead to some more interactive discussion (which might still lead to "it doesn't work", but with a solid "why"):

Both of our opnsense firewalls by themselves route and NAT traffic perfectly fine. Our sole issue is redundancy of the upstream gateway. Since we can not establish multpiple BGP Sessions on a single firewall, we need a different solution.
Would it be possible to tell the firewalls that they have two gateways, one being the BGP Neighbour of the ISP and the other, with a lower priority, the other firewall? So that in the case where firewall A is online and its BGP Neighbour is online, everything is routed through its configured BGP session. In case it is online but the BGP neighbour is not, it routes its traffic via the other firewall (since their BGP neighbour is most likely up).
Also we are aware that we have a state problem in the case where traffic flows asynchrounously inwards via Firewall B and outwards via Firewall A. Let's (for the sake of the discussion) ignore this fact for a second, because we would try to solve this problem as soon as we have the gateways sorted out.

Thanks again for your reply, we really hope to read again from you soon.
Best regards, arti

[mimugmail]

Look, the HA mechanism of OPNsense only runs stable when ALL interfaces use CARP. As you have 2 upstreams with /30 this will be hacky .. it's just hacky .. it wont run in a stable way.

I did this one time successfully with DFN ... I'd guess you also connect to them from a university?
You can run CARP on LAN and set a peer on each node, then you need to prioritize one line via local-pref AND ... AND .. AND you need to use MED to also force incoming. I only know the DFN which allows this, any usual provider prohibits using MED. Then you'll have a HA solution for your lines ..

You still might have problems with asymetric routing but it should work in general. You might also switch times to datacenter profile if you provider allows this, convergence will be much better.

[itngo]

Did you ever solve this? What you might need is an AS-Prepend on your "secondary", this will force traffic to the master and on failure it will go through slave....