Messages

1

General Discussion / Re: HA with 2 BGP sessions/transfer networks - asynchronous routing/NAT (configprob)

« on: August 19, 2022, 04:46:33 pm »

Hey mimugmail,

thanks for your reply. To be honest we kinda hoped for an answer from your side, because we read your name in countless other threads on the seek for help and you seemed to always have an answer. Besides that, we are both from munich
However, your answer is not a very productive one. It completly kills any discussion and gives everyone a nice reason not to interact with our problem, which is (at least in our opinion) not the way an "ask for help" should be answered.
We are aware, that opnsense is not the "one size fits them all" product and completly agree with the part that says "you could do that better/easier with other vendors". Sadly we currently don't really have this choice, since ressources and possibilites are not endless on our end.
So let me rephrase our question a little bit - hopefully this will lead to some more interactive discussion (which might still lead to "it doesn't work", but with a solid "why"):

Both of our opnsense firewalls by themselves route and NAT traffic perfectly fine. Our sole issue is redundancy of the upstream gateway. Since we can not establish multpiple BGP Sessions on a single firewall, we need a different solution.
Would it be possible to tell the firewalls that they have two gateways, one being the BGP Neighbour of the ISP and the other, with a lower priority, the other firewall? So that in the case where firewall A is online and its BGP Neighbour is online, everything is routed through its configured BGP session. In case it is online but the BGP neighbour is not, it routes its traffic via the other firewall (since their BGP neighbour is most likely up).
Also we are aware that we have a state problem in the case where traffic flows asynchrounously inwards via Firewall B and outwards via Firewall A. Let's (for the sake of the discussion) ignore this fact for a second, because we would try to solve this problem as soon as we have the gateways sorted out.

Thanks again for your reply, we really hope to read again from you soon.
Best regards, arti

2

General Discussion / HA with 2 BGP sessions/transfer networks - asynchronous routing/NAT (configprob)

« on: August 19, 2022, 02:35:44 pm »

Hi Guys,

short disclaimer: I posted this already in the german sub and decided to additionally try the broader mass here, because our problem seems quite specific.

the last few weeks we searched docs, the www, this forum here and had talks with different people to get into the right direction.

"Firewalls should firewall and routers route." The global shortage right now somehow forced us to also do the routing part...

Prerequisites
- 2 Transfer networks (1.2.10.12/30, 1.2.20.248/30)
- 1 ecternal network (2.3.30.0/24)
- 2 BGP Sessions (1 pro Firewall)
- HA OpnSense Master-Backup
- wokring CARP Failover on all interfaces except EXT
- No influence on provider side of this architecture

Interfaces
EXT - Our BGP Uplink (1 per FW)
PUB - Internal servers and switches and the external network (2.3.30.0/24)
OPT1-3 - different private networks and VLANs

High availability
For PUB and OPT interfaces there is a CARP-IP on both Firewalls which is used as default gateway for all servers.
Both firewalls have their own BGP session (and transfer IP) to announce the external network (2.3.30.0/24). Additionally both firewalls are NATing over their transfer net IP (1.2.10.13 & 1.2.20.249) all traffic (also the external network).

Problem description
(see attached png: BGP-Transfernetze.png)
(see attached png: asynchroner-traffic.png)
If both BGP sessions are running (our preferred scenario) there is a problem with the routing of the traffic. It seems like some packets take the RED route through FW2 in and take the route GREEN out over the default-GW. We don't really know if this form of asynchrounus traffic even is a problem or the cause that the packets are dropped. Additionally the states seem to play some sort of role - to exclude this, we already disabled the states on both EXT interfaces.
How is it possible to get into a state, where both firewalls have a running BGP session and the traffic is always routed to the www - even if one of the Uplinks (BGP sessions) is failing to reach real HA?

BGP-Sessions
Firewall-1

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.10.13
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.10.14
--------------------------
Firewall-2

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.20.249
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.20.250

BGP CARP Failover
Routing -> General -> Enable CARP Failover

Not possible since both EXT-Interfaces in both FWs have different transfer networks. Additionally the BGP failover takes a lot of time to create the new session.

This only works if another interface on which some CARP IP is configured fails. Since the EXT interface doesn't have CARP this is not feasible.


Both BGP Sessions
(see attached png: asynchroner-traffic.png)

VPN connection possible to both firewalls to their transfer-IP (1.2.10.14 / 1.2.20.250).
SSH, ping, curl... to VM only possible, if routed through FW1 (MASTER) (GREEN).

If we disconnect FW1-EXT (connection to BGP peer transfernet), there is no failover and there are no answers to the NAT-packets anymore. VPN to FW2 is of course still possible.

Both FWs are directly connected through a sync interface to enable CARP.

3

German - Deutsch / OPNsense HA mit BGP Transfernetzen und asynchronem Routing/NATing Konfig-Problem

« on: August 18, 2022, 11:16:34 am »

Hi Zusammen,

wir haben uns die letzten Wochen durch das Internet, Forum und einige Dokus gekämpft und sind leider bisher zu keiner Lösung gekommen. Daher hoffen wir auf eure Schwarmintelligenz

"Firewalls should firewall and routers route." Leider sind wir hier (wie viele andere) von Lieferengpässen betroffen.

Ausgangssituation
- 2 Transfernetze (1.2.10.12/30, 1.2.20.248/30)
- 1 Externes Netz (2.3.30.0/24)
- 2 BGP Sessions (1 pro Firewall)
- HA OpnSense Master-Backup
- Funktionierendes CARP Failover auf allen Interfaces außer EXT
- Auf die Provider-Seite haben wir keinen Einfluss

Interfaces
EXT - Hier hängt unser BGP Uplink
PUB - Hier hängen unsere Server und Switches und das externe Netz (2.3.30.0/24)
OPT1-3 - Hier hängen verschiedene private Netze und VLANs

High availability
Für PUB und die OPT Interfaces ist jeweils eine CARP IP über beide Firewalls eingerichtet, die als Standard-Gateway für alle Server dient.

Beide Firewalls nutzen ihre BGP Sessions (Transfer IPs) um darüber das externe Netz (2.3.30.0/24) zu announcen. Außerdem NATen beide Firewalls über ihre jeweilige Transfer-Netz IP (1.2.10.13 bzw. 1.2.20.249) allen Traffic (auch den des externen Netzes).

Problemstellungen
Laufen beide BGP Sessions (unser Wunschszenario) gibt es ein Problem mit dem Routing des Traffics. Es scheint als ob manche Pakete die Route ROT über Firewall 2 nach innen nehmen und dann vom CARP Standard-Gateway über die Route GRÜN der Firewall 1 wieder nach außen geroutet werden. Wir sind uns aber nicht sicher, ob diese Form von asynchronem Traffic überhaupt ein Problem ist bzw. die Ursache sein kann, dass Pakete gedroppt werden. Dazu spielen hier natürlich die States der Firewalls eine Rolle - um dies als Problemursache auszuschließen, haben wir die States auf beiden Firewalls auf dem entsprechenden EXT Interface vollständig deaktiviert.
Wie können wir einen Zustand erreichen, in dem beide BGP Sessions laufen und der Traffic immer erfolgreich nach draußen geroutet wird - sogar dann, wenn einer der beiden Uplinks (BGP Sessions) ausfällt, um echte Hochverfügbarkeit zu erreichen?

siehe Bild: BGP-Transfernetze.png

BGP-Sessions
Firewall-1

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.10.13
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.10.14
--------------------------
Firewall-2

General
BGP-AS: 23456
Route-distribution: Connected

Neighbor
Neighbor-Address: 1.2.20.249
Remote-AS: 1234

Interface-EXT
IPv4: 1.2.20.250

BGP CARP Failover
Routing -> General -> Enable CARP Failover

Nicht möglich, da die EXT-Interfaces an den beiden Firewalls in zwei verschiedenen Transfernetzen sind und entsprechend kein CARP ausgetauscht werden kann.

Weiterhin dauert der Failover (wenn er über eins der anderen IFs ausgelöst wird) deutlich zu lange, da die BGP Session neu verhandelt wird.

(Funktioniert generell, falls ein NICHT-EXT Interface abgesteckt wird. Hier funktioniert CARP wie gewünscht.)


Beide BGP Sessions

siehe Bild: asynchroner-traffic.png

VPN Verbindung auf beide Firewalls an die Transfer-Netz-IP (1.2.10.14 / 1.2.20.250) möglich.
SSH, Ping, etc. an VM nur möglich, wenn über FW1 (MASTER) rein kommend (GRÜN).

Falls wir FW1-ETH1 abstecken, passiert kein Failover und die Antworten auf NAT-Pakete kommen nicht mehr an. VPN nach FW2 funktioniert natürlich noch.

Die beiden Firewalls sind über ein SYNC Interface direkt miteinander verbunden, um CARP zu ermöglichen. Weitere Verbindungen sind nicht konfiguriert.