OPNsense Forum
English Forums => General Discussion => Topic started by: random1104 on September 17, 2021, 04:22:28 am
-
Hello!,
I'm trying to setup BGP with FRR and CARP, but couldn't find any mention about what is the expected way to integrate an HA configuration to external routers.
Would the peering be done towards the VIP?, or to the local/fixed interface IPs?
-
In general with BGP you set up two external and one internal peering and not use CARP at all. BGP is inherently "HA", so no need for that.
-
This is way too complex to answer, as you usually also get redundant feed from your upstream.
I have one customer with 2 feeds so CARP only on LAN and every machine has it's own peering, also with the option to have asymetric routing, but its working fine so far ( also with a local HAProxy instance 8) ).
You could also try CARP Failover to FRR is only started on the machine which is master, but this would mean in a failover event you would be offline until full feed is done.
-
The peers are internal routers (x4 in ECMP cluster), not the ISP. If I stablish the session with the "own" base IPs with the daemon only active on the master, the other team will get alarms regarding down BGP link (they don't like it).
Also, I cannot have them active/active because of asymetric traffic, other type of links are active only in the master too (VPN, IPv6 next hop, etc).
Is it correct/expected to use the CARP IP for the BGP/BFD sessions in a CARP/HA setup?
-
since you have 4 routers and 2 eBGP or iBGP, you would need to separate the CARP interfaces for all 4 routers. As in Pri.Sec A and Pri,Sec B. I would just make sure that you have a interior routing protocol if you are planning on using eBGP. you might use static routes, but it's not recommended.
-
Well, on my side, it would be a 2-node cluster with CARP running frr. My understanding from the configuration form is that the way to go is Active/Standby, with CARP asserting which node is the active one.
# Peering Interface
- Node 1: 192.168.0.2
- Node 2: 192.168.0.3
- CARP IP: 192.168.0.1
The question is still, should I ask the other party to integrate to my CARP IP (192.168.0.1 always active), or with the fixed IPs (192.168.0.2 & 192.168.0.3, one always down)?
The other side is sporting the ECMP cluster.
- Node A: 192.168.0.4
- Node B: 192.168.0.5
- Node C: 192.168.0.6
- Node D: 192.168.0.7
-
Update, decided to go with the CARP IP for peering, the only issue I see is the " Update-Source Interface" parameter for FRR in the GUI allows only to select a "physical"/fixed interface.
Any pointers on this?
-
Where is the problem here?
-
Hello!,
The other party expects to have all the sessions active. Peering through both nodes & using CARP failover, one session is always down.
Normal use case is:
A- We can setup Active/Active ECMP connection pools using multiple connections (we can't as that would break some traffic on the OPNSense side)
B- Have a single "always on" session that moves between the nodes.
-
I meant the update-source interface .. why does this matters with carp
-
I meant the update-source interface .. why does this matters with carp
Probably I missunderstood the purpose of the parameter. Since I'm not running IPv6 for this particular peer, probably it doesn't matter but for a CARP enabled link-local IPv6 seems to be needed. Am I correct?
-
@mimugmail the BGP peer would expect the OPNsense cluster to initiate the connection from a particular IP address. If FRR uses the interface address and not the CARP one, a connection will not be established.
@random1104 I still do not understand why you want to run a redundant dynamic routing protocol like BGP via CARP. CARP and VRRP exist to present a single IP address to client systems that rely on e.g. their default gateway to always be available. If you really failover a BGP session, you will need to re-establish the entire routing table with the standby node. Why not run two separate peerings and be done with it. At least that is what I do. I run two peerings to each of my three upstreams.
Anyway, your setup might still work. Because BGP does not send single "messages" over UDP like e.g. DNS but instead establishes a permanent TCP connection between peers. So while your active node will not be able to connect out to your peer, your peer will definitely try to connect *in*. If FRR does listen and accept the connection on the CARP address, the TCP connection will be established and everything will be "just working", with the only drawback that your side cannot *initiate* the connection.
Does that help?
Patrick
-
My concern is asymmetric routing, when there's no route back.
Assuming there's just one peering VLAN & I just setup four sessions (two destinations per firewall, as there are two routers on the other side). If an user on my LAN tries to reach a remote server behind the remote routers it should go first to my VIP gateway on LAN (owned by fw01), any of the external routers and then all the way to the destination server.
After that, the answer has to come back, if I have active/active routing in my firewall 2-node cluster, the traffic can come back through fw02 instead of fw01. Similar scenario would show up if connecting from roadwarrior VPN users that are connected to fw01, if the answer comes back to fw02, it doesn't reach the originating client.
Hence, is seems to me it would be simpler to just route everything through fw01 until it fails, and everything gets moved to fw02. With two interconnection VLANs it would look like something like the attachment.
It's sunday, maybe I'm missing something :D
Note: This is not pure Internet access, it's an internal integration to another network (NSX-T overlay with edge VMs as border) that runs internal services from other team.
-
My concern is asymmetric routing, when there's no route back.
[...]
It's sunday, maybe I'm missing something :D
Possibly. This is in general not a problem.
Peering 1 Peering 2
▲ ▲
│ │
│ │
┌────────────┐ ┌────────────┐
│ │ │ │
│ Router 1 │ │ Router 2 │
│ │ │ │
└────────────┘ └────────────┘
│ │
│ CARP address │
───────────▼─────────────────▲───────────────▼───────▶
│
│
┌────────────┐
│ │
│ Client │
│ │
└────────────┘
The client will send out any packet through the active (CARP) default gateway. This one will send out the packt to its peer. If a return packet arrives via the second BGP router, why should the client care? The intermediate router's IP address is nowhere in the packet. All layer 3 addressing is completely transparent. The packet will come from a different MAC address, but the IP (layer 3) source is off the local network, anyway. All of this is not a problem.
Unless, of course, you want to run a stateful firewall and a redundant BGP uplink on the same two-node cluster. Which I would discourage. Let routers route and firewalls firewall ...
But that is just me - you might come to different conclusions and, hence, configurations for your situation.
Kind regards,
Patrick
-
Yup, my OPNSense firewall cluster routes, "firewalls" & provide remote access through VPN. As updated in my last comment, this is not Internet access, but integration to an internal service network.
-
Some content regarding VIP and carp, I didnt read your last post yet
https://github.com/opnsense/plugins/issues/2540#issuecomment-922526058
-
Well, that seems the way to go for a OPNsense firewall regular usecase (firewall, VPN server, etc)
Active/Active would be nice, but doesn't play nice with the other services. As a pure router I wouldn't use opnsense, just linux with frr or bird for example.
-
You can always just install the pkg via CLI and don't use the UI, then you have the full power, with FRR or Bird
-
@random1104 you can still NAT the outbound BGP connection ...
-
@random1104 you can still NAT the outbound BGP connection ...
Yup, taken from the ticket:
1- Keep default of listening in all interfaces
2- Blocking undesired traffic to the base/fixed IP
3- Only allowing incomming connections to the VIP
4- Setting up outgoing NAT to the remote routers