OSPF as IGP for HA

[vgstech]

Hi there,


I'm coming here after several hundred hours spent trying to achieve HA using OSPF as IGP.

I'm working on a Data Center collocation project using two DEC3800 appliances. Each firewall has its own ISP provider, having that way Internet failure tolerance. That means each firewall has a direct cable to a different router (FW1 > Router 1 + FW2 > Router 2). Also, both FWs have a direct connection for HA using a fixed local IP(FW1 10.0.0.1 + FW2 10.0.0.2) on top of the interface called HA.

On the other hand, I've created a virtual IP in IP Alias mode. I cannot use CARP in this scenario since if one of the WANs is down and doesn't match the CARP master, the traffic isn't redistributed, that's why OSPF comes to the scene here.

I've seen this thread https://forum.opnsense.org/index.php?topic=34145.msg165886#msg165886 from Patrick and that's exactly what I'm trying to achieve.

• OSPF as IGP for LAN redistribution
• BGP for external routes redistribution

Till now, I've tested the following scenarios:

• OSPF v3, screenshots attached
• OSPF, screenshots attached

Questions:

• How should OSPF be configured to achieve that local routes redistribution?
• Should I use OSPF v2 or v3 for this purpose? If I'm not wrong, v3 should do the trick as well and it's easier to configure.
• Which interface should be used in the interfaces section? I've tested loopback and the HA interface.

Thanks in advance

[vgstech]

Adding extra screenshots in a separate message due to posting restrictions.

[Patrick M. Hausen]

Hi!

Quick answers that I can give without investing too much time which just right now I do not have:

OSPFv2: IPv4
OSPFv3: IPv6

You also asked about the servers behind the firewalls - for these you need to create a CARP address on LAN, of course, because "client" systems do not take part in the dynamic routing.

Infrastructure: OSPF, clients: CARP.

I do not understand your remark about the alias IP - on which interface is that?

I'll try to have a deeper look later. Still I might not be of much help if the issues are very OPNsense/FRR (?) specific. I claim to know OSPF inside out, but I have only been running an ISP backbone on Cisco gear

HTH
Patrick

[vgstech]

Hey Patrick,


Thanks for jumping in. If I'm not wrong, I should not use CARP on the LAN side since if I use CARP and the WAN of the master CARP node goes down, there is no way to redistribute the traffic to the other node since BGP cannot demote the node. Is that correct? So the plan is:

• Create a virtual IP using alias IP instead of CARP IP
• Redistribute the LAN traffic using OSPF as IGP
• Redistribute the WAN traffic using BGP

I'm open to new schemas or suggestions.

If you need an extra configuration description, I can share a schema of the network.


Thanks!

[Patrick M. Hausen]

The two nodes will of course talk iBGP with each other.

If the WAN of the CARP primary goes down it will lose all its routes to the outside world that it had learned via eBGP from the external peer.

The routes learned via iBGP from the partner node will take over and all the traffic received from clients on the CARP address will be handed over to the partner node.

A dedicated, preferably redundant, and high bandwidth connection between the two nodes for both OSPF and iBGP is necessary.

All internal VLANs can use CARP. For load distribution you can alternate primary/secondary roles over the VLANs. You do not have perfect control over which node serves as the exit node to the Internet, anyway, because the shortest path learned via BGP will be preferred so some AS will be routed via node 1 and some via node 2 - a lot of traffic between the nodes happens but this is not a problem.

At least that is how ISPs do it 

[vgstech]

Thanks for the clarification, Patrick. If I've understood correctly, that means that despite having a virtual IP in CARP mode, it's possible to redistribute the traffic to the backup node. I was confused about that part.

Since I'm struggling with the OSPF IGP configuration, do you have some configuration examples handy? Do you recommend OSPF v2 or v3?

BTW, I'm using two aggregated ports in LAGG mode between each FW, being that way fault tolerant on that side as well, as you've mentioned.


Thanks!

[Patrick M. Hausen]

I built this for a medium sized ISP on Cisco gear so I do not have an example for OPNsense, sorry.

The OSPF version question I already answered:

If you route IPv4 you must use OSPFv2.
If you route IPv6 you must use OSPFv3.

if you run dual stack you run both OSPFv2 and OSPFv3 ...

[vgstech]

I'm still a bit confused in the IGP part, since in this quote: https://forum.opnsense.org/index.php?topic=34145.msg165886#msg165886 it was mentioned that CARP should be used for firewall failover but not for route redundancy.

According to the previous message:

The routes learned via iBGP from the partner node will take over, and all the traffic received from clients on the CARP address will be handed over to the partner node.

If I've understood correctly, the proposed scenario is: CARP + OSPF for IGP (OSPF v2 in my case, since I deal only with IPv4). Based on that, I still wonder how a backup LAN node can become a master from the routing perspective without being a master LAN node. Hopefully, the question makes sense.

Regarding the remaining topics, roger, thanks!

[Patrick M. Hausen]

CARP presents a unique address to LAN clients. The clients will use one of the two nodes as their default gateway.

BGP to external peers tell each node where "the Internet" is.

iBGP - which is just BGP (not an IGP!) between two systems in the same AS - tells each node all routes the other node learned about "the Internet".

iBGP has a lower priority as the routes learned from external peers - also called eBGP.

So a packet gets to the CARP primary for e.g. destination the AS of German Telekom. If the AS path it received from external peer is shorter than the one it received from its partner node, it will send the packet through its uplink. Otherwise - even with both WAN links up and running - it will send the packet to its partner which will in turn send it along its WAN uplink.

The two nodes work in tandem.

If one WAN link fails all routes received from that external peer vanish.

If that was the WAN link of the active CARP primary, it will still receive the packets from the clients but now simply send all of them to its partner to use the other WAN uplink.

No failover for the clients ever (!) takes place as long as bot your nodes are up and running. The two nodes decide in a coordinated way if a packet for "the Internet" shall go "left" or "right" and act accordingly. The WAN links are completely opaque to the clients.

That's what you do with BGP.

Now OSPF is what you need if you have local networks inside your AS that should also be routed redundantly. So you have a remote office with VPN or leased lines and one connection to each of your nodes. That's where OSPF comes into play. Completely separate from all the BGP routing. The external BGP is only interested in a single route/prefix covering your entire AS.

Some people ditch OSPF altogether and use (i)BGP for internal routing, too. Leaner design but you must be careful not to leak more specific routes into the Internet.

Repeating myself: there is no failover ever for CARP if both nodes are running. Clients always send to the CARP primary. The primary then sends to the secondary in case it does not have an active uplink - or the secondary announces as better (i.e. shorter) route.

CARP is in place only for the case that one of your nodes explodes. In that case the secondary can take over.

[mimugmail]

Your screenshots look weird. Why not using carp and connect both units on both wans without a routing protocol

[vgstech]

CARP presents a unique address to LAN clients. The clients will use one of the two nodes as their default gateway.

BGP to external peers tell each node where "the Internet" is.

iBGP - which is just BGP (not an IGP!) between two systems in the same AS - tells each node all routes the other node learned about "the Internet".

iBGP has a lower priority as the routes learned from external peers - also called eBGP.

So a packet gets to the CARP primary for e.g. destination the AS of German Telekom. If the AS path it received from external peer is shorter than the one it received from its partner node, it will send the packet through its uplink. Otherwise - even with both WAN links up and running - it will send the packet to its partner which will in turn send it along its WAN uplink.

The two nodes work in tandem.

If one WAN link fails all routes received from that external peer vanish.

If that was the WAN link of the active CARP primary, it will still receive the packets from the clients but now simply send all of them to its partner to use the other WAN uplink.

No failover for the clients ever (!) takes place as long as bot your nodes are up and running. The two nodes decide in a coordinated way if a packet for "the Internet" shall go "left" or "right" and act accordingly. The WAN links are completely opaque to the clients.

That's what you do with BGP.

Now OSPF is what you need if you have local networks inside your AS that should also be routed redundantly. So you have a remote office with VPN or leased lines and one connection to each of your nodes. That's where OSPF comes into play. Completely separate from all the BGP routing. The external BGP is only interested in a single route/prefix covering your entire AS.

Some people ditch OSPF altogether and use (i)BGP for internal routing, too. Leaner design but you must be careful not to leak more specific routes into the Internet.

Repeating myself: there is no failover ever for CARP if both nodes are running. Clients always send to the CARP primary. The primary then sends to the secondary in case it does not have an active uplink - or the secondary announces as better (i.e. shorter) route.

CARP is in place only for the case that one of your nodes explodes. In that case the secondary can take over.

Thanks for this, Patrick; it makes much more sense to me now. I'll play around with OSPF to configure the internal redistribution part.

[vgstech]

Your screenshots look weird. Why not using carp and connect both units on both wans without a routing protocol

That's isn't possible in our scenario due to two things:

• In a Data Center world, each port you consume has a cost.
• The appliances have only two fiber channels; one is for the WAN, and the other is for the downstream network devices.

Anyway, thanks for your suggestion!

[bimbar]

Your screenshots look weird. Why not using carp and connect both units on both wans without a routing protocol

That's isn't possible in our scenario due to two things:

• In a Data Center world, each port you consume has a cost.
• The appliances have only two fiber channels; one is for the WAN, and the other is for the downstream network devices.

Anyway, thanks for your suggestion!

Your setup would usually look something like this in the DC / carrier world:

WAN ROUTER 1     ---------      WAN ROUTER 2
         |                                              |
       BGP                                         BGP
         |                                              |
     Router 1 ----- some IGP ------ Router 2
         |                                              |
       IP 1                CARP IP              IP 2
                                 |
                               LAN


Alternatively, the WAN could also be routed with some IGP, often IS-IS is used.

[vgstech]

Correct, bimbar. That's the schema we are developing, but instead of "WAN ROUTER," it will be an "Opnsense FW." The carriers provide routers 1 and 2, so I don't have to deal with that piece apart from configuring the BGP from the FWs to the routers.

Did you configure OPNSense in a similar scenario in the past?


Thanks!

[mimugmail]

Your screenshots look weird. Why not using carp and connect both units on both wans without a routing protocol

That's isn't possible in our scenario due to two things:

• In a Data Center world, each port you consume has a cost.
• The appliances have only two fiber channels; one is for the WAN, and the other is for the downstream network devices.

Anyway, thanks for your suggestion!

Your setup would usually look something like this in the DC / carrier world:

WAN ROUTER 1     ---------      WAN ROUTER 2
         |                                              |
       BGP                                         BGP
         |                                              |
     Router 1 ----- some IGP ------ Router 2
         |                                              |
       IP 1                CARP IP              IP 2
                                 |
                               LAN


Alternatively, the WAN could also be routed with some IGP, often IS-IS is used.

I set this up multiple times for universities in Germany exactly this way but without an IGP or just ibgp