Gone fishing for CARP.
I have a Qotom as my primary router and I have a Proxmox machine which is running several containers and VMs, and now an Opnsense instance too.
With nothing better to do, I decided to create a VM on the Proxmox running Opnsense and have a play with CARP. CARP requires two WAN connections, or does it? I only have one, so currently I am feeding that WAN into my Netgear switch, and then out to the Qotom on one port and out to the Proxmox Opnsense instance on another. I am going to use a script and use SNMP to toggle the WAN between the two outputs on the switch, triggered by whichever Opnsense instance goes master.
Now the question, is there a gaping hole in my theory, or should it work?
Hey,
Recommended setup for any CARP is 3 IPs, 1 per interface + 1 for VIP. But!
You can do this as well with one. Basically you just let the Interfaces without IP and configure the IP on the CARP VIP.
This will create a situation that only the primary device will have the IP, and if a failure occurred the secondary overtakes the connection and the IP will move to it. This is working pretty well.
Benefit:
+ You can have a HA setup with only 1 IP
Negative:
- Only one device at a time has access to internet
- In order to have the secondary device reach the internet you need to do some Internal Routing
- Its kinda a hack, but its working
Regards,
S.
OK, I am thinking a little differently. I had an instance a week or two ago where Unbound decided to go gaga, not for any reason I could see, it just did, it's not happened before. Now, I'm not sure CARP would pick this up, as the interfaces were still responding. I also have Home Assistant, that now watches the Unbound service, it also watches the gateways, and flags a warning etc. I'm going to use this flag to close one of the LAN ports on the switch so the primary router will be seen as down by the Proxmox instance and promote itself. This is the theory anyway, in practice we'll see what happens.
That would work how you describe it, but, its a bit overengineered. Like why not right.
But from pure NW perspective and OPNsense perspective. You can set the CARP to failover WAN + LAN if any of those fails be it LAN or WAN, one triggers the another.
I would create CARP for LAN/VLANs with 3 IPs (for that you have IPs). And WAN with a single IP (as you dont have more).
Have it implemented so a LAN failure triggers WAN failover and vise versa and sync between the FWs what is needed.
We tested this setup between two DECs in a LAB where DECs are connected to MLAGs towards Mikrotik and it works very fast and reliable.
Regards,
S.
In regards of the unbound "fail", the answer in regards of CARP is no. CARP does not track processes (keepalived can but that is not supported in FBSD).
Currently implementation of CARP in FBSD tracks only NIC state change, e.g is very static.
Which means if any of your processes, routing protocols or paths on OPN fail the CARP will not trigger a failover.
I personally run DNS outside of OPN, where I use keepalived between two DNS nodes (one node is on RPI other in PRX LXC) and VIP is given to the clients.
https://github.com/SeimusS/Pihole-HA
Just for info
https://github.com/opnsense/core/issues/10654#issue-5053589060
Regards,
S.