Hi Everyone,
I have an existing network, multiple VLANs and firewall rules, etc. I am trying to add a 2nd OPNsense firewall. The 2nd OPNsense firewall has its own dedicated internet connection from a separate ISP. I have two (2) goals: a) Make OPNsense fault tolerant. i.e. If `FW1` goes down, `FW2` will take over. And, b) Make our WAN highly available. So if `ISP1` (exclusively connected to `FW1`) goes down, internet will route through `ISP2` (exclusively connected to `FW2`). More details about my configuration can be found here (https://forum.opnsense.org/index.php?topic=48821.0).
Whilst I can appreciate the recommendation is to start fresh with two (2) new OPNsense firewalls, and recreate "everything" together on each OPNsense server; I don't have two (2) machines laying around. Thus, I am stuck with my current situation where I am trying to "add" this 2nd firewall into the existing network, configure OPNsense in HA, and make the necessary changes to the downstream networks. Because I'd imagine there must be a lot of people in that same situation, I figured I'd post here.
We've ran into a situation where I noticed that CARP traffic from the default gateways in the various subnets are trying to traverse across VLANs. I think I've discovered that's due to there being a mismatch on the "Identifiers" (under Assignments) on each firewall. For example, on `FW1`, Identifier `opt1` is pointing to "Network A". And on `FW2`, Identifier `opt2` is pointing to "Network C". That happened because we manually defined the Assignments in a different order on FW2, then we did when we initially setup FW1.
So to avoid me discovering any other important configuration like this "the hard way", does anyone know of a checklist or guide that walks through adding a 2nd OPNsense firewall into an existing network? And, if no guide exists, does anyone here with this sort of experience have any advice on what else I need to check or watch out for?
If anyone else runs into this problem where you are creating a cluster for the first time by adding an additional FW into your OPNsense network, the "Identifiers" for the interfaces must match. e.g. `VLAN X` must have `opt1` on both OPNsense firewalls.
That's a documented requirement:
https://docs.opnsense.org/manual/how-tos/carp.html#configuring-carp-for-ipv4
Warning
Make sure the interface assignments on both systems are identical!
Via Interfaces ‣ Overview you can check if e.g. DMZ is opt1 on both machines.
When the assignments differ you will have mixed Master and Backup IPs on both machines.