Messages

Hello all,

I am about to embark on clustering two OPNSense nodes and I have a couple questions:

1) I am going to have 3 internal interfaces and a Wireguard interface. Do I need to add a CARP firewall rule to each interface?
2) For the heartbeat network should the firewall rule be wide open or limited to the heartbeat network.

Thanks,
Steve

1) I believe the CARP firewall rule is automatically added and doesn't require manual rule creation. If you did want to make one though you could make a rule like the one below for every interface? Not sure about Wireguard but mine has been working without a custom rule.

Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
IPv4 CARP    carp_vlan10     *            carp_vlan10     *              *                     *

2) Most tutorials I've seen online for CARP create a wide open rule but I don't like that approach because what if somebody plugs in the wrong cable after poweroff/disconnect and they put your WAN link into the pfsync port? What I do is make a rule that allows  SOURCE PFSYNC network to DESTINATION PFSYNC network on both sides and it has worked for me. Maybe somebody else has a suggestion as to a better rule?

Protocol    Source            Port    Destination     Port    Gateway    Schedule
IPv4*    PFSYNC net    *            PFSYNC net     *            *                     *

Disclaimer: I am not an expert and just writing my experience. Perhaps someone more knowledgeable can chime in?

Am I correct in assuming that if Switch 1 were to go down that Firewall 1 would detect a dead connection and demote itself and so Firewall 2 take over?

I have a very similar setup to yours and that is how mine behaves. I did not check the "disable preempt" option under High Availability Settings.

On my Unifi switches (which do not support advanced bonding functions), I just configured Switch 1 to have a higher priority over Switch 2 (higher priority on Unifi means settings the actual priority numerical of Switch 1 to a lower value than that of Switch 2). And for anyone else with Unifi switches - I had to set the CARP frequency in OPNsense to 2 instead of 1 for it to be stable.

Edit: Just to add clarification, what I mean to say is that I didn't need to configure any LAG settings on my switches for it to work in my HA configuration which is identical to yours (except I have Unifi switches and Multi-WAN).

I have not had luck with any of the suggestions above to fix the issue on Android.

I did not have this issue while running 23.1 (and prior releases) with Wireguard-go but If I installed Wireguard-kernel I did have the issue. Since upgrading to 23.7, I get the issue whether I use Wireguard-go or Wireguard-Kernel. I will probably start another thread if there isn't one already.

What are the best practices to avoid downtime caused by firmware upgrade of OPNsense?
My current setup is similar to 'Diagarm 1'. Is the solution sketched in 'Diagarm 2' good, or is there any better solution?

The first diagram you posted is just Multi-WAN, this does not provide firewall redundancy so your network will be offline while OPNsense updates.

The second diagram you posted is Multi-WAN with High Availability. Since that setup has two firewalls, you can update the second firewall, temporarily make it the primary while you test functionality and stability, and then when satisfied you can update the first firewall and return it to the active/primary state. In my experience this is the best option with OPNsense. The documentation outlines this procedure in the link below.

https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster

Here https://docs.opnsense.org/manual/how-tos/carp.html#setup-interfaces-basic-firewall-rules are some text that I don't understand:
Because we're connecting both firewalls using a direct cable connection, we will add a single rule to accept all traffic on all protocols for that specific interface. Another option is to only accept traffic to the GUI port and pfSync protocol.

What does this "single rule" refer to? In this case, is there no need for a rule on the WAN and LAN, but just this one? However, it's not clear to me what this one should look like.

Does the LAN interface even need a rule that allows CARP? By default it already has one that allows all traffic into the LAN.

Please explain in more detail what these rules should be.

The interface used for pfsync needs a rule to allow traffic between the two firewalls. So this rule would go under Firewall --> Rules --> PFSYNC (or whatever you named your pfsync interface)

   Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
    IPv4 *      PFSYNC net    *            PFSYNC net    *             *                    *       

The recommendation is to just have it allow everything like below. If you want to harden things up you can modify if after you have everything up and running and see if everything still works.

   Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
    IPv4 *      *                       *            *                   *             *                    *       

I am not sure if you need to explicitly define rules for CARP on the other interfaces. I will try some things and report back.

I have setup CARP with a single WAN IP address that is assigned via DHCP. The way I did it was to place an intermediate "router" between my OPNsense firewall and the ISP ONT. This intermediate router takes the public WAN and then creates a private range where you can have as many IP addresses as you want. So my OPNsense firewalls see 192.168.1.251 (FW1),192.168.1.252 (FW2), and 192.168.1.1 (CARP) as the three "WAN" IP addresses.

Even though it is technically a double NAT, it does not have the challenges of a traditional double NAT because the intermediate router is setup to port forward all incoming traffic on the real WAN to the CARP WAN 192.168.1.1 address.

I realize it is impractical to put another "router" in between the ONT and OPNsense firewall but I used EdgeRouter ER-X which is tiny and very low on power draw. Mikrotik also has similar small devices in their range, such as the Hex. Just make sure you get something with RouterOS and not SwitchOS.

It does add another point of failure in the chain but I run Multi-WAN to negate that. Even if you didn't have the intermediate router the ISP ONT/modem would still be a single point of failure.

This is the best way I could think of doing it and based on my experiences to date, I would recommend it.