Messages

Historically wg-quick would do exactly that too. It can be disabled. It depends on what you want to achieve.


Cheers,
Franco

Is there an additional setting beyond the Wireguard Instance's Disable Routes checkbox? Checking that box and setting peer Allowed IPs to 0.0.0.0/0 fixed the problem of my static routes from System --> Routes being overwritten... but the issue of BGP learned routes being overwritten/removed still happens when the wireguard interface is restarted.

Basically looking to configure it so that wireguard does not touch the routing table so those BGP learned routes don't disappear (which requires FRR to be stopped and started to be re-added).

Thanks :)

Seems to be a longstanding issue that is not isolated to opnsense:

https://redmine.pfsense.org/issues/14668

https://redmine.pfsense.org/issues/14511

Per one of the comments in the issue I also believe disabling routes on the wg interface and setting the allowed IPs on the peer to 0.0.0.0/0 is the best approach since that assures that modifications will not be required later (i.e. added subnet for site so having to update and reload tunnel which will remove the routes per thread comment).

Restarting the wireguard tunnel interface also produces this behavior. After restarting (main menu reload button next to the wg interface) the static route from the Routes menu and the routes from BGP disappear. Need to click re-apply in the routes menu on opnsense and then stop and start FRR for the routes to get written again. Wireguard doesn't actually overwrite it but rather just removes it from the routing table.

Edit1: Based on this old bug report this would appear to be expected but I don't understand how it is bad design? how else would we do dynamic routing over redundant tunnels? https://redmine.pfsense.org/issues/11326 OpenVPN doesn't have this issue since using client specific overrides only adds iroutes and not kernel routes. Maybe the issue isn't wireguard but rather that when the wireguard interface is reloaded it removes any routes added by BGP which is what I'm seeing (the routes are in the BGP routes but not seen in the kernel routes).

Edit2: Could do an outbound SNAT on the wireguard interface src any  dest any NAT'ed it to the interface IP but then that would remove the ability to define granular firewall rules on the opposite rules as it would look like everything originates from the tunnel address.

Edit3: Is this expected? should I create a bug report? would it be an opnsense or frr bug?

Modifying or adding wireguard peers with a modified allowed IPs field causes some static routes configured in the GUI (or obtained via BGP) to disappear? I've already checked the box to disable adding routes in the wireguard instance but that doesn't seem to change the behavior of the peer setting? Has anyone been able to get around this? It only seems to happen when modifying a peer that has a conflicting route.

I don't need wireguard to install this route but wireguard still needs it since the allowed ips is also a form of authentication for wireguard to allow traffic to pass over its own interface. Anyone have any experience or workarounds?

For example I have a remote site 10.20.0.0/16 which is routed via BGP over a peering network that goes over the Wireguard tunnel. If I modify the peer of that wireguard tunnel containing any overlapping subnet it will delete the routes learned via BGP or static routes... and wireguard needs to have the allowed IP set (for example to 10.20.0.0/16) to permit traffic over the interface even though the route is not required since thats learned through BGP and the wireguard tunnel interface is a gateway... I have to re-apply any static routes (like to the peering instance) and stop and restart FRR.

I suspect the cause is that routes are added before the wireguard site-to-site tunnel is ready.

Actually now I believe the cause is something to do with the failed node giving an improper route because this does not happen when FRR is set to only be active on the master node and even then wireguard tunnel usually takes a minute to come back up.

Edit: Now I believe the issue may have been because I had an allowed IPs subnet in my Wireguard peer (from the site to site tunnel) that overlapped with the learned routes in FRR/BGP... Haven't had the chance to test but if anyone is doing something similar and finds this thread, try that. I will report back when I test.

Edit2: I believe it is related to this https://forum.opnsense.org/index.php?topic=50022.msg254431#msg254431 which also happens on active/backup FRR configuration

I recently setup iBGP for some internal services so I thought I would attempt this with eBGP between my two HA opnsense nodes. In the end I was able to achieve this with active/active BGP on each router (each having a unique router-id). However there seems to be an issue (bug? expected?) during failover and/or maintenance mode (mainly happen when one router is put into maintenance mode but not always) where an erroneous route is installed which not only breaks routing between sites but sends the traffic out of the WAN interface. The only way to fix it is to stop FRR and start it (restarting does not fix it).  I suspect the cause is that routes are added before the wireguard site-to-site tunnel is ready.

This is the output in the FRR routing table. The second entry is supposed to be the site-to-site wireguard interface with it's tunnel address.

CODE NETWORK ADMIN DISTANCE METRIC INTERFACE INTERFACE_NAME VIA
B>* 10.20.10.0/24 20 0 <blank> <blank> 192.168.20.251
B>* 10.20.10.0/24 20 0 igb1 wan01 <WAN-IP>

After multiple steps to troubleshoot I gave up and figured that the potential for unexpected behavior during failover/maintenance was not worth it and eventually reverted back. Active/backup BGP does not solve it since the FRR daemon does not run on the backup I cannot reach the services on the site like I originally sought out to do.

Any pointers on what such a rule would look like and how to get started? I'm not familiar with dynamic routing and it sounds like quite a lift when my main purpose is to just get logging/backups from the firewalls to opposite sites.

If IPSEC supports binding to a CARP VIP, would it just be simpler to move my site to site tunnels from wireguard to IPSEC?

I have two HA/CARP opnsense setups connected through a site to site Wireguard VPN. Since the Wireguard tunnel is only up on the active firewall, I cannot access the tunnel from the secondary/backup node. If I try to initiate a ping from the backup node at one site to the primary node at the other site, it does not work (ping: sendto: No route to host).

What is the best way to address this? Last I checked we couldn't assign a virtual IP to a wireguard interface.

I have an outbound rule so that VPN clients can access the backup node but I'm not sure how to go about it when the traffic is initiated at the backup node itself.

Hello all,

I am about to embark on clustering two OPNSense nodes and I have a couple questions:

1) I am going to have 3 internal interfaces and a Wireguard interface. Do I need to add a CARP firewall rule to each interface?
2) For the heartbeat network should the firewall rule be wide open or limited to the heartbeat network.

Thanks,
Steve

1) I believe the CARP firewall rule is automatically added and doesn't require manual rule creation. If you did want to make one though you could make a rule like the one below for every interface? Not sure about Wireguard but mine has been working without a custom rule.

Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
IPv4 CARP    carp_vlan10     *            carp_vlan10     *              *                     *

2) Most tutorials I've seen online for CARP create a wide open rule but I don't like that approach because what if somebody plugs in the wrong cable after poweroff/disconnect and they put your WAN link into the pfsync port? What I do is make a rule that allows  SOURCE PFSYNC network to DESTINATION PFSYNC network on both sides and it has worked for me. Maybe somebody else has a suggestion as to a better rule?

Protocol    Source            Port    Destination     Port    Gateway    Schedule
IPv4*    PFSYNC net    *            PFSYNC net     *            *                     *

Disclaimer: I am not an expert and just writing my experience. Perhaps someone more knowledgeable can chime in?

Am I correct in assuming that if Switch 1 were to go down that Firewall 1 would detect a dead connection and demote itself and so Firewall 2 take over?

I have a very similar setup to yours and that is how mine behaves. I did not check the "disable preempt" option under High Availability Settings.

On my Unifi switches (which do not support advanced bonding functions), I just configured Switch 1 to have a higher priority over Switch 2 (higher priority on Unifi means settings the actual priority numerical of Switch 1 to a lower value than that of Switch 2). And for anyone else with Unifi switches - I had to set the CARP frequency in OPNsense to 2 instead of 1 for it to be stable.

Edit: Just to add clarification, what I mean to say is that I didn't need to configure any LAG settings on my switches for it to work in my HA configuration which is identical to yours (except I have Unifi switches and Multi-WAN).

I have not had luck with any of the suggestions above to fix the issue on Android.

I did not have this issue while running 23.1 (and prior releases) with Wireguard-go but If I installed Wireguard-kernel I did have the issue. Since upgrading to 23.7, I get the issue whether I use Wireguard-go or Wireguard-Kernel. I will probably start another thread if there isn't one already.

What are the best practices to avoid downtime caused by firmware upgrade of OPNsense?
My current setup is similar to 'Diagarm 1'. Is the solution sketched in 'Diagarm 2' good, or is there any better solution?

The first diagram you posted is just Multi-WAN, this does not provide firewall redundancy so your network will be offline while OPNsense updates.

The second diagram you posted is Multi-WAN with High Availability. Since that setup has two firewalls, you can update the second firewall, temporarily make it the primary while you test functionality and stability, and then when satisfied you can update the first firewall and return it to the active/primary state. In my experience this is the best option with OPNsense. The documentation outlines this procedure in the link below.

https://docs.opnsense.org/manual/how-tos/carp.html#example-updating-a-carp-ha-cluster

Here https://docs.opnsense.org/manual/how-tos/carp.html#setup-interfaces-basic-firewall-rules are some text that I don't understand:
Because we're connecting both firewalls using a direct cable connection, we will add a single rule to accept all traffic on all protocols for that specific interface. Another option is to only accept traffic to the GUI port and pfSync protocol.

What does this "single rule" refer to? In this case, is there no need for a rule on the WAN and LAN, but just this one? However, it's not clear to me what this one should look like.

Does the LAN interface even need a rule that allows CARP? By default it already has one that allows all traffic into the LAN.

Please explain in more detail what these rules should be.

The interface used for pfsync needs a rule to allow traffic between the two firewalls. So this rule would go under Firewall --> Rules --> PFSYNC (or whatever you named your pfsync interface)

   Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
    IPv4 *      PFSYNC net    *            PFSYNC net    *             *                    *       

The recommendation is to just have it allow everything like below. If you want to harden things up you can modify if after you have everything up and running and see if everything still works.

   Protocol    Source            Port    Destination    Port    Gateway    Schedule       Description    
    IPv4 *      *                       *            *                   *             *                    *       

I am not sure if you need to explicitly define rules for CARP on the other interfaces. I will try some things and report back.

I have setup CARP with a single WAN IP address that is assigned via DHCP. The way I did it was to place an intermediate "router" between my OPNsense firewall and the ISP ONT. This intermediate router takes the public WAN and then creates a private range where you can have as many IP addresses as you want. So my OPNsense firewalls see 192.168.1.251 (FW1),192.168.1.252 (FW2), and 192.168.1.1 (CARP) as the three "WAN" IP addresses.

Even though it is technically a double NAT, it does not have the challenges of a traditional double NAT because the intermediate router is setup to port forward all incoming traffic on the real WAN to the CARP WAN 192.168.1.1 address.

I realize it is impractical to put another "router" in between the ONT and OPNsense firewall but I used EdgeRouter ER-X which is tiny and very low on power draw. Mikrotik also has similar small devices in their range, such as the Hex. Just make sure you get something with RouterOS and not SwitchOS.

It does add another point of failure in the chain but I run Multi-WAN to negate that. Even if you didn't have the intermediate router the ISP ONT/modem would still be a single point of failure.

This is the best way I could think of doing it and based on my experiences to date, I would recommend it.