CARP and WireGuard

[em.tie]

Dear All,

I have "upgraded" my single opnsense box to 2 opnsense boxes. Before upgrading I had wireguard running as VPN solution. With my 2 new boxes which are running in CARP / HA Mode wireguard seems not to work although I have followed the documentation. I have read a bit and found out, that in ha mode wireguard does not work. Is there a way to get wireguard working even it does not work with ha?

For clarification I have the following configuration / rules / etc.

WAN Router -> Fritzbox 192.168.1.1
opnsense1 (WAN) -> 192.168.1.10
opnsense 2(WAN) -> 192.168.1.11
opnsense carp (WAN) -> 192.168.1.20

internet -> WAN Router (fritzbox) with Portforwarding to opnsense carp -> opnsense 1 +2 -> LAN / ...

Firewall Rule on WAN Interface:

Interface - WAN
Direction - in
TCP/IP Version - IPv4
Protocol - UDP
Source - any
destination - WAN Address                                 ### is this right? should it be carp address?
destination port - WireGuard Server Port

Firewall outbound NAT

Interface - WAN
tCP / IP - IPv4
Protocol - any
source - WGUA Network        ### this is the created interface for wireguard according to the documentatino
source port - any
destination - any
destination port - any
Translation - CARP IP WAN Interface

WireGuard on opnsense 2 (backup box) -> Disabled


Thanks for any hints!!!

cu em.tie

[spali]

As I investigated a bit of time for getting DHCP WAN with single lease working (no CARP on WAN, only on LAN).
I can share some thoughts.
You could put a script in /usr/local/etc/rc.syshook.d/carp/ to stop wireguard service in BACKUP mode and start it in MASTER mode.

checkout the 20-openvpn script there.
Especially the part:

Code Select Expand


$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';
$type will either contain "MASTER" or "BACKUP".

Not sure, but I think the script does explicitly reconfigure the openvpn interface configuration. But in case of Wireguard, assuming you have a working wireguard setup, I think just stopping or starting the service should work to prevent connections from the BACKUP, due the fact that wireguard is listening on all interfaces by default.

You can also checkout my script for ideas, which does in fact disable WAN interface on BACKUP and enable it if it get's the master:
https://gist.github.com/spali/2da4f23e488219504b2ada12ac59a7dc

[berndtfromat]

Hi em.tie,

It took a while for me to set up a ha cluster with automatic failover in regards of wireguard, too...

fw rule and nat rule is correct, you should use your wan carp ip
I guess the following is missing on your setup:

change the wireguard implementation from go to kmod:
ssh into the shell:

Code Select Expand

pkg install wireguard-kmod
after a reboot it will be used instead of wireguard-go. "The wireguard-go service will show as stopped since the go implementation isn't being used, due to the kernel module, OPNsense will fix this in a later release."

you can sync wireguard settings via ha and wireguard seems to be fine running/enabled on both (!) nodes with the same (!) tunnel address.
When one node goes down carp ip will be switched and after a few (...) seconds the wireguard tunnels terminate on the new node.
Prerequisite is really a perfectly running carp setup, for example my isp modem blocks carp multicasts by default, so in my case my provider had to activate this to get things running.

BR
Berndt

[Patrick M. Hausen]

That's fantastic news, thanks for the report. I am going to try this in the next couple of days.

[bimbar]

This has been working for me for quite some time. wireguard-kmod handles this much better than wireguard-go.

[jrenken]

I had some trouble with two OPNsense nodes' WireGuard handshakes interfering with one another, even with wireguard-kmod, so I wrote a syshook based on @spali's suggestion: https://gist.github.com/jprenken/18ca7bf14ddae547ae0fdf6f56d72573

[taxilian]

The issue you run into with wireguard enabled on both is that if anything tries to use the wireguard interface on the backup -- or if you have keepalive on anything -- it will do a new handshake and mess up the vpn connection from the other one. You end up with the two routers fighting for control of any open wireguard connections.

In some cases that may not matter, but in many cases it does.

@jrenken awesome on this hook! I've been looking for something like this for awhile, but didn't know how to do it myself. I'll give it a try.

[toxic]

change the wireguard implementation from go to kmod:
ssh into the shell:

Code Select Expand

pkg install wireguard-kmod

That should really be added to the documentation, and make it possible to install this kmod from the GUI...
I was thinking as I can't find the package this thread must be obsolete but this solved my issue I had been working on for days prior to finding this kmod version. Now it's working like a charm!
Didn't realize carp would cause an issue.