Messages

No, I don't think so.


#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "102@igc0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         $config['OPNsense']['wireguard']['general']['enabled'] = '1';
         write_config("Enable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         $config['OPNsense']['wireguard']['general']['enabled'] = '0';
         write_config("Disable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');

I'm wondering if this is because of clustered firewalls and CARP and my WireGuard CARP script.

The script does get fired for start and stop quickly during a CARP standup and I wonder if the mechanism that creates the Firewall rule group "WireGuard (Group)" gets a bit lost.

Did you have the allow rules at the remote end again the "WireGuard (Group)", perhaps that firewall rule group is now missing?

[VPN > WireGuard > Settings > Endpoint]

Am I missing something?

VPN > WireGuard > Settings > Endpoint
You can specify a "Shared Secret"

On the remote site, where this Endpoint connects to:
VPN > WireGuard > Settings > Local/b]
I cannot see any way to add the "Shared Secret"

Or am I missing something?

[What happens]

I have had a really good go at trying to figure out the logic of when and how but I wasn't able to deteremine what's going on. What I can say is on 90% of the firewalls, the "WireGuard (Group) firewall rule group is missing despite rebooting, stopping and starting WireGuard, etc.

This is happening across many different firewalls, different hardware, Hyper-V based VMs, clustered and not clustered. I realised the problem when I had one way traffic, because the remote end had allow firewall rules only on the "WireGuard (Group)" that disappeared so blocked all inbound wg tunnel traffic.

What happens
If you're quick enough in the GUI, you see the "WireGuard (Group)" firewall rule group appear and then after a while, disappear.

Work around
1. Add every wq interface inside:
VPN > WireGuard > Settings > Local
(e.g. wg1, wg2, wg3...)

2. Assign these as an interface in:
Interfaces > Assignments

3. Then create allow firewall rules on these individual firewall interfaces

4. If you are running clustered firewalls
You need to start WireGuard on the backup firewall to be able to also add the interfaces to the backup firewall.

Maybe you can use less CARP addresses too.

You can "stack" addresses onto a single CARP VHID using IP Alias for the same interface. I use this for WAN a lot.

Example:
WAN CARP = 202.202.202.202/24 with VHID 100

You can add additional WAN addresses onto the same interface:
WAN IP Alias = 202.202.202.203/24 with VHID 100
WAN IP Alias = 202.202.202.204/24 with VHID 100


See the attachment for an example I added to my LAN for you to see....

See https://docs.opnsense.org/manual/firewall_vip.html

[Why?]

Have a master "Enable OpenVPN" that writes the 20-openvpn script into /usr/local/etc/rc.syshook.d/carp/ directory, otherwise the script does not exist.

If you unselect "OpenVPN" then the CARP script /usr/local/etc/rc.syshook.d/carp/20-openvpn is removed.

Why?
Because the script gets called and runs again and again and again for CARP events of INIT, BACKUP and MASTER events even when your are not running OpenVPN.

[You MUST change the line]

Here's my updated script following a single interface. You normally need to follow LAN since that's where your Wireguard VPN tunnels tunnel to and from...

This works better, especially if you have many CARP interfaces, some of my customers have 7 CARP interfaces:
LAN, WAN, WAN2, DMZ1, DMZ2, VoIP, UNTRUSTED

The issue is each of the CARP transitions fire the script, in my above example the script will get called 7 times!
With the locking to a single inetrface (LAN) the script exits quickly.

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "1@igb0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         $config['OPNsense']['wireguard']['general']['enabled'] = '1';
         write_config("Enable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         $config['OPNsense']['wireguard']['general']['enabled'] = '0';
         write_config("Disable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');

You MUST change the line "if ($subsystem != "1@igb0") exit;"

The "1" must be the VHID number and the "igb0" must equal the interface "Device" name.
If your interface is using a vlan, then it could look like this:

if ($subsystem != "2@vlan01") exit;

Thanks!

Good to hear.

I hear you... but in OPNsense you can can a master Alias which contains the individual aliases.


Alias: Allowed_oVPN_out

That master Alias contains a lot of single Aliases:

Alias: John
Alias: Mary_iPad
Alias: Mary_laptop

Etc... so you can do this today.

[Add Wireguard CARP awareness to the GUI and follow a single interface]

With Wireguard now baked into the core with 23.7.3, my Wireguard custom CARP script broke. I enlisted the help of a friend and together we built a new Wireguard CARP fail-over script.

It works absolutely brilliantly and it's ready for production.
I note that now as the CARP script gets called multiple times as multiple VHID's transition, but it no longer affects Wireguard forcing it to start and stop and often break as was the case. As this new script is called multiple times during a CARP transition from backup master, it starts up Wireguard once and remains stable, each time it is called again it doesn't restart Wireguard which is very good.

Add Wireguard CARP awareness to the GUI and follow a single interface
It would be very nice though, if the GUI had an option "Enable CARP Fail-over" (like FRR)
but!
Also have a drop down where you select a single interface to follow

Why?

• Because despite it being "impossible" over the many years of pfSense and now OPNsense experience I have seen many instances where CARP is misaligned between the backup and the primary firewall.
• Also, I would almost always follow the LAN interface, where the Wireguard VPN tunnels exit to and begin from and that's really the only CARP interface that we would want to start or stop Wireguard to follow.
• The new follow a single interface CARP script will exit quickly if the CARP event is not for the required interface, less work for the firewall to process.

I know I have seen many posts here and on GitHub requesting CARP for Wireguard (including me) and questions raised as to why. I thought I would summarise why Wireguard needs to follow CARP.

Why Wireguard with HA needs to stop and start and follow CARP

• Wireguard doesn't bind to VHID's, it binds to all interfaces (like the WAN firewall interface) and therefore on a CARP fail-over, the backup firewall Wireguard keeps running and interfering with the primary firewall.
• The above statement is especially true with Wireguard keepalives
• Because of the above two points, you need to have under HA sync Wireguard "unselected" to keep Wireguard "off" on the backup firewall but then changes to the Wireguard config are not sync'd anymore.
• This enables FRR to be left on, even on the backup firewall, no need for dynamic routing to be off on the backup firewall (because with Wireguard off, no traffic can pass) and thus on fail-over everything comes up super fast, it's very good.

I think Wireguard is now "prime time ready".


Here is my script.
I fully acknowledge that this is not my exclusive work, but follows the built-in CARP scripts.

Place this script here:

Code Select Expand

/usr/local/etc/rc.syshook.d/carp
Ensure the Rights are execute
Call the script "10-wireguard"

----- Script Start -----
#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if (!in_array($type, ['MASTER', 'BACKUP'])) {
      log_msg("Carp '$type' event unknown from source '{$subsystem}'");
      exit;
   }

if (!strstr($subsystem, '@')) {
        log_msg("Carp '$type' event triggered from wrong source '{$subsystem}'");
        exit;
    }

    switch ($type) {
        case 'MASTER':
         $config['OPNsense']['wireguard']['general']['enabled'] = '1';
         write_config("Enable WireGuard on this peer due to CARP event", false);
         log_msg("Starting WireGuard due to CARP event '$type'");
            break;
        case 'BACKUP':
         $config['OPNsense']['wireguard']['general']['enabled'] = '0';
         write_config("Disable WireGuard on this peer due to CARP event", false);
         log_msg("Stopping WireGuard due to CARP event '$type'");
            break;
    }

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');
----- Script Stop -----

(I couldn't get the script with "code" selected to look nicely, it keep adding lots of junk to the script...)

Please note, my script above doesn't follow a single interface like LAN, but, it does work!

I can create this on Github if appropriate. Comments please.

Force of habit re HA.

With HA and a CARP VIP, I find it best to bind HAPROXY to localhost and have a NAT for the WAN CARP VIP fwd to localhost.

[*** Update! ****]

I've been running WG client on 443 UDP for quite some time. I have the OPNsense GUI moved to another port and HAPROXY running, but listening on 127.0.0.1:44443 with a NAT port forward to TCP 443 to the localhost 127.0.0.1:44443 so 443 UDP is definitely free.

This has been running fine, for quite some time. I use 443 UDP because some places lock down outbound traffic and since the introduction of HTTPS over UDP, I find WG often works and a traditional WG port of say 51820 does not.

Anyway, all working great on an iPad, iPhone and Win 11 laptop. I don't use the client WG VPN much, but I just noticed if I connect, I get exactly 92 B received and nothing works.

Rebooted OPNsense, start and stop WG, try different client (iPad, iPhone and PC) but it just doesn't work. Move WG on OPNsense to UDP 1194 (I know, that's really oVPN, but I'm not running oVPN), move the client port to 1194 and voilia, it all works again.

Something changed upgrading from 23.7 to 23.7.1_3 that broke WG listening on 443 UDP.

I'm waiting until 23.7.3 before I upgrade further, unless 23.7.2 is known to fix this.

*** Update! ****
I changed the GUI to only listen on the LAN interface

System > Settings > Administration > Listen Interfaces > LAN

And now I can have WG working on 443 UDP again.

It looks like 23.7.1_3 binds the GUI somehow to 443 UDP... not to sure how that works....

You could continue to those 8.8.8.8 & 8.8.4.4 IP addresses for your gateway monitor and for your DNS use something different such as:

• Your ISPsrecommended / provided DNS servers

or

• CloudFlare's excellent public DNS servers 1.1.1.1 & 1.0.0.1

Edit: But now having another issue. When clicking on the + (add a static mapping) next to line with the lease, I get send back to the "Lobby"

Try:
Clearing your browser cache
Or
Using another browser
Or
Using private browser mode

Just to check that your issue is not browser caching related.