Messages

Same issued here - fresh install on two new 23.7.6 firewalls

/var/log/crowdsec/crowdsec-firewall-bouncer.log:

Code Select Expand

time="17-10-2023 16:07:54" level=warning msg="unexpected ${BACKEND} mode"
time="17-10-2023 16:07:54" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"

[Important notes]

My updated script 10 Sep 2023 - now working excellently to enable super fast fail-over for WireGuard for clustered firewalls.

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "1@igb0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard start");
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard stop");
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

Important notes

**** ONE ****
You MUST change the line "if ($subsystem != "1@igb0") exit;" to follow an interface.
This is normally your LAN since that's where for most people the WireGuard tunnels will be tunneling VPN traffic to and from.

The "1" must be the VHID number and the "igb0" must equal the interface "Device" name.
If your interface is using a vlan, then it could look like this:

if ($subsystem != "2@vlan01") exit;

**** TWO ****

• Place the script in this location "/usr/local/etc/rc.syshook.d/carp/"
• Call the script "10-wireguard" - no extension
• Make sure the script has execute permissions

**** THREE ****
Make sure the WireGuard package is set to replicate to the backup firewall.

Code Select Expand

System > High Availability > Settings > WireGuard "selected"

**** FOUR ****
If you are using a dynamic routing protocol, I recommend the FRR package and setup BGP.
Then FRR does NOT need to follow CARP, it can remain on and alive on both the MASTER and the BACKUP firewall at the same time.

I've tested the above and it seems very good.

Also, I no longer need to have
System > High Availability > Settings > WireGuard "unselected"

In fact, I need that set as selected and it is looking very, very good.

• The Firewall "WireGuard (Group)" now consistently appears.
• Transition for a failover is the loss of a single ping.
• I have FRR running on the primary and the backup firewall, alive, NOT following CARP and I rely on the WireGuard tunnels going up and down according to CARP and this is working excellently.

Thanks Franco!

Thanks.

So on transition to BACKUP, don't write

config['OPNsense']['wireguard']['general']['enabled'] = '0';

But, then, the code:

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');

Well, that won't actually stop WireGuard anymore because it's "enabled"...

Will this command from be the better fit?
https://forum.opnsense.org/index.php?topic=35578.msg172989#msg172989

pluginctl -s wireguard stop

Therefore my CARP script would become:

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "1@igb1") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard start");
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard stop");
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

Thanks.

Thanks, sounds good!

And, I've learnt a little more too.

[Naming consistency]

ok!

Thanks very much Patrick M. Hausen for the explanation.

So... peer side at each end is where you use a PSK, like this:
peer / endpoint for Site A > B
   and
peer / endpoint for Site B > A

But not the local "server" settings, because that's not really a server at all, it's actually just a wg interface.

Naming consistency
"Local" should be renamed to "Interface"
"Endpoints" should be renamed to "Peer"

This is in keeping with Wireguard terminology.
Reference: https://www.wireguard.com/#simple-network-interface

No, I don't think so.


#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "102@igc0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         $config['OPNsense']['wireguard']['general']['enabled'] = '1';
         write_config("Enable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         $config['OPNsense']['wireguard']['general']['enabled'] = '0';
         write_config("Disable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');

I'm wondering if this is because of clustered firewalls and CARP and my WireGuard CARP script.

The script does get fired for start and stop quickly during a CARP standup and I wonder if the mechanism that creates the Firewall rule group "WireGuard (Group)" gets a bit lost.

Did you have the allow rules at the remote end again the "WireGuard (Group)", perhaps that firewall rule group is now missing?

[VPN > WireGuard > Settings > Endpoint]

Am I missing something?

VPN > WireGuard > Settings > Endpoint
You can specify a "Shared Secret"

On the remote site, where this Endpoint connects to:
VPN > WireGuard > Settings > Local/b]
I cannot see any way to add the "Shared Secret"

Or am I missing something?

[What happens]

I have had a really good go at trying to figure out the logic of when and how but I wasn't able to deteremine what's going on. What I can say is on 90% of the firewalls, the "WireGuard (Group) firewall rule group is missing despite rebooting, stopping and starting WireGuard, etc.

This is happening across many different firewalls, different hardware, Hyper-V based VMs, clustered and not clustered. I realised the problem when I had one way traffic, because the remote end had allow firewall rules only on the "WireGuard (Group)" that disappeared so blocked all inbound wg tunnel traffic.

What happens
If you're quick enough in the GUI, you see the "WireGuard (Group)" firewall rule group appear and then after a while, disappear.

Work around
1. Add every wq interface inside:
VPN > WireGuard > Settings > Local
(e.g. wg1, wg2, wg3...)

2. Assign these as an interface in:
Interfaces > Assignments

3. Then create allow firewall rules on these individual firewall interfaces

4. If you are running clustered firewalls
You need to start WireGuard on the backup firewall to be able to also add the interfaces to the backup firewall.

Maybe you can use less CARP addresses too.

You can "stack" addresses onto a single CARP VHID using IP Alias for the same interface. I use this for WAN a lot.

Example:
WAN CARP = 202.202.202.202/24 with VHID 100

You can add additional WAN addresses onto the same interface:
WAN IP Alias = 202.202.202.203/24 with VHID 100
WAN IP Alias = 202.202.202.204/24 with VHID 100


See the attachment for an example I added to my LAN for you to see....

See https://docs.opnsense.org/manual/firewall_vip.html

[Why?]

Have a master "Enable OpenVPN" that writes the 20-openvpn script into /usr/local/etc/rc.syshook.d/carp/ directory, otherwise the script does not exist.

If you unselect "OpenVPN" then the CARP script /usr/local/etc/rc.syshook.d/carp/20-openvpn is removed.

Why?
Because the script gets called and runs again and again and again for CARP events of INIT, BACKUP and MASTER events even when your are not running OpenVPN.

[You MUST change the line]

Here's my updated script following a single interface. You normally need to follow LAN since that's where your Wireguard VPN tunnels tunnel to and from...

This works better, especially if you have many CARP interfaces, some of my customers have 7 CARP interfaces:
LAN, WAN, WAN2, DMZ1, DMZ2, VoIP, UNTRUSTED

The issue is each of the CARP transitions fire the script, in my above example the script will get called 7 times!
With the locking to a single inetrface (LAN) the script exits quickly.

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "1@igb0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         $config['OPNsense']['wireguard']['general']['enabled'] = '1';
         write_config("Enable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         $config['OPNsense']['wireguard']['general']['enabled'] = '0';
         write_config("Disable WireGuard due to CARP event on '{$subsystem}'", false);
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');

You MUST change the line "if ($subsystem != "1@igb0") exit;"

The "1" must be the VHID number and the "igb0" must equal the interface "Device" name.
If your interface is using a vlan, then it could look like this:

if ($subsystem != "2@vlan01") exit;

Thanks!

Good to hear.