I have had a really good go at trying to figure out the logic of when and how but I wasn't able to deteremine what's going on. What I can say is on 90% of the firewalls, the "WireGuard (Group) firewall rule group is missing despite rebooting, stopping and starting WireGuard, etc.
This is happening across many different firewalls, different hardware, Hyper-V based VMs, clustered and not clustered. I realised the problem when I had one way traffic, because the remote end had allow firewall rules only on the "WireGuard (Group)" that disappeared so blocked all inbound wg tunnel traffic.
What happens
If you're quick enough in the GUI, you see the "WireGuard (Group)" firewall rule group appear and then after a while, disappear.
Work around
1. Add every wq interface inside:
VPN > WireGuard > Settings > Local
(e.g. wg1, wg2, wg3...)
2. Assign these as an interface in:
Interfaces > Assignments
3. Then create allow firewall rules on these individual firewall interfaces
4. If you are running clustered firewalls
You need to start WireGuard on the backup firewall to be able to also add the interfaces to the backup firewall.
I'm wondering if this is because of clustered firewalls and CARP and my WireGuard CARP script.
The script does get fired for start and stop quickly during a CARP standup and I wonder if the mechanism that creates the Firewall rule group "WireGuard (Group)" gets a bit lost.
Is your script disabling WireGuard interfaces in the config.xml?
Cheers,
Franco
No, I don't think so.
#!/usr/local/bin/php
<?php
require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");
$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';
if ($subsystem != "102@igc0") exit;
if (!in_array($type, ['MASTER', 'BACKUP'])) exit;
switch ($type) {
case 'MASTER':
$config['OPNsense']['wireguard']['general']['enabled'] = '1';
write_config("Enable WireGuard due to CARP event on '{$subsystem}'", false);
log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
break;
case 'BACKUP':
$config['OPNsense']['wireguard']['general']['enabled'] = '0';
write_config("Disable WireGuard due to CARP event on '{$subsystem}'", false);
log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
break;
}
use OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');
> $config['OPNsense']['wireguard']['general']['enabled'] = '0';
Yeah... don't do this.
Cheers,
Franco
Thanks.
So on transition to BACKUP, don't write
Quoteconfig['OPNsense']['wireguard']['general']['enabled'] = '0';
But, then, the code:
Quoteuse OPNsense\Core\Backend;
$backend = new Backend();
$backend->configdRun('template reload OPNsense/Wireguard');
$backend->configdpRun('wireguard configure');
Well, that won't actually stop WireGuard anymore because it's "enabled"...
Will this command from be the better fit?
https://forum.opnsense.org/index.php?topic=35578.msg172989#msg172989 (https://forum.opnsense.org/index.php?topic=35578.msg172989#msg172989)
Quotepluginctl -s wireguard stop
Therefore my CARP script would become:
Quote#!/usr/local/bin/php
<?php
require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");
$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';
if ($subsystem != "1@igb1") exit;
if (!in_array($type, ['MASTER', 'BACKUP'])) exit;
switch ($type) {
case 'MASTER':
shell_exec("/usr/local/sbin/pluginctl -s wireguard start");
log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
break;
case 'BACKUP':
shell_exec("/usr/local/sbin/pluginctl -s wireguard stop");
log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
break;
}
Thanks.
I've tested the above and it seems very good.
Also, I no longer need to have
System > High Availability > Settings > WireGuard "unselected"
In fact, I need that set as selected and it is looking very, very good.
- The Firewall "WireGuard (Group)" now consistently appears.
- Transition for a failover is the loss of a single ping.
- I have FRR running on the primary and the backup firewall, alive, NOT following CARP and I rely on the WireGuard tunnels going up and down according to CARP and this is working excellently.
Thanks Franco!
Happy it works. FWIW, https://github.com/opnsense/plugins/issues/3579 will hit 24.1 or 23.7.x depending on when it is being rolled out.
Cheers,
Franco
To close off this topic:
Since OPNsense 23.7.8 and beyond with the built-in support of WireGuard to follow a CARP VHID, this issue and others have all been solved.
There's no longer any need to run custom scripts etc and WireGuard now works very well indeed!