Messages

Ok!

Working this morning after leaving it overnight.

Utterly no idea why. I rebooted multiple times both the remote and the main site firewall and yet I could not ping from main site to remote site the wg tunnel interface IP. Yet, after waiting overnight, it's working....


Very strange.

One site was working fine.

Main site:
WAN 202.202.202.202 (made up)
WG listens on 202.202.202.202 port 51820
tunnel IP for peer: 10.100.100.1/24

Remote site:
WAN DHCP / not static
tunnel IP for main site peer 10.100.100.2/24

It was all working fine:
I could ping 10.100.100.2 from main site...  all good!

Upgrade remote site 25.7.8... post reboot it came up, tunnel address answered about 5 pings and then gone.

Looks related to 27.7.8, main site and now remote sites all running 25.7.78 and multiple reboots but cannot ping wg tunnel addresses anymore.

• I have a bunch of firewalls I support which have dynamic IP.
• The main site (hub and spoke) has a static IP.
• The remote sites all make wg tunnels from remote to main site

.

I can no longer ping the remote site wg tunnel IP, and, I used to be able to. It's making a wg tunnel fine.
I can ping just fine any other wg VPN site to site tunnel IP where the other site has a static WAN IP.

Just wg to wg using the tunnel IP, it no longer works IF the remote side peer does not have a static WAN IP and port.

Error on attempting to remove IPv6 from LAN interface:

Code Select Expand

The following input errors were detected:

    The DHCPv6 Server is active on this interface and it can be used only with a static IPv6 configuration. Please disable the DHCPv6 Server service on this interface first, then change the interface configuration.
But it's not active.

I have disabled IPv6 completely. I use KEA for IPv4, no IPv6 on the WAN interface at all, and  "ISC DHCPv6" shows a lan interface with "Enable DHCPv6 server on LAN interface" not selected.

Kea DHCPv6 is disabled.
I have rebooted multiple times.

[Outbound static NAT]

I too run Plex.
I have two NAT's one for static NAT outbound and one for the incoming NAT (aka Port Forward).
I have a static WAN address assigned by my ISP.

Outbound static NAT

• I have my firewall NAT outbound in Hybrid mode.
  Firewall > NAT > Outbound
• I have added a manual static IP address for my Plex server (say 192.168.1.88)
• I have a static outbound NAT for Plex.

**** Like this ****
Interface: WAN
Source: 192.168.1.88 (my Plex server LAN IP address)
Destination: *
Destination Port: *
NAT Address: Interface Address
NAT Port: *
Static Port: YES
Description: Static NAT for Plex Out

Inbound port forward NAT

• Firewall > NAT > Port Forward
• I have added a port forward for me Plex server (say 192.168.1.88)

**** Like this ****
Interface: WAN
Proto: TCP
Address: *
Ports: *
Address: This Firewall
Ports: the static port number you set inside Plex, typically 32400
IP: 192.168.1.88 (my Plex server LAN IP address)
Ports: the static port number you set inside Plex, typically 32400, the same as the first "Ports"
NAT Address: Interface Address
Description: WAN to Plex IN


This works for me great! I do not used "One-to-One" NAT at all.

Renamed a "Network(s)" alias, didn't add or subtract any content, just renamed the alias.

Firewall rules in multiple interfaces using the alias failed to update to the new alias name thereby breaking the firewall rules.
Renamed a second alias and observed the same behavior.

[What is the Intel TCO Watchdog Timer]

I am running OPNsense on an ODroid-H4 and it works great. The H4 BIOS supports the The Intel TCO (Total Cost of Ownership) Watchdog Timer.

What is the Intel TCO Watchdog Timer
This is a hardware watchdog in the chipset and BIOS of the computer whose purpose is to reboot the computer if the system hangs. If the hardware watchdog does not receive a ping at a regular interval it will cause a hardware reset. To work, the software must run a software Watchdog daemon which regularly writes to the ACPI hardware tables. In this way, if the OS hangs then there is no update to the tables, the hardware watchdog sees this and reboots the computer.

I see there is support in FreeBSD for the Intel TCO Watchdog Timer, the "ichwd" driver.

The Intel TCO Watchdog Timer is not just ODROID-H4 specific, but many Intel based systems support this.


Feature Request
Add a package that enables support for the Intel TCO Watchdog Timer.

I image it would have a few simple settings that could be set:

• watchdog-timeout = 14
• realtime = yes

Plus the ability to:

• View watchdog logs

Reference
ODroid-H4 Intel TCO Watchdog on Linux/Ubuntu
FreeBSD - ichwd --   device driver for the Intel ICH   watchdog interrupt timer

[I had the same error - fixed by resetting DNS data]

I had the same error - fixed by resetting DNS data

• Reporting > Settings > Unbound DNS reporting "Reset DNS data"

Then do the upgrade again - worked flawlessly.

To close off this topic:

Since OPNsense 23.7.8 and beyond with the built-in support of WireGuard to follow a CARP VHID, this issue and others have all been solved.

There's no longer any need to run custom scripts etc and WireGuard now works very well indeed!

[Why so much trouble with stacked switches?]

My experience is that many switches do not handle CARP / VRRP correctly across "stacked" switches.

You can tell if you've got that problem, because, you get all sorts of weird  CARP issues occurring such as:

• MASTER or BACKUP firewall failing and disabling CARP due a a CARP problem
• CARP won't fail back
• CARP gets a huge demotion level number of 240 or 480 or higher
• Sometimes it works and sometimes it doesn't and you just can't figure out why
• You reboot the master or backup firewall and strange CARP issues occur
• Both the master and backup firewall report themselves as the CARP master (which should be impossible)

Check here to see the CARP demotion level number:
   Interfaces: Virtual IPs: Status

Ask the switch manufacturer if they properly support VRRP which is basically what CARP is. If you tell them your trying to run CARP they will say "What?????" and won't know what your talking about.

Even then, what you want is a switch stack that supports external VRRP / CARP, that is, not that the switch stack itself supports VRRP, but, that is supports another VRRP plugged into it. In our case firewall1 and firewall2 but it could be server1 and server2 share an IP address using VRRP (which is essentially CARP).

If they do support external VRRP then CARP will work.

Often I have found it simpler to have:

     Firewall1 > switch1
     Firewall2 > switch2
     Switch1 and switch2 cross connect as standalone switches and thus CARP behaves nicely.


Why so much trouble with stacked switches?
Because it's actually really complicated for the switch manufacturer. They have to maintain an ARP states across 2 switches and replicate certain things switch to switch and the whole time lie and pretend to be a single switch. CARP/VRRP complicates things because of the technical way it shares an IP address across two devices and the fact that VRRP / CARP are protocol 112 and not UDP (protocol number 17) and not TCP (protocol number 6).

I have found they work hard on getting stacked switches to work covering most stuff, but not things like VRRP / CARP.

I know I haven't cured the problem but hopefully my explanation helps it all make sense as to why sometimes you just can't get it to go.

[How to add GeoIP capability and run rules inside HAPROXY based on GeoIP]

Perhaps these instructions refer to a previous analysis, because I did not understand the need to establish paths that begin with /login/.
In any case, I don't see in the code where you check for /login/

What is the goal you aim to achieve?

No these instructions do not refer to previous analysis or set of instructions.

There is nothing in the code about /login/ - I am merely talking about the concept on using these instructions to satisfy a need to do something like:

• The URL requested starts with a path, say /login
• and the request is coming from a GeoIP of Australia
• Then allow the request, else block the request

In my concept above, you could then allow logins to a website only from Australia IP addresses.

The instruction are for:
How to add GeoIP capability and run rules inside HAPROXY based on GeoIP

Then, how to add the GeoIP capability for HAPROXY starts in my post:
*****************************************************
ONE: These two files need to get added to the system
*****************************************************
... and so on

You do need a high understanding of how HAPROXY works for any of this to make any sense.

[You can write conditions such as:]

Firstly, let me credit Brett Merrick for huge assistance. This is not all my own work.

Here's the steps to get GeoIP working inside HAPROXY, not at the firewall rule layer, but inside HAPROXY and still utilising OPNsense GeoIP alias function.

You can write conditions such as:
Condition: Paths starts with /login/
Condition: GeoIP matches Australia

Then write a rule that does things like:
Rule: Only permit login from Australia (Permit http_request if matches "Paths starts with /login/" and "GeoIP matches AU"

Very cool! Any many, many more possibilities for protection, reject excessive error rates or connection rates from certain countries, use Tarpitting on some countries and not others, and so on.

Ok - how:


*****************************************************
ONE: These two files need to get added to the system
*****************************************************

File1
filename: actions_custom.conf
location: /usr/local/opnsense/service/conf/actions.d/

Code Select Expand

[update]
command:/usr/local/opnsense/scripts/custom/haproxy-alias.sh
parameters:%s
type:script
message:Updating HAProxy Alias %s
description:Update HAProxy Alias

File2
filename: haproxy-alias.sh
location: /usr/local/opnsense/scripts/custom/

Code Select Expand

#!/bin/csh
if ( $#argv == 0 ) exit 1
configctl filter list table "$1" > "/var/haproxy/$1.lst"
chown 80:80 "/var/haproxy/$1.lst"
exit 0


************************************************************
TWO: Build the GeoIP alias "acl_geoip_au" (for say Australian IP addresses
************************************************************
If you haven't setup for the GeoIP downloads to get the GeoIP databnase list, then follow the OPNsense documentation first:
https://docs.opnsense.org/manual/how-tos/maxmind_geo_ip.html
Normal firewall GeoIP settings
Firewall > Aliases > GeoIP settings

*** make sure to only use ipv4 if you don't have ipv6 ***


*******************************************
THREE: Create the cron job and run it ONCE
*******************************************
Then set to run overnight after midnight sometime ideally running after GeoIP DB update and before HAPROXY reload. You will need an HAPROXY reload to pickup the new GeoIP tables.


************************
FOUR: Now setup a condition
************************
e.g.
Name: GEOIP_AU
Condition type: Source IP matches specified IP (from the drop down list)
Parameters: -f /var/haproxy/acl_geoip_au.lst

(see the acl_geoip_au.lst - that needs to match the firewall GeoIP alias name. In my example the alias name is acl_geoip_au)


*****************************************************************************
FIVE: Go make a rule that uses your condition and attach it a backend or frontend as appropriate
*****************************************************************************

Want more GeoIP ranges?
Start at step two and rinse and repeat for more aliases with different country combinations.
Each aliases needs a CRON job and don't forget you need to run the CRON job once to get the alias ready for HAPROXY to use.

Same issued here - fresh install on two new 23.7.6 firewalls

/var/log/crowdsec/crowdsec-firewall-bouncer.log:

Code Select Expand

time="17-10-2023 16:07:54" level=warning msg="unexpected ${BACKEND} mode"
time="17-10-2023 16:07:54" level=info msg="Starting crowdsec-firewall-bouncer v0.0.28-freebsd-af6e7e2"

[Important notes]

My updated script 10 Sep 2023 - now working excellently to enable super fast fail-over for WireGuard for clustered firewalls.

#!/usr/local/bin/php
<?php

require_once("config.inc");
require_once("util.inc");
require_once("interfaces.inc");

$subsystem = !empty($argv[1]) ? $argv[1] : '';
$type = !empty($argv[2]) ? $argv[2] : '';

if ($subsystem != "1@igb0") exit;

if (!in_array($type, ['MASTER', 'BACKUP'])) exit;

    switch ($type) {
        case 'MASTER':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard start");
         log_msg("Starting WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
        case 'BACKUP':
         shell_exec("/usr/local/sbin/pluginctl -s wireguard stop");
         log_msg("Stopping WireGuard due to CARP event '$type' on '{$subsystem}'");
            break;
    }

Important notes

**** ONE ****
You MUST change the line "if ($subsystem != "1@igb0") exit;" to follow an interface.
This is normally your LAN since that's where for most people the WireGuard tunnels will be tunneling VPN traffic to and from.

The "1" must be the VHID number and the "igb0" must equal the interface "Device" name.
If your interface is using a vlan, then it could look like this:

if ($subsystem != "2@vlan01") exit;

**** TWO ****

• Place the script in this location "/usr/local/etc/rc.syshook.d/carp/"
• Call the script "10-wireguard" - no extension
• Make sure the script has execute permissions

**** THREE ****
Make sure the WireGuard package is set to replicate to the backup firewall.

Code Select Expand

System > High Availability > Settings > WireGuard "selected"

**** FOUR ****
If you are using a dynamic routing protocol, I recommend the FRR package and setup BGP.
Then FRR does NOT need to follow CARP, it can remain on and alive on both the MASTER and the BACKUP firewall at the same time.

I've tested the above and it seems very good.

Also, I no longer need to have
System > High Availability > Settings > WireGuard "unselected"

In fact, I need that set as selected and it is looking very, very good.

• The Firewall "WireGuard (Group)" now consistently appears.
• Transition for a failover is the loss of a single ping.
• I have FRR running on the primary and the backup firewall, alive, NOT following CARP and I rely on the WireGuard tunnels going up and down according to CARP and this is working excellently.

Thanks Franco!