Messages

Correct.

I mean that it is probably the CPU waiting for I/O operations to finish because the storage subsystem is struggling. With NVMe storage, writing very small, frequent log lines leads to heavy 'write amplification' due to the large underlying flash block sizes, causing significant latency spikes.

The fact that reducing the logging activity resolved the issue completely confirms the storage bottleneck theory. When the logging was set to high, the frequent, small writes to the NVMe caused massive write amplification and continuous synchronous I/O blocks.

In FreeBSD/OPNsense, such a severe storage backlog causes 'Lock Contention' in the kernel and forces an influx of Software Interrupts (SWIs) to manage the blocked I/O buffers. This is why the IRQ handler threads (intr) ran hot: they weren't just processing network packets, but handling the massive overhead of a stalling filesystem trying to commit log rows to a struggling SSD. Turning down the logs broke this vicious cycle.

I would guess this really is more of a storage limitation because of RMW-cycles induced by ZFS.

That CPU is not the fastest, either. For each RMW cycle, it must calculate ZFS checksums. A proper centralized log server could probably buffer log messages more efficiently. If you have Monit running on top, each new message is also parsed and compared against triggers. You could use top and see which processes actually peg the CPU.

It only works for one specific monitoring IP, yes, not multiple destinations per gateway. There is no dynamic routing depending on "what is best".

On the other hand, that would also break your connections, because the routes (and IPs) would change. Normally, you would want "Failover States" to be enabled, too. If you want some destinations to use another (group of) interface(s), you can do PBR based on that.

You can use gateway groups for that and use specific trigger levels - it is in the official docs.

A system like that can be had from Quotom (Q20322G9) with passive cooling, plus it has 4x SFP+ ports on top of 5x 2.5 GbE ports instaed of 5x 1 GbE.

Das ist nicht wahr: Ich sehe "Hetzner DNS" und "Hetzner DNS Legacy (deprecated)" als Service in der GUI angeboten (26.1.11_5).

Do you guys use os-realtek-re or the native FreeBSD driver? You know that Realtek adapters are buggy under FreeBSD, don't you?

Where do you want to migrate the destination NAT rules to?

The only reason to migrate "Outbound NAT" to "SNAT" (aka "source NAT") was that this is the more common term. "Destination NAT" is already called correctly.

[not]

AFAIK, MacOS does requests in parallel, Windows does a fallback and goes back to preferred order. Unbound uses a sophisticated model preferring the fastest server - which is what many clients also do via the "Happy Eyeballs" approach. Whatever the case, this scheme does stop eventually once you factor in IPv6: Because you can only supply DNS server IPs of "your own kind" in any type of IP protocol, you have independent settings. So, which one will be preferred in the presence of both IPv4 and IPv6 DNS servers?

Before you say: "IPv6 is always preferred" - just today I had a case where I had augmented a DynDNS entry for a CG-NAT connection with a fixed IPv4, which has a reverse proxy to make the IPv6-only connection work for IPv4, too. Thus, the DynDNS name now refers to both an IPv4 and an IPv6.

And guess what? That broke a Wireguard PC client, who just happens to prefer IPv4 and thus did not reach the real IPv6 backend, but the IPv4 proxy (which did not have a Wireguard service). So there,

What can we learn from this? Patrick is right: Do not rely on DNS server ordering for ANY client. RFC 2132 is not helpful, either, because it only covers what an IPv4 server SHOULD do - i.e., it does not say:

a. what the server MUST do.
b. what the client SHOULD or MUST do.
c. what the client SHOULD or MUST do when an additional IPv6 DNS server list is provided.

[This setup only handles IPv4!]

There are lots of (mostly incomplete) instructions on how to setup NordVPN on OpnSense scattered all over the forum. Also, there are instructions from NordVPN themselves, but they do not focus on Wireguard.

Also, I found that my setup failed a few days ago, because NordVPN obviously cancelled one of their rented servers (de1019.nordvpn.com) which happened to be the one I used until then.

Therefore, I took the effort to create a setup with a failover for a fixed set of clients. This setup also will behave as a "kill switch", i.e. that no client targeted by it can bypass the NordVPN setup and expose itself accidentally.

1. You have to create a so-called "Access Token" in the NordVPN customer dashboard by clicking on NordVPN on the left. Then, the link to enter the access token menu will be at the bottom of the page. Create a token and be sure to note it, because it will not be accessible later on.

2. Once you have this token, you will need to create a private key from it. This can be done via 'curl -s -u token:<YOUR_TOKEN_HERE> https://api.nordvpn.com/v1/users/services/credentials | jq -r .nordlynx_private_key'.

3. For entry into WG instances, you will also need the corresponding public key, which you can get via 'echo <YOUR_PRIVATE_KEY_HERE> | wg pubkey'.

4. If you followed along, you should now be in possession of a private/public key pair pertaining to your NordVPN account.

5. Using this information, you can now add two Wireguard instances under VPN:Wireguard->Instances. Add these as "NORDVPN_INSTANCE1" and "NORDVPN_INSTANCE2", use different, free "Listen Ports" (these can be arbitrary, but must not be used anywhere else) and 10.5.0.2/16 as "Tunnel address" (yes, both are the same!). Be sure not to mix up the private and public keys. Both instances can, but do not strictly have to use the same key pair, should you have access to another one.

6. Obtain two NordVPN server names from their API via 'curl -s "https://api.nordvpn.com/v1/servers/recommendations?&filters\[servers_technologies\]\[identifier\]=wireguard_udp&limit=2" | jq -r ".[]|.hostname"'. This will give you two names like "xx999.nordvpn.com" for servers that are near you.

7. Next, create two Wireguard peers under VPN:Wireguard->Peers. Name them NORDVPN_PEER1 and NORDVPN_PEER2, and choose different NordVPN server names obtained in the previous step for both peers (e.g. 'deXXX.nordvpn.com'). Use "public key" = "m0tej5P6pYfBivkJc8yRV4KqQXmM81AChLlzlsOSjSs=" (this happens to be the public key for all NordVPN servers), "Allowed IPs" = "0.0.0.0/0", "Endpoint port" = 51820, "Keepalive Interval" = 30. Choose NORDVPN_INSTANCE1 as Wireguard instance for NORDVPN_PEER1 and NORDVPN_INSTANCE2 for NORDVPN_PEER2 - use only one per each peer!

8. Next, you can check VPN:WireGuard:Status. You should see both the peer and the instance come up as green for both instance/peer pairs if Wireguard is enabled.

9. Now, assign those two Wireguard interfaces names via Interfaces:Assignments. Name them NORDVPN1 and NORDVPN2. Select each of them under Interfaces and check the "Enable Interface" and "Prevent Interface Removal" boxes.

10. Create an alias with the definition of all clients, that should go through NordVPN only. Name that NORDVPN_CLIENTS. I prefer to use a network group alias that includes MAC aliases for individual machines, see also the hint about IPv6 at the end of this article.

11. Create two SNAT rules for each of the two NordVPN interfaces under Firewall:NAT:Source NAT with:

Code Select Expand

Description = "NAT for NordVPN"
Interface = NORDVPN1 / NORDVPN2
Version = IPv4
Protocol = any
Source Address = NORDVPN_CLIENTS
Translate Source IP = leave empty
12. Create two Gateways under System:Gateways:Configuration with

Code Select Expand

Name = NORDVPN_GW1
Interface = NORDVPN1
Address Family = IPv4
IP Address = 10.5.0.1
Far Gateway = checked
Failover States = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.96.100
Description = IPv4 Gateway for NordVPN


Code Select Expand

Name = NORDVPN_GW2
Interface = NORDVPN2
Address Family = IPv4
IP Address = 103.86.99.100
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.99.100
Description = IPv4 Gateway for NordVPN

The "IP address" of 103.86.99.100 in the second entry is not an error. It is a workaround because you cannot use the same gateway IP twice. WireGuard does not actually use this IP for routing.
Leave everything else on default.

13. Create a gateway group under System:Gateways:Group named NORDVPN_FAILOVER with "Trigger Level" = "Member Down", having NORDVPN_GW1 as Tier 1 and NordVPN_GW2 as Tier 2, with anything else as tier "Never". After having them created, both gateways should be listed as active (green) here.

14. Finally, create a firewall rule under Firewall:Rules[new] (it should precede most other rules):

Code Select Expand

Description = Force gateway for NordVPN clients
Interface = any
Quick = checked
Action = Pass
Direction = In
Version = IPv4
Protocol = any
Invert Source = unchecked
Source = NORDVPN_CLIENTS
Source Port = any
Invert Destination = checked
Destination = RFC1918
Destination Port = any
Gateway = NORDVPN_FAILOVER

This assumes that you have an RFC1918 alias. You could also use an inverted firewall group containing all of your local networks. What this is supposed to mean is "internet traffic", with the exception of "local traffic", should you want to access your NORDVPN_CLIENTS from any of your local networks. By explicitely setting a gateway for this rule, you effectively implement policy-based routing (PBR).



If you want to try the failover, disable the first Wireguard peer. You will see the gateway failing and after a few seconds, the client traffic will go over the second connection. Re-enable the peer and soon afterwards, the traffic will be routed over the first connection again.
You can verify this when you check the change of your visible external IP with e.g. https://www.whatismyip.com/ from one of your NordVPN clients.

Note: This setup only handles IPv4! Currently, NordVPN does not support IPv6 anyways. If you want to make sure that your NordVPN clients will not expose themselves via IPv6, disable it completely on those clients and / or block IPv6 completely via a firewall rule for NORDVPN_CLIENTS (hint: you can use the MAC addresses instead of IPs to define the alias, which will work for bot IPv4 and IPv6!).

P.S.: By default, OPNsense can take up to 60 seconds to detect a gateway failure and switch over. To make the failover happen faster, apply this adjustment:

Go to System:Gateways:Configuration, edit both gateways (`NORDVPN_GW1` and `NORDVPN_GW2`), scroll down, click "Advanced" under *Weight / Intervals* and change the values for both gateways to:
  - "Ping Interval" = 1 (pings every second)
  - "Time Period" = 3 (calculates average over 3 seconds)
  - "Down Interval" = 3 (marks the gateway as down after 3 missed pings)

[This setup only handles IPv4!]

There are lots of (mostly incomplete) instructions on how to setup NordVPN on OpnSense scattered all over the forum. Also, there are instructions from NordVPN themselves, but they do not focus on Wireguard.

Also, I found that my setup failed a few days ago, because NordVPN obviously cancelled one of their rented servers (de1019.nordvpn.com) which happened to be the one I used until then.

Therefore, I took the effort to create a setup with a failover for a fixed set of clients. This setup also will behave as a "kill switch", i.e. that no client targeted by it can bypass the NordVPN setup and expose itself accidentally.

1. You have to create not one, but two so-called "Access Token" in the NordVPN customer dashboard by clicking on NordVPN on the left. Then, the link to enter the access token menu will be at the bottom of the page. Create to different tokens and be sure to note them, because they will not be accessible later on.

2. Once you have those tokens, you will need to create a private key for both of them. This can be done via 'curl -s -u token:<YOUR_TOKEN_HERE> https://api.nordvpn.com/v1/users/services/credentials | jq -r .nordlynx_private_key'.

3. For entry into WG instances, you will also need the corresponding public keys, which you can get via 'echo <YOUR_PRIVATE_KEY_HERE> | wg pubkey'.

4. If you followed along, you should now be in posession of two private/public key pairs pertaining to your NordVPN account.

5. Using these informations, you can now add two Wireguard instances under VPN:Wireguard->Instances. Add these as "NORDVPN_INSTANCE1" and "NORDVPN_INSTANCE2", use different, free "Listen Ports" (these can be arbitrary, but must not be used anywhere else) and 10.5.0.2/16 as "Tunnel address" (yes, both are the same!).

6. Obtain two NordVPN server names from their API via 'curl -s "https://api.nordvpn.com/v1/servers/recommendations?&filters\[servers_technologies\]\[identifier\]=wireguard_udp&limit=2" | jq -r ".[]|.hostname"'.

7. Next, create two Wireguard peers under VPN:Wireguard->Peers. Name them NORDVPN_PEER1 and NORDVPN_PEER2, and choose different NordVPN server names obtained in the previous step for bot peers (e.g. 'deXXXX.nordvpn.com'). Use "public key" = "m0tej5P6pYfBivkJc8yRV4KqQXmM81AChLlzlsOSjSs=", "Allowed IPs" = "0.0.0.0/0", "Endpoint port" = 51820, "Keepalive Interval" = 30. Choose NORDVPN_INSTANCE1 as Wireguard instance for NORDVPN_PEER1 and NORDVPN_INSTANCE2 for NORDVPN_PEER2 - use only one per each peer!

8. Next, you can check VPN:WireGuard:Status. You should see both the peer and the instance come up as green for both instance/peer pairs if Wireguard is enabled.

9. Now, assign those two Wireguard interfaces names via Interfaces:Assignments. Name them NORDVPN1 and NORDVPN1. Select each of them under Interfaces and check the "Enable Interface" and "Prevent Interface Removal" boxes.

10. Create a "Host" alias with the definition of all clients, that should go through NordVPN only. Name that NORDVPN_CLIENTS.

11. Create two SNAT rules for each of the two NordVPN interfaces under Firewall:NAT:Source NAT with:

Code Select Expand

Description = "NAT for NordVPN"
Interface = NORDVPN1 / NORDVPN2
Version = IPv4
Protocol = any
Source Addres = VPN_CLIENTS
Translate Source IP = leave empty
12. Create two Gateways under System:Gateways:Configuration with

Code Select Expand

Name = NORDVPN_GW1
Interface = NORDVPN1
Address Family = IPv4
IP Address = 10.5.0.1
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.96.100
Description = IPv4 Gateway for NordVPN


Code Select Expand

Name = NORDVPN_GW2
Interface = NORDVPN2
Address Family = IPv4
IP Address = 103.86.99.100 [i](Note: This is not an error, but a trick!)[/i]
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.99.100
Description = IPv4 Gateway for NordVPN
Leave everything else on default.

13. Create a gateway group under System:Gateways:Group named NORDVPN_FAILOVER with "Trigger Level" = "Member Down", having NORDVPN_GW1 as Tier 1 and NordVPN_GW2 as Tier 2, with anything else as tier "Never". After having them created, both gateways should be listed as active (green) here.

14. Finally, create a firewall rule under Firewall:Rules[new]:

Code Select Expand

Description = Force gateway for NordVPN clients
Interface = any
Quick = checked
Action = Pass
Direction = In
Version = IPv4
Protocol = any
Invert Source = unchecked
Source = NORDVPN_CLIENTS
Source Port = any
Invert Destination = checked
Destination = RFC1918
Destination Port = any
Gateway = NORDVPN_FAILOVER

This assumes that you have an RFC1918 alias. You could also use an inverted firewall group containing all of your local networks. What this is supposed to mean is "internet traffic", with the exception of "local traffic", should you want to access your NORDVPN_CLIENTS from any of your local networks.



If you want to try the failover, disable the first Wireguard peer. You will see the gateway failing and after a few seconds, the client traffic will go over the second connection. Re-enable the peer and soon afterwards, the traffic will be routed over the first connection again.
You can verify this when you check the change of your visible external IP with e.g. https://www.whatismyip.com/ from one of your NordVPN clients.

Note: This setup only handles IPv4! If you want to make sure that your NordVPN clients will not expose themselves via IPv6, disable it completely on those clients.

There are at least three potential reasons for this:

1. Your ISP uses another IP on top of the PPPoE connection (like described in the thread Franco linked to.
2. Your ISP gives you an RFC1918 IP and does CG-NAT on that.
3. Your ISP hands out some RFC1918 IP when the WAN MAC changes and wants you to register the the MAC / router on a private site, before completing that step, you will only have limited access, but no internet.

Ask your ISP which is the case.

Either way, you probably need to uncheck "block RFC1918" on the WAN interface.

It's from their Rack Series : https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications

I know. I meant these specific Asrock Rack Mainboards are a chimera between consumer chipsets and server features and that never went that good, forums are full of failure reports. The stability of X470 / X570 was not stellar in the first place.

My remarks were not really meant to be questions, but purely rhetorical, which is to say:

If you only have two sides (or interfaces), doing a routed setup is no more complex than a transparent bridge - if you have more than two, a transparent bridge cannot be used at all (or at least I fail to see how). So, why use a transparent bridge in the first place?