Messages

Where do you want to migrate the destination NAT rules to?

The only reason to migrate "Outbound NAT" to "SNAT" (aka "source NAT") was that this is the more common term. "Destination NAT" is already called correctly.

[not]

AFAIK, MacOS does requests in parallel, Windows does a fallback and goes back to preferred order. Unbound uses a sophisticated model preferring the fastest server - which is what many clients also do via the "Happy Eyeballs" approach. Whatever the case, this scheme does stop eventually once you factor in IPv6: Because you can only supply DNS server IPs of "your own kind" in any type of IP protocol, you have independent settings. So, which one will be preferred in the presence of both IPv4 and IPv6 DNS servers?

Before you say: "IPv6 is always preferred" - just today I had a case where I had augmented a DynDNS entry for a CG-NAT connection with a fixed IPv4, which has a reverse proxy to make the IPv6-only connection work for IPv4, too. Thus, the DynDNS name now refers to both an IPv4 and an IPv6.

And guess what? That broke a Wireguard PC client, who just happens to prefer IPv4 and thus did not reach the real IPv6 backend, but the IPv4 proxy (which did not have a Wireguard service). So there,

What can we learn from this? Patrick is right: Do not rely on DNS server ordering for ANY client. RFC 2132 is not helpful, either, because it only covers what an IPv4 server SHOULD do - i.e., it does not say:

a. what the server MUST do.
b. what the client SHOULD or MUST do.
c. what the client SHOULD or MUST do when an additional IPv6 DNS server list is provided.

[This setup only handles IPv4!]

There are lots of (mostly incomplete) instructions on how to setup NordVPN on OpnSense scattered all over the forum. Also, there are instructions from NordVPN themselves, but they do not focus on Wireguard.

Also, I found that my setup failed a few days ago, because NordVPN obviously cancelled one of their rented servers (de1019.nordvpn.com) which happened to be the one I used until then.

Therefore, I took the effort to create a setup with a failover for a fixed set of clients. This setup also will behave as a "kill switch", i.e. that no client targeted by it can bypass the NordVPN setup and expose itself accidentally.

1. You have to create not one, but two so-called "Access Tokens" in the NordVPN customer dashboard by clicking on NordVPN on the left. Then, the link to enter the access token menu will be at the bottom of the page. Create two different tokens and be sure to note them, because they will not be accessible later on.

2. Once you have those tokens, you will need to create a private key for both of them. This can be done via 'curl -s -u token:<YOUR_TOKEN_HERE> https://api.nordvpn.com/v1/users/services/credentials | jq -r .nordlynx_private_key'.

3. For entry into WG instances, you will also need the corresponding public keys, which you can get via 'echo <YOUR_PRIVATE_KEY_HERE> | wg pubkey'.

4. If you followed along, you should now be in possession of two private/public key pairs pertaining to your NordVPN account. These are necessary to discriminate between the two connections, you cannot do this using only one key pair.

5. Using this information, you can now add two Wireguard instances under VPN:Wireguard->Instances. Add these as "NORDVPN_INSTANCE1" and "NORDVPN_INSTANCE2", use different, free "Listen Ports" (these can be arbitrary, but must not be used anywhere else) and 10.5.0.2/16 as "Tunnel address" (yes, both are the same!).

6. Obtain two NordVPN server names from their API via 'curl -s "https://api.nordvpn.com/v1/servers/recommendations?&filters\[servers_technologies\]\[identifier\]=wireguard_udp&limit=2" | jq -r ".[]|.hostname"'. This will give you two names like "xx999.nordvpn.com" for servers that are near you.

7. Next, create two Wireguard peers under VPN:Wireguard->Peers. Name them NORDVPN_PEER1 and NORDVPN_PEER2, and choose different NordVPN server names obtained in the previous step for both peers (e.g. 'deXXX.nordvpn.com'). Use "public key" = "m0tej5P6pYfBivkJc8yRV4KqQXmM81AChLlzlsOSjSs=" (this happens to be the public key for all NordVPN servers), "Allowed IPs" = "0.0.0.0/0", "Endpoint port" = 51820, "Keepalive Interval" = 30. Choose NORDVPN_INSTANCE1 as Wireguard instance for NORDVPN_PEER1 and NORDVPN_INSTANCE2 for NORDVPN_PEER2 - use only one per each peer!

8. Next, you can check VPN:WireGuard:Status. You should see both the peer and the instance come up as green for both instance/peer pairs if Wireguard is enabled.

9. Now, assign those two Wireguard interfaces names via Interfaces:Assignments. Name them NORDVPN1 and NORDVPN2. Select each of them under Interfaces and check the "Enable Interface" and "Prevent Interface Removal" boxes.

10. Create an alias with the definition of all clients, that should go through NordVPN only. Name that NORDVPN_CLIENTS. I prefer to use a network group alias that includes MAC aliases for individual machines, see also the hint about IPv6 at the end of this article.

11. Create two SNAT rules for each of the two NordVPN interfaces under Firewall:NAT:Source NAT with:

Code Select Expand

Description = "NAT for NordVPN"
Interface = NORDVPN1 / NORDVPN2
Version = IPv4
Protocol = any
Source Address = NORDVPN_CLIENTS
Translate Source IP = leave empty
12. Create two Gateways under System:Gateways:Configuration with

Code Select Expand

Name = NORDVPN_GW1
Interface = NORDVPN1
Address Family = IPv4
IP Address = 10.5.0.1
Far Gateway = checked
Failover States = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.96.100
Description = IPv4 Gateway for NordVPN


Code Select Expand

Name = NORDVPN_GW2
Interface = NORDVPN2
Address Family = IPv4
IP Address = 103.86.99.100
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.99.100
Description = IPv4 Gateway for NordVPN

The "IP address" of 103.86.99.100 in the second entry is not an error. It is a workaround because you cannot use the same gateway IP twice. WireGuard does not actually use this IP for routing.
Leave everything else on default.

13. Create a gateway group under System:Gateways:Group named NORDVPN_FAILOVER with "Trigger Level" = "Member Down", having NORDVPN_GW1 as Tier 1 and NordVPN_GW2 as Tier 2, with anything else as tier "Never". After having them created, both gateways should be listed as active (green) here.

14. Finally, create a firewall rule under Firewall:Rules[new] (it should precede most other rules):

Code Select Expand

Description = Force gateway for NordVPN clients
Interface = any
Quick = checked
Action = Pass
Direction = In
Version = IPv4
Protocol = any
Invert Source = unchecked
Source = NORDVPN_CLIENTS
Source Port = any
Invert Destination = checked
Destination = RFC1918
Destination Port = any
Gateway = NORDVPN_FAILOVER

This assumes that you have an RFC1918 alias. You could also use an inverted firewall group containing all of your local networks. What this is supposed to mean is "internet traffic", with the exception of "local traffic", should you want to access your NORDVPN_CLIENTS from any of your local networks. By explicitely setting a gateway for this rule, you effectively implement policy-based routing (PBR).



If you want to try the failover, disable the first Wireguard peer. You will see the gateway failing and after a few seconds, the client traffic will go over the second connection. Re-enable the peer and soon afterwards, the traffic will be routed over the first connection again.
You can verify this when you check the change of your visible external IP with e.g. https://www.whatismyip.com/ from one of your NordVPN clients.

Note: This setup only handles IPv4! Currently, NordVPN does not support IPv6 anyways. If you want to make sure that your NordVPN clients will not expose themselves via IPv6, disable it completely on those clients and / or block IPv6 completely via a firewall rule for NORDVPN_CLIENTS (hint: you can use the MAC addresses instead of IPs to define the alias, which will work for bot IPv4 and IPv6!).

P.S.: By default, OPNsense can take up to 60 seconds to detect a gateway failure and switch over. To make the failover happen faster, apply this adjustment:

Go to System:Gateways:Configuration, edit both gateways (`NORDVPN_GW1` and `NORDVPN_GW2`), scroll down, click "Advanced" under *Weight / Intervals* and change the values for both gateways to:
   - "Ping Interval" = 1 (pings every second)
   - "Time Period" = 3 (calculates average over 3 seconds)
   - "Down Interval" = 3 (marks the gateway as down after 3 missed pings)

[This setup only handles IPv4!]

There are lots of (mostly incomplete) instructions on how to setup NordVPN on OpnSense scattered all over the forum. Also, there are instructions from NordVPN themselves, but they do not focus on Wireguard.

Also, I found that my setup failed a few days ago, because NordVPN obviously cancelled one of their rented servers (de1019.nordvpn.com) which happened to be the one I used until then.

Therefore, I took the effort to create a setup with a failover for a fixed set of clients. This setup also will behave as a "kill switch", i.e. that no client targeted by it can bypass the NordVPN setup and expose itself accidentally.

1. You have to create not one, but two so-called "Access Token" in the NordVPN customer dashboard by clicking on NordVPN on the left. Then, the link to enter the access token menu will be at the bottom of the page. Create to different tokens and be sure to note them, because they will not be accessible later on.

2. Once you have those tokens, you will need to create a private key for both of them. This can be done via 'curl -s -u token:<YOUR_TOKEN_HERE> https://api.nordvpn.com/v1/users/services/credentials | jq -r .nordlynx_private_key'.

3. For entry into WG instances, you will also need the corresponding public keys, which you can get via 'echo <YOUR_PRIVATE_KEY_HERE> | wg pubkey'.

4. If you followed along, you should now be in posession of two private/public key pairs pertaining to your NordVPN account.

5. Using these informations, you can now add two Wireguard instances under VPN:Wireguard->Instances. Add these as "NORDVPN_INSTANCE1" and "NORDVPN_INSTANCE2", use different, free "Listen Ports" (these can be arbitrary, but must not be used anywhere else) and 10.5.0.2/16 as "Tunnel address" (yes, both are the same!).

6. Obtain two NordVPN server names from their API via 'curl -s "https://api.nordvpn.com/v1/servers/recommendations?&filters\[servers_technologies\]\[identifier\]=wireguard_udp&limit=2" | jq -r ".[]|.hostname"'.

7. Next, create two Wireguard peers under VPN:Wireguard->Peers. Name them NORDVPN_PEER1 and NORDVPN_PEER2, and choose different NordVPN server names obtained in the previous step for bot peers (e.g. 'deXXXX.nordvpn.com'). Use "public key" = "m0tej5P6pYfBivkJc8yRV4KqQXmM81AChLlzlsOSjSs=", "Allowed IPs" = "0.0.0.0/0", "Endpoint port" = 51820, "Keepalive Interval" = 30. Choose NORDVPN_INSTANCE1 as Wireguard instance for NORDVPN_PEER1 and NORDVPN_INSTANCE2 for NORDVPN_PEER2 - use only one per each peer!

8. Next, you can check VPN:WireGuard:Status. You should see both the peer and the instance come up as green for both instance/peer pairs if Wireguard is enabled.

9. Now, assign those two Wireguard interfaces names via Interfaces:Assignments. Name them NORDVPN1 and NORDVPN1. Select each of them under Interfaces and check the "Enable Interface" and "Prevent Interface Removal" boxes.

10. Create a "Host" alias with the definition of all clients, that should go through NordVPN only. Name that NORDVPN_CLIENTS.

11. Create two SNAT rules for each of the two NordVPN interfaces under Firewall:NAT:Source NAT with:

Code Select Expand

Description = "NAT for NordVPN"
Interface = NORDVPN1 / NORDVPN2
Version = IPv4
Protocol = any
Source Addres = VPN_CLIENTS
Translate Source IP = leave empty
12. Create two Gateways under System:Gateways:Configuration with

Code Select Expand

Name = NORDVPN_GW1
Interface = NORDVPN1
Address Family = IPv4
IP Address = 10.5.0.1
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.96.100
Description = IPv4 Gateway for NordVPN


Code Select Expand

Name = NORDVPN_GW2
Interface = NORDVPN2
Address Family = IPv4
IP Address = 103.86.99.100 [i](Note: This is not an error, but a trick!)[/i]
Far Gateway = checked
Disable Gateway Monitoring = unchecked
Monitor IP = 103.86.99.100
Description = IPv4 Gateway for NordVPN
Leave everything else on default.

13. Create a gateway group under System:Gateways:Group named NORDVPN_FAILOVER with "Trigger Level" = "Member Down", having NORDVPN_GW1 as Tier 1 and NordVPN_GW2 as Tier 2, with anything else as tier "Never". After having them created, both gateways should be listed as active (green) here.

14. Finally, create a firewall rule under Firewall:Rules[new]:

Code Select Expand

Description = Force gateway for NordVPN clients
Interface = any
Quick = checked
Action = Pass
Direction = In
Version = IPv4
Protocol = any
Invert Source = unchecked
Source = NORDVPN_CLIENTS
Source Port = any
Invert Destination = checked
Destination = RFC1918
Destination Port = any
Gateway = NORDVPN_FAILOVER

This assumes that you have an RFC1918 alias. You could also use an inverted firewall group containing all of your local networks. What this is supposed to mean is "internet traffic", with the exception of "local traffic", should you want to access your NORDVPN_CLIENTS from any of your local networks.



If you want to try the failover, disable the first Wireguard peer. You will see the gateway failing and after a few seconds, the client traffic will go over the second connection. Re-enable the peer and soon afterwards, the traffic will be routed over the first connection again.
You can verify this when you check the change of your visible external IP with e.g. https://www.whatismyip.com/ from one of your NordVPN clients.

Note: This setup only handles IPv4! If you want to make sure that your NordVPN clients will not expose themselves via IPv6, disable it completely on those clients.

There are at least three potential reasons for this:

1. Your ISP uses another IP on top of the PPPoE connection (like described in the thread Franco linked to.
2. Your ISP gives you an RFC1918 IP and does CG-NAT on that.
3. Your ISP hands out some RFC1918 IP when the WAN MAC changes and wants you to register the the MAC / router on a private site, before completing that step, you will only have limited access, but no internet.

Ask your ISP which is the case.

Either way, you probably need to uncheck "block RFC1918" on the WAN interface.

It's from their Rack Series : https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications

I know. I meant these specific Asrock Rack Mainboards are a chimera between consumer chipsets and server features and that never went that good, forums are full of failure reports. The stability of X470 / X570 was not stellar in the first place.

My remarks were not really meant to be questions, but purely rhetorical, which is to say:

If you only have two sides (or interfaces), doing a routed setup is no more complex than a transparent bridge - if you have more than two, a transparent bridge cannot be used at all (or at least I fail to see how). So, why use a transparent bridge in the first place?

X470 chips in general and the consumer platform based Asrock Mainboards are notorious for failing early, too.

What I always wondered about that transparent bridge setup: If you have only two sides between to filter traffic, then what would be so difficult as to use different subnets (aka routing)? And if you don't, like if you have separate VLANs, then how to you even get the traffic to pass your firewall?

Once you get to understand routing, it seems natural to choose that, which is possibly why < 1% of people here would be able to help if you don't.

All I know is that ASPM can be disabled globally or on a per-device basis. The latter normally needs BIOS or driver support, the former, AFAIK, does not.

I had freezing issues on my I226-V NICs on a Minisforum MS-01, the bug was fixed via a BIOS update. Also, there was this discussion of possible remedies: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279245

ASPM is causing this for I226 devices and I am not aware that updating the NIC firmware fixes that.

If there is an updated BIOS for the Protectl, try that first. You can actually make that go away with ASPM off, but AFAIK, you can only disable this for the whole machine under OpnSense if the BIOS does not set it selectively for your NICs.

The global setting is by done setting the tuneable hw.pci.enable_aspm=0. You should probably also set dev.igc.X.eee_control=0 with X=0,1.

There is now a proposed fix which can be tested via "opnsense-patch 67ea4ad". Applying the same patch again reverts it, whould the need arise.

Die Wahrscheinlichkeit ist hoch, dass die statische vergebene IPv6 noch im gecachten Lease-File für eine andere DUID (Frage: nutzt Du wirklich die MAC die Zuordnung?) steht, dann klappt die statische Nutzung nicht mehr. Das ist einer der Gründe, wieso man die statischen und dynamischen Bereiche bei DHCP strikt trennen sollte.

Ich nutze für IPv6 ohnehin kein DHCP. und adressiere Clients im Netz nur per IPv4 (IPv6 wird nur outbound verwendet), siehe: https://forum.opnsense.org/index.php?topic=45822.0

My clients are now only receiving the IPv4 addresses for my PiHole and OPNsense Unbound, in that order, and will fallback to the OPNsense server if the PiHole goes away for whatever reason.

AFAIK, this is a common misconception: There is no guaranteed order if you specify multiple DNS servers. A client may choose to send out the DNS queries in parallel and take the first answer. Thus, the order is arbitrary, so this is not a "fallback" in its strict sense. This exact behaviour can be detrimental for DNS blocking.

Why do you prefer the null routes solution?

That has been discussed in this thread: https://forum.opnsense.org/index.php?topic=50678 - essentially, with "out" rules, you may forget that you actually have RFC1918 IPs you want to reach on WAN, whereas with null routes, that works out of the box. Also, note the "disable reply-to" problem here: https://forum.opnsense.org/index.php?msg=258978

On the other hand, you are correct in that you will see nothing the logs, for this is routing, not a firewall rule.