Messages

I would be interested to see if you solve this. I have wanted to do the same. This is what I got to work:

WAN 1 = General Internet
WAN 2 = VPN Internet

WAN 1 Gateway got a weight of 200
WAN 2 Gateway got a weight of 100

Gateway Switching and such all configured. This allows Wireguard to use WAN 2 normally, and if WAN 2 goes down, it would then use WAN 1.

Created Gateway Group with WAN 1 Primary and WAN 2 Secondary.
Created a Firewall Rule on the LAN network using inverted destination so if it was not going to the internal or VPN networks, it would use the Gateway Group instead of the defaults.

Not sure this would work for you as it sounds like you might be hosting other services that require NAT. I also have not done this method on the latest version of OPNsense, so I do not know if this still works with the new rule and NAT structures.

An update to this. We are still trying to figure this out. In the mean time, the following has changed.

We moved a location, which resulted in a rebuild of a firewall. When we did the rebuild, most everything worked. While we were still unable to ping from the console (still able to ping from Web GUI) we were able to use DNS and LDAP. This prompted us to believe that it really might be something related to ZeroTier still lingering in the System. However, once we patched and rebooted the firewalls during our maintenance cycle, it stopped working. All the same symptoms.

We are now on 26.4 on all the firewalls. We did migrate the rules to the new format and there was no change.

We added a Outbound NAT rule that looked like this

Interface     Source          Source Port     Destination     Destination Port     NAT Address           Static Port
WG_STS        This Firewall   *               10.1.0.0/24     *                    Interface Address     No

This did not help as well.

We can ping and traceroute to external IPs just fine (1.1.1.1, 8.8.8.8, 4.2.2.1) and when we have live view open, it does not show that it is blocking the traffic, and we do see it pass, which is why we think it is something routing and WireGuard directly, not something to do with the rules.

There is one difference I did not put in the original post, which is that the HUB firewall has static routing, and the Site firewalls all use WG to push the routes. We did have one site on static routes as well, and saw the same issue.

We are still working on this, but it is kinda a low priority as other functions still work.

[From TAC (Spoke) to HUB -]

Ping and Trace Route

From TAC (Spoke) to HUB -
Pings:
10.1.0.1 (HUB LAN IP) from 10.2.0.1 (TAC LAN IP) via Shell = 100% Packet Loss
10.1.0.1 from 10.2.0.1 via GUI (Interface:Diagnostics:Ping) = 0% Packet Loss (Successful ping)
10.1.0.11 (DNS Server on HUB LAN) from 10.2.0.1 via Shell = 100% Packet Loss
10.1.0.11 from 10.2.0.1 via GUI = 0% Packet Loss
172.19.0.1 (HUB WG Interface) from 10.2.0.1 via Shell = 100% Packet Loss
172.19.0.1 from 10.2.0.1 via GUI = 0% Packet Loss

10.1.0.1 from 10.2.0.122 (TAC Client Workstation) = 0% Packet Loss
10.1.0.11 from 10.2.0.122 = 0% Packet Loss
172.19.0.1 from 10.20.122 = 0% Packet Loss

Traceroute:
10.1.0.1 (HUB LAN IP) from 10.2.0.1 via Shell = * * * for 10 hops (should never be more than 4 hops, so 10 should do it)
10.1.0.1 from 10.2.0.1 via GUI (Interface:Diagnostics:Traceroute) = Hop 1 is 172.19.0.1, Hop 2 is 10.1.0.1
10.1.0.11 from 10.2.0.1 via Shell = * * * for 10 hops
10.1.0.11 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.1, Hop 2 is 10.1.0.11
172.19.0.1 from 10.2.0.1 via Shell = * * * for 10 hops
172.19.0.1 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.1

10.1.0.1 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop 2 is 10.1.0.1
10.1.0.11 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop 2 is 172.19.0.1, Hop 3 is * * *, Hop 4 is 10.1.0.111
172.19.0.1 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop is 172.19.0.1

Interesting that from client to server, it drops the hop 3, which should be 10.1.0.1

From HUB to TAC -
Pings:
10.2.0.1 (TAC LAN IP) from 10.2.0.1 (HUB LAN IP) via Shell = 100% Packet Loss
10.2.0.1 from 10.1.0.1 via GUI = 0% Packet Loss (Successful ping)
10.2.0.50 (Printer on TAC LAN) from 10.2.0.1 via Shell = 100% Packet Loss
10.2.0.50 from 10.1.0.1 via GUI = 0% Packet Loss
172.19.0.2 (TAC WG Interface) from 10.1.0.1 via Shell = 100% Packet Loss
172.19.0.2 from 10.1.0.1 via GUI = 0% Packet Loss

10.2.0.1 from 10.1.0.11 (HUB DNS Server) = 0% Packet Loss
10.2.0.50 from 10.1.0.11 = 0% Packet Loss
172.19.0.2 from 10.1.0.11 = 0% Packet Loss

Traceroute:
10.2.0.1 (TAC LAN IP) from 10.1.0.1 via Shell = * * * for 10 hops (should never be more than 4 hops, so 10 should do it)
10.2.0.1 from 10.1.0.1 via GUI = Hop 1 is 172.19.0.2, Hop 2 is 10.2.0.1
10.2.0.50 from 10.1.0.1 via Shell = * * * for 10 hops
10.2.0.50 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.2, Hop 2 is 10.1.0.50
172.19.0.2 from 10.1.0.1 via Shell = * * * for 10 hops
172.19.0.2 from 10.1.0.1 via GUI = Hop 1 is 172.19.0.2

10.2.0.1 from 10.2.0.11 = Hop 1 is 10.1.0.1, Hop 2 is 10.2.0.1
10.2.0.50 from 10.2.0.11 = Hop 1 is 10.1.0.1, Hop 2 is * * *, Hop 3 is 172.19.0.2, Hop 4 is 10.2.0.50
172.19.0.2 from 10.1.0.11 = Hop 1 is 10.1.0.1, Hop is 172.19.0.2

Again, interesting that the WG HUB IP is not showing. I will look more on the HUB firewall, as it does appear to be something on that end. Still not sure though, as GUI seems to send traffic fine. But that could be explained that GUI may now use a different process since update.

Correct. Any client on the network can access everything, going over the tunnel just fine. If traffic originates from firewall, it does not go over the tunnel.

Last night I applied updates to the firewalls (25.10_2 for OPNsense Business), and now I can ping via the GUI, but not the shell (either from option 7 or via command line).

I also tried to do a policy based route. On Firewall:Rules:LAN, I created a rule like this

Protocol: IPv4*
Source: This Firewall
Destination: 10.1.0.0/16
Port: *
Gateway: WG_GW (Gateway for WG Tunnel, 172.19.0.1)

This did not help either.

I really think it is something simple I am missing. Just might be over complicating it a bit :D

Have 10 OPNsense firewalls configured in a Hub - Spoke Site to Site Wireguard VPN. 99% is functioning like it should. All LAN clients can talk to any other client on any other network, speed is great, adding new sites auto adds routes. The only issue I am having is that anything from the Firewall itself will not route over the WG tunnel. Things like LDAP, DNS, Ping, Traceroute, etc, will just fail with no route available. This only affects the firewalls. Anything behind the firewall works just fine, and the firewall can route over any other interface.

Configuration from hub to 1 spoke (other spokes are identical, just increment the second octet by 1)

Hub Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.1/24
Peers = TAC, BEL, SEA, TUK, LYN, SIL, POU, KNT, BAL
Disable Routes = Unchecked

Peer - TAC
Allowed IP = 172.19.0.2/24, 10.2.0.0/16
Endpoint Port = 51820
Keep Alive = 25

TAC Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.2/24 
Peers = HUB
Disable Routes = Unchecked

Peer - HUB
Allowed IP = 172.19.0.1/24, 10.1.0.0/16, 10.2.0.0/16, etc
Endpoint Port = 51820
Keep Alive = 25

Firewall Rules are fairly open right now as I am trying to troubleshoot. On the LAN and WG interfaces I have currently set to allow all.

What I see -
Pinging from TAC firewall to (Via Shell):
HUB Firewall LAN IP = 100% Packet Loss
HUB Firewall WG IP = 100% Packet Loss
Server on 10.1.0.0/16 Subnet = 100% Packet Loss
8.8.8.8 = 0 Packet Loss
Server on 10.2.0.0/16 Subnet = 0% Packet Loss
SEA Firewall LAN IP = 100% Packet Loss
SEA WG IP = 100% Packet Loss

Pinging from client on TAC LAN Subnet to any of the above gets 0 packet loss and low latency. Works as expected.

Traceroute from TAC Firewall to (via Shell):
HUB Firewall LAN IP = Timeout
HUB Firewall WG IP = Timeout
Server on 10.1.0.0/16 Subnet = Timeout
8.8.8.8 = Successfully traces
Server on 10.2.0.0/16 Subnet = Successfully traces
SEA Firewall LAN IP = Timeout
SEA WG IP = Timeout

Netstat -rn on either firewall shows correct routes over WG0

Firewall logs on both firewalls do not show blocking any traffic

Packet Capture on both firewalls show the packets going out and even over the tunnel, but never a response.

Using Web GUI to configure LDAP Server, I get a host can not be reached.

Using Web GUI to forward specific domains on Unbound DNS to HUB DNS server, I can add it, but does not get packets.


Things I have tried -
Rebooting :D
Changing from Disable Routes unchecked to checked and manually adding routes = No change
Making the firewall rules more open = No Change
Rebuilding WG = No Change

Not sure if this applies to me:
https://forum.opnsense.org/index.php?topic=41506.msg232809#msg232809
But willing to try if people think it will help.

I am kinda stumped. I have setup WireGuard before on OPNsense and have not had this issue. We were previously on ZeroTier, so I thought that might be causing issues, but we have added two new firewalls since then and they have the same problem.

Thanks in advance for any thoughts!