Messages

[From TAC (Spoke) to HUB -]

Ping and Trace Route

From TAC (Spoke) to HUB -
Pings:
10.1.0.1 (HUB LAN IP) from 10.2.0.1 (TAC LAN IP) via Shell = 100% Packet Loss
10.1.0.1 from 10.2.0.1 via GUI (Interface:Diagnostics:Ping) = 0% Packet Loss (Successful ping)
10.1.0.11 (DNS Server on HUB LAN) from 10.2.0.1 via Shell = 100% Packet Loss
10.1.0.11 from 10.2.0.1 via GUI = 0% Packet Loss
172.19.0.1 (HUB WG Interface) from 10.2.0.1 via Shell = 100% Packet Loss
172.19.0.1 from 10.2.0.1 via GUI = 0% Packet Loss

10.1.0.1 from 10.2.0.122 (TAC Client Workstation) = 0% Packet Loss
10.1.0.11 from 10.2.0.122 = 0% Packet Loss
172.19.0.1 from 10.20.122 = 0% Packet Loss

Traceroute:
10.1.0.1 (HUB LAN IP) from 10.2.0.1 via Shell = * * * for 10 hops (should never be more than 4 hops, so 10 should do it)
10.1.0.1 from 10.2.0.1 via GUI (Interface:Diagnostics:Traceroute) = Hop 1 is 172.19.0.1, Hop 2 is 10.1.0.1
10.1.0.11 from 10.2.0.1 via Shell = * * * for 10 hops
10.1.0.11 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.1, Hop 2 is 10.1.0.11
172.19.0.1 from 10.2.0.1 via Shell = * * * for 10 hops
172.19.0.1 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.1

10.1.0.1 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop 2 is 10.1.0.1
10.1.0.11 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop 2 is 172.19.0.1, Hop 3 is * * *, Hop 4 is 10.1.0.111
172.19.0.1 from 10.2.0.122 = Hop 1 is 10.2.0.1, Hop is 172.19.0.1

Interesting that from client to server, it drops the hop 3, which should be 10.1.0.1

From HUB to TAC -
Pings:
10.2.0.1 (TAC LAN IP) from 10.2.0.1 (HUB LAN IP) via Shell = 100% Packet Loss
10.2.0.1 from 10.1.0.1 via GUI = 0% Packet Loss (Successful ping)
10.2.0.50 (Printer on TAC LAN) from 10.2.0.1 via Shell = 100% Packet Loss
10.2.0.50 from 10.1.0.1 via GUI = 0% Packet Loss
172.19.0.2 (TAC WG Interface) from 10.1.0.1 via Shell = 100% Packet Loss
172.19.0.2 from 10.1.0.1 via GUI = 0% Packet Loss

10.2.0.1 from 10.1.0.11 (HUB DNS Server) = 0% Packet Loss
10.2.0.50 from 10.1.0.11 = 0% Packet Loss
172.19.0.2 from 10.1.0.11 = 0% Packet Loss

Traceroute:
10.2.0.1 (TAC LAN IP) from 10.1.0.1 via Shell = * * * for 10 hops (should never be more than 4 hops, so 10 should do it)
10.2.0.1 from 10.1.0.1 via GUI = Hop 1 is 172.19.0.2, Hop 2 is 10.2.0.1
10.2.0.50 from 10.1.0.1 via Shell = * * * for 10 hops
10.2.0.50 from 10.2.0.1 via GUI = Hop 1 is 172.19.0.2, Hop 2 is 10.1.0.50
172.19.0.2 from 10.1.0.1 via Shell = * * * for 10 hops
172.19.0.2 from 10.1.0.1 via GUI = Hop 1 is 172.19.0.2

10.2.0.1 from 10.2.0.11 = Hop 1 is 10.1.0.1, Hop 2 is 10.2.0.1
10.2.0.50 from 10.2.0.11 = Hop 1 is 10.1.0.1, Hop 2 is * * *, Hop 3 is 172.19.0.2, Hop 4 is 10.2.0.50
172.19.0.2 from 10.1.0.11 = Hop 1 is 10.1.0.1, Hop is 172.19.0.2

Again, interesting that the WG HUB IP is not showing. I will look more on the HUB firewall, as it does appear to be something on that end. Still not sure though, as GUI seems to send traffic fine. But that could be explained that GUI may now use a different process since update.

Correct. Any client on the network can access everything, going over the tunnel just fine. If traffic originates from firewall, it does not go over the tunnel.

Last night I applied updates to the firewalls (25.10_2 for OPNsense Business), and now I can ping via the GUI, but not the shell (either from option 7 or via command line).

I also tried to do a policy based route. On Firewall:Rules:LAN, I created a rule like this

Protocol: IPv4*
Source: This Firewall
Destination: 10.1.0.0/16
Port: *
Gateway: WG_GW (Gateway for WG Tunnel, 172.19.0.1)

This did not help either.

I really think it is something simple I am missing. Just might be over complicating it a bit :D

Have 10 OPNsense firewalls configured in a Hub - Spoke Site to Site Wireguard VPN. 99% is functioning like it should. All LAN clients can talk to any other client on any other network, speed is great, adding new sites auto adds routes. The only issue I am having is that anything from the Firewall itself will not route over the WG tunnel. Things like LDAP, DNS, Ping, Traceroute, etc, will just fail with no route available. This only affects the firewalls. Anything behind the firewall works just fine, and the firewall can route over any other interface.

Configuration from hub to 1 spoke (other spokes are identical, just increment the second octet by 1)

Hub Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.1/24
Peers = TAC, BEL, SEA, TUK, LYN, SIL, POU, KNT, BAL
Disable Routes = Unchecked

Peer - TAC
Allowed IP = 172.19.0.2/24, 10.2.0.0/16
Endpoint Port = 51820
Keep Alive = 25

TAC Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.2/24 
Peers = HUB
Disable Routes = Unchecked

Peer - HUB
Allowed IP = 172.19.0.1/24, 10.1.0.0/16, 10.2.0.0/16, etc
Endpoint Port = 51820
Keep Alive = 25

Firewall Rules are fairly open right now as I am trying to troubleshoot. On the LAN and WG interfaces I have currently set to allow all.

What I see -
Pinging from TAC firewall to (Via Shell):
HUB Firewall LAN IP = 100% Packet Loss
HUB Firewall WG IP = 100% Packet Loss
Server on 10.1.0.0/16 Subnet = 100% Packet Loss
8.8.8.8 = 0 Packet Loss
Server on 10.2.0.0/16 Subnet = 0% Packet Loss
SEA Firewall LAN IP = 100% Packet Loss
SEA WG IP = 100% Packet Loss

Pinging from client on TAC LAN Subnet to any of the above gets 0 packet loss and low latency. Works as expected.

Traceroute from TAC Firewall to (via Shell):
HUB Firewall LAN IP = Timeout
HUB Firewall WG IP = Timeout
Server on 10.1.0.0/16 Subnet = Timeout
8.8.8.8 = Successfully traces
Server on 10.2.0.0/16 Subnet = Successfully traces
SEA Firewall LAN IP = Timeout
SEA WG IP = Timeout

Netstat -rn on either firewall shows correct routes over WG0

Firewall logs on both firewalls do not show blocking any traffic

Packet Capture on both firewalls show the packets going out and even over the tunnel, but never a response.

Using Web GUI to configure LDAP Server, I get a host can not be reached.

Using Web GUI to forward specific domains on Unbound DNS to HUB DNS server, I can add it, but does not get packets.


Things I have tried -
Rebooting :D
Changing from Disable Routes unchecked to checked and manually adding routes = No change
Making the firewall rules more open = No Change
Rebuilding WG = No Change

Not sure if this applies to me:
https://forum.opnsense.org/index.php?topic=41506.msg232809#msg232809
But willing to try if people think it will help.

I am kinda stumped. I have setup WireGuard before on OPNsense and have not had this issue. We were previously on ZeroTier, so I thought that might be causing issues, but we have added two new firewalls since then and they have the same problem.

Thanks in advance for any thoughts!