Have 10 OPNsense firewalls configured in a Hub - Spoke Site to Site Wireguard VPN. 99% is functioning like it should. All LAN clients can talk to any other client on any other network, speed is great, adding new sites auto adds routes. The only issue I am having is that anything from the Firewall itself will not route over the WG tunnel. Things like LDAP, DNS, Ping, Traceroute, etc, will just fail with no route available. This only affects the firewalls. Anything behind the firewall works just fine, and the firewall can route over any other interface.
Configuration from hub to 1 spoke (other spokes are identical, just increment the second octet by 1)
Hub Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.1/24
Peers = TAC, BEL, SEA, TUK, LYN, SIL, POU, KNT, BAL
Disable Routes = Unchecked
Peer - TAC
Allowed IP = 172.19.0.2/24, 10.2.0.0/16
Endpoint Port = 51820
Keep Alive = 25
TAC Firewall
WireGuard Instance WG0
Listen Port = 51820
Tunnel Address = 172.19.0.2/24
Peers = HUB
Disable Routes = Unchecked
Peer - HUB
Allowed IP = 172.19.0.1/24, 10.1.0.0/16, 10.2.0.0/16, etc
Endpoint Port = 51820
Keep Alive = 25
Firewall Rules are fairly open right now as I am trying to troubleshoot. On the LAN and WG interfaces I have currently set to allow all.
What I see -
Pinging from TAC firewall to (Via Shell):
HUB Firewall LAN IP = 100% Packet Loss
HUB Firewall WG IP = 100% Packet Loss
Server on 10.1.0.0/16 Subnet = 100% Packet Loss
8.8.8.8 = 0 Packet Loss
Server on 10.2.0.0/16 Subnet = 0% Packet Loss
SEA Firewall LAN IP = 100% Packet Loss
SEA WG IP = 100% Packet Loss
Pinging from client on TAC LAN Subnet to any of the above gets 0 packet loss and low latency. Works as expected.
Traceroute from TAC Firewall to (via Shell):
HUB Firewall LAN IP = Timeout
HUB Firewall WG IP = Timeout
Server on 10.1.0.0/16 Subnet = Timeout
8.8.8.8 = Successfully traces
Server on 10.2.0.0/16 Subnet = Successfully traces
SEA Firewall LAN IP = Timeout
SEA WG IP = Timeout
Netstat -rn on either firewall shows correct routes over WG0
Firewall logs on both firewalls do not show blocking any traffic
Packet Capture on both firewalls show the packets going out and even over the tunnel, but never a response.
Using Web GUI to configure LDAP Server, I get a host can not be reached.
Using Web GUI to forward specific domains on Unbound DNS to HUB DNS server, I can add it, but does not get packets.
Things I have tried -
Rebooting :D
Changing from Disable Routes unchecked to checked and manually adding routes = No change
Making the firewall rules more open = No Change
Rebuilding WG = No Change
Not sure if this applies to me:
https://forum.opnsense.org/index.php?topic=41506.msg232809#msg232809
But willing to try if people think it will help.
I am kinda stumped. I have setup WireGuard before on OPNsense and have not had this issue. We were previously on ZeroTier, so I thought that might be causing issues, but we have added two new firewalls since then and they have the same problem.
Thanks in advance for any thoughts!