OPNsense routes local requests to subnet to WAN

[niraami]

To skip over the tedious explanation of my topology, I've attached it below. For now, I'm only focusing on the LAN_MGMT network/subnet.

Most of the setup works, host machine can ping any LAN or WAN address, and the firewall can ping the host (via Interfaces->Diagnostics->Ping). But no device from LAN can ping the host machine, but it's also not getting blocked by any firewall rules - as can be seen by the firewall logs attached below.
To achieve this, I've setup rules for both in & out on both interfaces to allow anything between those two interfaces.

Instead, what I'm seeing is that OPNsense is routing these requests to.. WAN? Why is that? I've not added any routes manually, but I can see that the HOST_MGMT (LAN_MGMT on the host machine) has been added automatically (screenshot attached).

Note: anything blurred out, is my public IP

edit:
One thing I forgot to note is that the HOST_MGMT interface in OPNsense is set to a static IP address of 10.9.254.1, if that matters.

[bartjsmit]

If traffic is going out of your WAN instead of back to the client, then it is more than likely a routing issue. Does OPNsense have a route back to your client? Check the routing table with netstat -r from the console.

Bart...

[niraami]

Does OPNsense have a route back to your client?

Could you elaborate, I'm quite new to routing rules, never really had to worry about it till now.
Do you mean 10.10.1.3/16 -> 10.9.247.1? If so, I don't know. All I can really do is show you the outputs, I don't quite know how it should look if it was correct.
I essentially thought that if the subnet is mentioned in the routing tables with the correct interface, then that's enough.

I've attached the output below, but it essentially says the same thing as the Route Status screenshot from earlier.

Code: [Select]

$ niraami@OPNsense:~ % netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            xxx-xxx-xxx-49.sta UGS      vtnet0
one.one.one.one    xxx-xxx-xxx-49.sta UGHS     vtnet0
one.one.one.one    xxx-xx-xxx.229     UGHS     vtnet1
dns.google         xxx-xx-xxx.229     UGHS     vtnet1
dns.google         xxx-xxx-xxx-49.sta UGHS     vtnet0
dns9.quad9.net     xxx-xxx-xxx-49.sta UGHS     vtnet0
10.9.0.0/16        link#5             U        vtnet4
OPNsense           link#5             UHS         lo0
10.10.0.0/16       link#3             U        vtnet2
OPNsense           link#3             UHS         lo0
OPNsense           link#2             UHS         lo0
xxx-xx-xxx.228/30  link#2             U        vtnet1
localhost          link#8             UH          lo0
rpz-public-resolve xxx-xx-xxx.229     UGHS     vtnet1
172.16.10.0/24     172.16.10.2        UGS      ovpns1
172.16.10.1        link#11            UHS         lo0
172.16.10.2        link#11            UH       ovpns1
xxx-xxx-xxx.0/24   link#1             U        vtnet0
OPNsense           link#1             UHS         lo0
resolver2.opendns. xxx-xxx-xxx-49.sta UGHS     vtnet0
resolver1.opendns. xxx-xx-xxx.229     UGHS     vtnet1

10.9.0.0/16 being the target subnet, where my "host" is & the 10.10.0.0/16 is, for now, the general subnet where essentially all other devices on the network are.

Also, when I say I believe the traffic is going out of WAN, I only assume that from that firewall log. If you think otherwise then definitely go ahead and correct me, I'm really at the edge of my knowledge right now

I've also decided to run traceroute from the firewall itself, and it doesn't have any issue routing correctly.

Code: [Select]

# /usr/sbin/traceroute -w 2 -I  -n  -m '18' -s '10.10.254.2'   '10.9.247.1'
traceroute to 10.9.247.1 (10.9.247.1) from 10.10.254.2, 18 hops max, 48 byte packets
 1  10.9.247.1  0.664 ms  0.542 ms  0.610 ms

Could this possibly be a switch issue? All that's between the client and the host on 10.9 is a unconfigured L3 switch connected via DHCP (for now) on 10.10.1.4/16 & subnet 10.10.254.2. That's the only thing that differs between pinging from the firewall directly, compared to pinging it from the client.
The switch itself also cannot ping the client.

[bartjsmit]

Do a packet trace on the firewall to catch the packets on their way in and on their way out. Interfaces, Diagnostics, Packet Capture.

The shark will help you visualise: https://www.wireshark.org/

Bart...

[niraami]

I guess I'll have to do that once I get home, because just now testing via my VPN I can ping that system just fine... Actually both of the hosts are reachable just fine.

Code: [Select]

$ ping 10.9.247.1
PING 10.9.247.1 (10.9.247.1) 56(84) bytes of data.
64 bytes from 10.9.247.1: icmp_seq=1 ttl=63 time=19.2 ms
64 bytes from 10.9.247.1: icmp_seq=2 ttl=63 time=46.6 ms
64 bytes from 10.9.247.1: icmp_seq=3 ttl=63 time=25.2 ms
64 bytes from 10.9.247.1: icmp_seq=4 ttl=63 time=19.5 ms
^C
--- 10.9.247.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 19.155/27.607/46.632/11.239 ms

$ ping 10.10.1.3
PING 10.10.1.3 (10.10.1.3) 56(84) bytes of data.
64 bytes from 10.10.1.3: icmp_seq=1 ttl=63 time=22.6 ms
64 bytes from 10.10.1.3: icmp_seq=2 ttl=63 time=23.3 ms
64 bytes from 10.10.1.3: icmp_seq=3 ttl=63 time=22.0 ms
64 bytes from 10.10.1.3: icmp_seq=4 ttl=63 time=54.2 ms
^C
--- 10.10.1.3 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 22.008/30.550/54.213/13.669 ms

My VPN is setup via OpenVPN on the subnet 172.16.10.0/24, rules setup as per the normal Road Warrior guides.
I'll report back later.

[niraami]

... i feel kinda stupid, cause I forgot that specific firewall rules should be above generic ones with destination set as "*". Moving the HOST_MGMT rule up above the redundant WAN rule fixed everything...
But without testing it on my VPN I probably wouldn't have thought about that.

Thanks for the help Bart! :)

i was able to test it, because I remembered I recently setup SSH to my local machine