Possible Routing Bug?

[utahbmxer]

I have 19.1 deployed in Hyper-V for testing, it's WAN interface is on VLAN10 (access) within Hyper-V, no tagging done in OPNSense (have tried no VLAN as well).  I am observing some strange behavior while trying to build an IPsec tunnel to another network appliance in the same WAN subnet.  After scratching my head for 2 days wondering why that other network appliance was not getting any response from the OPNsense appliance, i found the issue but have been unable to figure out the why.

Traffic from any host in the same /24 WAN network going to OPNsense WAN address, the response/return goes to the default gateway of the WAN interface (my actual home firewall).  The IP in the TCP/IP header is correct, but the MAC address is that of my home firewall.  ARP table looks correct, no static routes, no custom NAT, I am stumped.

root@OPNsense:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.10.1       UGS         hn1
127.0.0.1          link#2             UH          lo0
172.16.16.0/24     link#5             U           hn0
172.16.16.1        link#5             UHS         lo0
192.168.10.0/24    link#6             U           hn1
192.168.10.9       link#6             UHS         lo0

root@OPNsense:~ # arp -a -n
? (192.168.10.1) at a0:36:9f:28:75:24 on hn1 expires in 1132 seconds [ethernet]
? (192.168.10.9) at 00:15:5d:01:02:32 on hn1 permanent [ethernet]
? (192.168.10.13) at 00:15:5d:01:02:38 on hn1 expires in 1087 seconds [ethernet]
? (172.16.16.100) at 00:15:5d:01:02:33 on hn0 expires in 1041 seconds [ethernet]
? (172.16.16.1) at 00:15:5d:01:02:31 on hn0 permanent [ethernet]

Here is where it get's interesting.  A ping from the OPNsense appliance uses the correct MAC, but immediately after I ping from the other side and the last packet has the wrong MAC (that of the default gateway.

18:54:53.342957 00:15:5d:01:02:32 > 00:15:5d:01:02:38, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 27241, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.10.9 > 192.168.10.13: ICMP echo request, id 16234, seq 15, length 40
18:54:53.343594 00:15:5d:01:02:38 > 00:15:5d:01:02:32, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 55050, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.10.13 > 192.168.10.9: ICMP echo reply, id 16234, seq 15, length 40

18:55:03.320031 00:15:5d:01:02:38 > 00:15:5d:01:02:32, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 127, id 15972, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.10.13 > 192.168.10.9: ICMP echo request, id 1, seq 260, length 40
18:55:03.320259 00:15:5d:01:02:32 > a0:36:9f:28:75:24, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 63, id 5355, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.10.9 > 192.168.10.13: ICMP echo reply, id 1, seq 260, length 40

[mimugmail]

Interface : WAN set Upstream Gateway to auto-detect

[utahbmxer]

I have tried that setting when set to Static address.  "Out of the Box" it was using DHCP for WAN and automatically created the gateway, when DHCP is used that option isn't there.

The funny thing is I have setup this scenario many times using pfsense and even opnsense one other time, never seen this before.  Super strange.

[mimugmail]

So does it work now with auto detect?

[utahbmxer]

Nope, doesn't work.  I have been trying to test some stuff on a Sophos XG which is deployed to the same environment, same WAN subnet, etc.  It works just fine.  I deployed a pfsense appliance, it responds correctly.  I reinstalled opnsense, stepped through the wizard, added ICMP to the WAN rules, same behavior.  Response traffic is headed to the MAC of my home firewall.  Must be a bug.

Oh well, I have a urgent thing I am working on with Sophos, will have to come back to OONsense later.

[mimugmail]

This is exactly the behavior when you have set an upstream gateway in interface config

[utahbmxer]

What odd behavior as I can't specify an upstream when WAN in configured to DHCP.  Even though there is a direct/connected route for devices in that subnet, it still chooses the default gateway (upstream)?  The next "hop" (not really a hop) should be the device in the same subnet according to the route table.   Sounds like that needs to go back to the drawing board, never seen a firewall do that.  My ASA, SRX, SSG, pfSense, Sophos (UTM & XG) have never behaved that way.

[Maurice]

Try "Disable force gateway" in Firewall / Settings / Advanced.

Cheers

Maurice

[putt1ck]

Interface : WAN set Upstream Gateway to auto-detect

This fixes the issue for us, but means it is a bug. Or at least I can't think of a reason why telling a router which gateway to use would cause it not to route (sometimes) when it's the only gateway available.

[mimugmail]

Usually these gateways are only set in DHCP and in such setups it's highly unusual to have mutiple gateways on WAN side.

[putt1ck]

No further detail yet (not sure where to look) but we're seeing this issue flip-flop depending on changes made.

Scenario:
central office firewall with multiple internal subnets and public IPs. Single gateway (ISP "true" fibre router).

adding a new rule (LAN/SSN certainly, not observed with WAN rules - but it's not an everytime thing) and routing in one direction stops; changing the WAN gateway to auto and then back to manual gets everything back working.