Direct connected WAN Routing Bugged

Started by utahbmxer, June 08, 2021, 03:06:09 AM

Previous topic - Next topic
I've been using OPNsense for a little over a year as my home firewall (after switching from Sophos).  Been amazing.  I have my LAN interface setup with several VLANs, one of those VLANs is part of my lab.  I have a Cisco ASA plugged into that VLAN that's been working fine with a few VMs behind it.  I've been wanting to play around (troubleshoot for work) with some IPsec stuff on the ASA and so I put a OPNsense VM in the same lab VLAN, with a VM behind it just like the ASA.

ASA outside and OPNsense WAN are in the same /24 subnet.

ASA 192.168.10.11/24
OPNsense 192.168.10.22/24

Here's the bug, traffic leaves the Cisco ASA (or any other VM in the lab subnet), hits the WAN of OPNsense VM, but the response traffic is borked.  The dest IP is the Cisco ASA (great), but the MAC address of the Ethernet header has the gateway of my home OPNsense (physical one).

Route table on the OPNsense VM shows:

ipv4   default   192.168.10.11   UGS   56135   1500   hn1   wan       
ipv4   127.0.0.1   link#2   UH   74   16384   lo0   Loopback       
ipv4   192.168.10.0/24   link#6   U   172   1500   hn1   wan       
ipv4   192.168.10.1   00:15:5d:01:02:bd   UHS   6414   1500   hn1   wan       
ipv4   192.168.10.11/32   192.168.10.11   UGS   54731   1500   hn1   wan       
ipv4   192.168.10.22   link#6   UHS   0   16384   lo0   Loopback       
ipv4   192.168.100.0/24   link#5   U   79009   1500   hn0   lan       
ipv4   192.168.100.1   link#5   UHS   0   16384   lo0   Loopback

Why with the direct /24 route (as well as a /32 with a gateway IP of the Cisco) does the traffic use the wrong MAC address?  This makes OPNsense hard to use in a lab.  Sure I could put each in their own VLAN and route between them on my main OPNsense box, but I shouldn't have to.  Sophos (UTM and XG), pfsense, Cisco, Juniper and all my Linux and Windows VM work just fine in this scenario with no special config.  Is this a bug, or am I missing something?

June 09, 2021, 01:20:27 AM #1 Last Edit: June 09, 2021, 01:22:54 AM by utahbmxer
This doesn't make any sense.  From the shell, arp doesn't even show the ASA entry during active pings from it.

root@OPNsense:~ # tcpdump -i hn1 -ne icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on hn1, link-type EN10MB (Ethernet), capture size 262144 bytes
16:52:32.920731 44:d3:ca:12:15:c0 > 00:15:5d:01:02:bd, ethertype IPv4 (0x0800), length 74: 192.168.10.11 > 192.168.10.22: ICMP echo request, id 1, seq 15999, length 40
16:52:32.920891 00:15:5d:01:02:bd > a0:36:9f:28:75:1c, ethertype IPv4 (0x0800), length 74: 192.168.10.22 > 192.168.10.11: ICMP echo reply, id 1, seq 15999, length 40

root@OPNsense:~ # arp -na
? (192.168.10.1) at a0:36:9f:28:75:1c on hn1 expires in 943 seconds [ethernet]
? (192.168.10.22) at 00:15:5d:01:02:bd on hn1 permanent [ethernet]
? (192.168.100.10) at 00:15:5d:01:02:be on hn0 expires in 1075 seconds [ethernet]
? (192.168.100.1) at 00:15:5d:01:02:bc on hn0 permanent [ethernet]


As soon as I ping from the OPNsense VM, it's there.  However, the ping response still goes to the wrong MAC.

root@OPNsense:~ # ping 192.168.10.11
PING 192.168.10.11 (192.168.10.11): 56 data bytes
64 bytes from 192.168.10.11: icmp_seq=0 ttl=255 time=1.870 ms
64 bytes from 192.168.10.11: icmp_seq=1 ttl=255 time=1.305 ms

root@OPNsense:~ # arp -na
? (192.168.10.1) at a0:36:9f:28:75:1c on hn1 expires in 911 seconds [ethernet]
? (192.168.10.11) at 44:d3:ca:12:15:c0 on hn1 expires in 1197 seconds [ethernet]
? (192.168.10.22) at 00:15:5d:01:02:bd on hn1 permanent [ethernet]
? (192.168.100.10) at 00:15:5d:01:02:be on hn0 expires in 1171 seconds [ethernet]
? (192.168.100.1) at 00:15:5d:01:02:bc on hn0 permanent [ethernet]