[HOWTO] OpnSense under virtualisation (Proxmox et.al.)

[meyergru]

You must be very careful with dual-homed hosts:

a. they should not route packets between interfaces
b. that includes setting the gateway on the correct interface

Normally, you use this only in order to be able to reach the machine via a second "leg". I do that sometimes, when I have a VM that lies behind a reverse proxy with a "LAN" leg and I still have a direct IPv6 connection. In such cases, the LAN side has no gateway at all, because the reverse proxy accesses it via its own LAN IP.

[snulgy]

Thanks for the prompt answers meyergru and nero355.
I am well aware about the pitfalls of dual homed hosts, and I did of course double check those settings as I mentioned - packet forwarding is off for IPv4 & 6 (I actually disabled v6 entirely for now to rule out issues), there are only 3 routes which are correct (to each local network on the right interface, plus the default gateway which again points to the right interface). It does smell like asymmetric routing but I haven't yet figured out how this can possibly happen here.
I suspect it's somehow something wrong either at the proxmox ethernet bridge (somehow opnsense picks up packets not destined for its MAC) or maybe it's somewhere else in my small homelab network that something funny happens (it's just a few random inexpensive managed switches from different brands cobbled together).
when I find more time I'll look closer at some pcaps...

[nero355]

I did of course double check those settings as I mentioned - packet forwarding is off for IPv4 & 6 (I actually disabled v6 entirely for now to rule out issues)

Even when it's disabled it still occures because of :

there are only 3 routes which are correct (to each local network on the right interface, plus the default gateway which again points to the right interface).

When your Client/Server receives packets from Network X via Gateway Z the data won't go back via Gateway Z if the Client/Server has a NIC that it also connected to Network X and will use that NIC to send the data back to Network X.

It does smell like asymmetric routing but I haven't yet figured out how this can possibly happen here.

I am working on a document about this, but it's not finished yet because I am writing it many months after having dealt with such an issue so I need to double check a lot of things.

If it turns out you need to do something to fix A-Symmetric Routing let me know and I will try to help you as much as possible!

[snulgy]

Even when it's disabled it still occures because of :

there are only 3 routes which are correct (to each local network on the right interface, plus the default gateway which again points to the right interface).

When your Client/Server receives packets from Network X via Gateway Z the data won't go back via Gateway Z if the Client/Server has a NIC that it also connected to Network X and will use that NIC to send the data back to Network X.

That's an excellent point that I had not fully thought through but I think you're right about this being one of the factors I am dealing with. I actually have a few different dual homed machines on the network (a couple that still have an interface in the native VLAN in addition to their new home for example). And this should explain some of the random drops I have been seeing.

But in terms of the high volume of drops that has become very obvious recently, they are dropped by opnsense for invalid state and logs show those packets have both their source and destination IP in the same VLAN/subnet (and neither source nor destination are an opnsense interface obviously). Logs show those drops are for INcoming packets on the FW interface of that VLAN/subnet. But those shouldn't be routed to the firewall and given it's local traffic (ARP finds the neighbor, traceroute says it's one hop...), I still can't explain this by an asymmetric routing issue. I am missing something ...

another puzzling thing is that everything works. I fill up logs with drop events, but those flows all succeed. If I had a serious asymmetric routing issue I would expect to experience network problems, but I do not.

[snulgy]

more troubleshooting (still clueless). I picked the worst offender, 192.168.50.116 to 192.168.50.102 (the latter IP being one of the new VM dual home NICs). neither are an opnsense interface

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50  "src 192.168.50.116 and dst 192.168.50.102"-> shows tons of traffic as expected on a promiscuous & bridged interface in this network

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50 -p "(ether host $OPNSENSEMAC or ether broadcast) and src 192.168.50.116"-> no more promiscuous, this shows exactly what I would expect, traffic from this source host leaving (or trying to leave) this subnet, being sent to opnsense. destinations here are never in 192.168.50.0/24

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50 -p "(ether host $OPNSENSEMAC or ether broadcast) and src 192.168.50.116 and dst 192.168.50.102"-> again shows nothing, which makes sense as local traffic should not be addressed to the opnsense MAC address. yet during this test my opnsense logs keep filling up with drops for this exact source/destination/interface pattern?!