Messages

well, after way too much time spent on this I am giving up, for now at least. I don't absolutely need this dual home VM there but I hate not getting to the bottom of an issue like this.
It must be asymmetric routing related as some (?) return traffic "comes back" via the opnsense interface from other network devices despite staying in the same local VLAN. I put "come back" in quotes because it's more like "seen by opnsense" which causes all these drop logs, but the packets do arrive.
when I remove the VM's interface in that VLAN, causing all traffic to have to go through opnsense, no more state violations.
I tested everything I could think of, but the Linux bridge setup in proxmox is pretty simple and checks out and the interfaces in opnsense also show what I would expect. yet..
I might come back to this, and if anyone has theories I'm interested. Thanks!

more troubleshooting (still clueless). I picked the worst offender, 192.168.50.116 to 192.168.50.102 (the latter IP being one of the new VM dual home NICs). neither are an opnsense interface

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50  "src 192.168.50.116 and dst 192.168.50.102"-> shows tons of traffic as expected on a promiscuous & bridged interface in this network

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50 -p "(ether host $OPNSENSEMAC or ether broadcast) and src 192.168.50.116"-> no more promiscuous, this shows exactly what I would expect, traffic from this source host leaving (or trying to leave) this subnet, being sent to opnsense. destinations here are never in 192.168.50.0/24

Code Select Expand

root@OPNsense# tcpdump -ni vtnet0_vlan50 -p "(ether host $OPNSENSEMAC or ether broadcast) and src 192.168.50.116 and dst 192.168.50.102"-> again shows nothing, which makes sense as local traffic should not be addressed to the opnsense MAC address. yet during this test my opnsense logs keep filling up with drops for this exact source/destination/interface pattern?!

Even when it's disabled it still occures because of :

there are only 3 routes which are correct (to each local network on the right interface, plus the default gateway which again points to the right interface).

When your Client/Server receives packets from Network X via Gateway Z the data won't go back via Gateway Z if the Client/Server has a NIC that it also connected to Network X and will use that NIC to send the data back to Network X.

That's an excellent point that I had not fully thought through but I think you're right about this being one of the factors I am dealing with. I actually have a few different dual homed machines on the network (a couple that still have an interface in the native VLAN in addition to their new home for example). And this should explain some of the random drops I have been seeing.

But in terms of the high volume of drops that has become very obvious recently, they are dropped by opnsense for invalid state and logs show those packets have both their source and destination IP in the same VLAN/subnet (and neither source nor destination are an opnsense interface obviously). Logs show those drops are for INcoming packets on the FW interface of that VLAN/subnet. But those shouldn't be routed to the firewall and given it's local traffic (ARP finds the neighbor, traceroute says it's one hop...), I still can't explain this by an asymmetric routing issue. I am missing something ...

another puzzling thing is that everything works. I fill up logs with drop events, but those flows all succeed. If I had a serious asymmetric routing issue I would expect to experience network problems, but I do not.

Thanks for the prompt answers meyergru and nero355.
I am well aware about the pitfalls of dual homed hosts, and I did of course double check those settings as I mentioned - packet forwarding is off for IPv4 & 6 (I actually disabled v6 entirely for now to rule out issues), there are only 3 routes which are correct (to each local network on the right interface, plus the default gateway which again points to the right interface). It does smell like asymmetric routing but I haven't yet figured out how this can possibly happen here.
I suspect it's somehow something wrong either at the proxmox ethernet bridge (somehow opnsense picks up packets not destined for its MAC) or maybe it's somewhere else in my small homelab network that something funny happens (it's just a few random inexpensive managed switches from different brands cobbled together).
when I find more time I'll look closer at some pcaps...

Thank you very much meyergru, your tips have been invaluable. I've happily been running this setup for many months and it's proven really solid and reliable.

I just came across an issue recently which is not a show stopper but quite annoying. I noticed that since I recently moved a busy service to the same proxmox host than runs Opnsense, it has become apparent that I get tons of firewall drop events (and the associated log spam). Everything in my setup runs over VLANs on top of bridged interfaces passed to Opnsense (vtnet). And to be clear, everything actually works great, routing is fine, firewall does what it must etc.
It's just that now that I added a dual homed VM to the same proxmox host (attached to two of the bridges Opnsense uses),I notice somehow Opnsense *also* deals with packets that shouldn't be processed, so I get tons of drops. I get drops for intra VLAN activity (from/to the same local subnet). And it seems it's like return traffic only (like it's all ACKs, never a SYN). Obviously Opnsense drops these as invalid state/default deny. But again everything actually works and those local flows are fine, it's just the load and spam on the firewall which is annoying. routing wise (netmasks, gateways/static routes etc) are all correct on all hosts involved. It's not just this new VM either, it happens with random other clients/servers, it's just that this new box is chatty so the issue only now became very apparent.

I'm going to troubleshoot further but in case anyone has a clue that would be much appreciated.
thanks!