Baffled, traffic not passing between two LAN's

[lbh74]

So I have two LAN's, one called LAN (vtnet1 interface), one called K8S (vtnet2 interface). Hosts on any of these networks should be able to reach echt other. This worked a month ago, but I must have changed something.

I have drawn the issue in the included image. Basically I can send traffic from my host in LAN to another in K8S arrives there (proven with tcpdump), the reply traffic also arrives back on OPNsense, but then disappears, it doesn't come out of the LAN vtnet1 interface (also proven with tcpdump on both interfaces)



I only have one firewall rule on each interface: From: that_network, any, any allow. Besides the default rules that is.

If I tail -f /var/log/filter/latests.log I don't see any blocks (or check in the GUI).

Maybe it is because I'm sick at the moment, but I spend 3 hours on this and I feel like this should be simple :(

EDIT: FYI, if I disable the firewall I have the same behavior. Which makes me think this is a weird routing problem. But routing is very simple. I have no static routes, both subnets are connected directly, there is only one gateway, used by the WAN interface, pointing to the internet. Everything else points to the opnsense as a default gateway. This is the routing table of opnsense:

Code Select Expand

Internet:
Destination         Gateway            Flags     Netif Expire
default               10.0.0.1         UGS      vtnet0
10.0.0.0/24        link#1             U        vtnet0
10.0.0.2             link#7             UHS         lo0
10.0.1.0/24        link#2             U        vtnet1
10.0.1.1             link#7             UHS         lo0
192.168.0.0/24   link#3             U        vtnet2
192.168.0.1        link#7             UHS         lo0
127.0.0.1          link#7             UH          lo0


[BoodahsFever]

Without the TCPdump or logs it is hard to tell what is going on. But considering that one interface is called K8S i think there might be a kubernetes cluster on that side. There is then another network involved in the kubernetes cluster. Depending on the firewall rules that might be blocked. Maybe on the K8S side try to use a rule with source any to any instead of source network to any.

[lbh74]

Hi, I didn't install the k8s cluster yet,  It has not effect, it is just a name of the subnet/interface/bridge. I'm testing from two test VM's on each side. The tcpdump's were also quite clear, return traffic arrived on vtnet2, but didn't go out back to vtnet1. Same behavior if I turned of the firewall with pfctl. Routing was as shown in my original post, very basic. Fyi, I used tcpdump -i to specify the interface each time, had multiple tmux panes open for each interface on the opnsense, but also on each VM, so I could see the packets arrive and leave on each hop.

The firewall is a VM (running for almost a year), I turned it off and deployed a new VM and installed a fresh copy of OPNsense, recreated the config, same IP's and had no issues. I did upgrade the problematic one to 24.7 from 24.1 last night, to see if that helped, but it didn't.

For now I have to focus on a project that relies on this firewall to work, I will get back to troubleshooting the original one as I can't stand not knowing what the problem is :D

Thanks for commenting. I'll post back here if I figure out what was wrong.

[BoodahsFever]

Proxmox as Hypervisor? Sometimes when I have issues with IPv6 I open the vtnet interface affected, change nothing then save and apply and it starts working again. My WAN interface is PCI passthrough and works flawless. I am considering to also apply PCI Passthrough on my LAN interfaces to avoid weirdness (stuff i can't explain nor shows up in a log) like this. But I never had weirdness with IPv4.

Might be something with Proxmox or the VirtIO driver.