TCP stream stops being routed

[jbergler]

Hello, I'm having some connection issues between two vlans for which OPNsense is the router.
I believe (but am not sure) that this started with the upgrade to 22.1.

I'm running OPNsense as a VM, with a network card being passed in via PCI passthrough.
It's a router on a stick with a bunch of vlans on a trunk port. The relevant ones for this path are
 vlan 100 - 192.168.0.1/24
 vlan 102 - 192.168.2.1/24

Both vlans are dual stack, using a delegated prefix from upstream.

When I SSH from a device on vlan100 (192.168.0.53) to a device on vlan102 (192.168.2.23) using IPv4 the connection works for a little bit before hanging, and eventually disconnecting.

I have attached packet captures for both interfaces which show packets making it in both directions just fine for a while, but after frame 76, nothing makes it out vlan 102.

When I ssh over v6 between the same two hosts the connection seems unaffected.

At first I thought his might be arp related, but watching the arp table on opnsense shows no issues.

Any ideas for what I should be looking at next, or how I can debug this?

[cookiemonster]

Seems to be a one-way -only conversation ie. no dialogue.
I see no traffic back, so my guess is there's a firewall rule missing or getting in the way.

[jbergler]

That's a great observation - 192.168.2.23 also had an interface on vlan100, which resulted in the return traffic being sent directly on vlan100.

It makes sense that it works initially, but I guess something is happening on opnsense that results on the flow being invalidated maybe since it never sees the return traffic.

If I remove/disable the second interface and all the traffic goes through opnsense it works as expected.

[5SpeedFun]

I think if the traffic is only one way, and opnsense never sees the response, the TCP connection will eventually time out & the state will be closed/removed from OPNSense.  No state = no further traffic. If you time it from opening connection to connection dying, is it almost right at 30 seconds or 60 seconds or something?

This might be the issue you are having and wouldn't be specific to opensense.

[jbergler]

It seems to be more tied to total bytes sent than to time (definitely wasn't repeatable)

Either way, I don't need the flows to be asymmetrical so I can solve my problem by just routing through the box.