Unable to access self hosted websites across vlans

[viragomann]

Yes, I suspect, that the VLANs are leaking somewhere outside of OPNsense, so that not every packet passes it.

As mentioned, check out the TCP flag of the blocked packets. Presumably they are not SYN packets. This would mean, that the respective SYN packet of the connection didn't pass OPNsense.
Normally only SYN packets are logged. If they are allowed, OPNsense sets a state, which passes following belonging packets.

[Gryphon]

Normally only SYN packets are logged. If they are allowed, OPNsense sets a state, which passes following belonging packets.

From the image below (which I have re-posted for posterity), the first three packets that are allowed are SYN. Everything afterwards that is blocked are RST. At least I think so, they are 'S' and 'R' in the fields, not the full names.



I'm not sure where exactly the packets could be leaking, though this is definitely something I'm completely unfamiliar with.

[Patrick M. Hausen]

The passed packets are from client to server, probably HTTPS. The RST packets are a reaction of the client to a packet from the server that the client did not expect. So when the server answers the client, these packets seem to go past your OPNsense instead of through it.

Does the server have a network interface in the same network the client is connected to?

[Gryphon]

Does the server have a network interface in the same network the client is connected to?

Ah, when the last comment was mentioned leaking I had a thought that this might be the problem. Yes, 10.10.110.2 on VLAN110 is also on VLAN100 as 10.10.100.7. I can see how this might be an issue. Is there any way to avoid the problem without taking that machine off both interfaces?

[Patrick M. Hausen]

Does the server have a network interface in the same network the client is connected to?

Ah, when the last comment was mentioned leaking I had a thought that this might be the problem. Yes, 10.10.110.2 on VLAN110 is also on VLAN100 as 10.10.100.7. I can see how this might be an issue. Is there any way to avoid the problem without taking that machine off both interfaces?

If the client and the server share a network why do you want the client traffic to go through OPNsense in the first place?

So options are:

- remove VLAN100 from server
- use address on VLAN100 directly for communication without going through OPNsense
- create outbound NAT rule on OPNsense on VLAN110 to NAT the connections from VLAN100 to the interface address.

What is important to understand:

IP is a connectionless protocol, so called packet switching as opposed to circuit switching. There is no "connection". The illusion of a connection is created at higher layers, e.g. TCP.
Unless special policy routing is in place a packet is routed bases on it's destination IP address only. And this decision is made for each individual packet without taking any history into account.
Specifically the server system has no knowledge about on which interface the packets from the client arrived when sending the answer. It sees a packet destined for the client, in VLAN100, it has got a direct connection to that network, it sends the packet via that network.

We get the same topic frequently in the TrueNAS forum when people try to run e.g. the web UI and the services (SMB and the like) via different VLANs.

A generic Unix system without any additional firewall or policy routing mechanisms configured has absolutely no knowledge about on which interface a packet it received actually came in. That information is simply not stored anywhere. And it will send out each packet to any destination via the "best", i.e. shortest way.

HTH,
Patrick