Weird speed issue

[InstigatorX]

I have 5 subnets/vlans configured and 1Gbps cable Internet. I have this weird speed issue where if I'm on any subnet/vlan besides the default LAN (no vlan tag and the interface configured when installed from scratch) I get full speed a la speedtest. If I connect to LAN, I only get ~250Mbps. It's really weird. All the rules and settings are identical on the interfaces.

Thoughts?

Attaching some screenshot of the layout.

[pfry]

Heh. Does everybody try mixing tagged and untagged VLANs on an interface? And here I thought I and the other thousand folks here who've done it were special.

https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html

FreeBSD has trouble passing the correct entrance interface to pf when mixing tagged and untagged frames. What's surprising is your setup functions at all. I imagine it depends on your ruleset - a permissive one might work. But try switching LAN to a VLAN (tagged) interface (without locking yourself out...) and see if it helps. Also, virtual Ethernets can have some odd performance characteristics relating to "hardware" offloads, but not having used/planning to use such, I've paid limited attention.

[InstigatorX]

I put LAN onto a VLAN and no longer mixing tag/untagged and the speed problem persists on LAN only. All the other VLANs speed is normal. I've scrubbed all the FW/NAT rules (i.e., removed anything non-standard). Why would LAN only exhibit the speed issue? No shaping, either.

[meyergru]

Since this is a VM setup, this can be influenced by you bridging setup in the virtualisation host. Also, presumably, LAN is shared with the VM host on a bridge?

[dseven]

FreeBSD has trouble passing the correct entrance interface to pf when mixing tagged and untagged frames.

That's a new one on me. Could you point to any documentation (e.g. bug report) about this issue?

[pfry]

That's a new one on me. Could you point to any documentation (e.g. bug report) about this issue?

Reasonable question. I neglected the interface type: a bridge. As for other types, I have not tested them (nor do I plan to, as such is not required for my use cases).

[dseven]

Yeah, it's not supported to have an interface be a member of a bridge and also have tagged VLAN subinterfaces.

Otherwise there is no fundamental reason (yet been presented to me) for mixing untagged and tagged on the same interface being bad. There are some potential consequences to be aware of, such as stupid Windows NIC drivers blindly stripping VLAN tags and processing frames as if had been received untagged and causing things like IPv6 RA from (other) VLANs to get "leaked", but that has nothing to do with OPNsense or FreeBSD, and should be handled by only presenting the appropriate VLAN(s) to the Windows client at the switch port.