No connection to OPNSense over tagged VLAN interface

[Phiolin]

A bit of a strange issue here that I fail to understand.

Client is a MacBook which I mainly use for all kinds of admin stuff.
Client is connected via a switch port that has untagged/native VLAN 10 and tagged VLAN 99 configured.

OPNsense admin web-gui and SSH are configured to listen on all interfaces and of course OPNsense has interfaces configured in VLAN 10 and in VLAN 99. Firewall rules allow the relevant connections.

Client can reach OPNsense on VLAN 10: no problem, web-gui and SSH access working fine.
Client fails to reach OPNsense on VLAN 99: no access to web-gui and SSH.
Client can however reach other devices on VLAN 99 perfectly fine, just not OPNsense, so generally VLAN 99 connectivity seems to be working.

Now I switch the client to a native/untagged VLAN 99 switch port for verifying and connection immediately works fine.
Client can reach OPNsense on VLAN 99: web-gui and SSH access working fine.

In the first scenario with VLAN 10 untagged and VLAN 99 tagged, a packet capture on the OPNsense side sees a lot of TCP retransmissions. It looks like there is some kind of connectivity between the devices (TLS handshake), but something seems to fail.
I have attached an image of the packet capture and the pcap file from the session, if that helps.

The VLAN 99 interface on the client side is a virtual interface on the adapter that also holds the VLAN 10 connection - so both will share the same MAC address. Would that be an issue? I'd think switches can tell that apart and shouldn't have an issue with same MAC addresses in different VLANs and as connections to other devices on VLAN 99 work fine, I'd not think that would be an issue here?

[bartjsmit]

native VLAN 10 and tagged VLAN 99 configured.
Client can reach OPNsense on VLAN 10
Client fails to reach OPNsense on VLAN 99

This behaviour seems correct. The Mac does not tag packets but sends packets untagged to its switch port, and the switch then tags them with the native VLAN 10 before forwarding them to OPNsense which then routes them to the devices on VLAN 99.

You could trunk both VLAN 10 and 99 to the Mac https://support.apple.com/en-gb/guide/mac-help/mh15134/mac

A bit of a strange issue here that I fail to understand.

Install Wireshark on the Mac, grab packets locally and import traces from OPNsense. Evidence over speculation :) https://www.wireshark.org/

Bart...

[nzkiwi68]

I agree, wireshark and capture the packets.

But - if I had to guess, one side is tagging packets and the other side isn't, so you will strange things happening with only one way traffic flowing.

[Demusman]

A bit of a strange issue here that I fail to understand.

Client is a MacBook which I mainly use for all kinds of admin stuff.
Client is connected via a switch port that has untagged/native VLAN 10 and tagged VLAN 99 configured.

How are you testing the tagged vlan?
You would have to set the nic in the mac to tag vlan 99 on it.
An untagged nic can't access a tagged vlan.

[Phiolin]

Yes, of course a virtual VLAN adapter has been added for the VLAN 99 on the Mac.
Here's the network devices, en7 parent adapter and virtual vlan0 adapter.
That shouldn't really be the issue.
As I said - I can reach other devices on VLAN 99 just fine, just not the OPNsense device.
It goes so far, that I can even ping OPNsense from the Mac and will also get a DHCP address assigned, just cannot access the web gui via HTTPS or SSH.

When I switch the Mac to a native VLAN 99 access port and configure en7 with appropriate IP addresses, then everything works fine, so there's also no firewall rules in the way that would prevent the connections.

Code Select Expand

en7: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6467<RXCSUM,TXCSUM,VLAN_MTU,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether 80:6d:97:2b:1b:a6
inet6 fe80::94:1c33:8678:aba9%en7 prefixlen 64 secured scopeid 0xd
inet 10.0.11.168 netmask 0xfffffe00 broadcast 10.0.11.255
nd6 options=201<PERFORMNUD,DAD>
media: autoselect (1000baseT <full-duplex>)
status: active
vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
options=6063<RXCSUM,TXCSUM,TSO4,TSO6,PARTIAL_CSUM,ZEROINVERT_CSUM>
ether 80:6d:97:2b:1b:a6
inet6 fe80::77:1d43:f194:879d%vlan0 prefixlen 64 secured scopeid 0xf
inet 10.0.99.50 netmask 0xffffff00 broadcast 10.0.99.255
nd6 options=201<PERFORMNUD,DAD>
vlan: 99 parent interface: en7
media: autoselect (1000baseT <full-duplex>)
status: active


Here's a curl test to see that traffic goes through and I can hit the HTTP-redirect rule, but cannot successfully establish the HTTPS session:

Code Select Expand


curl -vvv http://10.0.99.1:80
*   Trying 10.0.99.1:80...
* Connected to 10.0.99.1 (10.0.99.1) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.0.99.1
> User-Agent: curl/7.87.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Location: https://10.0.99.1/
< Content-Length: 0
< Date: Tue, 04 Apr 2023 14:32:29 GMT
< Server: OPNsense
<
* Connection #0 to host 10.0.99.1 left intact


Code Select Expand


curl -vvv https://10.0.99.1:443
*   Trying 10.0.99.1:443...
* Connected to 10.0.99.1 (10.0.99.1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection 0
curl: (35) Recv failure: Connection reset by peer


[axsdenied]

Feels like an issue on the Mac side but let me ask a basic question.  Why are you plugging a client straight into a trunked port?

Why not just set it up as a native vlan and configure your clients access to wherever you want it to go, including access to other vlans, via OPNsense?