RESOVLED: Fresh Install OPNsense 26.7, unable to implement vlans with Netgear MS308E

[Schwermzilla]

Hello OPNsense forum, I apologize if this is a novice problem, but it has stumped me for days at this point and I have been searching here and at netgear's forum and not found anyone sharing the same problem, I have seen some related posts which I have tried to replicate the solutions of, to no success.

Scenario: I have OPNsense 26.7.4_1 running on a 5 port miniPC (Intel N150, 128gb ssd & 8GB ram), the pc has 3 2.5Gb ports and 2 10Gb SFP ports. Currently, I am just using the 2.5Gb ports, as I want to verify vlans work before I add more complexity. To keep things simple, I am working on a reduced complexity as things weren't working in a previous, more complex state either. So, I recently did a fresh install and full update because I was at wits end.
I am only able to upload one photo here because of image size, full folder is available here: https://drive.google.com/drive/folders/1YmCybrYoM4dVkMVTz_UjtMY8Dn6Zi6km      https://1drv.ms/f/c/676bc1fb105ced33/IgDdJIPRu_JdQJgC0L6t-j1OAQcqN-Abuw1LuiE72ZLj9BA?e=fjt0cN

Currently: LAN and WAN access work great, stable internet, DHCP leases working, Static IP config for the managed switch, wireless AP, and NAS.
Port 1 is WAN (igc0/Wide_Network in photos).
Port 2 is the LAN (igc1/Local_Network in photos).

Problem: I am trying to setup Port 3/igc2 as a truncated vlan port with only two tagged vlans on it;
tag 11 is the Guest_Network, tag 22 is the Surveillance_Network. I have setup the vlans, enabled them as interfaces, added static ipv4 addresses, allowed firewall access and allocated DHCP address pools (verifiable in the photos).
I have split out the Local_Network and vlans on separate ports, as I have seen others have issues with mixing tagged and untagged traffic on the same port with netgear/Opnsense.
The photo (netgear_ports) shares the mapping on the Netgear managed switch; ports 4/5 are connected to OPNSense, port 4 is paired with igc1 (Local), port 5 with igc2 (vlan Trunk).  Port 8 will go to an NVR which has POE ports for the cameras (haven't turned this on yet). Last relevant connection on the switch, port 1 goes to my wireless AP (Netgear Orbi Pro 6, SRX80) setup in AP mode, which works on local network access when tested. On the Orbi, it has been setup in Trunk mode since being in AP, I have two id's broadcasting there, one for 192.168.1.x (local) and the other for 192.168.11.x. (guest). I have tried running port 1 on the switch in trunk/uplink mode as well, but then I lose connection to the orbi in either Trunk or Access vlan 11 only (pictured). The only way it works is when it is set to access only local vlan 1; Then devices on the local wifi work fine, devices on the guest wifi connect to the Orbi but have an IP configuration error and aren't assigned DHCP leases, as expected.

My understanding of the problem, when in Trunk/Uplink mode, the Netgear switch is not seeing the incoming tagged traffic on igc2/port 5. Running in diagnostic mode shows zero "bytes received" on that port, the screenshot was captured after 30 mins or so. I am not confident OPNsense is sending anything, but I don't see any issues with my configuration from the OPNsense side.

Any help would be wonderful, I had spoken with my friend who is an SRE, and he gave me a bit of sanity check that I didn't have anything obviously setup wrong, but who knows, so I come to you all in hopes of finding a solution. THANK YOU!


UPDATE:

This was resolved, while there were settings that should have been changed, the main reason for ip silence was DNS.

[nero355]

Last relevant connection on the switch, port 1 goes to my wireless AP (Netgear Orbi Pro 6, SXR80) setup in AP mode, which works on local network access when tested.

On the Orbi, it has been setup in Trunk mode since being in AP, I have two id's broadcasting there, one for 192.168.1.x (local) and the other for 192.168.11.x. (guest).
I have tried running port 1 on the switch in trunk/uplink mode as well, but then I lose connection to the orbi in either Trunk or Access vlan 11 only (pictured).

The only way it works is when it is set to access only local vlan 1 : Then devices on the local wifi work fine, devices on the guest wifi connect to the Orbi but have an IP configuration error and aren't assigned DHCP leases, as expected.

So the problem is your WiFi Orbi Unit and not OPNsense then ?!

Do those things even understand VLANs on their Switch/LAN side ??
AFAIK they do not : Only on the WAN Port.

Also currently your Netgear Switch shows Port 3 as Access Port and those do not transport VLANs ofcourse...

[Schwermzilla]

So the problem is your WiFi Orbi Unit and not OPNsense then ?!

It may be a part of the current config problem, to test that, I have just plugged my laptop into port 1 and 8 to test vlan 11 and 22 and it also doesn't connect to the internet on either port in access mode for their respective vlans.
So, I am primarily trying to diagnose the managed switch and it's connection to OPNsense, so far, when I change any connection on the switch that connects to OPNsense to Trunk, I lose communication on that port (showing 0 bytes received over 30 mintues).
Not expanded above, but I have previously attempted to have the lan port assigned the non-vlan 192.168.1.x in addition to the tagged vlans of 11 and 22 (with lan/igc1 as parent), but I assumed it was sending the LAN as untagged traffic, and read elsewhere that the switch was expecting three vlans. When I was testing this I kept losing access to the OPNsense over the network when switching to Trunk mode on the switch, so I set one port, igc1, to be untagged LAN access and am attempting to diagnose the pathway of just vlans 11 and 22 from igc2 on the router, to port 5 on the switch in Trunk/Uplink mode, and then handing over vlan exclusive Access to 11 or to port 8 as exclusive access to 22, neither have worked. Essentially ignoring the AP for now.

The current connection of ports 2,3,4,6,7 are running on untagged Lan so I can continue to interface with network devices and troubleshoot the vlan path, do you think that is causing problems?

Do those things even understand VLANs on their Switch/LAN side ??
AFAIK they do not : Only on the WAN Port.

Those things being the wireless AP? and the NVR?
AP, yes I have its incoming port from the switch set to Trunk, and have the wireless network setup expecting tag 11 for the guest network wifi.
NVR, no, which is why I have it's port setup as Access - 22, expecting the netgear to provide untagged access exclusively to vlan 22. 

Also currently your Netgear Switch shows Port 3 as Access Port and those do not transport VLANs ofcourse...
Yeah, hope that is not what is causing the problem. as expanded above, other devices are runninng on LAN without tags. Hoping this would isolate the switch's Port 5 trunk port to the OPNSense igc2 connection, assigned just the two vlans and testing with the AP and a laptop for connection on either port 1 for vlan 11 or port 8 for vlan 22

[nero355]

Yeah, hope that is not what is causing the problem.

Well... if you want to transport Multiple VLANs to another Switch or Accesspoint then you need to use TAGGED VLANs and not ACCESS Mode :)

Just to be sure :

Are you aware of the following =>

OPNsense Interface (whole NIC basically) => Switch Port in ACCESS Mode
OPNsense VLAN Interface (VLAN Interface assigned to a NIC that's not used for anything else) => Switch Port in TAGGED Mode

Switch Port in ACCESS Mode => End user devices like PCs/TVs/Consoles/etc.
Switch Port in TAGGED Mode => Port of another Switch or Accesspoint that understands TAGGED VLAN traffic.

ACCESS MODE = Only 1 VLAN allowed.
TAGGED MODE = Multiple VLANs allowed.

Does your current setup look like this ??

[caplam]

it reminds me of the first time i enabled vlans on my switches.
if i understand correctly
igc1 (LAN) is linked to port4 of the switch
igc2 (VLAN11+VLAN22) is linked to port 5 of the switch.

port 4 needs to be set to access mode with PVID 1
port 5 needs to be set to trunk with vlan tag 11&22
but for this to work port5 can't have PVID 1.
I fixed things by creating a dummy vlan 99 (or whatever you want but not used ) and setting PVID 99 to port5.
and of course the port for the orbi needs pvid 1 and tagged vlan11 and vlan22 (if you have wifi cams)

[Schwermzilla]

Just to be sure :

Are you aware of the following =>

OPNsense Interface (whole NIC basically) => Switch Port in ACCESS Mode
OPNsense VLAN Interface (VLAN Interface assigned to a NIC that's not used for anything else) => Switch Port in TAGGED Mode

Switch Port in ACCESS Mode => End user devices like PCs/TVs/Consoles/etc.
Switch Port in TAGGED Mode => Port of another Switch or Accesspoint that understands TAGGED VLAN traffic.

ACCESS MODE = Only 1 VLAN allowed.
TAGGED MODE = Multiple VLANs allowed.

Does your current setup look like this ??

Yes x3, are you able to review the photos shared on google drive? https://drive.google.com/drive/folders/1YmCybrYoM4dVkMVTz_UjtMY8Dn6Zi6km
Verification of those things are visible there, but to reverify for my own sanity:

The only things in use that are connected in Access mode are the default LAN, which not setup as a vlan, so no tags from OPNsense igc1 to the switch port 4, which is set for access and connects my computer to OPNsense through the switch.

The only things connected in Trunk/tagged mode are the vlans 11 & 22 from OPNsense on igc2 to the switch port 5, which is set for Trunk/uplink. The vlan assignments are visible on the shared screenshot above, showing the interface assignments in OPNsense.

Do I need to have igc2, assigned as a "hardware" interface as well? Or does assigning the parent for the vlans as igc2 enable that device?

[Schwermzilla]

it reminds me of the first time i enabled vlans on my switches.
if i understand correctly
igc1 (LAN) is linked to port4 of the switch
igc2 (VLAN11+VLAN22) is linked to port 5 of the switch.

port 4 needs to be set to access mode with PVID 1
port 5 needs to be set to trunk with vlan tag 11&22
but for this to work port5 can't have PVID 1.
I fixed things by creating a dummy vlan 99 (or whatever you want but not used ) and setting PVID 99 to port5.
and of course the port for the orbi needs pvid 1 and tagged vlan11 and vlan22 (if you have wifi cams)

Yes that is exactly correct! Thank you for your response and assistance here.

I think I have it correct on OPNsense, igc1=port4 in access mode for lan (192.168.1.x subnet using PVID1?) While igc2 exclusively is handling the two vlans 11 & 22.
 
However, in matching the vlan settings on the switch, the pathway from igc2 to Port5 also cannot have any association Pvid/vlan 1? 
This might be the issue, Netgear's unmodifiable "default" uses vlan tag 1, would that cause conflict between PVID of OPNsense sending untagged traffic for PVID1 on Port4 vs Netgear expecting it tagged on Port5?

The Netgear switch Basic 802.1Q VLAN mode (which I have been using) seems to be always expecting tagged vlan 1 traffic in addition to the added vlans. Then, if I am following the logic correctly, the solution could be to move my untagged LAN to a different subnet location, say 192.168.99.x. Then create a new vlan 1, on subnet 192.168.1.x and run with it as my "main" network, running all three vlans (1,11,22) on port 5, with the matching vlans setup on the switch?

To make the options more confusing, the switch does have an Advanced 802.1Q VLAN support options, with port-by-port tag/untag/exclude options for each VLAN.
I initially felt like it was too overpowered for a simple test, but I may be able to the switch it to expect the mix of tagged and untagged traffic across the various ports. I will test this later tonight and update this post with the results.

[lmoore]

Netgear's unmodifiable "default" uses vlan tag 1

Referring to the User Manual for this switch (See page 45 - https://www.netgear.com/support/product/ms308e), you need to activate Advanced 802.1Q VLAN mode in order to change the PVID of a port.

[Schwermzilla]

Netgear's unmodifiable "default" uses vlan tag 1

Referring to the User Manual for this switch (See page 45 - https://www.netgear.com/support/product/ms308e), you need to activate Advanced 802.1Q VLAN mode in order to change the PVID of a port.

Thanks for taking a look and sharing the manual link! I was able to modify PVID's and I am attempting to rebuild this in the "Advanced 802.1Q VLAN" mode. I have added four new screenshots to the shared folder for visibility: https://drive.google.com/drive/folders/1YmCybrYoM4dVkMVTz_UjtMY8Dn6Zi6km   https://1drv.ms/f/c/676bc1fb105ced33/IgDdJIPRu_JdQJgC0L6t-j1OAQcqN-Abuw1LuiE72ZLj9BA?e=0TIz2N
The screenshots are prefixed with Advanced-vlan, I did move things around on the switch (sorry everyone, been organizing my cables while troubleshooting). Port 7 will be setup for vlan 22 with untagged traffic to the NVR and Port 8 is temporarily setup in the same way as port 8, untagged vlan 11 traffic for testing via my laptop. Once working, I will work on getting this port to have the three tagged vlans in trunk received by the AP, which I have removed temporarily from the setup/problem.

I have set things based on what you see in the screenshots, following what the linked Netgear manual said and I have learned in the last day. Including cleaning up the PVID table, but it still is not working. Tested on my laptop via Port 7 & 8.

The biggest issue I see is that my switch port statistics, according to the diagnostics tab for port 5, shows 0 bytes received vs 1,368,348 bytes sent. For comparison, port 4 (lan/Local untagged traffic to OPNsense) with 12,668,579 received and 9,916,805 sent.

If there is a config problem with the vlans on the switch would it show 0 bytes received? or could I have something wrong with my setup on OPNsense that isn't sending anything over port 5?

[nero355]

Are you able to review the photos shared on google drive? https://drive.google.com/drive/folders/1YmCybrYoM4dVkMVTz_UjtMY8Dn6Zi6km
Verification of those things are visible there

NOFI, but viewing the screenshots that way is horrible and seems to be optimized for their own Google Chrome browser, because LibreWolf (basically Firefox + uBlock Origin + Some Privacy stuff added) acts really weird when viewing them and I never had such issues anywhere else...

but to reverify for my own sanity:

The only things in use that are connected in Access mode are the default LAN, which not setup as a vlan, so no tags from OPNsense igc1 to the switch port 4, which is set for access and connects my computer to OPNsense through the switch.

Probably using VLAN 1 for both sides and that's just perfectly fine! :)

The only things connected in Trunk/tagged mode are the vlans 11 & 22 from OPNsense on igc2 to the switch port 5, which is set for Trunk/uplink.
The vlan assignments are visible on the shared screenshot above, showing the interface assignments in OPNsense.

That's correct too! :)

Do I need to have igc2, assigned as a "hardware" interface as well? Or does assigning the parent for the vlans as igc2 enable that device?

In all honesty : It's something that confuses me a bit too and conflicts with what I know from other systems and devices.

According to most people and from what I have seen here on the forum there is no need to Enable "the Main Interface" but in my case I have Enabled both the Interface that carries the WAN VLAN and the Interface that carries the LAN VLANs.

Reason : On for example a CISCO Router you need to Enable the Main Interface, leave it's IP configuration empty and just configure the sub-interface(s) and the VLAN configuration for them.

port 4 needs to be set to access mode with PVID 1
port 5 needs to be set to trunk with vlan tag 11&22
but for this to work port5 can't have PVID 1.
I fixed things by creating a dummy vlan 99 (or whatever you want but not used ) and setting PVID 99 to port5.

Can't you simply leave it empty ?!

I mean a dummy VLAN is fine, but a real Managed Switch allows you to simply assign the TAGGED VLANs and be done with it :)

and of course the port for the orbi needs pvid 1 and tagged vlan11 and vlan22 (if you have wifi cams)

Exactly! :)

I think I have it correct on OPNsense, igc1=port4 in access mode for lan (192.168.1.x subnet using PVID1?) While igc2 exclusively is handling the two vlans 11 & 22.

Good! :)

However, in matching the vlan settings on the switch, the pathway from igc2 to Port5 also cannot have any association Pvid/vlan 1? 
This might be the issue, Netgear's unmodifiable "default" uses vlan tag 1, would that cause conflict between PVID of OPNsense sending untagged traffic for PVID1 on Port4 vs Netgear expecting it tagged on Port5?

It's not a conflict : You can do whatever you need to do, but just make sure there is no network loop created by accident !! ;)

The Netgear switch Basic 802.1Q VLAN mode (which I have been using) seems to be always expecting tagged vlan 1 traffic in addition to the added vlans.
Then, if I am following the logic correctly, the solution could be to move my untagged LAN to a different subnet location, say 192.168.99.x. Then create a new vlan 1, on subnet 192.168.1.x and run with it as my "main" network, running all three vlans (1,11,22) on port 5, with the matching vlans setup on the switch?

You don't need an actual network : Just create the VLAN on the Switch and configure the Ports correctly and you are DONE!

Your VLAN 1 can stay as it is right now ;)

To make the options more confusing, the switch does have an Advanced 802.1Q VLAN support options, with port-by-port tag/untag/exclude options for each VLAN.
I initially felt like it was too overpowered for a simple test, but I may be able to the switch it to expect the mix of tagged and untagged traffic across the various ports. I will test this later tonight and update this post with the results.

Funny : I always first Enable any kind of Advanced/Expert Mode and from that moment on I only use the device that way! :P

[Schwermzilla]

You don't need an actual network : Just create the VLAN on the Switch and configure the Ports correctly and you are DONE!

Your VLAN 1 can stay as it is right now ;)

--

Funny : I always first Enable any kind of Advanced/Expert Mode and from that moment on I only use the device that way! :P

Thank you very much nero355! This is all helpful to my understanding and context on all of this. Since google drive doesn't play nice, maybe Microsoft works better? https://1drv.ms/f/c/676bc1fb105ced33/IgDdJIPRu_JdQJgC0L6t-j1OAQcqN-Abuw1LuiE72ZLj9BA

It seems like the best way to solve for this and allow for future updates is to have things more granularly setup in the advanced settings. In my previous post, caplam and lmoore also brought this up. However, after implementing "advanced mode" on the switch, I am still unable to access either vlan from my laptop when testing.

My immediate concern is the lack of traffic I can see from the OPNsense router. Do you have any insight into how/why I see 0 activity from OPNsesne on port5?

[nero355]

Thank you very much nero355!

I feel like you are very close to setting this up properly so it would be a shame not to help! :)

This is all helpful to my understanding and context on all of this.

COOL!

Since google drive doesn't play nice, maybe Microsoft works better? https://1drv.ms/f/c/676bc1fb105ced33/IgDdJIPRu_JdQJgC0L6t-j1OAQcqN-Abuw1LuiE72ZLj9BA

A little bit better...

It seems like the best way to solve for this and allow for future updates is to have things more granularly setup in the advanced settings.
In my previous post, lmoore also recommended this.

Unfortunately, after implementing this on the switch, I am still unable to access either vlan from my laptop when testing.

That sucks! :(

As far as I can see you have set it up correctly and I have no idea why it's not working...

- OPNsense stuff seems OK.
- Switch stuff now seems OK too.
But... I don't know this Switch software-wise so maybe there is more to it ?!

My immediate concern is the lack of traffic I can see from the OPNsense router.
Do you have any insight into how/why I see 0 activity from OPNsesne on port5?

Dunno...

Your Firewall Rules seem to be fine so that can't be it, can it ?!

To make sure it's not some stupid DNSmasq issue or the Firewall blocking communication to it you could assign the Laptop a Static IP Address and test that way : Ping the Gateway and see what happens...

To test if the Switch configuration is correct should be simple : Replace the OPNsense machine with another PC/Laptop and start pinging while both have Static IP Addresses configured.


Good luck! :)

[Schwermzilla]

Your Firewall Rules seem to be fine so that can't be it, can it ?!

To make sure it's not some stupid DNSmasq issue or the Firewall blocking communication to it you could assign the Laptop a Static IP Address and test that way : Ping the Gateway and see what happens...

To test if the Switch configuration is correct should be simple : Replace the OPNsense machine with another PC/Laptop and start pinging while both have Static IP Addresses configured.


Good luck! :)

You got it! memes be praised, it was DNS.

Following your thoughts, I was able to connect with a static ip on my laptop, received a DNS error trying to access any website. So I just deleted both DNS setups and reservations, and rebuilt/re-enabled everything in DNSmasq and it worked! So I clearly fumbled something in the DNS settings on my first... many attempts.

I will figure out how to mark this as Resolved!

Thanks again!