OPNsense and MikroTik CSS326 and Mgmt VLAN problem

[seki]

Hey!

So here's the problem. I want to set up a Management VLAN for the OPNsense and MikroTik CSS326-24G-2S+RM Switch.


OPNsense:

Iface	Physical iface	Virt. iface	DHCP
WAN	re0		IP assigned by ISP
LAN0	igb0		ISC-DHCP pool 10.0.0.1/24
LAN1	igb1		ISC-DHCP pool 10.1.0.1/24
Mgmt	Parent: igb0	vlan0005	ISC-DHCP pool 10.0.5.1/24
WiFi	Parent: igb0	vlan0100	ISC-DHCP pool 10.0.100.1/24

As you already see second octet is a physical interface, third one is VLAN (zero if not a VLAN).

CSS326 Switch (SwOS 2.17):

- Port 24 connected to LAN0(igb0)
- Port 22 connected to PC


CASE #1 (Switch has no config)
Starting both devices. My PC (connected to Port24) gets IP 10.0.0.2/24. I can access OPNsense GUI/SSH with no problem.
Switch only accessible if I manually assign IP address to my PC: 192.168.88.2 cause by default Switch is having 192.168.88.1

CASE #2 (Attempting to config Switch)
My intention is to have port 22 and 23 as Mgmt ports, one for my PC and the other one for the console. So in VLANs tab in SwOS I create two VLANs:

• ID: 1, Name: Default, all ports are members of this VLAN
• ID: 5, Name: Mgmt, all ports are members of this VLAN

Everything is working so far. I also need to go to VLAN tab where I have all ports listed with their default values:

• VLAN Mode: optional
• VLAN Receive: any
• Default VLAN ID: 1
• Force VLAN: unchecked

Since port 24 is going to be a trunk port then I set it up as follows:

• VLAN Mode: enabled
• VLAN Receive: any
• Default VLAN ID: 5
• Force VLAN: unchecked

After saving the config I lose my access to both OPNsense and switch


CASE #3 (Attempting to give Switch IP from Mgmt pool)
OPNsense has already everything configured. On SwOS I go to System tab and in the field named "Allow From VLAN" I put "5" cause thats the VLAN tag for the Mgmt interface on OPNsense, right?
Immediately after saving the config the connection is lost.

Questions:
I am upgrading my old Cisco switch where I was able to get IP from Mgmt DHCP pool. Port 22 and 23 was configured as Mgmt port on the switch but somehow I cannot understand how to do this on MikroTik?
What am I doing wrong?

How to get the following setup?

• Switch gets IP from Mgmt pool (10.0.5.2)
• Ports 22 and 23 are Mgmt ports
• Clients from Ports 1-16 get IPs from WiFi pool (10.0.100.1/24)

Seems like I don't understand how MikroTik switch is handling VLANs

[Seimus]

First of all, do not mix untagged and tagged VLANs on OPNsense. Just dont, it may work but it may bring problems down the road.
https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html

Second, In order to access a switch management in a VLAN, you need to tell that switch to have its management on a VLAN, for that consult the official Vendors documentation. (System TAB)

https://wiki.mikrotik.com/SwOS/CSS326

Allow From VLAN    VLAN ID from which the switch is accessible (VLAN Mode on ingress port must be other than disabled in order to connect)

The port towards OPNsense needs to be set as TRUNK, and have the VLANs allowed on the TRUNK port
The port towards PC needs to have PVID set to the VLAN you want to have the connection on, e.g 5 if its VLAN 5

Regards,
S.

[seki]

First of all, do not mix untagged and tagged VLANs on OPNsense. Just dont, it may work but it may bring problems down the road.
https://docs.opnsense.org/manual/how-tos/vlan_and_lagg.html

I apologize but I don't understand. Where do I mix VLANs on OPNsense?

Second, In order to access a switch management in a VLAN, you need to tell that switch to have its management on a VLAN, for that consult the official Vendors documentation. (System TAB)

Once I put value "5" in Allow From VLAN I immediately lose connectivity to the Switch, to OPNsense etc.

[cookiemonster]

> I apologize but I don't understand. Where do I mix VLANs on OPNsense?
Here:

Code Select Expand

Iface    Physical iface    Virt. iface    DHCP
LAN0    igb0        ISC-DHCP pool 10.0.0.1/24
Mgmt    Parent: igb0    vlan0005    ISC-DHCP pool 10.0.5.1/24
WiFi    Parent: igb0    vlan0100    ISC-DHCP pool 10.0.100.1/24[/td]igb0 has LAN untagged and vlan0100 tagged.

[cookiemonster]

Leet me get you a thread with my setup, same hardware.
There you go: https://forum.opnsense.org/index.php?topic=36530.msg178381#msg178381
and have a read on https://forum.opnsense.org/index.php?topic=36374.msg177867#msg177867

[cookiemonster]

So in sum, since you are using a single managed switch, you would create your VLANs on the switch, send all traffic tagged to OPN where you also set your VLANs to match as it will be doing the routing, and dedicate your designated management ports on the switch.
You'd want to check the latest documentation for management vlan ports https://help.mikrotik.com/docs/spaces/SWOS/pages/76415036/CRS3xx+and+CSS326-24G-2S+series+Manual#CRS3xxandCSS32624G2S%2BseriesManual-Managementaccess
From the pictures, the eth2 is the trunk and the one to go to OPN physically.
Then on OPN your VLANs are setup as per the forum links above.

[mooh]

Immediately after saving the config the connection is lost.

Of course it is if I understand the situation correctly.

• The switch will change its IP address when forcing it to listen on VLAN 5 only, depending on the "Address Acquisition" setting. You need to find its new address.
• Your PC (port 22) is not on the management network because only port 24 is on VLAN 5.

You need to make sure that either your PC and your switch are on the same network or that the firewall is configured to allow communication between your PC and the switch.

There's no need to declare a default VLAN in the "VLANs" tab. The name "default" for a VLAN doesn't mean anything to the switch. The VLAN tag used for untagged traffic is the number specified in "VLAN" "Default VLAN ID" setting.

[seki]

Hey cookiemonster!

Thank you for the great effort of talking to MikroTik community and explaining the way how MT understands and treats Native VLAN concept.

So in other words let me ask you if I understand this correctly:


Previously I had Cisco switch with following Port 24 config:

Code Select Expand

interface GigabitEthernet1/0/24
  description Link-to-OPN
  switchport trunk encapsulation dot1q
  switchport trunk native vlan 1337
  switchport mode trunk
  switchport trunk allowed-vlan 5,100,1337

And that basically was basically doing it's job. My Mgmt VLAN was "5" and I had:

Code Select Expand

interface GigabitEthernet1/0/22
  description Mgmt
  switchport access vlan 5
interface Vlan5
  ip address dhcp

With this my switch was getting IP from OPN's DHCP pool of vlan0005 (Parent: igb0, Tag 5). Then my PC connected to Gi1/0/22 was always getting next available IP from the 10.0.5.1/24 pool.

Today (thanks to your huge research and communication with MikroTik's community) it seems that SwOS on CSS326 doesn't have an option to set up a Native VLAN.
Instead you configure Port 24 on CSS326 (according to SwOS wiki) to this:

Code Select Expand

VLANs tab:
  VLAN 5, Name: Mgmt, Port Isolation: True, Learning: True, Mirror: False, IGMP Snooping: False, Members: 22, 24
  VLAN 100, Name: WiFi, Port Isolation: True, Learning: True, Mirror: False, IGMP Snooping: False, Members: 1,2,3,4,5,6,7,8,24  # let's say I have eight APs
  VLAN 1337, Name: Native, Port Isolation: True, Learning: True, Mirror: False, IGMP Snooping: False, Members: 22, 24

VLAN tab:
  Port 1-8, VLAN Mode: strict, VLAN Receive: only untagged, Default VLAN ID: 100
  Port 22, VLAN Mode: strict, VLAN Receive: only untagged, Default VLAN ID: 5
  Port 24, VLAN Mode: strict, VLAN Receive: only tagged, Default VLAN ID: 1337

Some additional questions regarding of how MikroTik understands ingress and egress:



For port 22 where my laptop is connected to CSS326 the ingres is red and egress is green, right?
Same thing for port 24 where OPNsense is connected. Ingress is blue, egress is purple, right?

Now is there a concept of ingress/egress distinguishing of traffic inside the switch and between the ports? In that case:
For port 22 the ingress from port 24 would be pink, and egress to port 24 would be orange, and vice versa from the port 24's perspective the ingress would be orange and egress would be pink. Am I understanding this right?

The reason why I ask is the configuration above (VLAN tab) says the following for the Port 22:

• VLAN Mode: I will filter all red and pink traffic and if it's not Tag=5 then I'll drop it. But PC is not member of VLAN5, only dude that is member of VLAN5 is Port 24 so I'll listen to the incoming traffic from him only.
• VLAN Receive: I will only allow incoming packets without VLAN tag. Since the only dude that is not sending me tags is PC - I will accept untagged traffic from him only.

This is very confusing for me in the MikroTik world. Contradictory terms. I would like to understand what's the point of VLAN Mode and VLAN Receive. Both explanations are talking about ingress and egress but in the VLAN Configuration Example Wiki Ether 6 would be my WiFi (VLAN100) case, Ether 8 would be my Mgmt (VLAN5) case.
Having a second thought I understand this MikroTik's VLAN Mode terms as follows:

• VLAN Mode optional: Don't care about VLAN tags but if your ID is wrong I'll give you a proper one.
  
  • You don't have VLAN Tag? I'll let you go through.
  • You have it and it matches my VLAN table? You're most welcome.
  • You have a tag that is not in my VLAN table? Oh that's not a problem, let's pretend that you're untagged and I'll tag you as my Default VLAN ID
• VLAN Mode disabled: VLAN table? Waaaat?
  
  • You don't have the Tag? I'll let you go through.
  • You have it? I'll let you go only if it matches my Default VLAN ID and my buddy VLAN Receive says "any" but your tag stays with me and you go naked from now on.
• VLAN Mode enabled: Only tags that I know go through.
  
  • You coming to me with the Tag I don't know, get out of here!
  • Your tag matches my Default VLAN Tag! That's awesome dude! Cause now I can act like a proper access port! But leave the Tag here and go naked, will you?
• VLAN Mode strict: Jokes are over - hand over your passport, reason for travelling, destination and where you come from?
  
  • I don't know your Tag, it's not on my VLAN table. Get lost!
  • Got the Tag? Good. It's on my list and the dude that sent you is also member of the VLAN of your Tag - you can go.

In other words VLAN Mode is an inside club security guy that checks your ID before letting you go into the party.

And for VLAN Receive terms I think it is like another security guy but standing outside filtering and funneling the incoming guests so the VLAN Mode guy have less work to do:

• VLAN Receive: any
• Oh whatever... Come in - VLAN Mode will handle you
• VLAN Receive: only tagged
• Only Tags! And tell the VLAN Mode guy that you already have one so no need to give you a Default VLAN ID
• VLAN Receive: only untagged
• Come in naked people - we'll give you a Default VLAN ID

Having said that in a humorous way I think that ingress from OPNsense is already tagged with VLAN 5 so Port 24 must be:

VLAN Mode: strict
VLAN Receive: only tagged
Default VLAN ID: doesn't really matter cause VLAN Receive and VLAN Mode handles only tagged frames.
Force VLAN ID: Also doesn't matter. Only if the ingress comes with a tag and Default VLAN=1 it will basically strip all egress from tags and we don't want that cause OPNsense needs the tags.


Please let me know how far I am from fully understanding it.



As for mixing VLANs in OPNsense. Do I understand it right that I need to disable LAN0(igb0) DHCP and rely only on Mgmt(VLAN5) and WiFi(VLAN100) DHCP since they are the children of this physical interface? In other words - physical interface is like a container that only has an IP of 10.0.0.1 and it can even be 10.0.0.1/32 so OPNsense can handle the traffic, right?

[cookiemonster]

Hi seki. I honestly walked away with only what I needed to know. I was lucky that the community engaged and gave me the information I requested for the case, but I think we agreed that the official documentation is short on details. They were great. Perhaps you would have as much luck.
As you have noticed it gives you examples but not a list of comprehensive scenarios so you end with gaps in the understanding. I wish I could tell I understand the inner workings of the settings but I don't.
The most important but lacking detail are VLAN Mode and VLAN Receive scenarios. It's a shame that it's left to the community heroes to answer queries. Maybe it has changed but at the time I didn't see engagement from the company there to explain nor to add to the docs.
Anyways, for the ingress/egress my understanding is that is from the switch perspective as you say, but only looking to its edge, not inside. So a
port is ingress when traffic coming into it from the outside only. Egress for the traffic leaving to the outside.
Again, my understanding: VLAN Mode - I think of it as the guard for traffic going out because selection applies to egress traffic on the port. Except for strict mode.
VLAN Receive is the guard for traffic coming in. But we need both options because each port is at any given time either ingress or egress, as you have realised. It's the documentation of the combinations that is quite unclear IMHO. Specially disabled and optional !
I don't speak Cisco language but I think I can make out your setup from the code. Looks clear enough, I think I get that you had a trunk with 3 VLANs and the native was 1337. Then for management you created VLAN 5 with DHCP and only port 22 on it. So far so good.

The current setup seems right for OPN to Mikrotik.
Port 24 is your trunk. Only tagged traffic and not mixed with untagged.Port 22 is also a member and seems your dedicated Mgmt port for the switch.
The only bit that you need to check I think is that you have allowed the management VLAN in the System tab. The "Management Access" part of the docs.
Otherwise you lock yourself out.

As for mixing VLANs in OPNsense. Do I understand it right that I need to disable LAN0(igb0) DHCP and rely only on Mgmt(VLAN5) and WiFi(VLAN100) DHCP since they are the children of this physical interface? In other words - physical interface is like a container that only has an IP of 10.0.0.1 and it can even be 10.0.0.1/32 so OPNsense can handle the traffic, right?

Yes you disable the parent and rely on the VLANs hanging off it. It is a container yes, but as unassigned, it does not need any atribute like an IP. See how it looks like on mine:
You cannot view this attachment.
You notice the igc1 is unnasigned. The two VLANs off it are assigned. As Herr Hausen says, the "Interface" in this OS distribution is a label only.
You put a lot of effort on your post, I'm sorry i didn't do it justice, but will be happy to help you get to the setup you want if I can.

[seki]

Hey cookiemonster!

They were great. Perhaps you would have as much luck.

So I talked to one of my ex-coworkers and he was kind of a MikroTik Guru (at least in my world) and I asked him to explain things like to a five year old child. So he liked the metaphor with club and bouncers/security guards. So here's what he told me:

TL;DR version - scroll down
Imagine a huge Club named "Swos" with multiple rooms where different music is played. One plays techno, other plays disco, another one metal and another one plays 50s-80s classics.
This club works 24/7 and collaborates with smaller clubs and they agree on working together on reusing the same tags/wristbands so visitors can use the bands together in a network and share the entry ticket profits. There's a small club called "PC" around the corner that doesn't use bands at all if a free-for-all club, there's also club two blocks away called "AP" which also has couple of music rooms but it's not as big as Swos capable of having 4096 rooms playing all together. And there's also one club called "Admin" - a rather small one but also uses wristbands and tags visitors.

This Swos club has bouncers and security guards all over the place and here is who is who:
VLAN Receive
You approach the Club and you notice a big bouncer guy - they call him by this nickname VLAN Receiver. He is in close contact with the Swos owner and lets people in based on the owner's instructions. On a normal, casual day like Tuesday they let in everyone cause people don't go to clubs on weekdays. On these days you can hear see the little badge on VLAN Receiver's jacket with word "any". On this day he will let anyone in. With the bands on their wrist coming from the AP Club or Admin Club and those without the bands coming from the PC Club too.

Usually on the weekend days Swos Club is very popular and everyone would like to come in but there's not enough space for these people so VLAN Receiver has the badge "only tagged" and accepts people from the AP and Admin Club only up until around midnight or so. After midnight there's a little more space in the club so usually he changes the badge back to "any" so those from PC Club can enter without the tickets too.

There was one time when Swos Club owner had a beef with AP Club owner. On that period VLAN Receiver was having a badge "only untagged" and he was accepting only those from PC Club so he wouldn't share profits with the AP Club dude. Unfortunately people from Admin Club were also affected by this decision.
VLAN Mode
This guy was a gate security guy inside the club after VLAN Receive did let you go through. He also was using badges with information.

When VLAN Mode had a badge "disabled" he basically didn't care for the tags/bands. He didn't check if your tag is on the list however he was taking off any tag you had on your wrist and did let you chose the music room you wanted to enter.

But most of the weekdays by default VLAN Mode had a badge called "optional" and you could chose any music room you wanted and he didn't took off your tag.

During weekend days before 6 PM and after midnight the traffic was higher but not critical. So VLAN Mode had a badge "enabled" and he was checking if your tag was on their so-called VLAN list. If your tag was on the list - you can enter any room you want.

But the worst moments were the weekend days between 6 PM and midnight. VLAN Mode had a badge "strict". They did let you through but they also checked if you're permitted to enter the most occupied music rooms. They were the most crowdy and VLAN Mode was checking if you're on their VLAN list and if you're even permitted to enter the room.
Default VLAN ID
One of the VLAN Mode's assistant was Default VLAN. He usually was giving you a new Default tag only when VLAN Mode was having badge "disabled". You could keep the old tag though.
Force VLAN ID
This guy was kind of easy if he was wearing "no" badge. Along with Default VLAN ID he was giving out new tag if Receiver and Mode let you in without a tag and told you to wear it.

But if he had "yes" badge? Oh boy... He took your tag away and gave a new one. No matter if you had a tag or not. No discussion - you had to had a new tag. Period.
TL;DR

• VLAN Receive comes first to analyze if the frame is tagged or not and based on the value
• ANY: Let any traffic in - tagged or not. No drops.
• ONLY TAGGED: Only tagged comes in - untagged traffic is dropped.
• ONLY UNTAGGED: Only untagged comes in - tagged traffic is dropped.
• VLAN Mode comes second to check if the tag is on the VLAN Table
• DISABLED: No checking against VLAN Table but removing all existing VLAN tags. Default VLAN ID becomes powerless as VLAN Mode disabled blocks all VLAN operations.
• OPTIONAL: Both tagged and untagged comes in (usually with Receive==any). Tagged traffic keeps the tag even if it's not on VLAN Table.
• ENABLED: Traffic needs to be on the VLAN Table.
• STRICT: Traffic needs to be on the VLAN Table and checks if this port actually belongs to the VLAN this traffic is tagged with.
• Default VLAN ID is only affecting untagged traffic
• If VLAN Receive==any: Tagged traffic is unaffected, Untagged gets the Default VLAN ID value.
• If VLAN Receive==only tagged: Tagged traffic is unaffected, Untagged doesn't even reach this point - it gets dropped by VLAN Receive
• If VLAN Receive==only untagged: Tagged traffic never gets processed - it gets dropped by VLAN Receive. Any other (untagged) is tagged now by Default VLAN ID value.
• Force VLAN ID works only with Default VLAN ID
• NO: Every tagged traffic keeps own original tag. Untagged is forcefully tagged with our Default VLAN ID but it's not Force VLAN ID's job
• YES: Everyone gets the new tag of Default VLAN ID. Any existing tags (or non existing if Receive==only tagged or Receive==any) are overwritten forcefully.

Yes you disable the parent and rely on the VLANs hanging off it. It is a container yes, but as unassigned, it does not need any atribute like an IP. See how it looks like on mine:

Awesome! Will try this one out. Thank you! <3

It is good to clash and exchange knowledge :)
MikroTik needs more documentation and more tutorials though...

[cookiemonster]

That is pretty useful for Mikrotik reference. I'll save this one.
Thanks for sharing!