[VLANs] Network segmentation with 6-port OPNsense box and no switch

[alirx]

Greetings. I'm a newbie in networking trying to set up my network with segmentation in mind. Guides, that I've checked, are all about ROaS concept, which implies having a managed L2 switch (which I don't, and have no option to buy one atm). I am aware that OPNsense is a router, not a switch and doing it comes with performance penalty, but with no money one has to improvise. So I did my research, read the docs and that's what I've come up with. It is probably wrong, and I have some questions, so I need your help.

TL/DR: Some broke guy tries to do some VLANs and switching with OPNsense and learn some networking.


Here is a network diagram of a desired state (sanitized):

No IPv6 at all.

And that is how it all connects (feel free to ask if something isn't clear) :

(Click for large version, it doesn't fit in the post)

Note: vlan interfaces placed under their parents. I've followed this: "Physical --> LAGG--> VLAN --> Bridge".
Interfaces igc2, lagg0, igc5 are not assigned (see question  1)

igc0 - WAN
igc1 - Access port (vlan 20)
igc2 - Trunk port (vlans 10, 20, 30) for AP
igc3 + igc4 = LAGG Trunk port (vlans 10, 100, 200) for Proxmox Server
igc5 = Access port (vlan 10)

Now to the questions:
1) I don't understand which interfaces should just be created, which should be assigned (but not enabled), and which should be enabled with no config (IPv4 Configuration Type = None)? Only thing I know for sure is that static IP and DHCP should be configured on the last interface in the chain (Physical -> LAGG -> VLAN -> Bridge).

2) Which is the right way to create access ports? As in igc1, where the interface itself is added to bridge with corresponding vlan? Or like igc5, where vlan-subinterface is created and added to the bridge?

3) Native network. I've read that it is good security practice to avoid using vlan 1 for data transfer, but it might stay on the network for system-lvl communications between network devices (discovery protocols etc.). I don't really understand where vlan 1 is in my setup. I believe it is disabled since no nets are configured on igcX interfaces, am I right?

4) (minor) Can these interfaces be hidden from menu, since they are not configured (subnets are configured on the bridges)? Same goes for Firewall > Rules section.

Code Select Expand

net.link.bridge.pfil_bridge = 1
net.link.bridge.pfil_member = 0

I've changed these tunables to disable filtering of individual bridge members and filter the bridge instead.



So, that's where I'm stuck and appreciate some help.

[cookiemonster]

I find the Mikrotik documentation for CSS326 quite newbie friendly. Please have a look and it might begin to make sense especially given that you seem to have one of their devices: https://help.mikrotik.com/docs/pages/viewpage.action?pageId=76415036

Now the important part for you. For VLANs use, something has to tag and untag the packets. The usual working assumption is that end devices (PCs, Mobile phones, etc) won't tag them themselves. That means their traffic coming from them into the network infrastructure needs to convert these untagged packets into tagged ones and from then on the VLAN flow begins. OPN will not tag those packets as it is a switch function; it will read them though and make routing decisions on them.
Now with this knowledge,  for instance for question 2: By default a port on a switch is an access port. Once you have your switch to tag the incoming packets, they leave the switch tagged, normally on a trunk port. That is what arrives at OPN.
I'm afraid you can't get away without a switch that can tag packets to create VLANs.

[Patrick M. Hausen]

You can get away without a switch and still have some network segregation. A 6 port appliance is a good start - more ports are good, right?

To be able to help you I need a better understanding of your network, which I do not quite get from the diagram you posted. There are a heck of a lot more devices than 6 in there - how are you going to connect them without a switch?

Please take a step back and try to describe

- how many different internal networks?
- how many devices in each?

before you dive into VLANs. Without a switch you might not even need them ...

Kind regards,
Patrick

[alirx]

Well, I've mostly figured this stuff out.

@cookiemonster - OPNsense can act as a switch, but a slow one coz instead of dedicated chip it will use CPU. It is not recomended, but it is possible. As for tagged and untagged ports... I've found that if VLAN interfaces a belong to a bridge (e.g. bridge0 in my case), and network is configured on the bridge itself you can mark certain physical as untagged by assigning the parent interface to that bridge.

See the second diagram:
Again, bridge0 as an example: bridge0 connects all interfaces of VLAN 10 subnet, physical interface is igc5, VLAN interface for VLAN 10 under parent igc5 - vlan0.5.10.
If i need any device unaware of VLANs connect to VLAN 10th subnet, i remove vlan0.5.10 from the bridge, and add igc5 to a bridge. So there we have igc5 as an access port.
If I leave things as they were - where igc5 is enabled, but not configured (IPv4 config = none) and vlan0.5.10 is a part of bridge0 - any plugged device will not get any IP address from DHCP unless said device is VLAN-aware and configured to VLAN 10.

If we need to add another VLAN to the mix: create VLAN interface for VLAN 20 with igc5 as parent interface - vlan0.5.20 and add it to a bridge1, this will make igc5 a trunk port for vlan 10 and vlan 20.
And that is without native network at all, as no interfaces configured to it.


As you can see both networks are connected on my laptop through one eth adapter through vlans.

And for hybrid port: vlan 10 untagged, vlan 20 tagged we'll need a following state: Bridge0 members: ... ... igc5, Bridge1: ... ... vlan0.5.20.

Text above is based on my experiments with my box and Pop_Os machine. Firewall rules for inter-vlan communication is out of the scope of this topic.

[Patrick M. Hausen]

And for hybrid port: vlan 10 untagged, vlan 20 tagged we'll need a following state: Bridge0 members: ... ... igc5, Bridge1: ... ... vlan0.5.20.

Don't do this - tagged and untagged on the same port - in OPNsense. It leads to unexpected and "interesting" failure scenarios, e.g. DHCP server on untagged also answering for your tagged VLANs etc.

[alirx]

@Patrick M. Hausen 5 different networks.

1) NSFW Net ( 5 devices, some wired some wireless + wireless guests)
2) Management net (4 devices)
3) Server net (1 physical device, lots of VMs)
4) DMZ (1 physical device, lots of VMs)
5) IoT (wired smart tv, wireless devices connects through microtik)

I know i could just assign each different Ethernet port each own subnet without diving to VLANs at all. And probably with Server Net and DMZ i should've gone this route coz for now i have only 1 physical server with dual 2.5Gb NIC in it.

As for amount of devices - most of them wireless and don't need high bandwidth, that's where my old microtik router comes in. It has wireless adapter and 4 100Mb eth ports. And it is the reason why i turned to VLANs. Coz most of devices will connect through that microtik and need to be assigned to different networks.

And plus I'd like to have management interfaces for some VMs, network equipment sitting on it's own isolated network.

Hence vlans.

[Patrick M. Hausen]

I see. But that's hardly "no switch". Your Mikrotik router is a switch  ;) The "switch" part of it at least.

That's why your initial post confused me - how were you going to connect more than 5 internal devices, right?

Kind regards,
Patrick

[alirx]

Don't do this - tagged and untagged on the same port - in OPNsense. It leads to unexpected and "interesting" failure scenarios, e.g. DHCP server on untagged also answering for your tagged VLANs etc.

Oh, I did not expect that, thanks for the heads-up.

Anyway, it was an interesting experience learning all this. I've decided to bite the bullet and bought a 2.5Gb switch (MikroTik CRS310-8G+2S+IN). Should arrive in a few days, and let's see where it takes me.

Thanks for your replies, guys, appreciate it.

[mattlach]

OPNsense can act as a switch, but a slow one coz instead of dedicated chip it will use CPU. It is not recomended, but it is possible.

You can say that again.  It is generally not recommended if you need switching between those ports.  I tend to think of traffic on multiple ports on OPNSense (and other PC based hardware) as best used for routing only.   If you need any switching you are better off using VLANS on a single port (or multiple link aggregated ports if you need the redundancy or capacity) and a switch in between.

[Patrick M. Hausen]

You can say that again.  It is generally not recommended if you need switching between those ports.  I tend to think of traffic on multiple ports on OPNSense (and other PC based hardware) as best used for routing only.   If you need any switching you are better off using VLANS on a single port (or multiple link aggregated ports if you need the redundancy or capacity) and a switch in between.

Fundamentally correct, but I also understand users with a small home network who (rightfully, IMHO) expect the same as from any consumer DSL router - 1 uplink, 3 or 4 LAN ports.

And OPNsense can do that, the FreeBSD bridge has been greatly improved and can easily switch 1 Gbit/s and more. Foundation sponsored project.

If you have one fixed PC at your desk, one printer, one WiFi access point - why an extra switch?

The fun starts - and lots of endless forum threads and disappointments and debugging ... when people want to get fancy with VLANs and the whole kitchen sink and still don't want to simply buy a switch.

I have a Mikrotik CRS326-24G-2S+IN - 24 Gbit ports, 2 SFP+ at < 200€/$. No-brainer.

Kind regards,
Patrick

[mattlach]

You can say that again.  It is generally not recommended if you need switching between those ports.  I tend to think of traffic on multiple ports on OPNSense (and other PC based hardware) as best used for routing only.   If you need any switching you are better off using VLANS on a single port (or multiple link aggregated ports if you need the redundancy or capacity) and a switch in between.

Fundamentally correct, but I also understand users with a small home network who (rightfully, IMHO) expect the same as from any consumer DSL router - 1 uplink, 3 or 4 LAN ports.

And OPNsense can do that, the FreeBSD bridge has been greatly improved and can easily switch 1 Gbit/s and more. Foundation sponsored project.

If you have one fixed PC at your desk, one printer, one WiFi access point - why an extra switch?

The fun starts - and lots of endless forum threads and disappointments and debugging ... when people want to get fancy with VLANs and the whole kitchen sink and still don't want to simply buy a switch.

I have a Mikrotik CRS326-24G-2S+IN - 24 Gbit ports, 2 SFP+ at < 200€/$. No-brainer.

Kind regards,
Patrick

Yeah, Mikrotik switches offer a fantastic bargain for managed Layer2 switching.   I have several of them myself.



Their SwOS is fantastic, but their RouterOS is IMHO a bit much.   It requires you to do everyhting manually, in some pretty terrible ways, and documentation is bad, and often is a nightmare to get working.   It feels a bit like using a beta product to me.

And while they support layer3, apart from a few dedicated router product, the CPU's in their devices lack the horsepower to really do any kind of real routing.

So, Mikrotik is awesome, as long as you use SwOS and stick to layer2 :p

[Patrick M. Hausen]

I use RouterOS and stick to layer 2. Couldn't find out how to assign ports to 4 different lagg interfaces statically in SwOS. Also practically no SNMP ...

[cookiemonster]

Or, try to use the ACL against the manual/wiki to apply different vlan tags to traffic. I didn't figure it out when I tried.