Looking for a practical worked example for starting with VLANS

[sparticle]

Hello,

We are a long time user of OPNsense. The time has come I feel for us to be a little more security conscious and start to logically segment the network.

VLANS seem to be the answer. But I am a little confused on the practicalities. It seems its like a magic trick, someone who knows the trick makes it seems simple. YOu can sort of understand it, it makes sense but the practical implementation alludes us.

We can create VLANS in OPNsense. Configure DHCP on the VLAN. But are missing knowledge of the detail. For instance. We have OPNSense with a WAN with a /29 and a LAN /24. The LAN is 10.0.0.0/24 we have firewall rules and port forwards to various servers and all is well.

OPNsense is connected to our main network switch which has a number of servers for Virtualisation, NAS etc. also has security cameras, office PC's, WAP's a wireless link to another building which also has a WAP and security cameras, a link to another building which has 5 WAPS, Security Cameras, Smart TV's, a number of internet connected devices. All on the same /24 network.

Ideally in future world we would want to segment some of these devices.

When configuring a VLAN it asks for the DNS server and gateway. If I put the DNS server address as the OPNSense LAN address it can't see that as it is on a different network.

As a simple example to get started. We have a VLAN capable WAP in the office.

IT would be good to configure 2 networks on it. The default network called Office_Wifi and a Guest network on a different network.
Office_Wifi
Default 10.0.0.0/24
Guest_Wifi 10.0.10.0/24

I can configure the Guest_WiFi network in the WAP with VLAN id of 10 and set it's address as 10.0.10.1
DHCP would come from the OPNSense DHCP service.

So the WAP would have 2 networks. 10.0.0.0/24 and 10.0.10.0/24

At the OPNSense end I can configure a VLAN with an ID of 10 and setup the network as 10.0.10.0/24 But what DNS do I specify and what gateway address? The address of the OPNsense LAN 10.0.0.1 for both?

Then at the network switch I would need to setup VLAN 10 and add the port that the WAP is connected to? Would it need to be set as a trunk port as it could be carrying traffic from either of the 2 wifi domains and networks?

Do I need to configure the port that the OPNSense LAN is connected to as a Trunk port. Do I need to add that port to VLAN 10 also?

As a basic starter, I would love someone to assist with setting up this first VLAN to get the WAP serving the 2 groups; office staff and guests. With connected office staff being able to see the default network 10.0.0.0/24 and Guests being served a DHCP address by OPNsense and only able to access the internet.

Any help is appreciated. And apologies for the ramble just trying to get this stuff out of my head.

Cheers

[bartjsmit]

You need to start on your 'main network switch'. That is where you define the VLAN's and assign them to ports. It sits in the middle of all your traffic flows.

Since both your user groups need access to the internet, your office and guest VLAN's need to be trunked to OPNsense. This gives you the layer-2 separation. On top of that, you set up separate subnets. The IP address of the OPNsense interfaces on each subnet will be the gateway for that subnet, so that they can reach the internet.

DNS and potentially DHCP are separate services. You can provide them on OPNsense, but also on a pi-hole, Windows Server or a multitude of other platforms.

Bart...

P.S. an old but good article on El Reg gives a good intro into the practicalities https://www.theregister.com/2017/06/30/vlans_at_20/

[sparticle]

Bart,

Many thanks. I am a little stuck at present.

I have configured OPNsense to have VLAN 10 with the parent interface on the LAN copied the same firewall rules that the LAN has to VLAN 10 and setup DHCP for VLAN10.

On the WAP I have configured 2 networks one on the default VLAN 1 and one on VLAN 10

On the switch I have configured the port the WAP is connected to as belonging to VLAN 10 and the port is marked as a hybrid port. I also configured the switch with an additional address on the VLAN 10 network.

The port connected to the OPNsense LAN is also configured as hybrid.

I can connect to the office_wifi and all is well and I have internet access as well as lan access to the main network and all is well on that network.

If I connect to the Guest network I do not get a DHCP address. However, if I configure a static IP on the connection from my phone. I can connect to the switch on its VLAN 10 address. But I cannot get to the OPNsense VLAN 10 address or the internet.

I am sure I am missing something fundamental but can't see it.

The switch is a HP A5120 fully managed.

Cheers

[bartjsmit]

You haven't made things easy for yourself - that's a layer-2/layer-3 switch  ;)

You have the firewall and WAP on hybrid ports, so far so good. If you have a spare port, configure it as an access port on VLAN 10 and plug a laptop into it. Phones over WiFi are a bit limited for troubleshooting.

If you have a toy laptop without ethernet, use a desktop or a USB/Thunderbolt ethernet adapter.

Set a static on the laptop ethernet (e.g. 10.0.10.101) and set the VLAN 10 interface on OPNsense with a static of 10.0.10.254. Confirm you can ping both the laptop and the WAP from OPNsense. If not, add a temporary floating rule for all v4 ICMP anywhere.

Now that your cabled VLAN is working, connect the laptop to the guest WiFi with a static of 10.0.10.102, disconnect its network cable and ping from OPNsense. This confirms your layer-3 connections.

DHCP and DNS are next but see if you can get basic connectivity sorted first.

Bart...

[sparticle]

Bart,

Thank you for this.

I can confirm that everything in OPNsense looks good. I can see the VLAN 10 interface it is configured correctly. The DHCP service and DNS services are all set to include the VLAN 10 interface etc.

Worth mentioning is that OPNSense runs in an ESXI VM. The PG connected to the VM on the LAN interface is set to VLAN ID 4095 (meaning all) and works perfectly for the rest of the main network.

I setup a basic VM and gave it an address in the VLAN 10 network 10.0.10.40 with a new PG with VLAN 10 I can ping the OPNSense VLAN 10 LAN from it and it gets DNS services from OPNSense.  I can also ping out to the OPNSense LAN network. 10.0.0.0/24 and all devices are accessible! I am sure this should not be possible.

So inside the ESXI VM environment, all seems fine.

I setup 2 laptops configured with a VLAN 10 static addresses 10.0.10.120 and 10.0.10.130 and also setup ports(9 and 10) on the physical switch and set it to Untagged 1,10 and PVID 10 it is not possible to config it as an access port. As soon as you tag it, it changes to a hybrid port.

Port 1 on the switch is configured as untagged 1 (default) and tagged 10 with a PVID of 1 and is automatically made a hybrid port.

port 17 is where the office WAP is connected and that is configured as untagged 1 tagged 10 PVID 1

I can connect to the WAP OFFICE and all is well. All services are accessible and I can talk to anything on the main (default) network and get to the internet.

If I connect to the WAP Guest network I can also get a DNCP address from the VLAN10 network and connect to the internet. But I can also access all of the resources on the OPNSense LAN network 10.0.0.0./24

I try to connect via either laptop I can ping the other and all network resources on both the OPNSense LAN and OPNSense VLAN10 networks. But I cannot ping into the VM 10.0.10.40 from outside of the VM environment even though from the VM I can ping everywhere.

It is working sort of but no network seperation and really I don't know why it is working. If I change the 2 laptop ports to be untagged 1 Tagged 10 PVID 10 I can get nowhere from them.

Also I can connect to both the OPNSense LAN (Main network) and the VLAN 10 network from any pc connected to the switch and ping all resources.

[bartjsmit]

ESXi actually makes things simpler. Add another vNIC to the firewall VM connected to the VLAN 10 port group.

I have OPNsense on VMware with six such interfaces  8)

[sparticle]

ESXi actually makes things simpler. Add another vNIC to the firewall VM connected to the VLAN 10 port group.

I have OPNsense on VMware with six such interfaces  8)

Hello Bart,

Did you have any comment on the above? I am still a little baffled as to why it is sort of working.

Setting the PG in ESXI for the main network servers to 4095 was probably a mistake. Once you have set it to anything other than the 0 it has by default you cannot set it back to 0.

[bartjsmit]

Setting the PG in ESXI for the main network servers to 4095 was probably a mistake.

Back up any VM's you want to keep (Veeam have a good free CE tier) and wipe the host to start fresh. If you have an external datastore, just unregister the VM's and register them after the rebuild.

I do have a default 'VM Network' PG without a VLAN tag which connects to the LAN

[sparticle]

Setting the PG in ESXI for the main network servers to 4095 was probably a mistake.

Back up any VM's you want to keep (Veeam have a good free CE tier) and wipe the host to start fresh. If you have an external datastore, just unregister the VM's and register them after the rebuild.

I do have a default 'VM Network' PG without a VLAN tag which connects to the LAN

I am just trying to get one vlan working. I do not want to break the existing network at this time.

My instinct is it is something to do with the port config on the HP.

Untagged vs tagged and PVID is confusing. I think it means if the traffic is untagged then default to the VLAN equal to PVID.

I suppose the test VM I setup with an address in the VLAN 10 network would be sending untagged traffic to the PG it is connected to which is set as VLAN 10. Is this the same as a physical switch port being set as an access port on VLAN10?

So in theory all traffic from this VM would be tagged as VLAN 10 by the PG. But why then can I ping the OPNsense LAN interface on the default untagged network? Is it also allowing teh default traffic which is untagged? I can't see any way of being more specific there is no option to set untagged vlan number. Just the VLAN number.

In fact from one of the office pc's on the LAN network 10.0.0.0/24 I can ping the test VM on the VLAN10 network!

The PG on VLAN 10 that the test VM is connected to should not be pingable from a pc on the LAN network surely...right? I can ssh into it so it is like both networks are the same.

[bartjsmit]

VLAN's should be consistent throughout your network. Since your hypervisor(s) have an internal virtual switch, they need to have the same VLAN's as on your physical switch(es). You may get away with configuring your vswitch with just the VLAN's used on your VM's, but it's bad practice.

That means that you need to configure the switch to send the VLAN tags to the hypervisor since it is a VLAN-aware device. The same would be true if you run OPNsense on bare metal. You only use untagged traffic on access ports which connect to 'dumb' devices from a VLAN perspective - e.g. laptops running Windows.

To get connectivity on VLAN 10 between a VM (OPNsense or otherwise) you need to trunk VLAN 10 to ESXi and have other devices on VLAN 10 hanging off the switch - either access ports for laptops or trunk ports to WiFi access points.

To avoid having to run a default VLAN with hundreds of devices, you can normally mix tagged and untagged frames on the same port - that will be the hybrid port in HP parlance. That allows a big old LAN network with pockets of a handful devices that need to be separate (guests, DMZ servers, etc.)

If you have a VM on VLAN 10 and only VLAN 10, it should not connect to devices on the LAN, only to devices on VLAN 10.

Bart...

[sparticle]

Bart,

Many thanks for your additional insights. I think I need a brain reset on this.

I will digest this and other notes and see if I can work out a way forward. At this stage I can't redo the entire network.

I was hoping it would be simpleish to introduce a number of VLANS to start to learn more.   

I think it might be a long journey :)

Cheers

[bartjsmit]

If you are going to muck about with the switch, you may want to do (automated) config backups.

Rancid was my go-to project for this before I joined the Ubiquiti cult. There is an HP Comware plugin:

https://shrubbery.net/rancid/
https://sites.google.com/site/jrbinks/code/rancid/cmwrancid

[bimbar]

This seems like a bit of overkill. You don't need to redo your network. This can be migrated fairly seemlessly. In fact, I'm doing such a migration right now for a customer where we have a parallel tagged/untagged setup that needs to go in the long run.

I imagine you have one cable running from the firewall to the main switch at the moment. On that connection there is untagged vlan 1 and your new vlan 10. I have found that this does not work, you can't really mix tagged and untagged traffic with opnsense, neither can you use vlan 1 tagged, it just doesn't seem to work well.

So connect a second interface for the new tagged stuff, you can do the same for the esxi. the new interface should be a pure trunk port, use the old port untagged for vlan 1, you can migrate that later. if it's a cisco, or cisco like, the port should look like this:

interface <name>
switchport mode trunk
description firewall

same on the new port for the esxi. On untagged ports, you would do

interface <name>
switchport mode access
switchport access vlan 10

OR, in case it's the pvid thing

interface <name>
switchport mode general
switchport general allowed vlan add 10 untagged
switchport general pvid 10

OR with the unfortunate WAP, we usually need to do

interface <name>
switchport mode general
switchport general allowed vlan add 10 tagged
switchport general allowed vlan add 1 untagged
switchport general pvid 1

I would advise you to try to migrate away from vlan 1, because it tends to cause trouble.

You do not need to reinstall the esxi, you can reconfigure the network stuff, and you can change the vlans on it. I would even tell you how, but I don't have one anymore, switched to proxmox.
The gist is, you have a vswitch, which connects to the physical port. It doesn't know about vlans. On this, you create networks, which can have a vlan tag, then you don't need to untag on the vm level. You would create one of those for each of your vlans and then connect the vm to it.
Alternatively you can use vlan 4095 for trunking, but I don't imagine you would need that, except if your opnsense itself is virtualized. In that case, do the above, only with virtual connections.

Can't think of anything else right now, hope that helps.

[sparticle]

BArt and Bimbar,

Many thanks for this. Yes, I had concluded that part of my issue was the VLAN 1 thing. I have started separating my VMS onto their own PG in ESXI so I can add vlans at some point.

You are right I have a port on the primary network switch that is connected to the OPNSense LAN. That is currently set as Hybrid Untagged 1 Tagged 10 PVID 1

I will persevere for a while and learn some more basics.

Thank you for you help and support.

Cheers

[bimbar]

Best make that into 2 ports.