Learning OPNsense: Can't reach OPNsense VM on VLAN

[emsknock]

Hi there!

I'm trying to learn OPNsense by running it as a VM on Proxmox and I can't seem to get VLANs working. My set up is as follows:

I've a switch where I've connected the Proxmox host on port 1 and two other machines on ports 2 and 3. Port 1 is set to be untagged VLAN 1 and tagged VID 2. Port 2 is untagged VID 2, and port 3 is untagged VID 1. The switch is a TP-Link one where apparently "untagged" doesn't automatically mark incoming traffic, but I've set the PVID there so that the machines on ports 2 and 3 do actually have their incoming frames marked with the correct VIDs.

The Proxmox host has a static IP of 10.10.10.10/24. The machine on port 2 of the switch has 10.10.20.2/24 and the machine on port 3 has 10.10.10.2/24.

Then, in Proxmox, the OPNsense VM has a single VirtIO NIC device that's bridged (via a VLAN-aware bridge named vmbr0) to the host's physical NIC. The virtual device has no firewall and no VLAN tag set. I realise I could just handle the VLANs through adding another virtual NIC to the VM but I'm specifically wanting to learn OPNsense here, so I'd like to know how to set this up through it.

The virtual network card shows up as a device named vtnet0 in OPNsense. I can assign it as the WAN interface through the console on Proxmox and give it the IP 10.10.10.1/24. This means I can open the OPNsense Web UI from the machine on port 3.

Navigating the UI, I've created a VLAN device, vlan2. The parent is vtnet0 and the Tag is 2. I've then gone and assigned vlan2 to interface opt1, enabled it, and given it the static IP 10.10.20.1/24; saved and applied changes. Then I've gone to the Firewall section, and (during two different attempts at getting this working) tried both creating a new rule passing any incoming traffic to OPT1 and cloning the default allow-rule from WAN to OPT1.

At this point, to my understanding, the machine on port 2 (10.10.20.2) should be able to ping the OPNsense VM (through vlan2 [opt1] at 10.10.20.1) but this doesn't work. The OPNsense VM can't ping the machine on port 2 either. Pings have 100% loss, and ARP tables don't populate with MAC addresses for IPs. To make sure the switch configuration is correct, I set a fourth port on it to VLAN 2 too, and moved the computer on port 3 to it, changing its IP to 10.10.20.3/24. Sure enough, 10.10.20.2 could ping 10.10.20.3 just fine! I also tried this through adding a new virtual NIC on Proxmox to OPNsense, this time tagging the NIC to VID 2 through Proxmox, and using the NIC as-is on OPNsense (i.e. without creating a VLAN device there). This time OPNsense could ping 10.10.20.2! The machine couldn't ping back but I assume this was because of the firewall — I started from scratch and didn't make a pass rule for incoming traffic.

To my understanding, this all points to my configuration in OPNsense being somehow faulty but I'm at my wit's end now as to what the issue can be. I'd be really grateful for any pointers or help debugging!

[EricPerl]

Hmm, learning OPN using a virtualized instance and a single NIC is ambitious...
I started ~6 months ago and I went for a simpler setup first: 1 NIC for PVE, 2 more for OPN's WAN & LAN.
You might not even need VLANs for this if you have a free port on your existing router.

If you insist on a 1 NIC setup, I wrote this a few days back:
https://forum.opnsense.org/index.php?topic=45437.msg227451#msg227451
It's not exactly simple from a VLAN perspective.

Do you have a smart easy switch, or is it a fully managed switch?

[emsknock]

Hmm, learning OPN using a virtualized instance and a single NIC is ambitious...
I started ~6 months ago and I went for a simpler setup first: 1 NIC for PVE, 2 more for OPN's WAN & LAN.
You might not even need VLANs for this if you have a free port on your existing router.

Sure, but one doesn't learn new things without doing something they haven't done before ;) Since the set up works with an extra vNIC tagged to the VLAN, in my view the next step to learn is how to do this without the extra vNIC!

Good question about the switch; it is indeed one of those easy smart switches, a tl-sg108e to be precise. I don't immediately see how that'd be a problem in this set up thoguh, again especially since the Proxmox-configured vNIC works.

Actually now that I wrote that down, I realise it could of course be that Proxmox is somehow stripping the VLAN bits from the frames before OPNsense can read them; not sure how I'd go about checking whether that's what's happening or not though...

[dseven]

Get a shell on OPNsense and run `tcpdump -nnei vtnet0 vlan` - if there are any tagged frames, you should see them...

[EricPerl]

The interface with default allow all is LAN (not WAN).
You can't access the WebGUI on WAN out of the box unless WAN is the only interface that exists.
And adding a new interface could cause you to lose access to the web GUI because the anti-lockout rule will become lan or opt1.

In another thread with similar gear (especially the switch), the OP had to restart the switch to get things working.

I assume you intend to connect OPN WAN to your existing network at some point.
Decide on which side you want promox to be (existing LAN or OPN LAN). The latter would make more sense if OPN were to become your edge router.

OPN WAN could be left untagged (I haven't tried that because I didn't want my OPN LAN experiments access my existing LAN).
It could also be tagged (VLAN 3). You'd have to allow 3 tagged on the trunk (and 3 untagged on the uplink to the existing network, OR tagged all the way to your existing router).

OPN LAN should be tagged (eg VLAN 2). As long as VLAN 2 is not allowed to the uplink on the switch, OPN LAN is constrained to the ports of the switch configured accordingly.