VLAN devices are on LAN IPs

[tonys]

Since upgrading to 26.7.x.x, all my VLAN devices (192.168.20.x, 192.168.40.x) now have LAN IP addresses. Here's my configuration (ssh'd into the Protectli VP2420 box):

*** OPNsense.home.lan: OPNsense 26.7.1_1 (amd64) ***

 DMZ (igc2)      -> v4: 192.168.30.1/24  (Protectli hardware port 3, DMZ addresses are correct)
 Guest (vlan01)  -> v4: 192.168.20.1/24 
 IoT (vlan02)    -> v4: 192.168.40.1/24
 LAN (bridge0)   -> v4: 192.168.1.1/24.  (Protectli hardware port 1, routes to WiFi)
 OPT4 (igc3)     -> (Bridged to LAN for external devices)
 WAN (igc1)      -> v4/DHCP4.            (Protectli hardware port 2)

------------------------------------------------------------------------------------

Interfaces: Assignments

wan  WAN hardware  igc1 
lan  LAN bridge    bridge0 (LAN->OPT4 Bridge) 
opt1 DMZ hardware  igc2 
opt2 Guest vlan    vlan01 Guest_VLAN (Parent: igc0, Tag: 20) 
opt3 IoT   vlan    vlan02   IoT_VLAN (Parent: igc0, Tag: 40) 
opt4 OPT4 hardware igc3 
opt5 OPT5 hardware igc0

BACK PANEL OF VP2420
  ___________________________________

 |                                   |
 |  [Port 4]  [Port 3]  [Port 2]  [Port 1] <-- (Physical Labels)
 |   igc3       igc2      igc1      igc0   <-- (OPNsense OS Names)

 |___________________________________|

Prior to the upgrade, all guest devices were assigned IP addresses in the 192.168.20.x space and all IoT devices were assigned IP addresses in the 192.168.40.x space. Now the ARP table shows all devices that are part of vlan01 (Guest) and vlan02 (IoT) having LAN IPs in 192.168.1.x space. I've tried rebooting several times but no luck. How do I get my VLAN IPs back? I don't want guests and IOT devices on my LAN.

[viragomann]

The devices might get their IPs from the DHCP on OPNsense. That they get IPs assigned from the wrong subnet indicates, that they are able to communicate with the DHCP on the other subnet.
If you think, your OPNsense configuration is correct, check the subnet separation outside.
How did you do this? Which device is connected to port 1?

[dseven]

What is the purpose of the bridge? What are the member ports?

What are the parent interfaces for the VLAN interfaces?

Mixing bridges and VLAN tagging can be tricky...

[tonys]

The devices might get their IPs from the DHCP on OPNsense. That they get IPs assigned from the wrong subnet indicates, that they are able to communicate with the DHCP on the other subnet.
If you think, your OPNsense configuration is correct, check the subnet separation outside.
How did you do this? Which device is connected to port 1?

The configuration listed above worked fine under OPNSense 21.x. I am now aware that 26.7 doesn't like mixing an untagged LAN with tagged VLANs on the same interface port (igc0 is on Protectli's port 1). I had also bridged the LAN to igc3 on port 4 under 21.x after getting locked out of the wireless LAN during setup of the wireless AP. It's nice having a free external ethernet port available in case the wireless network goes down again but it's certainly not mandatory.

[Patrick M. Hausen]

Can you post the output of "ifconfig -a", please? Redact external IP addresses. Private ones are not a security concern, but if you like, change them, too. I am after a full view of VLAN parent devices and bridge memberships.

Kind regards,
Patrick

[tonys]

What is the purpose of the bridge? What are the member ports?

What are the parent interfaces for the VLAN interfaces?

Mixing bridges and VLAN tagging can be tricky...

What is the purpose of the bridge? What are the member ports?

What are the parent interfaces for the VLAN interfaces?

Mixing bridges and VLAN tagging can be tricky...

The purpose of the bridge is to give LAN access to future hard-wired clients. Interface assignments are shown in the original post. The untagged LAN + both tagged VLANs are on igc0 (port 1 of the Protecli) and go ONLY to the Unifi AP. This worked fine up to and including OPNSense 21.x but I now understand that 26.7 doesn't seem to like mixing untagged and tagged frames on one port. I'm trying to setup a VLAN for the LAN so all three will be VLANs on igc0. I'm not sure if this is compatible with the bridge setup though.

Planned reconfiguration:

VLANs on igc0: vlan00 (LAN) + vlan01 (Guest) + vlan02 (IoT)

Must I remove the LAN bridge to make this work?

Can you post the output of "ifconfig -a", please? Redact external IP addresses. Private ones are not a security concern, but if you like, change them, too. I am after a full view of VLAN parent devices and bridge memberships.

Kind regards,
Patrick

Hi Patrick, here you go:

igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 64:62:66:22:4c:ab
   media: Ethernet autoselect (2500Base-T <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: WAN (wan)
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 64:62:66:22:4c:ac
   inet xx.xx.xx.xx netmask 0xfffff800 broadcast 255.255.255.255
   inet6 x
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
igc2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: DMZ (opt1)
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 00:e0:4c:68:1c:12
   hwaddr 64:62:66:22:4c:ad
   inet 192.168.30.1 netmask 0xffffff00 broadcast 192.168.30.255
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igc3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: OPT4 (opt4)
   options=4802028<VLAN_MTU,JUMBO_MTU,WOL_MAGIC,HWSTATS,MEXTPG>
   ether 64:62:66:22:4c:ae
   media: Ethernet autoselect
   status: no carrier
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
   options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
   inet 127.0.0.1 netmask 0xff000000
   inet6 ::1 prefixlen 128
   inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
   groups: lo
   nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
   options=0
   groups: enc
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
pfsync0: flags=0 metric 0 mtu 1500
   options=0
   maxupd: 128 defer: off version: 1500
   syncok: 1
   groups: pfsync
pflog0: flags=1000141<UP,RUNNING,PROMISC,LOWER_UP> metric 0 mtu 33152
   options=0
   groups: pflog
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: Guest (opt2)
   options=4000000<MEXTPG>
   ether 64:62:66:22:4c:ab
   inet 192.168.20.1 netmask 0xffffff00 broadcast 192.168.20.255
   groups: vlan
   vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
   media: Ethernet autoselect (2500Base-T <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan02: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: IoT (opt3)
   options=4000000<MEXTPG>
   ether 64:62:66:22:4c:ab
   inet 192.168.40.1 netmask 0xffffff00 broadcast 192.168.40.255
   groups: vlan
   vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
   media: Ethernet autoselect (2500Base-T <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: LAN (lan)
   options=10<VLAN_HWTAGGING>
   ether 58:9c:fc:10:4f:48
   inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   bridge flags=0<>
   member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           port 1 priority 128 path cost 55 vlan protocol 802.1q
   member: igc3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           port 4 priority 128 path cost 2000000 vlan protocol 802.1q
   groups: bridge
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan00: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   options=4000000<MEXTPG>
   ether 64:62:66:22:4c:ab
   groups: vlan
   vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
   media: Ethernet autoselect (2500Base-T <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

[dseven]

The purpose of the bridge is to give LAN access to future hard-wired clients. Interface assignments are shown in the original post. The untagged LAN + both tagged VLANs are on igc0 (port 1 of the Protecli) and go ONLY to the Unifi AP.

You didn't say anything about what's connected to igc0 in your original post.

This worked fine up to and including OPNSense 21.x but I now understand that 26.7 doesn't seem to like mixing untagged and tagged frames on one port.

That's not true - I'm still doing it, and it works fine - although some people here will try to convince you that it's a terrible idea. Nothing changed in this regard between 26.1 and 26.7 AFAIK anyway.

Must I remove the LAN bridge to make this work?

I'm pretty sure you can't put a bridge on igc0 and also have VLAN interfaces on igc0 (and expect it to work) - do one or the other.

Life will generally be better if you just get rid of the bridge, IMO.

[Patrick M. Hausen]

That would be readable much better in code tags like so - I trimmed it down to the essential parts.

Code Select Expand

igc0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: WAN (wan)
igc2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: DMZ (opt1)
igc3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   description: OPT4 (opt4)
vlan01: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: Guest (opt2)
   vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
vlan02: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: IoT (opt3)
   vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   description: LAN (lan)
   member: igc0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           port 1 priority 128 path cost 55 vlan protocol 802.1q
   member: igc3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           port 4 priority 128 path cost 2000000 vlan protocol 802.1q
vlan00: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
   vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: igc0

Your problem is that even back in FreeBSD 14 (OPNsense 26.1) and earlier this is a forbidden configuration. A bridge member interface cannot have VLAN subinterfaces. Period. If it worked for your use case then by accident.

In FreeBSD 15 (OPNsense 26.7) the bridge now supports VLAN subinterfaces directly on the bridge interface. OPNsense does not use that feature yet, but the new bridge capability now triggers all sorts of weird behaviour with your faulty setup.

Recommendation: move LAN to an unused port and do not mix tagged and untagged interfaces in the same port. Remove igc0 from the bridge.
This is the only way to fix your problem.

Kind regards,
Patrick

[Patrick M. Hausen]

That's not true - I'm still doing it, and it works fine - although some people here will try to convince you that it's a terrible idea. Nothing changed in this regard between 26.1 and 26.7 AFAIK anyway.

You are right that it can and does work in specific situations. As you can see from this thread and my last reply the shit hits the fan when you add other features and e.g. make the untagged port member of a bridge. Then suddenly and very surprisingly all hell breaks loose.

So it is generally not recommended.

Once the new VLAN aware bridge code in FreeBSD 15 and up

- has stabilized
- is fully integrated into OPNsense including the UI
- is tested to work flawlessly with pf

there will be more up to date ways to turn your OPNsense box into a "software switch" but until then not mixing tagged and untagged is considered best practice unless you know very well what you are doing.

Kind regards,
Patrick

[dseven]

Perhaps, but I think the issues here are a bit orthogonal - someone could still fall into the trap of trying to put a bridge across multiple physical interfaces and then doing VLANs on top even if they weren't trying to mix tagged and untagged frames - that still won't work (until a possible future, where OPNsense harnesses the new "VLAN filtering bridge" functionality, as you say)

[nero355]

The untagged LAN + both tagged VLANs are on igc0 (port 1 of the Protecli) and go ONLY to the Unifi AP.

If you need the VLAN 1 stuff for Ubiquiti UniFi equipment then this would be the right setup :

OPNsense :
- NIC #1 = LAN Untagged
- NIC #2 = Additional Tagged VLANs

Add a Switch between OPNsense and the UniFi UAP :
- Switch Port #1 = Untagged and connected to OPNsense NIC #1
- Switch Port #2 = Tagged and connected to OPNsense NIC #2

- Switch Port #3 that has PoE or PoE+ Output and carries :
Untagged LAN network from OPNsense NIC #1 coming from Switch Port #1
Tagged Additional VLANs from OPNsense NIC #2 coming from Switch Port #2

UniFi UAP :
- Connected to Switch Port #3 that has PoE or PoE+ Output and is then reachable via VLAN 1 which will be the OPNsense LAN network from NIC #1.
- The WiFi SSIDs will then use all the Tagged Additional VLANs from OPNsense NIC #2.


I hope the above is clear enough and if you have questions then let me know :)