VLAN & Bridging issue

[kiwieater]

Hi there,

I'm facing a weird VLAN issue. Perhaps someone here would be able to shed some light on this?:

My hardware is a PCengines APU4 with 4 NICS where igb0 is used for WAN (working fine). igb1-3 are meant to be used as LAN resp. VLAN ports. To keep things simple, I'm only looking at a single VLAN (ID 120) for now.

I've set up VLAN interfaces for ibg1-3:

igb1_vlan120: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=4000000<NOMAP>
   ether 00:0d:b9:55:b6:cd
   groups: vlan
   vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: igb1
   media: Ethernet autoselect
   status: no carrier
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2_vlan120: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=4000000<NOMAP>
   ether 00:0d:b9:55:b6:ce
   groups: vlan
   vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: igb2
   media: Ethernet autoselect (1000baseT <full-duplex>)
   status: active
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb3_vlan120: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=4000000<NOMAP>
   ether 00:0d:b9:55:b6:cf
   groups: vlan
   vlan: 120 vlanproto: 802.1q vlanpcp: 0 parent interface: igb3
   media: Ethernet autoselect
   status: no carrier
   nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Only one cable is currently plugged into port igb2 ("status: active").

Furthermore, I bridged those VLAN interfaces and configured a static IP 172.19.26.254 on that bridge:

bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 58:9c:fc:10:bc:0d
   inet 172.19.26.254 netmask 0xffffff00 broadcast 172.19.26.255
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   member: ath0_wlan2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 10 priority 128 path cost 33333
   member: igb3_vlan120 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 17 priority 128 path cost 2000000
   member: igb2_vlan120 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 16 priority 128 path cost 20000
   member: igb1_vlan120 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 15 priority 128 path cost 2000000
   groups: bridge
   nd6 options=9<PERFORMNUD,IFDISABLED>

To configure a guest LAN, I did made another bridge with the 'naked' (non VLAN) interfaces igb1-3, with a different IP, 172.20.26.254:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
   ether 58:9c:fc:10:ff:cc
   inet 172.20.26.254 netmask 0xffffff00 broadcast 172.20.26.255
   id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
   maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
   root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
   member: ath0_wlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 9 priority 128 path cost 33333
   member: igb3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 4 priority 128 path cost 2000000
   member: igb2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 3 priority 128 path cost 20000
   member: igb1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
           ifmaxaddr 0 port 2 priority 128 path cost 2000000
   groups: bridge
   nd6 options=9<PERFORMNUD,IFDISABLED>

There's a DHCP server running on both bridges.

At the other end of that single Ethernet wire, a Linux machine is connected. That machine has two configurations for the NIC, one with VLAN 120, the other without:

3: enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 9c:8e:99:2a:e9:a6 brd ff:ff:ff:ff:ff:ff
    inet 172.20.26.199/24 brd 172.20.26.255 scope global dynamic noprefixroute enp1s0f1
       valid_lft 3993sec preferred_lft 3993sec
    inet6 fe80::9e8e:99ff:fe2a:e9a6/64 scope link
       valid_lft forever preferred_lft forever
5: enp1s0f1.120@enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 9c:8e:99:2a:e9:a6 brd ff:ff:ff:ff:ff:ff
    inet 172.19.26.199/24 brd 172.19.26.255 scope global dynamic noprefixroute enp1s0f1.120
       valid_lft 3993sec preferred_lft 3993sec
    inet6 fe80::9e8e:99ff:fe2a:e9a6/64 scope link
       valid_lft forever preferred_lft forever

Both interfaces receive the assigned IP 199 from the DCHP servers.

The Linux computer can only ping the non-VLAN bridge, however:

# ping 172.20.26.254
PING 172.20.26.254 (172.20.26.254) 56(84) bytes of data.
64 bytes from 172.20.26.254: icmp_seq=1 ttl=64 time=0.493 ms
64 bytes from 172.20.26.254: icmp_seq=2 ttl=64 time=0.418 ms
...

# ping 172.19.26.254
PING 172.19.26.254 (172.19.26.254) 56(84) bytes of data.
^C
--- 172.19.26.254 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3070ms

So I started drilling a bit deeper using tcpdump. I kept running the second ping above from the computer to the firewall's bridged VLAN interface (i.e., 172.19.26.254) and observed what I get to see on the VLAN,Non-VLAN interfaces on both the computer and the firewall. I'll start with the computer side:

# tcpdump -XXnni enp1s0f1 icmp
21:49:57.915044 IP 172.19.26.199 > 172.19.26.254: ICMP echo request, id 2843, seq 3203, length 64
        0x0000:  589c fc10 bc0d 9c8e 992a e9a6 8100 0078  X........*.....x
        0x0010:  0800 4500 0054 b12f 4000 4001 fb8d ac13  ..E..T./@.@.....
        0x0020:  1ac7 ac13 1afe 0800 dbbf 0b1b 0c83 b576  ...............v
        0x0030:  4162 0000 0000 41f6 0d00 0000 0000 1011  Ab....A.........
        0x0040:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021  ...............!
        0x0050:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031  "#$%&'()*+,-./01
        0x0060:  3233 3435 3637                           234567

This looks ok to me: I was expecting to see the "8100 0078" bit: 8100 indicating this is a VLAN frame and 0078 being the hexadecimal version of VLAN ID 120. So far so good.

Here's the same tcpdump, this time on the computer's VLAN interface:

# tcpdump -XXnni enp1s0f1.120 icmp
21:52:46.875030 IP 172.19.26.199 > 172.19.26.254: ICMP echo request, id 2843, seq 3368, length 64
        0x0000:  589c fc10 bc0d 9c8e 992a e9a6 0800 4500  X........*....E.
        0x0010:  0054 f4d8 4000 4001 b7e4 ac13 1ac7 ac13  .T..@.@.........
        0x0020:  1afe 0800 77b6 0b1b 0d28 5e77 4162 0000  ....w....(^wAb..
        0x0030:  0000 fc59 0d00 0000 0000 1011 1213 1415  ...Y............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
        0x0060:  3637                                     67

This is also what I would expect to see: The OS has stripped away the VLAN information "8100 0078" from the previous dump, just leaving "0800 4500", the normal Ethertype followed by the beginning of the IP header.



Now here's the OPNSense side of things, same tcpdump again, first on the non-VLAN interface igb2:

# tcpdump -XXnni igb2
21:56:06.554127 IP 172.19.26.199 > 172.19.26.254: ICMP echo request, id 2843, seq 3563, length 64
   0x0000:  589c fc10 bc0d 9c8e 992a e9a6 8100 0078  X........*.....x
   0x0010:  0800 4500 0054 6680 4000 4001 463d ac13  ..E..Tf.@.@.F=..
   0x0020:  1ac7 ac13 1afe 0800 a8d4 0b1b 0deb 2678  ..............&x
   0x0030:  4162 0000 0000 0778 0800 0000 0000 1011  Ab.....x........
   0x0040:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021  ...............!
   0x0050:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031  "#$%&'()*+,-./01
   0x0060:  3233 3435 3637                           234567

Again, looks good, the packets from 172.19.26.199 arrive apparently. However, the VLAN stripping observed on the Linux side does not seem to happen on the OPNSense side, as the ICMP packes do not appear on igb2's VLAN 120 sub-interface, only occasionally some ARP traffic appears there:

# tcpdump -XXnni igb2_vlan120
21:58:57.562510 ARP, Request who-has 172.19.26.254 tell 172.19.26.199, length 42
   0x0000:  ffff ffff ffff 9c8e 992a e9a6 0806 0001  .........*......
   0x0010:  0800 0604 0001 9c8e 992a e9a6 ac13 1ac7  .........*......
   0x0020:  0000 0000 0000 ac13 1afe 0000 0000 0000  ................
   0x0030:  0000 0000 0000 0000                      ........
21:58:57.562604 ARP, Reply 172.19.26.254 is-at 58:9c:fc:10:bc:0d, length 28
   0x0000:  9c8e 992a e9a6 589c fc10 bc0d 0806 0001  ...*..X.........
   0x0010:  0800 0604 0002 589c fc10 bc0d ac13 1afe  ......X.........
   0x0020:  9c8e 992a e9a6 ac13 1ac7                 ...*......

The firewall is completely open on igb2, igb2_vlan120 and bridge1, so I do not suspect it to interfere here. I even tried disabling the packet filter completly using pfctl -d to be 100% sure the filter is not getting me here...

So my questions would be:

  * Am I doing something obviously wrong here?
  * Why do I not get to see any ICMP traffic on the firewall's igb2_vlan120 interface?
  * While ICMP traffic does not appear to work, the Linux machine still gets its DHCP configuration successfully on both interface, so DHCP at least seems fine.

Best,
Kiwieater

[lilsense]

you cannot have a tagged and untagged. Either all are tagged or none at all.

[kiwieater]

  Thanks, that did the trick!