One VLAN is faster than the others (and than LAN, too)

[Brink7564]

Hi,

I noticed one of my VLANs is getting full throughput (the full 250 Mbit/s from my ISP), while both the LAN and other VLANs are only getting around 100 Mbit/s. I verified this via Ethernet connected to the managed switch that sits behind the OPNsense.

When I plug the cable into a port of VLAN20, I get 250 MBit/s. When I plug it into a port of LAN or VLAN10, I get the aforementioned 100 MBit/s. I would assume that, when there isn't any traffic on VLAN20, VLAN10 and LAN should get the same speed as VLAN20 does, right? Is there anything I can do to gain the full 250 MBit/s on those interfaces that aren't getting the full speed?

I didn't (knowingly) create any rules or set any settings that would cause this behaviour, especially since the VLANs are all configured pretty much the same (except for obvious things like IP addresses etc.).

[cookiemonster]

There's no reason they would normally have different speeds in terms of OPN setup by default.
Maybe the interfaces aren't being identified with their right speeds, duplex, etc.
Can you see them appear OK with ifconfig ?

[Brink7564]

Ok good to hear; so there's probably something wrong with my config or potentially hardware (though I believe the HW should be fine). And yeah, I can see the interfaces fine and they don't seem to have any issues. They're all recognized as 1000baseT full-duplex, which is expected:

Code Select Expand

igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: LAN (lan)
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
ether [REDACTED]
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
igb2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: VLANs (opt5)
options=4800028<VLAN_MTU,JUMBO_MTU,NOMAP>
ether [REDACTED]
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan02: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: PERSONAL (opt2)
options=4000000<NOMAP>
ether [REDACTED]
inet 192.168.101.1 netmask 0xffffff00 broadcast 192.168.101.255
groups: vlan
vlan: 10 vlanproto: 802.1q vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan03: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: WORK (opt3)
options=4000000<NOMAP>
ether [REDACTED]
inet 192.168.102.1 netmask 0xfffffffc broadcast 192.168.102.3
groups: vlan
vlan: 20 vlanproto: 802.1q vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan04: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: GUEST (opt4)
options=4000000<NOMAP>
ether [REDACTED]
inet 192.168.103.1 netmask 0xffffff00 broadcast 192.168.103.255
groups: vlan
vlan: 30 vlanproto: 802.1q vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vlan05: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: DMZ (opt6)
options=4000000<NOMAP>
ether [REDACTED]
inet 192.168.104.1 netmask 0xffffff00 broadcast 192.168.104.255
groups: vlan
vlan: 40 vlanproto: 802.1q vlanpcp: 0 parent interface: igb2
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Could this have something to do with different-size netmasks? I wouldn't think so, but that's the only apparent difference I can think of (VLAN20 is a /30 net, while the others are /24).

[cookiemonster]

HW is linking correctly. Masks have no bearing on throughput, no.
100 mbps is quite specific. My guess is that the device in question is limited to that, like voip adapters for instance ship with those interfaces; that's all they need.
An iperf between interfaces should clear the situation, or use another device to verify.

[Brink7564]

That could be the case.. I'm on a macBook with only USB-C, so I use an adapter for Ethernet. This is supposed to be a Gigabit device (https://en.sharkoon.com/product/TypeCCombo#specs), but indeed when I connect this to the port which is definitely receiving/giving 250 MBit/s, it maxes out just under 100 MBit/s. I had already tried it with a different device, but smarty-pants me used an old laptop and after a quick ifconfig, I noticed that it only supports 100 MBit/s -.-

So I checked again using iperf, and sure enough I'm getting fairly bad connection speeds over WiFi:

Code Select Expand

~ % iperf3 -c 192.168.1.1 -p 58447
Connecting to host 192.168.1.1, port 58447
[  5] local 192.168.101.104 port 50626 connected to 192.168.1.1 port 58447
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  10.8 MBytes  89.7 Mbits/sec
[  5]   1.01-2.00   sec  9.00 MBytes  75.6 Mbits/sec
[  5]   2.00-3.01   sec  9.88 MBytes  82.8 Mbits/sec
[  5]   3.01-4.01   sec  10.4 MBytes  87.0 Mbits/sec
[  5]   4.01-5.01   sec  9.38 MBytes  78.6 Mbits/sec
[  5]   5.01-6.01   sec  8.00 MBytes  67.1 Mbits/sec
[  5]   6.01-7.01   sec  8.75 MBytes  73.4 Mbits/sec
[  5]   7.01-8.00   sec  8.38 MBytes  70.5 Mbits/sec
[  5]   8.00-9.00   sec  10.0 MBytes  83.9 Mbits/sec
[  5]   9.00-10.00  sec  6.88 MBytes  57.7 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  91.4 MBytes  76.6 Mbits/sec                  sender
[  5]   0.00-10.04  sec  91.2 MBytes  76.2 Mbits/sec                  receiver


Similarly, when doing the same test via Ethernet, I get only slightly better and definitely sub-par results:

Code Select Expand

~ % iperf3 -c 192.168.1.1 -p 42017
Connecting to host 192.168.1.1, port 42017
[  5] local 192.168.1.190 port 50704 connected to 192.168.1.1 port 42017
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  11.2 MBytes  94.2 Mbits/sec
[  5]   1.00-2.00   sec  11.2 MBytes  94.4 Mbits/sec
[  5]   2.00-3.00   sec  11.2 MBytes  94.3 Mbits/sec
[  5]   3.00-4.00   sec  11.2 MBytes  94.5 Mbits/sec
[  5]   4.00-5.00   sec  11.1 MBytes  93.3 Mbits/sec
[  5]   5.00-6.01   sec  11.2 MBytes  94.0 Mbits/sec
[  5]   6.01-7.00   sec  11.2 MBytes  94.4 Mbits/sec
[  5]   7.00-8.00   sec  11.2 MBytes  94.7 Mbits/sec
[  5]   8.00-9.00   sec  11.1 MBytes  93.4 Mbits/sec
[  5]   9.00-10.01  sec  11.4 MBytes  95.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   112 MBytes  94.2 Mbits/sec                  sender
[  5]   0.00-10.01  sec   112 MBytes  94.1 Mbits/sec                  receiver

I'm not sure if I did this correctly though; ideally I'd want to check the connection between LAN and VLAN, right? The tests above were done from my laptop (in LAN) to the respective IP.

I also tried testing from my OPNsense LAN to VLAN30, but the test results for this in the Iperf tab on the webGUI state 192.168.103.1 as both the local and remote host, despite me having logged into 192.168.1.1 via SSH, where 192.168.1.1 is the LAN IP of my OPNsense. So I'm not sure the results are worth anything:

Code Select Expand

~ $ iperf3 -c 192.168.103.1 -p 9006
Connecting to host 192.168.103.1, port 9006
[  5] local 192.168.103.1 port 7345 connected to 192.168.103.1 port 9006
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec  2.85 GBytes  24.3 Gbits/sec    0   1.99 MBytes
[  5]   1.01-2.01   sec  2.84 GBytes  24.3 Gbits/sec    0   1.99 MBytes
[  5]   2.01-3.00   sec  2.81 GBytes  24.3 Gbits/sec    0   1.99 MBytes
[  5]   3.00-4.00   sec  2.85 GBytes  24.6 Gbits/sec    0   2.01 MBytes
[  5]   4.00-5.01   sec  2.83 GBytes  24.2 Gbits/sec    0   2.01 MBytes
[  5]   5.01-6.00   sec  2.82 GBytes  24.3 Gbits/sec    0   2.01 MBytes
[  5]   6.00-7.01   sec  2.85 GBytes  24.4 Gbits/sec    0   2.01 MBytes
[  5]   7.01-8.00   sec  2.81 GBytes  24.3 Gbits/sec    0   2.01 MBytes
[  5]   8.00-9.00   sec  2.84 GBytes  24.3 Gbits/sec    0   2.01 MBytes
[  5]   9.00-10.00  sec  2.82 GBytes  24.3 Gbits/sec    0   2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  28.3 GBytes  24.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  28.3 GBytes  24.3 Gbits/sec                  receiver

How do I properly test the connection between the LAN and a VLAN? (Assuming that is indeed what I want to test with iperf...)

[Saarbremer]

Check your involved network devices, switches and cabling. These run on 100MBit/s, you're pretty close to the limit.

If link speed is indicated as 100MBit (Fast Speed) you either configured a NIC wrong or your cable lacks some connections / is subject to interference from others. The speed is indicated in system information and refers to the used transmission protocol, no actual speed test involved.

[Brink7564]

The devices are all 1000 MBit/s; I made sure of that when purchasing them. I hadn't thought about the cables yet though - good call! I'll check all of those to make sure they're not a bottleneck.

As for the speed indications, do you mean somewhere in the OPNsense GUI? I looked for something like that under Reporting, System, and in the Diagnostics section under Interfaces but couldn't find any speed indications anywhere. Googling didn't yield any useful results either. The only mention of speed indications I could find is in the Interfaces section of my dashboard, where all ports show up as 1000baseT <full-duplex> (as is the case in the code snippets below).

[cookiemonster]

your vlans appear with the same media type 1 G, children of igb2 which also is 1 G. So the interfaces appear OK on the OPN side.
You need to now for completeness that it shows the same on the switch.
I imagine the "issue" to be discovered is either the device is 100 M, the switch is syncing at that speed with the device in question, or cables, etc.; or even port throttling on the switch.