Packet loss in local network, when OPNsense is connected to switch

[TechHome]

Hi,

i did some iperf3 tests the last days and figured out, that packet loss appears in all networks, when the OPNSense MiniPC (i226-V) is connected to my Unifi Switch. The interesting this is, that iperf3 only shows packet loss in one direction.

The problem immediately vanishes, when I disconnect the OPNSense from the switch.
Connecting the switch to an unconfigured port on the OPNSense, doesn't cause the issues.

The two servers and notebooks I used for the test are in the same network & vlan.

Code Select Expand

root@TrueNAS[~]# iperf3 -c 192.168.1.100
Connecting to host 192.168.1.100, port 5201
[  5] local 192.168.1.46 port 52000 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   945 Mbits/sec  199    242 KBytes       
[  5]   1.00-2.00   sec   110 MBytes   922 Mbits/sec  151    195 KBytes       
[  5]   2.00-3.00   sec   110 MBytes   924 Mbits/sec  121    264 KBytes       
[  5]   3.00-4.00   sec   111 MBytes   930 Mbits/sec  157    236 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   931 Mbits/sec  117    294 KBytes       
[  5]   5.00-6.00   sec   109 MBytes   914 Mbits/sec  145    267 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   930 Mbits/sec  150    243 KBytes       
[  5]   7.00-8.00   sec   110 MBytes   923 Mbits/sec  118    276 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec  114    310 KBytes       
[  5]   9.00-10.00  sec   109 MBytes   918 Mbits/sec  160    270 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec  1432             sender
[  5]   0.00-10.00  sec  1.08 GBytes   925 Mbits/sec                  receiver

iperf Done.
root@TrueNAS[~]# iperf3 -c 192.168.1.100 -R
Connecting to host 192.168.1.100, port 5201
Reverse mode, remote host 192.168.1.100 is sending
[  5] local 192.168.1.46 port 56234 connected to 192.168.1.100 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   111 MBytes   932 Mbits/sec                 
[  5]   1.00-2.00   sec   112 MBytes   940 Mbits/sec                 
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   4.00-5.00   sec   112 MBytes   940 Mbits/sec                 
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   8.00-9.00   sec   112 MBytes   941 Mbits/sec                 
[  5]   9.00-10.00  sec   112 MBytes   939 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec                  receiver

iperf Done.

Code Select Expand

root@pangolin:~# ip -s link show vmbr0
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:a9:21:8f brd ff:ff:ff:ff:ff:ff
    RX:    bytes  packets errors dropped  missed   mcast           
    188181120880 32354906      0  525486       0 2137344
    TX:    bytes  packets errors dropped carrier collsns           
    133855832187 16058567      0       0       0       0

Code Select Expand

6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 5a:b6:1d:01:b2:34 brd ff:ff:ff:ff:ff:ff
    RX:    bytes  packets errors dropped  missed   mcast           
    109392092671 25760900      0  540315       0 2409010
    TX:    bytes  packets errors dropped carrier collsns           
     98245425617 10384107      0       0       0       0


What could cause this?
Thanks in advance!


~May

[bartjsmit]

Bad cable, bad NIC, or even bad earth (electrically speaking) - it's not likely to be a software issue

Test with a different OS instead of OPNsense.

Bart...

[chemlud]

"bad earth" -> bad earthing. ;-)

This world is a bad place in many ways, but not responsible for lost packages in iperf (yet). :-D

[TechHome]

I did replace the cables already. Didn't change anything.
I also don't understand in which aspect the OPNsense/NIC could negatively influence the traffic on the switch. I mean, the iperf devices were in the same vlan and same network. Thus routing doesn't happen.
I'm also not sure how much a different os could bring me forward, because I'd somehow have to replicate the network config.

I have a usb nic. Maybe I should try this one first?

[cookiemonster]

very likely candidate

[meyergru]

USB NICs are notoriously unreliable.

The I226 series has lots of problems, but as far as I understand, the RX errors occur on the other end of the connection, not at the switch port that connects to the OpnSense.

If that is the case and it happens only when the OpnSense is the target, it can only be some kind of CRC errors in the packets that are caused by the OpnSense. If this were framing errors, the packets would get dropped by the switch already.

Since the I226 has severe problems, I would suggest that you check that no hardware acceleration features are enabled on the OpnSense NIC. Maybe the hardware miscalculates something.

[TechHome]

It doesn't only happen, when the OPNsense is the target.
The test happens in a local network from two devices which are directly connected to the switch, no routing happens.
If the OPNsense is unplugged from the switch, the packet loss immediately vanishes.
Disabled hardware acceleration and rebooted afterwards. Changed nothing.

[chemlud]

would try with a different switch...

and do a package capture at the OPNsense interface during iperf to see if any packages reach the interface.

[TechHome]

I have some news!
Did some tests with the usb network card and figured out that the packet loss happens when vlan 30 is mapped to the nic.

Since the USB nic quickly reached its limits (ping: sendto: No buffer space available), I removed it and was able to determine the same behavior with the built-in nic.
Sounds like a switch issue, I'll dig into tho and report back.

Thanks for help so far!

[Patrick M. Hausen]

You probably built a bridging loop that involves this particular VLAN somehow.

[TechHome]

Yep that was it...
Found the device, a security camera...

Now theres some packet loss in the management network (vlan 2) to solve, I will look into that.
Thanks so far!

[TechHome]

Okay this one is weird.

Unplugged every device on the switch and had packet loss from both servers to the OPNsense. I plugged my laptop with the usb nic into the switch and ran an iperf3 test.
No packet loss there. Same switch port configuration.

What could it be?😪