Hi,
i did some iperf3 tests the last days and figured out, that packet loss appears in all networks, when the OPNSense MiniPC (i226-V) is connected to my Unifi Switch. The interesting this is, that iperf3 only shows packet loss in one direction.
The problem immediately vanishes, when I disconnect the OPNSense from the switch.
Connecting the switch to an unconfigured port on the OPNSense, doesn't cause the issues.
The two servers and notebooks I used for the test are in the same network & vlan.
root@TrueNAS[~]# iperf3 -c 192.168.1.100
Connecting to host 192.168.1.100, port 5201
[ 5] local 192.168.1.46 port 52000 connected to 192.168.1.100 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 945 Mbits/sec 199 242 KBytes
[ 5] 1.00-2.00 sec 110 MBytes 922 Mbits/sec 151 195 KBytes
[ 5] 2.00-3.00 sec 110 MBytes 924 Mbits/sec 121 264 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 930 Mbits/sec 157 236 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 931 Mbits/sec 117 294 KBytes
[ 5] 5.00-6.00 sec 109 MBytes 914 Mbits/sec 145 267 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 930 Mbits/sec 150 243 KBytes
[ 5] 7.00-8.00 sec 110 MBytes 923 Mbits/sec 118 276 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 114 310 KBytes
[ 5] 9.00-10.00 sec 109 MBytes 918 Mbits/sec 160 270 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.08 GBytes 927 Mbits/sec 1432 sender
[ 5] 0.00-10.00 sec 1.08 GBytes 925 Mbits/sec receiver
iperf Done.
root@TrueNAS[~]# iperf3 -c 192.168.1.100 -R
Connecting to host 192.168.1.100, port 5201
Reverse mode, remote host 192.168.1.100 is sending
[ 5] local 192.168.1.46 port 56234 connected to 192.168.1.100 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 932 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 940 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 940 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 939 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver
iperf Done.
root@pangolin:~# ip -s link show vmbr0
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 0c:c4:7a:a9:21:8f brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
188181120880 32354906 0 525486 0 2137344
TX: bytes packets errors dropped carrier collsns
133855832187 16058567 0 0 0 0
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 5a:b6:1d:01:b2:34 brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
109392092671 25760900 0 540315 0 2409010
TX: bytes packets errors dropped carrier collsns
98245425617 10384107 0 0 0 0
What could cause this?
Thanks in advance!
~May
Bad cable, bad NIC, or even bad earth (electrically speaking) - it's not likely to be a software issue
Test with a different OS instead of OPNsense.
Bart...
"bad earth" -> bad earthing. ;-)
This world is a bad place in many ways, but not responsible for lost packages in iperf (yet). :-D
I did replace the cables already. Didn't change anything.
I also don't understand in which aspect the OPNsense/NIC could negatively influence the traffic on the switch. I mean, the iperf devices were in the same vlan and same network. Thus routing doesn't happen.
I'm also not sure how much a different os could bring me forward, because I'd somehow have to replicate the network config.
I have a usb nic. Maybe I should try this one first?
very likely candidate
USB NICs are notoriously unreliable.
The I226 series has lots of problems, but as far as I understand, the RX errors occur on the other end of the connection, not at the switch port that connects to the OpnSense.
If that is the case and it happens only when the OpnSense is the target, it can only be some kind of CRC errors in the packets that are caused by the OpnSense. If this were framing errors, the packets would get dropped by the switch already.
Since the I226 has severe problems, I would suggest that you check that no hardware acceleration features are enabled on the OpnSense NIC. Maybe the hardware miscalculates something.
It doesn't only happen, when the OPNsense is the target.
The test happens in a local network from two devices which are directly connected to the switch, no routing happens.
If the OPNsense is unplugged from the switch, the packet loss immediately vanishes.
Disabled hardware acceleration and rebooted afterwards. Changed nothing.
would try with a different switch...
and do a package capture at the OPNsense interface during iperf to see if any packages reach the interface.
I have some news!
Did some tests with the usb network card and figured out that the packet loss happens when vlan 30 is mapped to the nic.
Since the USB nic quickly reached its limits (ping: sendto: No buffer space available), I removed it and was able to determine the same behavior with the built-in nic.
Sounds like a switch issue, I'll dig into tho and report back.
Thanks for help so far!
You probably built a bridging loop that involves this particular VLAN somehow.
Yep that was it...
Found the device, a security camera...
Now theres some packet loss in the management network (vlan 2) to solve, I will look into that.
Thanks so far!
Okay this one is weird.
Unplugged every device on the switch and had packet loss from both servers to the OPNsense. I plugged my laptop with the usb nic into the switch and ran an iperf3 test.
No packet loss there. Same switch port configuration.
What could it be?😪