Slow one way iperf3 inter vlan routing

[phrreakk]

Hello,

IT/Network engineer...fairly simple network I would say...
OPNsense 23.1.11_1-amd64
Chinese knockoff PC (Intel N5105, 16GB, 4x 2.5Gb NICS I226-V)
2.5 Gb Engenius POE switch
2 x 2.5 Gb LAGG with 6 VLANs (only worried about the main VLAN and Storage VLAN)
The routing is working correctly between the VLANs
TrueNAS NAS on DELL hardware with 10Gb NICs in LAGG config

Problem: Slow network speed and iperf3 results going from Home VLAN to Storage VLAN in one direction, reverse works fine.

OPNsense > TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG

Code Select Expand


-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.33.50.254, port 10119
[  5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 35518
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   268 MBytes  2.25 Gbits/sec                 
[  5]   1.00-2.00   sec   270 MBytes  2.26 Gbits/sec                 
[  5]   2.00-3.00   sec   268 MBytes  2.25 Gbits/sec                 
[  5]   3.00-4.00   sec   275 MBytes  2.30 Gbits/sec                 
[  5]   4.00-5.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   5.00-6.00   sec   266 MBytes  2.23 Gbits/sec                 
[  5]   6.00-7.00   sec   167 MBytes  1.40 Gbits/sec                 
[  5]   7.00-8.00   sec   272 MBytes  2.28 Gbits/sec                 
[  5]   8.00-9.00   sec   270 MBytes  2.26 Gbits/sec                 
[  5]   9.00-10.00  sec   270 MBytes  2.27 Gbits/sec                 
[  5]  10.00-10.00  sec  41.0 KBytes  1.56 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.53 GBytes  2.18 Gbits/sec                  receiver


OPNsense < TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG

Code Select Expand


Accepted connection from 10.33.50.254, port 1266
[  5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 46896
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   248 MBytes  2.08 Gbits/sec    0   1.69 MBytes       
[  5]   1.00-2.00   sec   271 MBytes  2.28 Gbits/sec    0   2.02 MBytes       
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0   2.02 MBytes       
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec    0   2.02 MBytes       
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   2.02 MBytes       
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   9.00-10.00  sec   248 MBytes  2.08 Gbits/sec  728   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.66 GBytes  2.29 Gbits/sec  728             sender



Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN)
Windows > OPNsense (routing) > TrueNAS (StorageVLAN):

Code Select Expand


Connecting to host 10.33.50.11, port 5201
[  4] local 10.33.10.55 port 50146 connected to 10.33.50.11 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   256 KBytes  2.07 Mbits/sec
[  4]   1.01-2.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.01-4.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   4.01-5.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   5.00-6.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   6.01-7.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.01-8.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   8.01-9.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   256 KBytes   210 Kbits/sec                  sender
[  4]   0.00-10.01  sec  64.2 KBytes  52.5 Kbits/sec                  receiver

iperf Done.



Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN) with the Reverse flag
Windows > OPNsense (routing) > TrueNAS (StorageVLAN) REVERSE:

Code Select Expand


Connecting to host 10.33.50.11, port 5201
Reverse mode, remote host 10.33.50.11 is sending
[  4] local 10.33.10.55 port 50200 connected to 10.33.50.11 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   950 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   948 Mbits/sec
[  4]   3.00-4.00   sec   110 MBytes   920 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
[  4]   6.00-7.00   sec   112 MBytes   943 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec                  receiver

iperf Done.


We have line speed...which is what I would expect.

Now I have run the tests below:
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN): Line speed ~900Mb/s
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN) REVERSE: Line speed ~900Mb/s

I can't for the life of me figure out what I am missing to allow line speed traffic to flow from my HomeVLAN to the StorageVLAN.

My rules are very simple:
StorageVLAN: All traffic out goes down a VPN gateway
HomeVLAN: Allow any any

[phrreakk]

I have a new bit of information.

Firewall > Settings > Advanced > Disable Firewall (Obviously off by default)

Turning this to ON, now I'm getting full line speed.  So it is definitely a rule/FW issue.

[Knogle]

Were you able to solve this issue? I'm having the same problem.

[cs278]

I've just solved about 20 CAPTCHA's to post this so hopefully it's useful! In a week of troubleshooting (admittedly I started at layer 7/disk IO) this is about the only information I've found of somebody seeing the same pattern, so I thought it best to reply even if it's only for my own reference if I see the problem again. :D

I was seeing the same behaviour as your 3rd set of iperf results between two of my hosts in the end I managed to identify what the problem was for me, perhaps it's the same for you.

In my situation I was running iperf from my desktop on VLAN 30 to a server on VLAN 20 and getting this:

Code Select Expand


[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   126 KBytes  1.03 Mbits/sec
[  4]   1.00-2.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   2.01-3.01   sec  63.0 KBytes   519 Kbits/sec
[  4]   3.01-4.02   sec  0.00 Bytes  0.00 bits/sec
[  4]   4.02-5.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   5.01-6.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   6.00-7.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.01-8.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   8.01-9.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   189 KBytes   155 Kbits/sec                  sender
[  4]   0.00-10.01  sec  64.2 KBytes  52.5 Kbits/sec                  receiver


Running a reverse test was absolutely fine and ran at about 500Mbps.

The key to this was the server was configured with two NICs one on VLAN 20 and one on VLAN 30; my layer 3 IP network maps one to one with VLANs. I'd originally done this so the servers management interface could be accessed from other hosts on VLAN 30 without having to hop through the firewall.

So from my understanding of what was happening in my case is the TCP packets were flowing:

Code Select Expand


Desktop (VLAN 30) -> Firewall -> Server (VLAN 20)
Desktop (VLAN 30) <- Server (VLAN 30)


Forgive me if the technical details are wrong but this but basically the replies were coming directly to my desktop as they were actually on the same L3 network.

In the end I realised placing the server on both networks was daft as it allowed it to reach into my more secure network without going through any firewall rules so I just removed the additional interface.

FWIWW Running wireshark on both sides of the connection was a tremendous help in tracking this down.

Edit: I didn't try disabling the firewall - I wasn't sure of the implications of that so stayed well away - so I can't say if it's exactly the same problem.