Hello,
IT/Network engineer...fairly simple network I would say...
OPNsense 23.1.11_1-amd64
Chinese knockoff PC (Intel N5105, 16GB, 4x 2.5Gb NICS I226-V)
2.5 Gb Engenius POE switch
2 x 2.5 Gb LAGG with 6 VLANs (only worried about the main VLAN and Storage VLAN)
The routing is working correctly between the VLANs
TrueNAS NAS on DELL hardware with 10Gb NICs in LAGG config
Problem: Slow network speed and iperf3 results going from Home VLAN to Storage VLAN in one direction, reverse works fine.
OPNsense > TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.33.50.254, port 10119
[ 5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 35518
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 268 MBytes 2.25 Gbits/sec
[ 5] 1.00-2.00 sec 270 MBytes 2.26 Gbits/sec
[ 5] 2.00-3.00 sec 268 MBytes 2.25 Gbits/sec
[ 5] 3.00-4.00 sec 275 MBytes 2.30 Gbits/sec
[ 5] 4.00-5.00 sec 267 MBytes 2.24 Gbits/sec
[ 5] 5.00-6.00 sec 266 MBytes 2.23 Gbits/sec
[ 5] 6.00-7.00 sec 167 MBytes 1.40 Gbits/sec
[ 5] 7.00-8.00 sec 272 MBytes 2.28 Gbits/sec
[ 5] 8.00-9.00 sec 270 MBytes 2.26 Gbits/sec
[ 5] 9.00-10.00 sec 270 MBytes 2.27 Gbits/sec
[ 5] 10.00-10.00 sec 41.0 KBytes 1.56 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 2.53 GBytes 2.18 Gbits/sec receiver
OPNsense < TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG
Accepted connection from 10.33.50.254, port 1266
[ 5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 46896
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 248 MBytes 2.08 Gbits/sec 0 1.69 MBytes
[ 5] 1.00-2.00 sec 271 MBytes 2.28 Gbits/sec 0 2.02 MBytes
[ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 2.02 MBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 2.02 MBytes
[ 5] 4.00-5.00 sec 279 MBytes 2.34 Gbits/sec 0 2.02 MBytes
[ 5] 5.00-6.00 sec 281 MBytes 2.36 Gbits/sec 0 2.02 MBytes
[ 5] 6.00-7.00 sec 279 MBytes 2.34 Gbits/sec 0 2.02 MBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 2.02 MBytes
[ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec 0 2.02 MBytes
[ 5] 9.00-10.00 sec 248 MBytes 2.08 Gbits/sec 728 1.06 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.66 GBytes 2.29 Gbits/sec 728 sender
Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN)
Windows > OPNsense (routing) > TrueNAS (StorageVLAN):
Connecting to host 10.33.50.11, port 5201
[ 4] local 10.33.10.55 port 50146 connected to 10.33.50.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.01 sec 256 KBytes 2.07 Mbits/sec
[ 4] 1.01-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 3.01-4.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.01-5.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.00-6.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.01-7.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.01-8.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.01-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 256 KBytes 210 Kbits/sec sender
[ 4] 0.00-10.01 sec 64.2 KBytes 52.5 Kbits/sec receiver
iperf Done.
Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN) with the Reverse flag
Windows > OPNsense (routing) > TrueNAS (StorageVLAN) REVERSE:
Connecting to host 10.33.50.11, port 5201
Reverse mode, remote host 10.33.50.11 is sending
[ 4] local 10.33.10.55 port 50200 connected to 10.33.50.11 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 113 MBytes 950 Mbits/sec
[ 4] 1.00-2.00 sec 113 MBytes 949 Mbits/sec
[ 4] 2.00-3.00 sec 113 MBytes 948 Mbits/sec
[ 4] 3.00-4.00 sec 110 MBytes 920 Mbits/sec
[ 4] 4.00-5.00 sec 113 MBytes 949 Mbits/sec
[ 4] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 4] 6.00-7.00 sec 112 MBytes 943 Mbits/sec
[ 4] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 4] 8.00-9.00 sec 113 MBytes 949 Mbits/sec
[ 4] 9.00-10.00 sec 113 MBytes 949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.10 GBytes 946 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.10 GBytes 946 Mbits/sec receiver
iperf Done.
We have line speed...which is what I would expect.
Now I have run the tests below:
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN): Line speed ~900Mb/s
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN) REVERSE: Line speed ~900Mb/s
I can't for the life of me figure out what I am missing to allow line speed traffic to flow from my HomeVLAN to the StorageVLAN.
My rules are very simple:
StorageVLAN: All traffic out goes down a VPN gateway
HomeVLAN: Allow any any
I have a new bit of information.
Firewall > Settings > Advanced > Disable Firewall (Obviously off by default)
Turning this to ON, now I'm getting full line speed. So it is definitely a rule/FW issue.
Were you able to solve this issue? I'm having the same problem.
I've just solved about 20 CAPTCHA's to post this so hopefully it's useful! In a week of troubleshooting (admittedly I started at layer 7/disk IO) this is about the only information I've found of somebody seeing the same pattern, so I thought it best to reply even if it's only for my own reference if I see the problem again. :D
I was seeing the same behaviour as your 3rd set of iperf results between two of my hosts in the end I managed to identify what the problem was for me, perhaps it's the same for you.
In my situation I was running iperf from my desktop on VLAN 30 to a server on VLAN 20 and getting this:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 126 KBytes 1.03 Mbits/sec
[ 4] 1.00-2.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 2.01-3.01 sec 63.0 KBytes 519 Kbits/sec
[ 4] 3.01-4.02 sec 0.00 Bytes 0.00 bits/sec
[ 4] 4.02-5.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 5.01-6.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 6.00-7.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 7.01-8.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 8.01-9.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 9.01-10.01 sec 0.00 Bytes 0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.01 sec 189 KBytes 155 Kbits/sec sender
[ 4] 0.00-10.01 sec 64.2 KBytes 52.5 Kbits/sec receiver
Running a reverse test was absolutely fine and ran at about 500Mbps.
The key to this was the server was configured with two NICs one on VLAN 20 and one on VLAN 30; my layer 3 IP network maps one to one with VLANs. I'd originally done this so the servers management interface could be accessed from other hosts on VLAN 30 without having to hop through the firewall.
So from my understanding of what was happening in my case is the TCP packets were flowing:
Desktop (VLAN 30) -> Firewall -> Server (VLAN 20)
Desktop (VLAN 30) <- Server (VLAN 30)
Forgive me if the technical details are wrong but this but basically the replies were coming directly to my desktop as they were actually on the same L3 network.
In the end I realised placing the server on both networks was daft as it allowed it to reach into my more secure network without going through any firewall rules so I just removed the additional interface.
FWIWW Running wireshark on both sides of the connection was a tremendous help in tracking this down.
Edit: I didn't try disabling the firewall - I wasn't sure of the implications of that so stayed well away - so I can't say if it's exactly the same problem.