Messages

I have a new bit of information.

Firewall > Settings > Advanced > Disable Firewall (Obviously off by default)

Turning this to ON, now I'm getting full line speed.  So it is definitely a rule/FW issue.

Hello,

IT/Network engineer...fairly simple network I would say...
OPNsense 23.1.11_1-amd64
Chinese knockoff PC (Intel N5105, 16GB, 4x 2.5Gb NICS I226-V)
2.5 Gb Engenius POE switch
2 x 2.5 Gb LAGG with 6 VLANs (only worried about the main VLAN and Storage VLAN)
The routing is working correctly between the VLANs
TrueNAS NAS on DELL hardware with 10Gb NICs in LAGG config

Problem: Slow network speed and iperf3 results going from Home VLAN to Storage VLAN in one direction, reverse works fine.

OPNsense > TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG

Code Select Expand


-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.33.50.254, port 10119
[  5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 35518
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   268 MBytes  2.25 Gbits/sec                 
[  5]   1.00-2.00   sec   270 MBytes  2.26 Gbits/sec                 
[  5]   2.00-3.00   sec   268 MBytes  2.25 Gbits/sec                 
[  5]   3.00-4.00   sec   275 MBytes  2.30 Gbits/sec                 
[  5]   4.00-5.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   5.00-6.00   sec   266 MBytes  2.23 Gbits/sec                 
[  5]   6.00-7.00   sec   167 MBytes  1.40 Gbits/sec                 
[  5]   7.00-8.00   sec   272 MBytes  2.28 Gbits/sec                 
[  5]   8.00-9.00   sec   270 MBytes  2.26 Gbits/sec                 
[  5]   9.00-10.00  sec   270 MBytes  2.27 Gbits/sec                 
[  5]  10.00-10.00  sec  41.0 KBytes  1.56 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  2.53 GBytes  2.18 Gbits/sec                  receiver


OPNsense < TrueNAS (Storage VLAN, same VLAN): Shows working 10Gb and 2.5Gb LAGG

Code Select Expand


Accepted connection from 10.33.50.254, port 1266
[  5] local 10.33.50.11 port 5201 connected to 10.33.50.254 port 46896
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   248 MBytes  2.08 Gbits/sec    0   1.69 MBytes       
[  5]   1.00-2.00   sec   271 MBytes  2.28 Gbits/sec    0   2.02 MBytes       
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec    0   2.02 MBytes       
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec    0   2.02 MBytes       
[  5]   6.00-7.00   sec   279 MBytes  2.34 Gbits/sec    0   2.02 MBytes       
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec    0   2.02 MBytes       
[  5]   9.00-10.00  sec   248 MBytes  2.08 Gbits/sec  728   1.06 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.66 GBytes  2.29 Gbits/sec  728             sender



Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN)
Windows > OPNsense (routing) > TrueNAS (StorageVLAN):

Code Select Expand


Connecting to host 10.33.50.11, port 5201
[  4] local 10.33.10.55 port 50146 connected to 10.33.50.11 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.01   sec   256 KBytes  2.07 Mbits/sec
[  4]   1.01-2.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   2.01-3.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   3.01-4.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   4.01-5.00   sec  0.00 Bytes  0.00 bits/sec
[  4]   5.00-6.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   6.01-7.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   7.01-8.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   8.01-9.01   sec  0.00 Bytes  0.00 bits/sec
[  4]   9.01-10.01  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.01  sec   256 KBytes   210 Kbits/sec                  sender
[  4]   0.00-10.01  sec  64.2 KBytes  52.5 Kbits/sec                  receiver

iperf Done.



Now, if I run the iperf3 command from my Windows box to TrueNAS (StorageVLAN) with the Reverse flag
Windows > OPNsense (routing) > TrueNAS (StorageVLAN) REVERSE:

Code Select Expand


Connecting to host 10.33.50.11, port 5201
Reverse mode, remote host 10.33.50.11 is sending
[  4] local 10.33.10.55 port 50200 connected to 10.33.50.11 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   113 MBytes   950 Mbits/sec
[  4]   1.00-2.00   sec   113 MBytes   949 Mbits/sec
[  4]   2.00-3.00   sec   113 MBytes   948 Mbits/sec
[  4]   3.00-4.00   sec   110 MBytes   920 Mbits/sec
[  4]   4.00-5.00   sec   113 MBytes   949 Mbits/sec
[  4]   5.00-6.00   sec   113 MBytes   949 Mbits/sec
[  4]   6.00-7.00   sec   112 MBytes   943 Mbits/sec
[  4]   7.00-8.00   sec   113 MBytes   949 Mbits/sec
[  4]   8.00-9.00   sec   113 MBytes   949 Mbits/sec
[  4]   9.00-10.00  sec   113 MBytes   949 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec    0             sender
[  4]   0.00-10.00  sec  1.10 GBytes   946 Mbits/sec                  receiver

iperf Done.


We have line speed...which is what I would expect.

Now I have run the tests below:
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN): Line speed ~900Mb/s
TrueNAS (StorageVLAN) > OPNsense > Windows (HomeVLAN) REVERSE: Line speed ~900Mb/s

I can't for the life of me figure out what I am missing to allow line speed traffic to flow from my HomeVLAN to the StorageVLAN.

My rules are very simple:
StorageVLAN: All traffic out goes down a VPN gateway
HomeVLAN: Allow any any