Routing between VLans is a little bit slow

[SaschaVolk]

Hi

I have built an Opnsense on a Supermicro motherboard A2SDi-4C-HLN4F with an Intel Atom processor C3558. In addition an Intel X722-DA2 as Lan port.
The whole thing is connected to a Ubiquiti UniFi Usw-Pro aggregation with 32 ports. Unfortunately, the speed between the VLan is only max 2GBit/s. Measured with IPerf3 between a client with 10GBit/s and TruNAS also with 10GBit/s.

When i start IPERF direct on the Router i get only 1 GBit

Accepted connection from 192.168.254.167, port 65202
[  5] local 192.168.254.1 port 5201 connected to 192.168.254.167 port 65203
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   133 MBytes  1.11 Gbits/sec                 
[  5]   1.00-2.00   sec   127 MBytes  1.06 Gbits/sec                 
[  5]   2.00-3.00   sec   127 MBytes  1.06 Gbits/sec                 
[  5]   3.00-4.00   sec   131 MBytes  1.10 Gbits/sec                 
[  5]   4.00-5.00   sec   129 MBytes  1.09 Gbits/sec                 
[  5]   5.00-6.00   sec   128 MBytes  1.08 Gbits/sec                 
[  5]   6.00-7.00   sec   131 MBytes  1.10 Gbits/sec                 
[  5]   7.00-8.00   sec   130 MBytes  1.09 Gbits/sec                 
[  5]   8.00-9.00   sec   125 MBytes  1.05 Gbits/sec                 
[  5]   9.00-10.00  sec   129 MBytes  1.08 Gbits/sec                 
[  5]  10.00-10.00  sec   486 KBytes  1.10 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.26 GBytes  1.08 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------

If the client is in the same VLan I get about 9 GBit/s. What can be the problem here?

[meyergru]

When the client is in the same VLAN, the traffic does not get routed? Looks like abysmal routing performance over OpnSense. An Intel Atom C3558 is likely not enough to route at 10 GBit/s, at least not without optimizing, like hardware offloading and RSS.

[SaschaVolk]

Ok - RSS i have enabled. Hardware Offloading is Disabled. How i can check it ??

Only Enable Hardware CRC, Hardware TSO, Hardware LRO and VLAN Hardware Filtering ??

[meyergru]

Enabling hardware offloading often has its own quirks because some features might or might not work, depending on the hardware. The general advice is to leave it off unless absolutely neccessary.

That being said, if you do not find any documented experience on this forum, you are essentially out on a limb as to which setting reaps which benefits - this differs a lot depending on hardware as well. OpnSense being a FreeBSD variant, you might be able to find experiences based on that on the internet.

Basically, it is hard work to try different settings, measure effects and note the results for comparison. And as I wrote, your CPU will be a quite limiting factor anyway.

[SaschaVolk]

Ok thank you. I will try some other Hardware next Week. I have some other Mainboards with Xeons and also other 10 + 25 GBit Nics for testing.