10Gbit performance problems with Chelsio T520-SO-CR (solved)

[JamesFrisch]

Because of the apparently good Chelsio FreeBSD driver support, I bought some Chelsio T520-SO-CR NICs.
Unfortunately they seem to max out at 6Gbit/s when using FreeBSD. This problem has come up multiple times in similar threads, but I was unable to find an answer.

https://forum.opnsense.org/index.php?topic=25263
https://forum.opnsense.org/index.php?topic=25844


Did anyone actually managed to run this card close to line speed and would not mind sharing the configs?

[rungekutta]

One of those threads were mine and no, afraid not. Moved to another solution for now.

[j_s]

@JamesFrisch

Can you provide specs on your opnsense system?  Is it virtualized?

[JamesFrisch]

One of those threads were mine and no, afraid not. Moved to another solution for now.

To bad, makes me worry that there is no solution to this problem.

Can you provide specs on your opnsense system?  Is it virtualized?

Both. I first tried it in a VM on a Proxmox host with an Xeon E-2276G. Linux Bridge, Q35, hardware offloading disabled. Only got 6Gbit. Thought this has to be some virtualization overhead. Now I run it bare-metal on a i3-8100 CPU @ 3.60GHz with 8GB RAM, but the problem is still there. Tried updating to the newest firmware, enable hardware offloading and so on. Nothing helped.

What I did to was loading the Chelsio driver with loader.conf and not with a tunable, because there are no other NICs.

After some digging, I also added the t4 and t6 line to the loader.conf (although not sure if needed, because it is a t5 card) and added the following lines to tunable:

hw.cxgbe.fcoecaps_allowed 0   
hw.cxgbe.iscsicaps_allowed 0   
hw.cxgbe.rdmacaps_allowed 0   
hw.cxgbe.toecaps_allowed 0

That did not help with performance but brought down the T5 temp by 10 degrees in the dashboard 

Performance with ipferf3 is 4.5GBit down and 6Gbit up.

[eneerge]

I had performance issues as well (https://forum.opnsense.org/index.php?topic=31680.0). I've since moved to a Linux based product. I experienced the same issue on pfSense as well.

My issue occurs only with 1gbit up/down. Initially, I get full 1gbit up/down. However, it will eventually go into "slow mode" and the download will be like 500-600mbit and the upload only 50mbit. I only get this when I use a pf based firewall. Updated all ethernet firmware as well. Now that I've swapped to something netfilter based, I have 0 hickups.

The best way I could reproduce the issue was to go to youtube and then click on several videos in quick succession for about 5-10 minutes. Eventually, it would just slow down. If I rebooted the system, speed would come back. If I disabled the interface and reenabled it, the speed would come back. Tried various tweaks/tunes to no avail.

What I'm running now doesn't have the features that OpnSense does. I miss it, but I can't deal with the major slow down.

[CJ]

How are you testing your speed?  Single or multiple connections?

IIRC, when testing with iperf against TrueNAS I was able to get line speed using multiple connections.  I can't recall what I got when I was testing iperf against OPNSense.  I haven't looked into it too much because I don't have anything on the other side that could support that high a speed in order to do throughput testing.

[134]

*sense isn't great at routing. However the odd thing i find is that my Atom C3558 and Intel X710 hit a wall at 5-6Gbps of iperf3 traffic while Netgate claim 6100 with same CPU running pfsense plus can push almost 10G. Does anybody know what '10k ACLs' mean on their specs?

At the end I gave up trying 10G inter-VLAN routing with FreeBSD firewall, mainly because i don't actually need 10G routing. I'm moving from Supermicro 1U appliance to a Chinese fanless mini PC with new Intel N100 SoC and 5x 2.5GbE port.

If you really need 10G routing, try VyOS. It's CLI only and require steep learning curve but it's linux based and very decent at routing.

[CJ]

*sense isn't great at routing.

It's a FreeBSD thing rather than specific to *sense.  I forget the reasons why Linux performs better on the same hardware.

[rungekutta]

*sense isn't great at routing.

It's a FreeBSD thing rather than specific to *sense.  I forget the reasons why Linux performs better on the same hardware.

While this is probably true, like you also mentioned I also got much better results with TrueNas (on FreeBSD). So there seems to be something going on in additions which is specific to OpnSense. Haven’t tried pfsense.

I looked at VyOS but wasn’t for me. Only marginally more convenient than rolling your own nftables config file on top of a minimal Debian install, but comes with the downside of vendor lock-in and faff with getting the ISOs etc.

[lilsense]

*sense isn't great at routing.

It's a FreeBSD thing rather than specific to *sense.  I forget the reasons why Linux performs better on the same hardware.

This is incorrect in so many ways... Have you heard of Juniper routers or firewalls? They use FreeBSD. Juniper is not the only one there are many high performance network routers that use FreeBSD, Force10 and Extreme comes to mind.

[CJ]

While this is probably true, like you also mentioned I also got much better results with TrueNas (on FreeBSD). So there seems to be something going on in additions which is specific to OpnSense. Haven’t tried pfsense.

TrueNAS doesn't do routing, though.

This is incorrect in so many ways... Have you heard of Juniper routers or firewalls? They use FreeBSD. Juniper is not the only one there are many high performance network routers that use FreeBSD, Force10 and Extreme comes to mind.

I'll admit that I haven't looked into it.  I just recall there being something different about how FreeBSD vs Linux handles things that causes the performance differences.  It's been a few years since I've seen discussions on it so perhaps the situation has changed.

[JamesFrisch]

Guys, can we please stay on the topic? I don't care what Juniper or what VyOS does!

I am wondering, is 10Gbit achievable with OPNsense and Chelsio?
If yes, how?
If no, what NICs do?

[lilsense]

FreeBSD is far more stable than say a linux distro... Most edge testings of 100GigE+ are done on FreeBSD as it no longer has a software limitation as much as the hardware itself.

Linux Kernel does not have this issue which this cannot be said on various distros.

here you go:

https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99

[134]

This is incorrect in so many ways... Have you heard of Juniper routers or firewalls? They use FreeBSD. Juniper is not the only one there are many high performance network routers that use FreeBSD, Force10 and Extreme comes to mind.

Junos OS Evolved is now Linux-based, they are moving away. Force10's FTOS 10 is also Linux now under the name Dell Networking OS. Also these OSes run on networking gears equipped with ASIC or FPGA to boost performance, so we can't say anything by pointing at these companies.

FreeBSD is far more stable than say a linux distro... Most edge testings of 100GigE+ are done on FreeBSD as it no longer has a software limitation as much as the hardware itself.

Linux Kernel does not have this issue which this cannot be said on various distros.

here you go:

https://netflixtechblog.com/serving-100-gbps-from-an-open-connect-appliance-cdb51dda3b99

Linux can also be very stable, depending on the kernel you choose. In that Netflix case study, FreeBSD was used as file server and not router or firewall. The problem presented in that post was not the networking stack itself but feeding the data to networking stack of FreeBSD.

In context of pure software routing/firewalling, this is the contest between Linux's iptables/nftables and FreeBSD's pf. It's no secret that nftables is not only faster but also scales better with number of cores:

https://matteocroce.medium.com/linux-and-freebsd-networking-cbadcdb15ddd

And then there's this new toy called eBPF which is used by Google, Cloudflare, Netflix, Alibaba .... for packet processing. The developments around Linux is just much more active and it's big reason for transition.

[134]

Guys, can we please stay on the topic? I don't care what Juniper or what VyOS does!

I am wondering, is 10Gbit achievable with OPNsense and Chelsio?
If yes, how?
If no, what NICs do?

Is that 6Gbps result done with single stream or multiple stream of iperf3?

I doubt the NIC is bottleneck. You can try turning pf off, but it would mean that a no firewall or ACL on any interface.