WAN performance issue

[FrAllard]

I have 1.5 Gbps internet. My NIC host the GPON from my provider and I have a modded driver that allow OPNsense to link at 2500 Mbps. All that is working, but my speed isn't on par with what I was easily getting with pfSense with the same exact hardware. I'm wondering which settings I need to adjust to get my speed back.

Is can do a speedtest that will show 1400 Mbps and the next one seconds after the first will show 1000 Mbps. The speed is very inconsistent. The speed also ramps up slowly, where with pfSense the speed was instantaneous.

When I download on Usenet, I was getting consistently 1.5 Gbps speed, but with OPNsense I cannot get faster than 1 Gbps.

I have no IDS and anything other than basic firewall rules enabled. I did try to create a trafic shaper with fq_codel, but I deleted eveything in that section since I was still having speed problem.

Here is the dmesg regarding that nic.

root@OPNsense:~ # dmesg | grep bxe0
bxe0: <QLogic NetXtreme II BCM57810 10GbE [BELL BYPASS] (B0) BXE v:1.7> mem 0xdd800000-0xddffffff,0xdd000000-0xdd7fffff,0xde010000-0xde01ffff irq 17 at device 0.0 on pci2
bxe0: PCI BAR0 [10] memory allocated: 0xdd800000-0xddffffff (8388608) -> 0xfffff800dd800000
bxe0: PCI BAR2 [18] memory allocated: 0xdd000000-0xdd7fffff (8388608) -> 0xfffff800dd000000
bxe0: PCI BAR4 [20] memory allocated: 0xde010000-0xde01ffff (65536) -> 0xfffff800de010000
bxe0: Found 10Gb Fiber media.
bxe0: IFMEDIA flags : 20
bxe0: Using defaults for TSO: 65518/35/2048
bxe0: Ethernet address: 00:0e:1e:82:cd:d0
bxe0: MSI-X vectors Requested 5 and Allocated 5
vlan0: changing name to 'bxe0_vlan35'
bxe0:
bxe0: link state changed to UP
bxe0_vlan35: link state changed to UP
bxe0: link state changed to DOWN
bxe0_vlan35: link state changed to DOWN
bxe0: ERROR: Changing VLAN_HWTAGGING is not supported!
bxe0: ERROR: Changing VLAN_HWFILTER is not supported!
bxe0: NIC Link is Up, 2500 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link state changed to UP
bxe0_vlan35: link state changed to UP
bxe0: ERROR: Changing VLAN_HWCSUM is not supported!
bxe0: link state changed to DOWN
bxe0_vlan35: link state changed to DOWN
bxe0: NIC Link is Up, 2500 Mbps full duplex, Flow control: ON - receive & transmit
bxe0: link state changed to UP
bxe0_vlan35: link state changed to UP
bxe0: ERROR: Changing VLAN_HWTAGGING is not supported!
bxe0: ERROR: Changing VLAN_HWFILTER is not supported!
bxe0: ERROR: Changing VLAN_HWCSUM is not supported!

The same nic has two ports avec un LAN port has a 10 Gbps connection to my 10 Gb switch and tested the speed between my server and the LAN port and I was only getting 1.5 Gbps. SOmething in the optimization of that nic is missing.

Please help!

[rungekutta]

Not just you. As per the other current thread I am also struggling to get more than 1-1.5Gb out of my 10Gb card on OpnSense. A minimal Linux install in the same environment for comparison easily saturates 10Gb. https://forum.opnsense.org/index.php?topic=25263.0

What card have you got?

[FrAllard]

Yeah I saw your post, since it was for a Chelsea, if I remember correctly, I didn't want to hijack your thread.

I have a QLogic NetXtreme II BCM57810 10GbE. It's one of those card that is "compatible" with the sfp my isp install in their equipment. So I took it out installed it in that card, modified the bios of the card a little to allow 2.5 GBps link and modified and recompiled the driver to allow the driver to link at 2.5 Gbps also.

If I could find a good Linux equivalent to OPNsense/pfSense I'd go with that probably. I've learned that pppoe under FreeBSD is single threaded. I have a TrueNAS also and it too is struggeling with 10 Gbps. I can only hit about 5 Gbps with it.

I really hope we can find a solution to this performance problem. I kind of really like OPNsense. I've spent too many years to be able to count with pfSense, but since they changed the way they handle their business I wanted to change for OPNsense, when I crashed my pfSense I saw the perfect opportunity to switch. Loosing more than 25% of performance is not an option though.

[rungekutta]

Interesting. Your case is also different to mine in that you’ve had better performance with pfSense in the past as comparison. So I guess you could then compare the differences (as they are siblings, so to speak). Could the difference lie in FreeBSD vs HardenedBSD? That difference will disappear when OpnSense moves back to vanilla FreeBSD (Q1 2022?). Have you looked at Spectre mitigations and such? I think OpnSense applies stricter settings per default than pfSense. See https://docs.opnsense.org/troubleshooting/hardening.html

I like OpnSense too and would like to stay on it. Will try with a more recent Chelsio card and probably see through the migration off HardenedBSD, but if neither of those things help I’m moving to a Linux based fw, either Vyos or roll my own from a minimal Debian. Don’t want to go there though.

[testo_cz]

IMHO, the best is to iperf3 benchmark network throughput of your OPNsense box without your ISP;s connect -- only  with use of a LAN PC (iperf3 clients) and WAN PC (iperf3 server). And observe CPU usage within these tests. And optionally observe effects if tunables within these tests.

After you'll be happy with TCP performance, connect via ISP's box, because there might be need to adjust OPNsense w.r.t. to ISP.

I'd say the checklist of fully initialized network card is simple:
- HW driver queues for RX/TX
- netmap driver queues for RX/TX
- TSO, LRO all off based on OPNsense default settings

I don't see bxe driver or the Broadcom cards to be natively supported by netmap (4).
@rungekutta might soon report the difference in throughput between natively supported Chelsio NIC and NIC supported only in emulated netmapmode.

[rungekutta]

I don't see bxe driver or the Broadcom cards to be natively supported by netmap (4).
@rungekutta might soon report the difference in throughput between natively supported Chelsio NIC and NIC supported only in emulated netmapmode.

The difference was from approx 1Gb/s to approx 6Gb/s. So quite a big difference. Netmap in emulated mode (ie without native support in NIC drivers) seem to struggle to break 1Gb/s no matter the hardware. At least in Hardened/FreeBSD.