Performance Issues on 26.7 and 26.1.11, 3Gbps down to 350-450Mbps

[dg6464]

Hello All,

I've been running OPNsense for many years (thank you for the great firewall product) and have symmetrical 3Gbps internet via PPPoE passthrough... rock solid for many years.

I've noticed some instability the last few months and slowness, but just started investigating the last couple of days as it hasn't been "family affecting".

I noticed initially in the 26.1.8, then upgraded to 26.1.11 and subsequently upgraded to 26.7.1, all with the same performance issue, just slower overall internet performance than usual.

This is an Optiplex with i5-6500 and an Intel X710-DA2 NIC and plenty of RAM, so lots of horsepower when not running anything advanced.

There was a reddit post from someone explaining that in the latest 26.1.11 release they had gone from 8Gbps to 300Mbps performance on an Intel X710:
https://www.reddit.com/r/opnsense/comments/1v2fwhz/comment/ozqs5w8/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

I did some troubleshooting and decided to migrate to an Intel X520-DA2 NIC that I had lying around.

I've swapped SFP's to DAC's with no resolve... still performance issues.

When tested directly connected to the modem, or on a VLAN connected directly to the modem (via same switching uplinks/infrastructure... bypassing the OPNsense, going directly through the modem), I get the full 3Gbps symmetrical.

OPNsense has been running amazing for years in this exact environment, so this has me a bit puzzled.

Wondering if I should be swapping back to the 24.7 release when older X520 drivers were utilized by FreeBSD 14.1:
https://www.thomas-krenn.com/en/wiki/OPNsense_network_card_driver

I'm at a bit of a loss as to how to troubleshoot this...

Everything seems to have migrated fine... no bootloader issues, or issues with firewall rules or DHCP; but please let me know if I need to consider anything that could be causing a performance issue in these newer versions.

I'm not running anything out of the ordinary besides Tailscale (no ZenArmor, or advanced IPS / NGFW L7 stuff).

Any guidance you can provide as to how I go about troubleshooting this would be appreciated.

I was honestly hoping it was just a tempermental X710, as I've heard they haven't been as rock solid from a reliability perspective as the X520's are.

Thanks in advance!

[meyergru]

I am using X520 connected via DACs and I just tested with 26.7.1_1. I can get >5 GBit/s with "iperf3 -P4".

[notspam]

stupid questions:
- which tunables are used ?
- is a cpu microcode addon loaded ?

[meyergru]

Oh, yes. Of course you have to both enable RSS as per documentation and also check with multiple streams...

[dg6464]

Just curious what sort of configuration you have for tunables if you can be more specific for the X520 or X710?

I haven't enabled the Microcode plugin stuff, as I've seen most people removing it or disabling recently.

Just following ChatGPT guidance and the fact that my previous configuration and setup on older versions was rock solid and performant.

So unless there has been a change in the way OPNsense uses the hardware or drivers in a major way... I'm not sure the tunables and such should make a difference if I haven't changed them in years.

But I am happy to follow any specified best practices and troubleshooting steps for Intel X520 or X710 configuration tunables and CPU microcode.

The one factor I do want to verify is if you are running PPPoE on your OPNsense setup... as that has been a reason for historical performance grief as FreeBSD has changed over time.

Any guidance or links you recommend are appreciated.

Happy to provide configuration details or snippets if it will help.

Thanks!

[meyergru]

I use PPPoE and enabled microcode updates and RSS. Otherwise, see this, point 10.

[dg6464]

Understood - so... I'm actually leaning towards this being an ISP issue now. As I stated, this has been running well for a long time, the hardware and config has been fine.

I understand your post, RSS and point 10.

I have RSS configured properly (was disabled for a bit as I was testing using a VM on Proxmox), as well as CPU microcode (also... Proxmox testiing, was disabled). I have since configured and tested that all CPU cores / threads are being used properly and during a speedtest of 350-400Mbps, all cores are less than 2% utilized overall with no real spikes, or NIC card errors, etc). This isn't an N100 machine, it's an i5-6500 processor, which is absolutely plenty of power, even with a single core (unless I had a ton of advanced IDS stuff or ZenArmor, etc).

The ISP in my area has reports on their forums of exactly this issue when specifically using PPPoE passthrough.

The reports are on both Gigahub and Gigahub 2.0 modems... that connections directly through the modem post full speeds, but when using a downstream device with PPPoE passthrough (OPNsense, another Router, etc)... the speeds are limited in the 300-450Mbps range, which is exactly what I am experiencing.

Users complain that in testing they move temporarily to a double-NAT scenario, swapping from PPPoE on the WAN to DHCP (so leveraging the ISP Modem's own PPPoE tunnel)... and get full speeds (albeit double-NAT).

They also state that using the "Advanced DMZ" function, which essentially provides the ISP Modem's IP address directly to the OPNsense DHCP interface, sharing the IP... also provides full speeds (but has always been "flaky" or "less reliable").

I've swapped the NIC and verified the full infrastructure.

The tests I am about to perform is using my secondary Optiplex machine, with the X710-DA2 NIC, on the exact same 10G switch and infrastructure setup and configuration.

I'll test PPPoE Passthrough (and likely see the same speed issue).
I'll test temporary double-NAT'd DHCP to the WAN instead of PPPoE (and likely see proper speeds).
I'll test Advanced DMZ on the ISP modem to the WAN interface MAC of the OPNsense machine (and likely see proper speeds, but have the internet drop out regularly).

I'll likely call my ISP and ask for a new modem and complain to their tech support of this issue and start posting on their forums and other common forums that they look at decribing the testing and issue in depth.

Thanks for your assistance - I'm pretty sure this is not an OPNsense issue and now an ISP Modem Firmware issue.

The only other testing I will do from an OPNsense perspective if the ISP Modem firmware proves to be working fine wuold be checking if the latest FreeBSD update has caused a PPPoE performance regression or something... would be to move back to an older OPNsense release in the 24.x days when an older FreeBSD instance used an older ix driver - but that is unlikely because X710 and X520 use different drivers and both are having issues here.

Posting these threads below more for other users that come across this issue (and to aggregate all threads with similar discussions or issues and hopefully prove this is not an OPNsense issue and avoid future posts like mine):

https://forum.bell.ca/t5/internet/giga-hub-2-0-pppoe-passthrough-2nd-pppoe-session-capped-average/td-p/100998
https://www.reddit.com/r/bell/comments/1q2p3tu/pppoe_passthrough_capped_330_mbps_on_giga_hub_20/
https://www.reddit.com/r/bell/comments/1oyo8ji/bell_gigahub_20_pppoe_passthrough_throttling/
https://www.reddit.com/r/bell/comments/1ti7lgc/gigahub_20_pppoe_issues/
https://hardwarecanucks.com/forum/threads/bell-gigahub-can-it-be-replaced.87726/page-3
https://forum.bell.ca/t5/internet/pppoe-passthrough-is-now-broken-on-open-bsd-router/td-p/75316
https://forum.bell.ca/t5/internet/pppoe-not-working-after-a-recent-firmware-update/td-p/109253
https://www.reddit.com/r/bell/comments/1uh22wi/giga_hub_and_ubiquiti_udmp_woes/
https://community.tp-link.com/en/home/threads/topic/863758
https://forum.bell.ca/t5/internet/using-pppoe-and-dmz-advanced-dmz-for-bridge-mode-use-of-3rd/td-p/705
https://forum.bell.ca/t5/internet/advanced-dmz-issue-of-giga-hub-hh4000/td-p/13364
https://forum.bell.ca/t5/internet/giga-hub-advanced-dmz-issue-wired-connection-to-asus-router/td-p/15059
https://www.reddit.com/r/bell/comments/1tptkwe/troubleshooting_advanced_dmz_on_bell_giga_hub_20/
https://forum.opnsense.org/index.php?topic=29194.0
https://forum.bell.ca/t5/internet/home-hub-4000-bridge-drops-internet-when-using-the-full-3gbps/td-p/5821
(Previous thread, reported by ChatGPT): https://www.dslreports.com/forum/r33444694-Internet-Home-Hub-4000-bridge-drops-internet-at-3Gbps