Upload speed dropping on Netgate SG5100 with FQ Codel enabled

[vicking]

Hi All,

I have a strange issue where my upload speed drops almost directly after I run a speedtest.

I am running the latest OPNsense version on a Netgate SG5100 with a Intel Atom C3558, 8GB ram and 128GB nvme SSD. Everything is running fine except this strange issue.

To prevent bufferbloat I also use SqM with FQ-Codel enabled which seems to work fine for download and upload but somehow the upload speed drops to around 650Mbit when enabled altough when I start the speedtest it starts @ 780Mbit which is the same speed the Download is capable of (Pipes are set @ 825Mbit).

I use a 1Gbit up/down PPPoE connection.
As far as I can see the hardware is capable because when running htop and check CPU I can see that none of the cores are reaches 100%...

Any ideas?

Edit: to clearify, when I disable the shaper I get a consistent 930/930mbit results, so somehow with shaper on for upload @ 825mbit it cuts the speed to 650Mbit instead of 780Mbit which is what happens for the download speed...

Another question I have is about Powerd. As far as I know this CPU doesn't have Intel Speedstep and in the bios not using any C-states.. should I enable Powerd or does it has no effect on this CPU?

[dinguz]

Please follow this configuration guide: https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html
And if it still doesn't work, you really need to post screenshots of your settings.

[vicking]

I did follow that guide

[vicking]

Got the same behavior after configuring it on my new N100 build..
Just configured everything following this guide:

https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html

Didn’t apply any tweaks or optimization.. but as soon as it is enabled @ 850/850 which is 85% of my 1Gb connection the download is holding nicely @ +-819mbit but upload drops to 750mbit after starting @ +-819mbit like the download speed…

Disabling shaper and both up and download reach 930mbit again…
Can’t remember this on older builds, could it be a bug??

[Seimus]

Is ti possible you have set the "limit" other than the default?

Regards,
S.

[vicking]

Is ti possible you have set the "limit" other than the default?

Regards,
S.

Nope, didn't touch the limit to get a good baseline. Wanted to tune later on, but this strange behavior is holding me back..

[Seimus]

Alright than try to disable ECN in the scheduler (Pipe)

https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html#ecn
https://docs.opnsense.org/manual/how-tos/shaper_bufferbloat.html#create-pipe-for-download

And test again.

Also what is your WAN MTU set to?

Regards,
S.

[vicking]

I will try that!

I am using a PPPoE connection which requires a Vlan tag 6 for internet.
My MTU is set @ 1508 so the calculated PPP MTU is 1500.

[pfScrub]

I will try that!

I am using a PPPoE connection which requires a Vlan tag 6 for internet.
My MTU is set @ 1508 so the calculated PPP MTU is 1500.

Here are some more possible issues:
Maybe Netgate specific related-

mbuf exhaustion

msix
RSS

Intel Management Engine bios firmware and mitigations (sometimes my AMD PSP chips may be to blame for issues with multi-operating systems)

Intel Quick Assist incompatibility

Not exactly netgate related-
Improper dscp between ISP and firewall
IPv6 bugging out
DNS

ECN settings
(maybe) rx and tx file descriptors (some people say to set rx twice the tx)

Auto interframe spacing intel drivers (and other drivers) not correctly working between OS and hardware

Not perfect ifconfig settings. Have you tried
ifconfig ifdisabled
or disabling ipv6?
A tcpdump?

Some people say lowering the quantum setting to 300 in opnsense makes a huge difference. Others say to set the quantum to the mtu.

[pfScrub]

And maybe the netgate ssd is optimized for zfs and OPNsense is ufs. I have reformatted home lab ssds as xfs and tried zfs before. And it worked but maybe broke things.

[pfScrub]

Could also be the ISP sending IEEE 1905.1 frames. And maybe spanning tree protocol. Supposedly it invisibly negotiates QoS with interrupts and routes bluetooth and WiFi around ISP meshes.  🤮 not okay if you are traffic shaping yourself.

"Scope
This standard defines an abstraction layer for multiple home network technologies. The abstraction layer provides a common data and control service access point to the heterogeneous home network technologies described in the following specifications: IEEE Std 1901(TM)-2010, IEEE Std 802.11(TM)-2012,IEEE Std 802.3(R)-2008, and MoCA(R) 1.1.1, 2 Other network technologies are supported by an extensible mechanism using an IEEE OUI and an XML-formatted document.
Purpose
The abstraction layer's common interface allows applications and upper layer protocols to be agnostic to the underlying network technologies. The purpose of this standard is to facilitate the integration of IEEE 1901 with other home network technologies. Additionally, the purpose of this standard is to define an abstraction layer that allows the following: -- Common network setup among heterogeneous network technologies -- Providing the same user experience in the process of adding a device to the network and the same user experience while setting an encryption key -- Intelligent network interface selection for delivery of packets that provides improved coverage performance, improved data rate on the poorest link, improved network capacity, improved network reliability and QoS, and support for end-to-end QoS for different traffic classes -- Seamless/transparent interface switching -- Real-time mapping of connection links and interfaces for each traffic class/stream -- Green energy management"



Or have devices that don't use that protocol and have special ed ethernet ports that are also USB polling and connected to an HDMI adapter (ie a Nintendo)

ISP may be sending spanning tree at you too.

To be honest, my dumb ISP's router can't identify devices sometimes over LAN. But with the help of an external Chinese WAP mac addresses started being correctly identified by the router. (one of the reasons to use a freebsd router anyways)

[pfScrub]

Really though, it is an all-in-one piece of garbage.