Help needed with upload speed loss in a 1 Gbit/s PPPoE connection [SOLVED]

[Boomshiko]

Hello all. I have a symmetrical 1 Gbit/s FttH connection at home.
I switched from my ISP provided router to a mini PC running Opnsense this week. Let me first mention the issue I'm encountering before detailing my setup and what I've tried to determine the cause.

Issue
On my ISP provided router, whenever I ran the Ookla speedtest I would get 930/930 Mbit/s down/up results. Since I switched to Opnsense, I consistently get 930/810 Mbit/s results.

Setup
My mini PC is an MSI Cubi N ADL. It has an Intel N100 processor, 8 GB RAM, 512 GB SSD and two Intel 1 Gbit/s capable NICs.

My ISP splits the network on the WAN side. It uses PPPOE on VLAN6 for regular internet, VLAN4 for IPTV and VLAN7 for VoIP.

I have wired connections throughout the house.

Root cause analysis, so far
iperf3
I ran iperf3 (regular and reverse mode) with Opnsense as server and my laptop as client. There are two switches between the Opnsense box and my laptop. I get consistent 950/950 Mbit/s results. This tells me the issue is not on the LAN side.

ONT
I connected my ISP provided router to the ONT again and plugged my laptop into the ISP router. I get (again) consistent 930/930 Mbit/s results. So the issue is not with the ONT or the ethernet cables.

NICs
I connected the MSI/Opnsense box to the ISP provided router and set up the WAN interface of Opnsense in DHCP mode. I plugged the laptop into the MSI/Opnsense box. I get consistent 930/930 Mbit/s results. So the issue is not in the NICs of my MSI mini PC.

Speculative stuff: Tunables
I reverted to the ONT <-> MSI/Opnsense <-> Rest of my network setting with the VLANs setup for the WAN side.

I've found some posts on this forum, other forums suggesting all kinds of tunables settings for enabling multi-threading and whatnot. These settings didn't make a difference. I get 930/810 Mbit/s with all permutations of these settings.

Speculative stuff, MTU
I've tried changing the MTU from default (defaults to 1492) to 1500 (calculates 1492), 1508 (calculates 1500) and 1512 (calculates 1504). Doesn't make a difference for the speedtest, except that on 1512 the WAN side breaks down completely.

Request
Does anyone have any idea what could be the cause of the speed loss on the upstream? I'm pretty sure it's related to the PPPoE configuration in some way, but I don't know how to pinpoint the problem now.

Any help and or suggestions are appreciated. Thanks in advance.

[Seimus]

Do you have a shaper configured?

Regards,
S.

[Boomshiko]

Do you have a shaper configured?

Regards,
S.

I do not. Is it something I should have in place?

[Seimus]

It helps with bufferbloat, but reduces throughput a bit. Its a tradeoff.

So that's why I am asking, you would see a bit decrease in throughput like you describe if you would run shaper.

But re-reading your input, when you put OPNsense WAN on DHCP you have seen the targeted throughput. When its PPPoE you see a slight decrease. All of this without Shaper enabled.

Not sure if I recall correctly but PPPoE is a bit tricky, I know there are posts on the forum where people were dealing with something similar in regards of PPPoE. You tried as well tunables, specifically did you enable RSS? And did you have it properly configured?

Regards,
S.

[Boomshiko]

Not sure if I recall correctly but PPPoE is a bit tricky, I know there are posts on the forum where people were dealing with something similar in regards of PPPoE. You tried as well tunables, specifically did you enable RSS? And did you have it properly configured?

Regards,
S.

I did try enabling RSS. But whether I did correctly, I'm not sure. Not at home right now, when I'm home and free I'll enable it again and paste the settings here.

[Boomshiko]

I did try enabling RSS. But whether I did correctly, I'm not sure. Not at home right now, when I'm home and free I'll enable it again and paste the settings here.

I'm somehow getting 930/930 Mbit/s results now, but I wouldn't be able to tell you why. I was going through all my settings to make sure everything was back to default. I noticed that in Interfaces -> Settings, I had unchecked the disable hardware offloading settings and enabled hardware VLAN filtering (I did this after testing 930/810).

So I checked the boxes again, disabled VLAN hardware filtering and rebooted Opnsense. I rebooted Opnsense. Afterwards I tested 930/930.

To verify that my speed loss was related to these settings, I undid what I just did (unchecked the boxes again and enabled hardware VLAN filtering) and rebooted again. I again tested 930/930.

So something happened when I rebooted Opnsense. But I really don't understand what changed since I did reboot Opnsense many times in the last few days when changing and reverting some of the ISR and RSS related tunables for example.

So now I'm happy my setup performs as it should, but upset that I can't reproduce the issue. Still don't know what's going on!

[Seimus]

Well glad it works for you now as expected.

And yest those offload settings and VLAN HW filtering should be like always disabled. It tents to cause more problems.


P.S. Change your topic description with  [SOLVED] from of it. Lets keep the forum clean ;)

Regards,
S.

[Boomshiko]

The thing is, they're enabled now and all still works. It's very vague to me what's going on and would still like to know.

[meyergru]

The thing is, they're enabled now and all still works. It's very vague to me what's going on and would still like to know.

You must have changed your config somehow and you can track every difference in the configuration backup section. You can compare your current (working) configuration to a known bad configuration if you want to know.

[Boomshiko]

You must have changed your config somehow and you can track every difference in the configuration backup section. You can compare your current (working) configuration to a known bad configuration if you want to know.

Wasn't aware of this section, thanks for that hint! Pinpointed the problem now and leaving this behind for any future readers:

The problem was in my MTU setting on the PPPoE interface. My ISP claims that the PPPoE tunnel has an MTU of 1500, therefore many Opnsense×my ISP-guides claim a 1508 MTU needs to be inputted. This is incorrect. The MTU setting should just be left blank.

Now, I did try many different settings of the MTU, saved and hit apply changes. System -> Routes -> Status will show that that interface has the new MTU. But the new MTU setting isn't in effect. For some reason (I do not know what, maybe driver related??) my setup needs a reboot after changing the MTU setting. Only then the new MTU setting is in effect.

[meyergru]

Found that reboot neccessary, too. Depending on how you PPPoE is set up (via VLAN or directly), in order to reach MTU 1500 over PPPoE, it can be needed to set the physical interface's MTU to 1508 or even 1512, first.

If you leave the PPPoE interface's MTU blank, it will be calculated from what will be left from the underlying interface's MTU after substracting the PPPoE header of 8 bytes. Thus, you probably have 1492 (or 1488 if your ISP uses VLAN).

I found I had to set the physical interface to 1512 and the PPPoE interface to 1508 in order to really get 1500. You cannot set the VLAN interface's MTU directly, BTW.