Poor(?) performance on a decent supermicro N3700 setup

[msowka]

I've done basic due diligence in searching for posts with similar setup, and at this point will add mine:

I just got setup with symmetric 1G FTTH service and was consistently hitting 960/960Mbps with the provided consumer grade Sagem router, connected to a Nokia ONT.

Having acquired what I thought was a decent low powered machine I got to ditching the Sagem:
- supermicro X11SBA-LN4F, https://www.supermicro.com/en/products/motherboard/X11SBA-LN4F
- Quad Intel i210-AT nics igb(4), max of 8G of Crucial RAM, Kingston 240GB UV500 SSD mSATA

The opnSense install went without a hitch, save for insisting kern.vty=sc to not have the vga hang (something to do with the embedded graphics, don't fully understand this <shrug>), and rebooting to complete the 20.7.4 update.

As per our local ISP (Bell Canada) config I have VLAN 35 created on that igb0, then pppoe on top of that as the WAN. The WAN iface goes up... but the speedtests.net leave something to be desired, reporting consistently: ~320/440Mbps (interesting that UP is faster).

So far I have started tuning the following, according to some docs & random hearsay:
- kern.ipc.nmbclusters=1000000   
- kern.ipc.nmbjumbop= 1000000 (interstingly this helped to bring UP to 500Mbps)
- hw.igb.fc_setting=0
- set VLAN priority (pcp) to 6... stab in the dark (?)
- set the pppoe MTU size to 1600 from default 1492... again, stab in the dark

I admit, perhaps I had expected too much out of this 10W setup, but like to think it can do better!

Thanks for any suggestions, Mike

[msowka]

Update here, rather than in the original post: the culprit seems to be the pppoe interface, as when I 'double nat' opnsense through the provided sagem router, it then taking care of the pppoe connection, I get a respectable 920/830!

What should I be investigating other than the two arbitrary tweaks on pppoe & vlan I mentioned above?

[mimugmail]

Pppoe doesnt perform well on undersized hardware and single streams

[msowka]

Oh, well that sounds like a cue for some optimization!... I wonder what are the technical bottlenecks?

Oh damn, FUN: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203856

[mimugmail]

Yes, but when you have multiple streams you can reach full wirespeed