New build, very slow - i7-8700, x710-da2, rtl8125

[myth74]

I've been fighting this setup a few days now.  I should be able to get 2000/400 from my ISP but the max speed I've seen so far is 400/400 (usually much slower).    Most of the settings are at their default.  First opnsense box.  I've tried using the onboard 1GB intel nic but the results were similar, I could try again though.

Specs:
Optiplex SFF 5060
i7-8700
8GB RAM
m.2 ssd
intel x710-da2 10G LAN
realtek rtl8125 2.5G WAN

[myth74]

iperf shows that I can hit 6gbps from a different x710 host to the opnsense x710. 

Another clue is the opnsense webui is pretty slow as well.

CPU usage rarely goes above 3% usage. 

Seems like it will always hit my 400mbps upload max speed, and is usually less than 100mpbs down

[myth74]

Tried swapping to the onboard nic and it's even slower.

[meyergru]

Realtek driver support is abysmal on FreeBSD, so probably, you need the os-realtek-re plugin. Realtek chips are know to have problems connecting to some ONTs. With your onboard 1 GBps adapter, you obviously will not get 2000 MBps anyway.

Probably, you also need to play around with MTU settings (depending on how your internet is connected - PPPoE, VLAN?) and RSS.

[myth74]

I flashed the intel x710 Lenovo card with the latest intel driver and I was able to hit 909mbps down and 400mbps up.  A lot better but not quite there yet. I also forgot to mention I already installed the realtek plugin.

I believe I set up RSS correctly.  Although I'm not sure how many bits to set with the 6 core cpu. 

I'll have to look into setting the MTU tomorrow.  Any documentation for setting that optimally?

[pfry]

I flashed the intel x710 Lenovo card with the latest intel driver [...]

Aside: A Lenovo card let you install an Intel NVM image? (Lenovo usually supplies their own via Intel OEM tools.)

I believe I set up RSS correctly.  Although I'm not sure how many bits to set with the 6 core cpu. 

I'll have to look into setting the MTU tomorrow.  Any documentation for setting that optimally?

Someone here posted OPNsense Performance Tuning for Multi-Gigabit Internet a bit ago, which links to FreeBSD Tuning and Optimization. You can go nuts, if you're bored. Most of the settings will help with sessions terminating on the firewall rather than traversing it, but hey.

The rss.bits setting is an odd one - it appears to be more of an entropy setting than a core hint or limit. Recommendations I've seen include "number of real cores", "total cores", and "double the number cores". "netstat -Q" shows stats, which will probably look OK at most any setting. OPNsense docs recommend a lower setting, by the way (which I use); Calomel doesn't even mention the setting.

[myth74]

Aside: A Lenovo card let you install an Intel NVM image? (Lenovo usually supplies their own via Intel OEM tools.)

I followed https://gist.github.com/mietzen/736583d37a1d370273c0775aaaa57aa5 which allows you to flash any available firmware.

I also followed the Multi-Gigabit guide you linked.... but then I remembered I was testing from a gigabit host.  When I tested from a 10G host, I was able to hit 1441/349 from the waveform buffer bloat test.

All of the other "normal" speed tests report around 400 down, and slowly climb to that number.  Is the that indicative of another setting I need to modify?  I don't think I've seen higher than 8% CPU usage.

[pfry]

Just as a data point, I'm currently (still) testing OPNsense, and my junk Win10 workstation here is connected (currently) via an RTL8125 on the firewall, at 1Gb. Through a random Netgear switch. I have a 500/500 Internet service, and it tests out fine. I still don't have any 10Gb machines running, and I just haven't bothered to poke the firewall at 1Gb/s.

How does your latency look? I wouldn't expect that to be an issue in local testing; I'd expect any decent benchmark to report packet loss, so that should be unlikely. A slow climb is usually one of those two, or a (very) slow/limited TCP window ramp. Which also shouldn't be an issue.

What Internet tests are you using?