10Gbit (NAT) Throughput with Intel X710

[Mitzsch]

Hello everyone,

I just wanted to give an update on a post I made almost a year ago. (https://forum.opnsense.org/index.php?topic=22477 - it´s archived, not possible to post there anymore) 
The issue was not getting the throughput I was able to get with pfsense on the same hardware. I thought this was related to the ixl driver not being up to date in OPNsense but this was probably not the case. Now I had time to have a look at it again and I found the culprit. It´s hw.ibrs_disable = 0 which activates the IBRS-based mitigation. When set to "= 1" (pfsense default value, IBRS-based mitigation disabled) 10gbit throughput was easily achievable with the E3 1230v5 on OPnsense 22.1 (Netflow disabled, no Suricata). I never thought that the SpectreV2 mitigation would impact performance that much but well... I hope this helps someone with the same problem!

Setup:

Code: [Select]

[Linux PC1 - iperf client] <---"LAN - 10g"---> [OPNsense] <---"WAN - 10g"---> [Linux PC2 - iperf server]
Even the docs mention the huge performance hit! -> https://docs.opnsense.org/troubleshooting/hardening.html

[RamSense]

so curious question is: Is it safe to set hw.ibrs_disable = 1, while pfsense has it as default setting, or because of possible security issues on Intel systems with vulnerability to spectre and meltdown, opnsense set it to hw.ibrs_disable = 0 to "patch" this vulnerability?

[mimugmail]

If you have your BIOS updated and are on the latest version and you trust that all vendors fixed all still possible attack vectors ... its safe to disable 

[franco]

https://docs.opnsense.org/troubleshooting/hardening.html


Cheers,
Franco

[Mitzsch]

May I ask what the default setting is right now in OPNsense 22.1? I updated two firewalls to 22.1 and both had a different value set. First 22.1 beta to 22.1 final - value set to 0, Second 21.7 to 22.1 final - value set to 1 - on both systems I have not changed anything in the "Tunables" tab.

[franco]

Historically it depends how old your installation is since the config.xml from that time contains the tunable (which defaults to 0 for us) or not (which defaults to 1 on FreeBSD). The tunable hw.ibrs_disable was added to the default config.xml some time back in 2018 so one machine is older than the, the other is younger.


Cheers,
Franco

[RamSense]

For my perception, has there ever been a real exploit for these CPU vulnerabilities in opnsense / routers?
Or is this only hypothetical for external attack? And therefore safe to set to 1?
e.g. what made that opnsense team set it to 0 by default, trying to understand the considerations made to this, and me following with that.

[franco]

When we were using HardenedBSD the recommendation was to enable it by default. We might change the default in a future release, but it's been documented (see above) and really easy to decide for yourself what you need.

There's newer microcode you can install as well to make this obsolete.


Cheers,
Franco

[RamSense]

thanks for explaining. I will read more into it, but also wanting to keep my system as close as possible to the opnsense default hardening settings for safety

[Mitzsch]

Historically it depends how old your installation is since the config.xml from that time contains the tunable

Just wanted to say thank you, Franco! (sorry for the late reply) Indeed the config.xml that does set it to 1 is from an install made in 2016.