WAN throughput performance

[SonjaS]

Hello everyone! I read a lot on this forum and other websites about opnsense and performance issues, but nothing seems to help my setup. And I am not on a virtual environment like proxmox as in several posts. OPNsense is running directly on hardware.

My setup is as folllows:

[ISP modem/router] ------- (NIC2)[OPNsense](NIC1)------vlan trunk------ [L2 switch] ------access vlan------ [Client]

Software
OPNsense 23.1.9-amd64
FreeBSD 13.1-RELEASE-p7
OpenSSL 1.1.1t 7 Feb 2023
Hardware
Intel(R) Core(TM) i5-7500T CPU @ 2.70GHz (4 cores, 4 threads)

On NIC1 there are all internal vlan interfaces and trunked to the switch. On NIC2 there is no vlan tagging and is connected to the ISPs-device. The CPU usage is between 0 and 10%, mostly 0%.

Issue
On speedtest I get the full upload performance my ISP is offering (50 mbit/s). But the download is only getting 90 mbit/s, when my ISP offers 1000 mbit/s. When I skip the OPNsense I can get true 1050 mbit/s.

I tried:
* keeping the parent interfaces activated (as officially reported)
* activating / deactivating hardware CRC,TSO,LRO global and each interface
* activating / deactivating hardware vlan filtering global and each interface (this logged me out :D)
* using automatic, hybrid or no NAT
* activating / deactivating IDS
* using several tunables suggested (most of them were marked unsupppported)
* using traffic shappping suggestions

Is there someone with the same issue and can help?

[meyergru]

Assuming you use a true multi-threaded test (like iperf -P8 or speedtest.net), have you tried if the interface speed is really 1000 MBit/s? I just had this fallback to 100 MBit/s on an otherwise 1000 MBit/s link on a Nokia ONT. A reboot fixed it. You can check this in the interface overview or on the dashboard.

Also, you can try flow control: https://forum.opnsense.org/index.php?topic=27992.0