Hi,
I'm running opnSense on a china-box with intel 1Gbit/s NICs (igbn). opnsense is virtualized using VMware as hypervisor. The opnSense VM NICs ar configured as VMXNET3.
The Box in question was using a "Deutsche Glasfaser" FTTH connection. WAN received its IP via DHCP. The throughput was 400/200 Mbps (the same speed the ISP offered in their contract).
The ISP was now changed to "1&1" using the same FTTH line. 1&1 requires a connection via PPPoE. From other PPPoE based connections I use with other opnSense gear I know that PPPoE and higher bandwidths on FreeBSD is a bit of an issue.
Im my case switching from DHCP to PPPoE reduced the downstream from 400Mbps to only ~30Mbps. I then remembered some tweak-settings within opnSense. The setting that provided the biggest improvement was "Hardware TSO" within the Interface Settings. Once I "unchecked" Disable hardware TCP segmentation offload, I instantly received 600Mbps in Downstream (the new bandwidth the ISP offers). The settings had no effect towards my upstream which always worked like the contract specified (200 old contract and 300Mbps new contract).
I wonder why moving the TCP sgementation towards "hardware" improves the throughput so much. And why it didn't matter when using DHCP instead of PPPoE. MTU size is 1492 (auto configured by opnSense).