Weird Asynchronous LAN<->Firewall Performance (10G) / i7-8700 / Intel X710

[jm3s]

Hi there, I am having some weird performance issue with my bare-metal OPNsense installation at home.

When the traffic is passing from the LAN to the firewall, I can achieve something around 7-9 Gbit/s, but from the firewall to LAN, I am consistently only getting about 1.3 Gbit/s. The reason I discovered this is that I have a 10G WAN connection, but can only get roughly the same throughput of 1.3 Gbit/s, but figured it would be better to start debugging the issue locally.

The firewall is an oldish desktop PC with an Intel i7-8700 and an Intel X710 NIC (2x 10G). The client is an R9 3950X with a Mellanox ConnectX-4 Lx 2x 25G.
Sadly, I do not have two 10G clients, so I could put them on either side of the firewall, so I am measuring with iperf3 on the firewall itself.
It does not look like the CPU is a bottleneck in this case (attached image). I also attached a rough overview of the network hardware as a diagram.

Some other settings that might be relevant:
- The LAN interface has seven VLANs on it
- Hardware CRC/TSO/LRO offloading is enabled, but I've also tried without them, unsuccessfully
- Hardware VLAN filtering is on default
- Spectre mitigations are disabled as a test (hw.ibrs_disable = 1), but I did not observe any changes
- IDS is not active
- Firewall rules: there is a single allow all rule from the LAN, and the default rules including "let out anything from the firewall host itself"

I am not really sure what else to try, or how to identify the bottleneck exactly, and would be thrilled about any pointers :)
If any other information would be helpful, I am happy to provide it, but I wasn't sure what exactly would be helpful.

[pfry]

Any buffer delays/failures in "netstat -m"?

[jm3s]

Hi,

Does not look like it to me:

Code Select Expand

root@igw:~ # netstat -m
23465/16930/40395 mbufs in use (current/cache/total)
18615/15011/33626/1006134 mbuf clusters in use (current/cache/total/max)
42/4276 mbuf+clusters out of packet secondary zone in use (current/cache)
0/5334/5334/503067 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/149056 9k jumbo clusters in use (current/cache/total/max)
0/0/0/83844 16k jumbo clusters in use (current/cache/total/max)
43104K/55590K/98695K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 sendfile syscalls
0 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
0 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed