Assymetric poor performance on Sophos SG 105

[bruceg]

I am investigating a performance issue with my router as a new opnsense user. As configured out of the box, I could not get better than 480Mbit/sec in either direction when tested using iperf3 over the local LAN. I did discover the hardware CRC, TSO, and LRO settings, and un-disabled them. That has boosted my sending speed up to 800-950Mbit/sec, but the receive speed is unchanged.

While sending, top shows >90% system time and iperf3 CPU time. While receiving, the numbers are a little less but still close to 90%.

For comparison, I booted up Linux on the same system, and achieved 950-980Mbit/sec sending and receiving out of the box on the same interface. Other systems on the same LAN also get upwards of 950Mbit/sec in both directions.

The hardware is an Atom E3826 at 1.46GHz with 2GB RAM and 4 Intel GbE ports. I am running OPNsense version 22.1.2 (just updated and retested).

What else can I look at to improve the performance? Could some of the routing features cause that much drag on the performance? I'm not doing any content scanning.