Wireguard site to site tunnel slow, high CPU usage, internet ok

[jmarefit88]

Looking for some assistance in where to go next on troubleshooting this issue that I seem to be having.

Running an identical pair of Dell R620 servers at a local and remote site.  Both sites are connected via 1Gb fiber internet connections.  Have built a Wireguard site to site tunnel on top of that connection.

When running a speedtest.net speed test, from either site, I can get near gig speed up and down (~980Mbps).  When running an OpenSpeedTest or iPerf3 test across the tunnel from local to remote or remote to local, it maxes at about 180Mbps.  When running the standard internet speed test, the CPU stays "idle" around 2-3%.  When running it across the site to site Wireguard tunnel, the CPU spikes to ~70% for the duration of the test.

We were running an old version of OpnSense (21.x if I recall correctly) and recently upgraded, but we were seeing those slow speeds before as well and were hoping the upgrade resolved whatever issue might be lingering there.  At one point post reboot, we briefly saw almost line rate across the wireguard, but then within an hour it had disappeared and went back to the seemingly capped speeds.

I have went in and set the tunable for hw.ibrs_disable to 1 on both ends.  No difference seemed to be there.

I've tried turning on powerd and using hiadaptive and no change.

The NICs are Broadcom onboard NICs and the CPU is a dual socket Intel Xeon E5-2430 @ 2.2Ghz.  (12 cores total, 24 threads)

Any suggestions of what to try next?  I've debated trying an IPsec based tunnel to see if that handles it.

Appreciate any guidance and support on how to debug this one!