FireWall speed

[zebr]

Good afternoon. Caught such a problem.
OPNSENSE (22.7.10_2) is deployed on a physical Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (40 cores, 80 threads) server. The server is connected to the switch with 2x10Gb/s links. There are servers behind the switch that are also connected with 2x10Gb/s links. VMs are deployed on them. VMs are located in different subnets. And here's the problem:
if I have any source and any destination specified in the firewall rules, then when measuring iperf
iperf3 -c X.X.X.X -l 9000 -t 120 -P40 outputs a speed of 14-15 GB/s. And if I start prescribing strict rules to specific hosts in the firewall, then the iperf3 -c X.X.X.X -l 9000 -t 120 -P40 test does not pass. And only perf3 -c X.X.X.X -l 9000 -t 120 passes, and with such testing, the speed floats from 2.5 to 7 GB / s. Ie, iperf tests in several threads do not pass. Returning firewall settings all to all. Everything is restored and the speed is approaching 15Gb/s. When checking the channel by transferring traffic from VM to VM, we see the same situation. The processor is loaded by 12-15% no more during testing.

[vpx]

The default server port for iperf3 is 5201. I don't know if the 40 parallel connections (-P) need additional ports to be opened.

Check the firewall log for the server IP, if there are any blocked ports.

[abulafia]

Try a CPU with less cores but higher per-core ooomph (higher IPC, higher CPU frequency, etc.)

And try it without using HyperThreading (SMT).