DEC4280

[DMartinsson]

Hello all

We have implemented a few DEC4280 (currently on 25.10.1_2) in our company. After some application performance issues we checked with iperf3 what the firewalls can throughput and we are far from the specs:

Firewall Throughput 60Gpbs
Firewall Packets Per Second 5000Kpps
Firewall Port to Port Throughput 21Gpbs

Our internal DEC4280, which sits between the server and client vlans, are connected with 2x25Gbit lagg0 to the primary server vlan and 2x25gbit lagg1 to about 20 client and developer vlans.

iperf3 results between 2 servers in the server vlan are normal with 23-24gbit as all our hardware servers are also connected with 2x25gbit but between the vlans where the DEC4280 is we get:

iperf3 -c x.x.x.x -p 1234 -P1 3-4 Gbit
iperf3 -c x.x.x.x -p 1234 -P4 8-13 Gbit
iperf3 -c x.x.x.x -p 1234 -P8 10-22 Gbit

And you can see on the opnsense with top -CHIPS that you have 1 or 4 or 8 cores at 99% with iperf3 -P1, -P4, -P8.

And the iperf3 -P4 and -P8 runs vary. Sometimes -P8 has 10 Gbit, sometimes it has 16, sometimes 22. It varies between runs and not in a run.
You can see with top -CHIPS that the number of CPUs at 99% correlates with it. The higher it gets the more CPUs are at 99%.

The low numbers at single stream are also visible at file copy over cifs between windows clients and windows server. Here we get 250-300 MByte/s which would corespondent to the 3-4 Gbit above.


I have already tried rss enable or disable.
net.isr.maxthreads=-1
net.isr.bindthreads = 1
net.isr.dispatch = deferred
net.inet.rss.enabled = 1
net.inet.rss.bits = 4
enable or disable does not make a difference at single stream. At multi streams the cpus are more even utilized.

Power Savings are set to maximum, that made the difference that the iperf3 tests dont begin at low numbers and increase.

hw.ibrs_disable = 1 made light difference
ice_ddp_load = yes made a hugh difference, bevor we had 1 gbit in single stream.


I find 3-4 Gbit for a single stream to low for this big box with 25gbit connectivity and the spec "Firewall Port to Port Throughput 21Gpbs"

Any suggestions?
Has anyone with a DEC4280 also run iperf3 tests?

br
Daniel

[Monviech (Cedrik)]

Did you upgrade to the latest available BIOS?

https://forum.opnsense.org/index.php?topic=48449.0

[DMartinsson]

Did you upgrade to the latest available BIOS?

https://forum.opnsense.org/index.php?topic=48449.0

A few months ago. It made a difference that multi stream get over 8gbit.

[pfry]

I'm not surprised that the quoted performance is a bit hard to achieve. The device is a throughput device - did you try it with 12 or 16 threads?

[DMartinsson]

I'm not surprised that the quoted performance is a bit hard to achieve. The device is a throughput device - did you try it with 12 or 16 threads?

Yes, sometimes it gets 14, sometimes 23 gbit.

But the point is the performance for single threads which results in only 3-4gbit when one client speaks to one server.