Hi,
I'm using DEC4280 appliances from Deciso with OPNsense preinstalled. The goal is to use them in HA with two link aggregations: two SFP28 interfaces for the internal VLANs and two SFP+ interfaces for the uplink.
Unfortunately during the tests, it was difficult to obtain multi-gigabit speeds. So I'm back using one appliance with a very basic configuration to perform tests:
- igc0 (assigned to LAN): just for management purpose
- ice0 (assigned opt1): plugged to a switch in access mode on a VLAN
- ice1 (assigned opt2): plugged to a switch in access mode on another VLAN
On interfaces opt1 and opt2, only one rule allow all traffic to pass. There is only one machine in each VLAN. With iperf3, I got this result:
# iperf3 -c 10.2.2.12 -p 5201
Connecting to host 10.2.2.12, port 5201
[ 5] local 10.1.1.11 port 38024 connected to 10.2.2.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 129 MBytes 1.08 Gbits/sec 52 1.17 MBytes
[ 5] 1.00-2.00 sec 125 MBytes 1.05 Gbits/sec 0 1.28 MBytes
[ 5] 2.00-3.00 sec 126 MBytes 1.06 Gbits/sec 0 1.36 MBytes
[ 5] 3.00-4.00 sec 126 MBytes 1.06 Gbits/sec 0 1.43 MBytes
[ 5] 4.00-5.00 sec 126 MBytes 1.06 Gbits/sec 1 1.08 MBytes
[ 5] 5.00-6.00 sec 125 MBytes 1.05 Gbits/sec 0 1.16 MBytes
[ 5] 6.00-7.00 sec 126 MBytes 1.06 Gbits/sec 0 1.24 MBytes
[ 5] 7.00-8.00 sec 126 MBytes 1.06 Gbits/sec 0 1.31 MBytes
[ 5] 8.00-9.00 sec 126 MBytes 1.06 Gbits/sec 0 1.38 MBytes
[ 5] 9.00-10.00 sec 125 MBytes 1.05 Gbits/sec 3 1.02 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.23 GBytes 1.06 Gbits/sec 56 sender
[ 5] 0.00-10.00 sec 1.23 GBytes 1.06 Gbits/sec receiver
iperf Done.
But if I plug the two machines directly on the same VLAN, bypassing the firewall, I got this result:
# iperf3 -c 10.1.1.12 -p 5201
Connecting to host 10.1.1.12, port 5201
[ 5] local 10.1.1.11 port 40454 connected to 10.1.1.12 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 2.63 GBytes 22.6 Gbits/sec 61 2.53 MBytes
[ 5] 1.00-2.00 sec 2.66 GBytes 22.8 Gbits/sec 0 2.91 MBytes
[ 5] 2.00-3.00 sec 2.66 GBytes 22.9 Gbits/sec 0 2.96 MBytes
[ 5] 3.00-4.00 sec 2.68 GBytes 23.0 Gbits/sec 0 3.14 MBytes
[ 5] 4.00-5.00 sec 2.52 GBytes 21.6 Gbits/sec 47 2.43 MBytes
[ 5] 5.00-6.00 sec 2.66 GBytes 22.8 Gbits/sec 0 2.48 MBytes
[ 5] 6.00-7.00 sec 2.66 GBytes 22.8 Gbits/sec 0 2.55 MBytes
[ 5] 7.00-8.00 sec 2.69 GBytes 23.1 Gbits/sec 0 2.55 MBytes
[ 5] 8.00-9.00 sec 2.67 GBytes 22.9 Gbits/sec 0 2.66 MBytes
[ 5] 9.00-10.00 sec 2.66 GBytes 22.9 Gbits/sec 0 2.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 26.5 GBytes 22.7 Gbits/sec 108 sender
[ 5] 0.00-10.00 sec 26.5 GBytes 22.7 Gbits/sec receiver
Increasing the number of parallel streams increased a bit the bandwidth but we are far from 10Gb/s (and we are using 25Gb/s interfaces). The only time I reached near 10Gb/s was with firewall disabled and a lot of parallel streams.
I tried multiple tunable settings found on this forum, tried to enable/disable HW offloading and I also tried to use the SFP+ interfaces, but same results.
The interfaces looks healthy and the counters for errors and collisions are at 0.
I'm running out of ideas to troubleshoot this. Anyone got similar issues?
Thanks.
EDIT: The new BIOS version (https://docs.opnsense.org/hardware/bios.html#dec4200-series) fixed the issue.
The speed is so ridiculously close to exactly 1 Gbit/s that I would believe the negotiated link speed is at 1 GBit/s for one or both interfaces. Did you inspect the settings? With these types of interfaces, often times it is neccessary to manually set link speeds.
Quote from: l1lz on July 17, 2025, 05:30:31 PM[...]
I tried multiple tunable settings found on this forum [...]
So long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput. How does CPU utilization look while running the tests?
If you weren't using a Deciso device, I'd recommend looking at the ice devices, e.g. "dmesg | grep ice" and "pciconf -lvV ice0" for any oddities (mine really wanted to set up as x4 v3; you want x4 v4 or x8 v3 at a minimum).
That speed does look suspicious, like a per-flow/session shaper.
Quote from: meyergru on July 17, 2025, 06:47:25 PMThe speed is so ridiculously close to exactly 1 Gbit/s that I would believe the negotiated link speed is at 1 GBit/s for one or both interfaces.
That was also my first thought, but they negotiate 25G, and if I use parallel streams it can get up to 5 Gb/s. But a single flow seems capped at around 1 Gb/s. I also changed settings about error corrections and flow control, but it has no effect.
ice0: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice0: link state changed to UP
ice1: Link is up, 25 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: RS-FEC, Autoneg: False, Flow Control: None
ice1: link state changed to UP
Quote from: pfry on July 17, 2025, 10:19:43 PMSo long as you're not terminating iperf sessions on the firewall, about the only tunables/sysctls that'll help you are RSS: net.isr.bindthreads, net.isr.maxthreads, net.inet.rss.bits, net.inet.rss.enabled. Have a look at "netstat -Q" (IIRC) to check. I believe OPNsense sets most other necessary sysctls reasonably. But I believe RSS is mainly good for throughput. How does CPU utilization look while running the tests?
I just reverted all the tunables to the default config that came with the device and I enabled this:
- net.isr.bindthreads = 1
- net.isr.maxthreads = -1
- net.inet.rss.enabled = 1
- net.inet.rss.bits = 4
After reboot, results are the same. The command
netstat -Q shows a lot of workstreams now. If I use iperf with the option -P 16, I get up to 4.3 Gbit/s and a single stream is around 1.5Gbit/s. During transfer with 16 streams, some CPU cores are indeed used at 100%, global usage is:
CPU: 0.0% user, 0.0% nice, 23.6% system, 30.0% interrupt, 46.4% idle.
I noticed an error regarding the ice interfaces while running
sysctl command, do you think it's related?
root@OPNsense:~ # sysctl -a | grep rss
ice0: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice1: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice2: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
ice3: ice_add_rss_cfg on VSI 0 could not configure every requested hash type
net.inet.rss.bucket_mapping: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7 8:8 9:9 10:10 11:11 12:12 13:13 14:14 15:15
net.inet.rss.enabled: 1
net.inet.rss.debug: 0
net.inet.rss.basecpu: 0
net.inet.rss.buckets: 16
net.inet.rss.maxcpus: 64
net.inet.rss.ncpus: 32
net.inet.rss.maxbits: 7
net.inet.rss.mask: 15
net.inet.rss.bits: 4
net.inet.rss.hashalgo: 2
hw.bxe.udp_rss: 0
hw.ix.enable_rss: 1
dev.ax.3.rss_enabled: 1
dev.ax.2.rss_enabled: 1
dev.ax.1.rss_enabled: 1
dev.ax.0.rss_enabled: 1
root@OPNsense:~ # dmesg | grep vectors
igc0: Using MSI-X interrupts with 5 vectors
igc1: Using MSI-X interrupts with 5 vectors
igc2: Using MSI-X interrupts with 5 vectors
igc3: Using MSI-X interrupts with 5 vectors
ice0: Using MSI-X interrupts with 33 vectors
ice1: Using MSI-X interrupts with 33 vectors
ice2: Using MSI-X interrupts with 33 vectors
ice3: Using MSI-X interrupts with 33 vectors
ax0: Using MSI-X interrupts with 16 vectors
ax1: Using MSI-X interrupts with 16 vectors
ax2: Using MSI-X interrupts with 6 vectors
ax3: Using MSI-X interrupts with 6 vectors
As far as I understand, RSS only helps with multiple flows. Is 1Gbit/s the best we can expect from this device for a single flow?
Quote from: l1lz on July 18, 2025, 10:03:10 AM[...]
I noticed an error regarding the ice interfaces while running sysctl command, do you think it's related?
I do not, but of course I could be wrong. I don't recall the error on my machine (offline, so I can't look), but I believe I had had both virtualization and SR-IOV disabled in the BIOS (regular PC, bare metal FreeBSD install).
Speaking of that, I quoted the wrong pciconf command: should be "-lvc [interface]".
How about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd? (I have an anomalous Idrop stat on ixl interfaces and lots of unaccountable Oerrs on one bridge.)
Quote[...] Is 1Gbit/s the best we can expect from this device for a single flow?
It shouldn't be, but I don't have similar hardware to test.
Assuming you want fast opt1 to opt2 file transfers, have you tried copying a large file from one computer to the other?
@l1lz just to be sure, you are testing with the firewall in between iperf3 machines and with our default config. running iperf3 on your firewall is a common testing issue as it influences your measurements (a lot).
our default test setup (which reaches reaches the advertised speeds), is setup like:
[client] --> [firewall] --> [server]
The client uses an iperf3 command like:
iperf3 -c <other-host> -P 8 -Z -t 5 -M 1500
Somethings to consider when numbers don't match:
* ice needs firmware loaded, which is in our default config, the tunable is called "ice_ddp_load"
* make sure to disable the default reply-to rule (Firewall: Settings: Advanced), when a gateway is accidentally configured it may send you packets in the wrong direction.
For regular routing in my experience rss doesn't really help much as most network cards already distribute the load properly over different cores. When results are less than expected, always check `top -CHIPS` as it helps figuring out if flows are distributed as one would expect.
Best regards,
Ad
Quote from: AdSchellevis on July 24, 2025, 09:21:19 PMour default test setup (which reaches reaches the advertised speeds), is setup like:
[client] --> [firewall] --> [server]
I confirm that we are not using the firewall's iperf server but two machines connecting to the firewall as described in your message. There was a switch initially but I just did the test again without any switch and the result is about the same (around 5 Gbit/s with parallel threads).
root@PROXMOX4:~# iperf3 -c 10.1.1.12 -p 5201 -P 8 -Z -t 5 -M 1500
Connecting to host 10.1.1.12, port 5201
[...]
[SUM] 0.00-5.00 sec 2.79 GBytes 4.79 Gbits/sec 873 sender
[SUM] 0.00-5.02 sec 2.77 GBytes 4.75 Gbits/sec receiver
iperf Done.
I checked "Disable reply-to on WAN rules" but it changed nothing. The tunable `ice_ddp_load` is set to `YES`.
Quote from: pfryHow about some more "netstat"s: "-m", "-i", "-sp ip" - anything look odd?
I'm definitely not an expert, but I don't have packets drops or errors here. Same for the pciconf command:
root@OPNsense:~ # pciconf -lvc ice0
ice0@pci0:6:0:0: class=0x020000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x1593 subvendor=0x8086 subdevice=0x0005
vendor = 'Intel Corporation'
device = 'Ethernet Controller E810-C for SFP'
class = network
subclass = ethernet
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 512 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x8000]
cap 10[a0] = PCI-Express 2 endpoint max data 512(512) FLR RO
max read 512
link x16(x16) speed 8.0(16.0)
cap 03[e0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 000e[148] = ARI 1
ecap 0003[150] = Serial 1 000100ffff000000
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0100, VF RID Stride 0x0001
VF Device ID 0x1889
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
P2P Direct Translated unavailable, Enhanced Capability unavailable
ecap 0019[1d0] = PCIe Sec 1 lane errors 0
ecap 0025[200] = Data Link Feature 1
ecap 0026[210] = Physical Layer 16.0 GT/s 1
ecap 0027[250] = Lane Margining at Receiver 1
Hello,
we experience the same Problem with two DEC4280. We tried all the things mentioned in this thread and more. At the start we were below 500 MBit/s, but we can reach now 1 GBit/s.
The test iperf3 setup is also:
client -> firewall -> server
But with only the 10GBit interfaces. To troubleshoot the issue even more we tried a selection of SFP+ modules including fiber, DAC and RJ45.
The speed seems to be normal with the RJ45 (igc0 - igc3) port on the firewall if connected via a 1G switch.
Hello,
I also wanted to chime in — we are experiencing the reduced speeds as well. When connecting two computers through the firewall, we see speeds of approximately 1.5–2 Gbit/s, whereas directly connecting them yields the expected 10 Gbit/s. The problem only occurs when using the 10/25 Gbit ports; the 1 Gbit ports work flawlessly.
The issue only persists when the firewall is directly involved in packet inspection. When running iperf on the firewall as a client (with a computer acting as the server), we achieve the expected performance. Another observation is the abysmal performance when running iperf locally on the firewall. Hosting and connecting to iperf3 locally yields about 5 Gbit/s, whereas my i7-10850H in WSL achieves around 30 Gbit/s.
A non-comprehensive list of things we have already tried:
- Disabled firewalling completely
- Adjusted various tunables
- Enabled RSS
- Disabled virtualization
- Configured LACP
- Various BIOS tweaks / complete BIOS reflash
- Reinstallation of OPNsense
- Various different OPNsense versions (24.7–26.1dev)
- Different SFP modules
There are no visible errors — neither packet drops nor any dmesg errors.
Another point to note is the lack of a second NUMA node in the output of lstopo, contrary to what is shown in the ServeTheHome AMD EPYC 3451 review (https://www.servethehome.com/amd-epyc-3451-benchmarks-and-review/#:~:text=late%20November%202019%2C%20the%20top,to%20cover%20in%20our%20review)
At this point, we are at our wits' end and hope to find any answer.
@DEC4280Problem, @User65192 it might be better to contact us directly if you purchased a machine and having difficulties with your setup. Creating new accounts (yes, I checked) every week reporting the same challenges usually isn't very helpful.
We have a test setup over here available and do run these tests quite often showing the expected results, when needed we also offer excellent commercial support to debug your setup.
Best regards,
Ad
@AdSchellevis I would like to open a support case, but unfortunately our reseller didn't get us support when we bought the appliances... So we are now in the process of acquiring support, and in the meantime I opened this topic. It seems that I'm not the only one facing this problem, so if a solution is find, it would be great to share it.
One questions to all of the affected.
Do you run Suricata or ZenArmor?
Regard,
S.
Quote from: Seimus on August 08, 2025, 01:52:45 PMDo you run Suricata or ZenArmor?
No, IDS is not enabled. I got the problem with the default configuration out of the box, with just two interfaces configured and basic firewall rules.
@l1lz can you drop me an email? there is a chance this is bios related, in which case I can offer something to test/validate on your end.
I've been struggling with this too. Not sure if its the same issue, but I feel its helpful to share. My hardware is a Qotom with the C3808 Atom and 4 10G SFP ports, so maybe this isn't related, but it sure sounds like it might be.
Where I see slowdown is in receiving on the interface. Just a simple iperf test, I can get 9.4Gb sending firewall->server. But all sending from server->firewall are 1G-1.5G. Between servers i get 9+Gb both direction, on same switch. Like others, testing with direct connection, connect to switch, using fiber , DAC, or CAT6a, different SFP modules, different ports, different server, different NIC's, same results. And tried all kinds of tunnables. But I'm seeing some more in other posts I'd like to test, and I'll double check my bios.
@AdSchellevis, Sorry, I understand me creating a new account just to chime in isn't too much help, but I find it helpful. ;) If there's anything I can test I'd be happy to report back.
Edit:
I wonder if this has something to do with this? Link PCIe speed is 1x?
pciconf -lbcevV pci0:11:0:0
cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
max read 512
link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
Quote from: pizza_shrapnel on August 08, 2025, 09:05:08 PM[...]
I wonder if this has something to do with this? Link PCIe speed is 1x? [...]
Could be. The SOC could have some oddities, but x1 v1 for a 10Gb interface seems unlikely. I'd run the query against the device mnemonic rather than the PCI address.
If that link speed is correct (and pciconf usually does report it correctly), then the speed would be limited at 500 MBit/s with 1 * PCIe 2.x.
My OpnSense show this for an Intel 82599ES:
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
max read 512
link x4(x8) speed 5.0(5.0) ASPM disabled(L0s)
Yours seems to even be a toned-down PCIe 2.x version (because the base speed of PCIe 2 should be 5 GT/s, not 2.5 GT/s, which would be PCIe 1.0).
Was that for the correct device? You can list all by leaving out the pci bus id.
Can you dump sysctl -a into a post. Feel free to mask out any settings you don't want seen publicly.
Does that dec have HT enabled?
Hello @BrandyWine,
Here is our sysctl -a output:
privatebin.net (https://privatebin.net/?bd10aac1f1a94413#B6SkdGbiGbfmMrsTMwjB9XFzPhZbuYrJU4Rvq3WVysYU)
Hyperthreading is enabled.
Kind regards
Hi,
We just uploaded a new bios version, which can be downloaded from https://docs.opnsense.org/hardware/bios.html#dec4200-series and should resolve the throughput issues we have seen.
Best regards,
Ad
I confirm that the new BIOS version solved the issue for me. Many thanks!