"
Hi Daniel,
we're seeing the exact same issue on our DEC3862 (EPYC 3201, ax driver, 10G SFP+). Single-stream tops out at 3–4 Gbit/s (or even less) through the firewall, multi-stream varies wildly. The appliance was purchased for 10G inter-VLAN routing and right now is clearly not delivering.
We spent quite some time isolating the problem the last couple of weeks and found something important: it seems to be a firmware regression. On our second DEC3862 (vanilla install, no plugins, no custom config, no tunables, just one single vlan) we benchmarked with iperf3 before and after updating:
25.10: 9.45 Gbit/s receive, 6.56 Gbit/s send (essentially line speed)
25.10.2: 3.35 Gbit/s receive, 1.77 Gbit/s send
Rollback to 25.10 via ZFS snapshot: line speed restored immediately
We also tested OPNsense on completely different hardware (i9-9900K with Intel X520-DA2, ix driver). Full line speed right from the start, no issues. So it seems to be specific to the Deciso appliances and the firmware versions after 25.10.
What's even more alarming is that a UDP test on our production DEC3862 (25.10.2) showed 48% packet loss at 10G. The appliance is silently dropping nearly half of all packets under load. With UDP some packet loss is expectable but nearly 50% is unusual. With TCP you don't notice it as clearly because retransmits mask the drops, but the underlying loss rate is severe.
Since you're on 25.10.1_2 and seeing the same symptoms on a DEC4280 with ice driver (I guess?) this confirms it's not a NIC-driver-specific problem but rather something in the FreeBSD kernel or pf packet path that was introduced after 25.10 (for example from what I can see from the changelogs, the way how checksums are handled has changed).
We're opening a support ticket with Deciso including all our benchmark data. Happy to share our findings in more detail if that helps. If you have a ZFS snapshot from before the update, it might be worth testing a rollback on your side as well to confirm.
Best regards,
Robin
we're seeing the exact same issue on our DEC3862 (EPYC 3201, ax driver, 10G SFP+). Single-stream tops out at 3–4 Gbit/s (or even less) through the firewall, multi-stream varies wildly. The appliance was purchased for 10G inter-VLAN routing and right now is clearly not delivering.
We spent quite some time isolating the problem the last couple of weeks and found something important: it seems to be a firmware regression. On our second DEC3862 (vanilla install, no plugins, no custom config, no tunables, just one single vlan) we benchmarked with iperf3 before and after updating:
25.10: 9.45 Gbit/s receive, 6.56 Gbit/s send (essentially line speed)
25.10.2: 3.35 Gbit/s receive, 1.77 Gbit/s send
Rollback to 25.10 via ZFS snapshot: line speed restored immediately
We also tested OPNsense on completely different hardware (i9-9900K with Intel X520-DA2, ix driver). Full line speed right from the start, no issues. So it seems to be specific to the Deciso appliances and the firmware versions after 25.10.
What's even more alarming is that a UDP test on our production DEC3862 (25.10.2) showed 48% packet loss at 10G. The appliance is silently dropping nearly half of all packets under load. With UDP some packet loss is expectable but nearly 50% is unusual. With TCP you don't notice it as clearly because retransmits mask the drops, but the underlying loss rate is severe.
Since you're on 25.10.1_2 and seeing the same symptoms on a DEC4280 with ice driver (I guess?) this confirms it's not a NIC-driver-specific problem but rather something in the FreeBSD kernel or pf packet path that was introduced after 25.10 (for example from what I can see from the changelogs, the way how checksums are handled has changed).
We're opening a support ticket with Deciso including all our benchmark data. Happy to share our findings in more detail if that helps. If you have a ZFS snapshot from before the update, it might be worth testing a rollback on your side as well to confirm.
Best regards,
Robin
