DEC840 slow upload test when using SFP+

[marcog]

Hi everyone,
I'm dealing with a frustrating upload speed issue on my DEC840 that I can't seem to figure out.

Currently, my WAN is connected to my ISP's ONT using an FS SFP-10G-T (SFP+ to RJ45) transceiver in one of the DEC840's 10G ports. When I run a speed test, I get around 2 Gbps download, but my upload is capped at roughly 500 Mbps.

Suspecting the copper transceiver was the bottleneck, I moved the WAN connection to one of the DEC840's native 1 Gbps RJ45 ports. As expected, my download speed dropped to ~940 Mbps, but surprisingly, my upload actually increased to ~700 Mbps.

Thinking I was on the right track and just dealing with a issue on the transceiver, I bought an unmanaged switch with 2.5 Gbps RJ45 ports and 10 Gbps SFP+ ports bypass it.
- I connected the ONT to the switch's 2.5G port.
- I connected the DEC840 to the switch's 10G port via a 10Gtek SFP+ DAC cable, completely bypassing the FS copper adapter.

To my surprise, when I retried the speed test, I was right back to where I started: 2 Gbps down, and stuck at ~500 Mbps upload.

It seems like there is an underlying issue with how OPNsense/FreeBSD is handling outbound traffic on the SFP+ interface itself, regardless of whether I use a transceiver or a switch.

I should also mention that I'm dealing with the infamous PPPoE for this connection. While I am seeing packet loss during the speed tests, that still doesn't explain the 200 Mbps discrepancy. If PPPoE single-core performance was the only bottleneck, the 1G RJ45 port shouldn't be outperforming the 10G SFP+ port

Has anyone experienced this or have any ideas on what tunables/settings I should check next?

Thanks in advance!

[tedhughes]

Reviving this because I have the same symptom on the same hardware, and I've been able to rule out a lot of the variables you couldn't.

Setup

• DEC840, WAN on ax0, LAN on ax1 (8 tagged VLANs + untagged LAN)
• 8 Gbps symmetrical fiber, static /26, no PPPoE (key difference from OP's setup)
• WAN transceiver: FS SFP-10G-T (30m), on Deciso's tested-modules list for the DEC800 series
• LAN: passive SFP+ DAC to a UniFi 10G switch

Symptom

Download 3.5-4.7 Gbps. Upload pinned at ~600-870 Mbps, consistently, regardless of test method or target. Notably it has twice briefly exceeded 1.1 Gbps mid-test before collapsing back — so it behaves like a dynamic loss mechanism, not a static cap.

The control test that matters

A Mac Studio connected directly to the ONT — same static IP, same cable, same ONT port — gets ~6 Gbps in both directions. Circuit, provisioning, and the ISP upload policer are all demonstrably fine. The cap only appears when traffic passes through the DEC840.

What I've ruled out (all measured, not assumed)

• Forwarding capacity — iperf3 Mac <-> OPNsense, 8 streams: 9.11 Gbit/s aggregate
• LAN segment / DAC / switch — single stream: 2.65 Gbps up / 4.09 Gbps down, ~0 retransmits
• Interface errors — netstat -i -I ax0 -w 1 during sustained upload: zero errs, zero idrops, both directions
• CPU — top -aSH during upload: all cores essentially idle
• Shaper — pfctl -sq empty, no pipes/queues configured
• IDS/IPS/DPI — Suricata, Zenarmor, ntopng all stopped, nothing attached via netmap
• Hardware offload — CRC/TSO/LRO disabled, verified absent from ifconfig flags: no change
• pf scrub — disabled: no change
• MSS — SYN-ACKs show mss 1460, not clamped
• VPN policy routing — LAN clients egress the correct static IP
• Flow control — dev.ax.0.rx_pause/tx_pause=1 + reboot: no change, media line reports rxpause,txpause regardless

Packet-level finding

tcpdump on ax0 during upload shows SACK blocks in the returning ACKs across multiple independent flows:

Code Select Expand

sack 1 {73849:78193}
sack 1 {53577:75297}
sack 1 {73849:101361}
Receivers are reporting gaps in the upload stream — frames leave ax0 and don't all arrive, while every counter on my side stays clean. Advertised receive windows on those flows also shrink under load (2045 -> 2003 -> 1972). Per-flow throughput works out to roughly 120-160 Mbit/s; 4-5 parallel flows sums to the observed ceiling.

Version ladder (tested identically at each step)

Version	FreeBSD base	Upload
24.7.7	14.1	~600 Mbps
24.7.12	14.1	~600 Mbps
25.1.12	14.2	~600 Mbps

Continuing up the ladder; will update with results.

Tunables — factory Deciso defaults, unmodified:

Code Select Expand

dev.ax.0/1.iflib.override_nrxds = 2048 x8
dev.ax.0/1.iflib.override_ntxds = 2048 x8
dev.ax.0/1.link_workaround = 1
dev.ax.sph_enable = 0
Where this leaves it

The OP's switch-bypass test (ONT -> switch -> DAC -> SFP+, no copper transceiver in the path) produced no change, which points away from the transceiver and toward the ax TX path itself. I'm running the equivalent test on my side and will report back.

Two questions for anyone who's been here:

1. Has anyone with a DEC840/DEC850 achieved sustained multi-gig upload on ax0 with a real WAN — and if so, on what version?
2. Is there any known asymmetry in how axgbe handles TX for forwarded/NAT'd traffic vs locally-originated? Everything I can measure says the interface is healthy and the packets simply don't all survive the trip.

Happy to provide full pcaps or run any test that would help.