DEC840 slow upload test when using SFP+

[marcog]

Hi everyone,
I'm dealing with a frustrating upload speed issue on my DEC840 that I can't seem to figure out.

Currently, my WAN is connected to my ISP's ONT using an FS SFP-10G-T (SFP+ to RJ45) transceiver in one of the DEC840's 10G ports. When I run a speed test, I get around 2 Gbps download, but my upload is capped at roughly 500 Mbps.

Suspecting the copper transceiver was the bottleneck, I moved the WAN connection to one of the DEC840's native 1 Gbps RJ45 ports. As expected, my download speed dropped to ~940 Mbps, but surprisingly, my upload actually increased to ~700 Mbps.

Thinking I was on the right track and just dealing with a issue on the transceiver, I bought an unmanaged switch with 2.5 Gbps RJ45 ports and 10 Gbps SFP+ ports bypass it.
- I connected the ONT to the switch's 2.5G port.
- I connected the DEC840 to the switch's 10G port via a 10Gtek SFP+ DAC cable, completely bypassing the FS copper adapter.

To my surprise, when I retried the speed test, I was right back to where I started: 2 Gbps down, and stuck at ~500 Mbps upload.

It seems like there is an underlying issue with how OPNsense/FreeBSD is handling outbound traffic on the SFP+ interface itself, regardless of whether I use a transceiver or a switch.

I should also mention that I'm dealing with the infamous PPPoE for this connection. While I am seeing packet loss during the speed tests, that still doesn't explain the 200 Mbps discrepancy. If PPPoE single-core performance was the only bottleneck, the 1G RJ45 port shouldn't be outperforming the 10G SFP+ port

Has anyone experienced this or have any ideas on what tunables/settings I should check next?

Thanks in advance!

[tedhughes]

Reviving this because I have the same symptom on the same hardware, and I've been able to rule out a lot of the variables you couldn't.

Setup

• DEC840, WAN on ax0, LAN on ax1 (8 tagged VLANs + untagged LAN)
• 8 Gbps symmetrical fiber, static /26, no PPPoE (key difference from OP's setup)
• WAN transceiver: FS SFP-10G-T (30m), on Deciso's tested-modules list for the DEC800 series
• LAN: passive SFP+ DAC to a UniFi 10G switch

Symptom

Download 3.5-4.7 Gbps. Upload pinned at ~600-870 Mbps, consistently, regardless of test method or target. Notably it has twice briefly exceeded 1.1 Gbps mid-test before collapsing back — so it behaves like a dynamic loss mechanism, not a static cap.

The control test that matters

A Mac Studio connected directly to the ONT — same static IP, same cable, same ONT port — gets ~6 Gbps in both directions. Circuit, provisioning, and the ISP upload policer are all demonstrably fine. The cap only appears when traffic passes through the DEC840.

What I've ruled out (all measured, not assumed)

• Forwarding capacity — iperf3 Mac <-> OPNsense, 8 streams: 9.11 Gbit/s aggregate
• LAN segment / DAC / switch — single stream: 2.65 Gbps up / 4.09 Gbps down, ~0 retransmits
• Interface errors — netstat -i -I ax0 -w 1 during sustained upload: zero errs, zero idrops, both directions
• CPU — top -aSH during upload: all cores essentially idle
• Shaper — pfctl -sq empty, no pipes/queues configured
• IDS/IPS/DPI — Suricata, Zenarmor, ntopng all stopped, nothing attached via netmap
• Hardware offload — CRC/TSO/LRO disabled, verified absent from ifconfig flags: no change
• pf scrub — disabled: no change
• MSS — SYN-ACKs show mss 1460, not clamped
• VPN policy routing — LAN clients egress the correct static IP
• Flow control — dev.ax.0.rx_pause/tx_pause=1 + reboot: no change, media line reports rxpause,txpause regardless

Packet-level finding

tcpdump on ax0 during upload shows SACK blocks in the returning ACKs across multiple independent flows:

Code Select Expand

sack 1 {73849:78193}
sack 1 {53577:75297}
sack 1 {73849:101361}
Receivers are reporting gaps in the upload stream — frames leave ax0 and don't all arrive, while every counter on my side stays clean. Advertised receive windows on those flows also shrink under load (2045 -> 2003 -> 1972). Per-flow throughput works out to roughly 120-160 Mbit/s; 4-5 parallel flows sums to the observed ceiling.

Version ladder (tested identically at each step)

Version	FreeBSD base	Upload
24.7.7	14.1	~600 Mbps
24.7.12	14.1	~600 Mbps
25.1.12	14.2	~600 Mbps

Continuing up the ladder; will update with results.

Tunables — factory Deciso defaults, unmodified:

Code Select Expand

dev.ax.0/1.iflib.override_nrxds = 2048 x8
dev.ax.0/1.iflib.override_ntxds = 2048 x8
dev.ax.0/1.link_workaround = 1
dev.ax.sph_enable = 0
Where this leaves it

The OP's switch-bypass test (ONT -> switch -> DAC -> SFP+, no copper transceiver in the path) produced no change, which points away from the transceiver and toward the ax TX path itself. I'm running the equivalent test on my side and will report back.

Two questions for anyone who's been here:

1. Has anyone with a DEC840/DEC850 achieved sustained multi-gig upload on ax0 with a real WAN — and if so, on what version?
2. Is there any known asymmetry in how axgbe handles TX for forwarded/NAT'd traffic vs locally-originated? Everything I can measure says the interface is healthy and the packets simply don't all survive the trip.

Happy to provide full pcaps or run any test that would help.

[tedhughes]

SOLVED on my end — and the answer might apply to your setup too, so posting the full findings.

TL;DR: It was the 10GBASE-T SFP+ transceiver's link to the ISP's ONT. Not the DEC840, not axgbe, not OPNsense. Bypassing the transceiver by landing the ONT on a switch's native 10GBASE-T port and trunking WAN to the DEC took me from ~600 Mbps upload to 4.4 Gbps.

My setup (for comparison with OP)

• DEC840, 8 Gbps symmetrical fiber, static /26, no PPPoE
• WAN was: FS SFP-10G-T (30m, on Deciso's tested list) in ax0, Cat6a to the ISP ONT's RJ45 handoff
• LAN: SFP+ DAC from ax1 to a UniFi switch, untagged LAN + 8 tagged VLANs

Symptom — same as OP

Download 3.5–4.7 Gbps, upload pinned at ~600–870 Mbps regardless of test method. Occasionally it would briefly spike past 1.1 Gbps mid-test and collapse — so a dynamic loss mechanism, not a static cap.

What made it maddening: every measurement on the firewall was clean

• iperf3 LAN client <-> DEC840: 9.11 Gbit/s aggregate, so forwarding capacity was fine
• netstat -i during sustained upload: zero errors, zero drops, both interfaces
• CPU essentially idle during the cap
• No shaper, no IDS/IPS/DPI running, offload disabled and verified, scrub disabled as a test, MSS unclamped (SYN-ACKs showed mss 1460)
• Tried versions 24.7.7 -> 24.7.12 -> 25.1.12: identical results on all

The evidence that pointed at the wire

tcpdump on the WAN during uploads showed SACK blocks in the returning ACKs across multiple independent flows — remote receivers reporting gaps in the upload stream. So frames were leaving ax0 and dying somewhere past the demarc, while every counter on my side stayed clean. Per-flow throughput worked out to ~120–160 Mbit/s, and 4–5 parallel flows summed to exactly the ceiling I was seeing.

Meanwhile a Mac Studio plugged directly into the ONT — same static IP, same cable, same port — did ~6 Gbps both directions. Circuit and provisioning fine. The loss only existed on the transceiver's link.

Best explanation: the FS module trained a link that looked perfect (10G, clean counters, stable carrier) but silently shed frames in the transmit direction toward the ONT. 10GBASE-T is a negotiated DSP relationship between two PHYs — echo cancellation, per-pair equalization — and this particular Marvell-module-to-ONT pairing was marginal in one direction. The Mac's Aquantia PHY and (see below) a switch-grade PHY both handled the same ONT fine.

The fix

• ISP ONT handoff -> native 10GBASE-T RJ45 port on a UniFi 10G switch, that port as an access port on an isolated WAN VLAN (no DHCP, no L3, exists nowhere else)
• WAN VLAN tagged up the existing SFP+ DAC to ax1
• On OPNsense: new VLAN on ax1 with the WAN tag, WAN reassigned to it. Transceiver removed entirely.

Result, immediately: 6.9 Gbps down / 4.4 Gbps up, 2 ms loaded latency.

Why this might matter for OP's case

Your switch-bypass test used a DAC to the SFP+ port and didn't help — but your 2.5G switch port to the ONT and PPPoE are both still in that path. My result shows the DEC840/axgbe TX path is NOT inherently capped: it pushes 4.4G+ of real NAT'd upload fine once the copper link is clean. If you haven't already, I'd look hard at the ONT-facing copper link (different PHY, different cable) and at PPPoE single-thread limits separately. The "1G RJ45 port uploads faster than the 10G SFP+" result you saw is also consistent with a marginal 10G copper link — the 1G PHY negotiation is far more forgiving.

Practical notes for anyone landing here from search

• A 10GBASE-T SFP+ module can hold a "perfect" link (full speed, zero interface errors, stable carrier) and still lose frames one-directionally. Nothing on the firewall will show it. The tells are SACKs in return ACKs and single-stream throughput far below multi-stream aggregate.
• The definitive test is swapping the PHY talking to the ONT — direct-connect a laptop/desktop with its own 10GBASE-T NIC, or land the handoff on a switch. A different module of the same type is a weaker test (you might re-roll the same bad pairing).
• If your ISP offers a fiber/SFP handoff, take it — it removes this entire failure class.
• Interesting aside: an Intel X722 10GBASE-T port couldn't hold a stable link to this same ONT at all (constant up/down flapping, local_faults incrementing) while the DEC's module and the switch PHY both could. 10GBASE-T interop across PHY vendors is genuinely inconsistent — worth knowing before buying hardware to "fix" a throughput problem.

Happy to share pcaps or more detail if useful. Versions tested: 24.7.7, 24.7.12, 25.1.12 — behavior identical on all, because it was never the software.

[meyergru]

One of the potential problems with those 10GbE transceivers is that they get awfully hot, especially when they are in a passively cooled unit and the neigboring SFP+-slot is occupied as well. Under these conditions, higher error rates will occur. I had the same on an X540 adapter, which are made for rack servers with high airflow. Those are not suited for installation in a desktop case.