Messages

ahh, no?

https://www.fs.com/products/185594.html

And bevor you ask, friend of mine his XGS-PON with that and OPNsense ;)

@tedhugehes I you would at least only slop your own thread, instead of someone elses. *old man yelling at cloud* Back in my days, we called that thread hijacking.

"Thread hijacking" - of which there were zero replies to a thread created 5 months ago on what looked like a similar topic. Keep on beclowning yourself, though. And please show me another transceiver that I can't plug into my ISP's ONT. Please? A third might help!

So much noise and wasted energy.
And for what? To realize that it is stupid not to directly input your ISP's fiber into your OPNsense? And that a unnecessary transceiver makes no sense?

Sorry if I am wrong, I just skipped it, not going to read that AI slop.

Super helpful response. And - shocker! - you're also wrong. ISP doesn't offer a fiber handoff.

I had just completed hours of troubleshooting - yeah, I didn't feel like writing a novel, so I used a tool to help me summarize what was attempted. The only noise and wasted energy is your participation in this thread.

[TL;DR: It was the 10GBASE-T SFP+ transceiver's link to the ISP's ONT. Not the DEC840, not axgbe, not OPNsense. Bypassing the transceiver by landing the ONT on a switch's native 10GBASE-T port and trunking WAN to the DEC took me from ~600 Mbps upload to 4.4 Gbps.]

SOLVED on my end — and the answer might apply to your setup too, so posting the full findings.

TL;DR: It was the 10GBASE-T SFP+ transceiver's link to the ISP's ONT. Not the DEC840, not axgbe, not OPNsense. Bypassing the transceiver by landing the ONT on a switch's native 10GBASE-T port and trunking WAN to the DEC took me from ~600 Mbps upload to 4.4 Gbps.

My setup (for comparison with OP)

• DEC840, 8 Gbps symmetrical fiber, static /26, no PPPoE
• WAN was: FS SFP-10G-T (30m, on Deciso's tested list) in ax0, Cat6a to the ISP ONT's RJ45 handoff
• LAN: SFP+ DAC from ax1 to a UniFi switch, untagged LAN + 8 tagged VLANs

Symptom — same as OP

Download 3.5–4.7 Gbps, upload pinned at ~600–870 Mbps regardless of test method. Occasionally it would briefly spike past 1.1 Gbps mid-test and collapse — so a dynamic loss mechanism, not a static cap.

What made it maddening: every measurement on the firewall was clean

• iperf3 LAN client <-> DEC840: 9.11 Gbit/s aggregate, so forwarding capacity was fine
• netstat -i during sustained upload: zero errors, zero drops, both interfaces
• CPU essentially idle during the cap
• No shaper, no IDS/IPS/DPI running, offload disabled and verified, scrub disabled as a test, MSS unclamped (SYN-ACKs showed mss 1460)
• Tried versions 24.7.7 -> 24.7.12 -> 25.1.12: identical results on all

The evidence that pointed at the wire

tcpdump on the WAN during uploads showed SACK blocks in the returning ACKs across multiple independent flows — remote receivers reporting gaps in the upload stream. So frames were leaving ax0 and dying somewhere past the demarc, while every counter on my side stayed clean. Per-flow throughput worked out to ~120–160 Mbit/s, and 4–5 parallel flows summed to exactly the ceiling I was seeing.

Meanwhile a Mac Studio plugged directly into the ONT — same static IP, same cable, same port — did ~6 Gbps both directions. Circuit and provisioning fine. The loss only existed on the transceiver's link.

Best explanation: the FS module trained a link that looked perfect (10G, clean counters, stable carrier) but silently shed frames in the transmit direction toward the ONT. 10GBASE-T is a negotiated DSP relationship between two PHYs — echo cancellation, per-pair equalization — and this particular Marvell-module-to-ONT pairing was marginal in one direction. The Mac's Aquantia PHY and (see below) a switch-grade PHY both handled the same ONT fine.

The fix

• ISP ONT handoff -> native 10GBASE-T RJ45 port on a UniFi 10G switch, that port as an access port on an isolated WAN VLAN (no DHCP, no L3, exists nowhere else)
• WAN VLAN tagged up the existing SFP+ DAC to ax1
• On OPNsense: new VLAN on ax1 with the WAN tag, WAN reassigned to it. Transceiver removed entirely.

Result, immediately: 6.9 Gbps down / 4.4 Gbps up, 2 ms loaded latency.

Why this might matter for OP's case

Your switch-bypass test used a DAC to the SFP+ port and didn't help — but your 2.5G switch port to the ONT and PPPoE are both still in that path. My result shows the DEC840/axgbe TX path is NOT inherently capped: it pushes 4.4G+ of real NAT'd upload fine once the copper link is clean. If you haven't already, I'd look hard at the ONT-facing copper link (different PHY, different cable) and at PPPoE single-thread limits separately. The "1G RJ45 port uploads faster than the 10G SFP+" result you saw is also consistent with a marginal 10G copper link — the 1G PHY negotiation is far more forgiving.

Practical notes for anyone landing here from search

• A 10GBASE-T SFP+ module can hold a "perfect" link (full speed, zero interface errors, stable carrier) and still lose frames one-directionally. Nothing on the firewall will show it. The tells are SACKs in return ACKs and single-stream throughput far below multi-stream aggregate.
• The definitive test is swapping the PHY talking to the ONT — direct-connect a laptop/desktop with its own 10GBASE-T NIC, or land the handoff on a switch. A different module of the same type is a weaker test (you might re-roll the same bad pairing).
• If your ISP offers a fiber/SFP handoff, take it — it removes this entire failure class.
• Interesting aside: an Intel X722 10GBASE-T port couldn't hold a stable link to this same ONT at all (constant up/down flapping, local_faults incrementing) while the DEC's module and the switch PHY both could. 10GBASE-T interop across PHY vendors is genuinely inconsistent — worth knowing before buying hardware to "fix" a throughput problem.

Happy to share pcaps or more detail if useful. Versions tested: 24.7.7, 24.7.12, 25.1.12 — behavior identical on all, because it was never the software.

[Setup]

Reviving this because I have the same symptom on the same hardware, and I've been able to rule out a lot of the variables you couldn't.

Setup

• DEC840, WAN on ax0, LAN on ax1 (8 tagged VLANs + untagged LAN)
• 8 Gbps symmetrical fiber, static /26, no PPPoE (key difference from OP's setup)
• WAN transceiver: FS SFP-10G-T (30m), on Deciso's tested-modules list for the DEC800 series
• LAN: passive SFP+ DAC to a UniFi 10G switch

Symptom

Download 3.5-4.7 Gbps. Upload pinned at ~600-870 Mbps, consistently, regardless of test method or target. Notably it has twice briefly exceeded 1.1 Gbps mid-test before collapsing back — so it behaves like a dynamic loss mechanism, not a static cap.

The control test that matters

A Mac Studio connected directly to the ONT — same static IP, same cable, same ONT port — gets ~6 Gbps in both directions. Circuit, provisioning, and the ISP upload policer are all demonstrably fine. The cap only appears when traffic passes through the DEC840.

What I've ruled out (all measured, not assumed)

• Forwarding capacity — iperf3 Mac <-> OPNsense, 8 streams: 9.11 Gbit/s aggregate
• LAN segment / DAC / switch — single stream: 2.65 Gbps up / 4.09 Gbps down, ~0 retransmits
• Interface errors — netstat -i -I ax0 -w 1 during sustained upload: zero errs, zero idrops, both directions
• CPU — top -aSH during upload: all cores essentially idle
• Shaper — pfctl -sq empty, no pipes/queues configured
• IDS/IPS/DPI — Suricata, Zenarmor, ntopng all stopped, nothing attached via netmap
• Hardware offload — CRC/TSO/LRO disabled, verified absent from ifconfig flags: no change
• pf scrub — disabled: no change
• MSS — SYN-ACKs show mss 1460, not clamped
• VPN policy routing — LAN clients egress the correct static IP
• Flow control — dev.ax.0.rx_pause/tx_pause=1 + reboot: no change, media line reports rxpause,txpause regardless

Packet-level finding

tcpdump on ax0 during upload shows SACK blocks in the returning ACKs across multiple independent flows:

Code Select Expand

sack 1 {73849:78193}
sack 1 {53577:75297}
sack 1 {73849:101361}
Receivers are reporting gaps in the upload stream — frames leave ax0 and don't all arrive, while every counter on my side stays clean. Advertised receive windows on those flows also shrink under load (2045 -> 2003 -> 1972). Per-flow throughput works out to roughly 120-160 Mbit/s; 4-5 parallel flows sums to the observed ceiling.

Version ladder (tested identically at each step)

Version	FreeBSD base	Upload
24.7.7	14.1	~600 Mbps
24.7.12	14.1	~600 Mbps
25.1.12	14.2	~600 Mbps

Continuing up the ladder; will update with results.

Tunables — factory Deciso defaults, unmodified:

Code Select Expand

dev.ax.0/1.iflib.override_nrxds = 2048 x8
dev.ax.0/1.iflib.override_ntxds = 2048 x8
dev.ax.0/1.link_workaround = 1
dev.ax.sph_enable = 0
Where this leaves it

The OP's switch-bypass test (ONT -> switch -> DAC -> SFP+, no copper transceiver in the path) produced no change, which points away from the transceiver and toward the ax TX path itself. I'm running the equivalent test on my side and will report back.

Two questions for anyone who's been here:

1. Has anyone with a DEC840/DEC850 achieved sustained multi-gig upload on ax0 with a real WAN — and if so, on what version?
2. Is there any known asymmetry in how axgbe handles TX for forwarded/NAT'd traffic vs locally-originated? Everything I can measure says the interface is healthy and the packets simply don't all survive the trip.

Happy to provide full pcaps or run any test that would help.

Pretty straightforward - been using this for months, things *seem* to be working, but troubleshooting some odd stuff all of a sudden with my August Bridge not working. I goto Live View, and I see a BUNCH of default deny rules being triggered. For example, here's a live log entry:

Interface: VLAN33
Source: 192.168.33.x:y
Destination: <public IP>:443 (so, a website)
Proto: tcp
Label: Default Deny rule

Here's what I have for VLAN33 under my rules:
1. Protocol: IPv4, all other items wildcard
2. Protocol: IPv6, all other items wildcard

That's it. No other rules. Why in the hell is a default deny rule hitting *at all* on IPv4 traffic for this interface? It's happening on others, as well. Inexplicable as things are pretty explicitly allowed.

I'm running the latest OPNsense (21.7.3_1 LibreSSL) with the wireguard-go plugin installed.

I'm using WG as a site to site link with a VM running wireguard at a remote datacenter. I was running a VM locally with WG to great success, but with the recent introduction of my new OPNsense hardware firewall, wanted to migrate this endpoint from the local VM to the firewall itself. Got it setup thanks to the docs and various blogs, and thought I was in good shape.

As some point, I was tinkering with things and created an interface assignment for the wg0 interface. From that point on I got into a state where, if I do anything with the interface (enable/disable/assign/unassign), it breaks my WG tunnel, where if I restart the service it will not come back up, until I remove the allowed IPs from the remote endpoint (aka, my remote networks I want to reach over the tunnel). Once I remove all but the /32 private WG IP from my remote endpoint, the tunnel comes back up, and then I can re-add my remote networks to the endpoint. It's all very tempermental.

One thing I'm unclear on is if I need any firewall rules defined for the tunnel. It would appear that almost everything passes, but on one of my remote networks, I get a default deny rule trigger. Yet, on my wireguard ruleset, I have an allow everything inbound. But the live log view shows a default deny rule triggering on the wg0 interface. Since I have since un-done the interface assignment, I cannot filter based on the wg0 interface in the live view.

So - I guess I'm stuck on the relationship of interfaces in WG - how much is done automatically [don't touch it!] - and how much I have control over. I do have a "WireGuard" ruleset, but I don't know if they're being honored. I do see evaluations/states/packets/bytes when inspecting the one allow all rule I've created, so I'm confused on why *anything* would hit the default deny at that point.

Any help or advice would be greatly appreciated. Thanks much!