OPNsense on Sophos XGS: drivers for every front port (XGS 136 / XGS 3300)

[abdelmonem]

Hi everyone,

Anyone who has tried OPNsense on a Sophos XGS knows how it goes: it installs, it boots, and there isn't a single network port to assign. That's where I started too. The reason is that these boxes are two computers in one case. The x86 board runs the firewall, and every front port hangs off a Marvell coprocessor sitting behind a PCIe link. Sophos only has drivers for its own OS.

I've been at this for close to two years now, first just figuring out how the two halves talk to each other, then writing FreeBSD drivers for the coprocessor and an OPNsense plugin around them. It's finally at the point where I'm comfortable showing it.

On my two boxes:

• XGS 3300 (Cavium OCTEON TX): all 12 front ports, plus the 4x10G expansion module
• XGS 136 (Marvell ARMADA): all 14 front ports

The part I'm happiest with is the offload. Once pf has passed a connection, the coprocessor forwards it on its own, and IPsec set up by strongSwan runs on the coprocessor's crypto engine (IPv6 and ESN included). pf stays in charge, so a block list, a JA4 fingerprint on a list or an IPsec policy that starts covering the traffic will still cut a connection that's already in hardware.

A few numbers from the bench. Every one of them has a page with how it was measured and a photo of the run:

• 9,837 Mbit/s through an XGS 3300 with 1,032 connections in hardware, CPUs at 7.8 %
• 16 IPsec tunnels between two 3300s, 9,019 Mbit/s together
• the XGS 136 doing a gigabit in hardware with its CPUs at 0.4 %
• 3,186 GiB of TLS through both boxes without a single byte changed

The other models in the two families (XGS 87, 107, 116, 126, 138, 2100, 2300, 3100) will install, but I don't own any of them, so they're marked as untested and check their own ports on first boot. Someone is trying it on an XGS 126 right now. If you have one of these sitting around, I'd really like to hear how it goes. There's a short form on the site for that.

Before you try it: it's alpha (0.1.0-alpha.5.4) and needs OPNsense 26.7, and you'll be wiping the Sophos firmware. The easiest route is the image builder. The image has everything the driver needs, so the ports come up on first boot without internet. Installing on top of a running OPNsense works as well, but the box needs its kernel sources first, because without the driver you have no ports. The docs walk through it.


One more thing I'd rather say up front: by default it sends problem reports and a once-a-day count of the appliance. They're scrubbed of anything that identifies the box or your network, there's a banner about it until you choose, and the same page lets you set reports to manual or turn both off.

Edit: Since 0.1.0-alpha.5.5, nothing is sent until you choose. Problem reports stay on the box until you send them, the daily count is off, and a card on XGS: Overview asks once. This change came out of the discussion with Sergio below and the feedback here. Input like that has done more to improve this project than anything I could have decided on my own, so thank you, and please keep it coming.



Site and docs: https://npuforge.org
All the models with their ports drawn: https://npuforge.org/devices/
Image builder (runs in your own GitHub fork): https://build.npuforge.org
Source, BSD-2-Clause: https://github.com/AbdelmonemAwad/os-xgs-npu

This is a personal, independent project. It has nothing to do with Sophos or Deciso and doesn't ship any Sophos software. Questions are welcome, I'll be around.

[BoNetTech]

Hi abdelmonem,

Thank you for sharing this, and for being so clear about the alpha status and what the installation involves. A few questions that matter for anyone who would use this in front of real networks:

1) Intrusion detection. If the coprocessor forwards a connection on its own after pf has passed it, I assume Suricata no longer sees the packets of that connection. Is that right? If so, is there a way to keep chosen traffic out of the offload, so it still goes through the IPS, and what happens to a connection that is already offloaded when the rules change?

2) Telemetry. You mention problem reports and a daily count that are scrubbed of anything identifying. Could you describe exactly which fields are sent, where they go, and whether the receiving side is documented? For many users the default matters, and an opt-in would be easier to accept than an opt-out.

3) Trust in the image. Since the image builder runs in the user's own fork and the driver sits in the kernel, how can a user check that the image matches the published source? Signed releases or reproducible builds would help a lot.

I do not own any of these units, so I cannot test, but I wish you good luck with the project.

Sergio

[abdelmonem]

Hi Sergio,

Thank you, these are exactly the questions I'd ask too.

1) Intrusion detection. You're right. The flow maker hands a connection to the coprocessor after its first data, so Suricata sees how each connection starts (the TLS ClientHello, enough for a JA4 fingerprint) and nothing after that. Inline IPS mode hasn't been tested with the offload yet, so for now I'd treat the two as either/or.

To keep traffic out of the offload:
- XGS: Settings has "Never offloaded": addresses, networks and ports, up to 64 entries, and either end of a connection can match. Those connections always stay with pf, and Suricata sees all of them.
- Any single connection can be sent back with "Return to host" on the Offloaded Flows page.
- Or turn acceleration off, and every packet goes through the host like on any OPNsense box.
Connections the Traffic Shaper or the Captive Portal names stay on the host by themselves.

When the rules change: every ruleset reload empties the coprocessor's whole flow table before the new ruleset loads. Every connection comes back through pf, and the flow maker only offloads again what pf still passes. pf keeps existing states across a reload, as usual, so a connection that has to stop needs its state killed. That's what the block-list and JA4 cuts do, and both are measured: https://npuforge.org/docs/measurements/lists-cut-2026-10-19/

2) Telemetry. The full list is here: https://npuforge.org/docs/problem-reports/ In short:
- A problem report carries the problem's own log lines (or a panic's backtrace) and a description of the machine: model, board revision, BIOS, CPU and memory, the coprocessor and NICs by PCI id, the OPNsense and kernel versions, and the installed commit of each plugin. IP and MAC addresses, e-mail addresses, serials, the UUID, the host name and domain, keys and long hex strings are removed before the report is even stored. The appliance is named by 12 hex digits of a hash of its serial and UUID. "View" on the page shows the exact text that would be sent.
- It goes to a small relay, reports.npuforge.org, a Cloudflare Worker whose code is in the repo (https://github.com/AbdelmonemAwad/os-xgs-npu/blob/main/relay/worker.js), and it files the report as a public GitHub issue. The relay keeps the sending address for two days for its rate limit, and nothing else.
- The daily count sends three things: a random ID the box made for itself (nothing read from the hardware), the model as SMBIOS names it, and the installed release. No address is kept, and count.py --show prints it without sending anything.

On the default: it's on because there are eight models I've never had in my hands, and a report is the only way I see what breaks on them. But you're not the first to say opt-in would be easier to accept, so I'm weighing it seriously.

3) Trust in the image. It isn't signed or bit-for-bit reproducible yet, but here's what you can check today:
- The OPNsense image underneath is the official one, checked against the SHA-256 its release publishes. The builder refuses to build from anything it can't verify.
- The driver isn't in the image as a binary. Its source is, at a recorded commit, and the appliance compiles it at first boot against its own kernel sources, so what ends up in your kernel is source you can diff against the tag. Every other component is fetched at a commit listed in the build's summary.
- The build runs in your own fork, from a workflow you can read, and its log stays in your account.
- Releases are published immutable, with GitHub's release attestation: gh release verify v0.1.0-alpha.5.4 -R AbdelmonemAwad/os-xgs-npu. Updates from the GUI fetch that tag's commit and confirm it with GitHub directly.

A signed build attestation for the image itself is a fair ask, and I'll look at adding it.

Thanks again for reading it this carefully.

Abdelmonem

[BoNetTech]

Hi Abdelmonem,

Thank you for the detailed and honest answers. Knowing that the IPS and the offload are, for now, either one or the other is exactly what people need to know before they choose, so it may be worth stating it clearly on the project page. The same goes for telemetry being on by default: an opt-in would make it easier to adopt this in places where reports must not leave the network.

Good luck with the project.

Sergio

[abdelmonem]

Hi Sergio,

Fair on both counts, and thank you for coming back to it.

IPS and the offload. I'll say it plainly on the project page, in the FAQ and on XGS: Settings itself: for now it's one or the other. Anything Suricata should see in full goes on the "Never offloaded" list, or you run with acceleration off.

Whether the two can work together is under research now, and for a reason I didn't have when I designed the offload. A sweep of the vendor's published GPL sources yesterday turned up things nobody here knew were there: besides "forward", a connection's verdict can be "drop" or "send to inspection", where an inspector sees how a connection starts and the coprocessor carries the rest. That's very close to what an IPS needs, and it changes the question from "one or the other" to "how to hand over". The XGS 3300's coprocessor even runs a deep packet inspection engine of its own. None of it is tried yet. What was found, and where each piece stands:
https://npuforge.org/docs/findings/discoveries-2026-10-22/

So it's research for now, not a promise. The first step is simply to run OPNsense's inline IPS on these ports with the offload off, since that hasn't been done at all. I'll only call the two compatible once it's measured, like everything else on the site.

Telemetry. You're not the only one who has said it, and I agree. The next release turns it around: problem reports stay on the box until you send them, the daily count is off, and a one-time card on the Overview asks, with the model's tested status beside it. No report and no count leaves the box unless you say so.

And honestly, I'm really glad you raised all this. I needed it. Building something alone for this long, every decision gets made in your own head, and a conversation like this one changes how you see things, even the ones made for good reasons. The telemetry default is one of them: there was a real reason behind it, and it still wasn't the right default.

Thanks again for taking the time,
Abdelmonem

[abdelmonem]

Hi Sergio,

A follow-up, since you asked the right question: the two work together now, since 0.1.0-alpha.5.6.

XGS: Settings has "Inspect, then hardware", off by default. With it on, each new connection goes to OPNsense's own Intrusion Detection in IPS mode, through an interface the driver makes for it, oxips0, until Suricata has let its first megabyte through. That's Suricata's own default stream depth. After that the coprocessor carries the rest, and a stream that a drop rule names never gets through at all.

Measured through the Intrusion Detection service on both boxes: a new TCP stream alerted on, then carried in hardware at 3.98 Gbit/s on the XGS 3300 and 941 Mbit/s on the XGS 136, its link full; a UDP stream with a drop rule, nothing let through.
https://npuforge.org/docs/measurements/inspect-2026-10-23/

So the honest limit now is the depth: past the first megabyte of a connection, Suricata sees nothing more of it.

Thanks again for pushing on this. It's a better project for it.

Abdelmonem