I didn't see this flying around, and someone had to start it, so I figured I might as well.
An NXP-based router (https://mono.si/) that... should support OPNsense (https://opnsense.mono.si/). I wonder about the hardware offloading, as I figure it would play hell with monitoring/logging.
It looks like the first development kit sale has closed, awaiting a second or production run. Hopefully they make it, but it's a tough market.
Maurice, do you have one?
Yes, I have one and it does indeed run OPNsense. Hardware offloading is supported and really sets it apart from anything I've seen before. And yes, it can offload connections which are firewalled by pf. Pretty impressive.
I recommend watching Tomaž's latest video on YouTube (https://www.youtube.com/watch?v=bSZYNP41xqA).
Cheers
Maurice
Full disclosure: I've been contracted by Mono to maintain their OPNsense update server.
Tomaž's videos would show up in my algorithm from time to time. Awesome development w.r.t to OPNsense. Very cool, @Maurice!
I did cringe a bit when he mentioned Claude, but those are very appreciable gains that he discussed. I guess the question I have is whether the AI produces fewer bugs and vulnerabilities than a team of humans would.
I just placed a pre order :-)
I followed Tomasz'sYT video series for a while now, and noted early that I would like to see OpnSense on it, instead of OpenWRT - this seemed infeasible at the time...
However, apart from the entertainment and enthusiast aspect of this effort, which in itself deserves praise, I see three problems:
1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.
2. The price is now 600€ (the finished product will even be more, AFAIU), which is more than I would pay for an x64-based appliance with 3x 2.5 GbE and 2x SFP+. AFAIK, the routing speed is 10 Gbps as well for those.
3. As it appears, there are some legal obstacles with conflicts of the GPL v2 and commercially licensed code, which Tomasz has acknowledged in a pinned comment under his video and now seeks legal counsel as to if and how it will be possible to do it they way he intended.
Quote from: Maurice on March 29, 2026, 04:26:25 AMYes, I have one and it does indeed run OPNsense. Hardware offloading is supported and really sets it apart from anything I've seen before. And yes, it can offload connections which are firewalled by pf. Pretty impressive.
But where does one put this SoC based on it's performance ?
Similar to ARM64 A3x/A5x/A7x or maybe even X1 and the likes ?
Similar to Intel Atom N97/100/150/305 ?
Of is this like the 500 MHz MIPS SoC was in my old Router that pretty much completely depended on it's Offloading Features ?!
And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!
This is also something I have no experience with :
Quote64 MB NOR flash for Bootloader
Is that like :
- /boot ?
- UEFI Boot Partition ?
And how much do we currently need for OPNsense ?
How "futureproof" is it sizewise ?
Quote from: OPNenthu on March 29, 2026, 06:48:39 AMI did cringe a bit when he mentioned Claude, but those are very appreciable gains that he discussed.
I guess the question I have is whether the AI produces fewer bugs and vulnerabilities than a team of humans would.
+1 :)
I like stuff actual humans actually thought about!
And especially when it costs € 600+ !!!
Quote from: meyergru on March 29, 2026, 04:56:23 PMI would like to see OpnSense on it, instead of OpenWRT
I don't mind OpenWRT as long as there is active development and good community support for the port for a specific device.
But that's not always the case sadly...
Quote1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.
2. The price is now 600€ (the finished product will even be more, AFAIU), which is more than I would pay for an x64-based appliance with 3x 2.5 GbE and 2x SFP+. AFAIK, the routing speed is 10 Gbps as well for those.
+1 :)
If you are asking € 600 for a device then just give me 4 x 10 Gbps RJ45 NICs that can work at 1/2,5/5/10 Gbps and be done with it!
(This is based on 4 x € 100 for the NICs and another € 200 for the AARCH64 SoC Mainboard which is IMHO more or less the average price for those parts sold seperately.)
Quote from: nero355 on March 29, 2026, 08:03:51 PMAnd why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!
Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces?
If we have an SOC with a switching fabric and vendor support for OPNsense - great!
There is nothing wrong with offloading, only that the features supported by generic server hardware don't buy you much in the first place in terms of forwarding speed and sometimes plain don't work in FreeBSD as soon as pf, NAT and friends come into play.
Kind regards,
Patrick
I really recommend Tomaž's videos for the hardware offloading deep dive. He's the expert on this, I'm not.
Quote from: nero355 on March 29, 2026, 08:03:51 PMBut where does one put this SoC based on it's performance ?
It has 4 Cortex A72 cores. But most packets never touch these cores.
Quote from: nero355 on March 29, 2026, 08:03:51 PMAnd why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!
What you're probably thinking of is offloading basic packet processing like checksums to the NICs.
Gateway doesn't have NICs in the traditional sense. The PHYs connect directly to the SoC, which handles routing and other frame and packet processing (VLANs, NAT, PPP, ...) in dedicated hardware. This means routing at wire speed with essentially no CPU load, like on a switch. The CPU cycles are available for other stuff that can't be offloaded.
The 64 MB NOR flash is for U-Boot and a small recovery Linux. The main OS (originally OpenWrt, now OPNsense) is installed on the 32 GB eMMC.
The OPNsense image we've just made available uses GPT and ZFS, but Gateway currently doesn't use FreeBSD's UEFI kernel loader. Instead, U-Boot loads the kernel directly.
Keep in mind that this is ongoing development and things may and probably will change in the final production version.
Cheers
Maurice
Quote from: meyergru on March 29, 2026, 04:56:23 PM1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.
It's kind of an odd spot for those with ISP plans above 1 Gbps and only have an RJ45 interface to the modem.
The largest terrestrial ISPs where I live have "multi-gigabit" plans now, which really just means 2+ Gbps. A common one is Comcast/Xfinity with their XB7 and similar DOCSIS customer gateways that can be put into bridge mode. Optionally customers can bring their down DOCSIS modem, like this one:
https://www.amazon.com/ARRIS-Surfboard-S33-Multi-Gigabit-Ethernet/dp/B08FMSC5WZ
Those only have 2.5GbE ports. Maybe they got up to 10GbE now (I'm not sure). The main concern is that the latency with DOCSIS is already quite high, then you add the RJ45 SFP+ transceiver into the mix. How much latency does it add and is it negligible?
Quote from: OPNenthu on March 29, 2026, 11:56:43 PMThe SFP+ cage is there, but how much latency does the RJ45 transceiver add?
Essentially none (less than a microsecond?), that's just a PHY.
Cool, then I can stop avoiding them :)
Quote from: Patrick M. Hausen on March 29, 2026, 10:51:26 PMQuote from: nero355 on March 29, 2026, 08:03:51 PMAnd why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!
Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces?
There is nothing wrong with offloading, only that the features supported by generic server hardware don't buy you much in the first place in terms of forwarding speed and sometimes plain don't work in FreeBSD as soon as pf, NAT and friends come into play.
So to avoid bugs just like I remember from a very long time ago :)
But who says this will always work correctly ?
Or not need "in-chip" fixing that can't be done via firmware updates ?
Tricky...
Quote from: Maurice on March 29, 2026, 11:04:03 PMI really recommend Tomaž's videos for the hardware offloading deep dive. He's the expert on this, I'm not.
I like to avoid YouTube whenever I can when it comes to this kind of stuff : Reading about it is more my style :)
QuoteIt has 4 Cortex A72 cores. But most packets never touch these cores.
Still old hardware then basically...
QuoteWhat you're probably thinking of is offloading basic packet processing like checksums to the NICs.
Gateway doesn't have NICs in the traditional sense. The PHYs connect directly to the SoC, which handles routing and other frame and packet processing (VLANs, NAT, PPP, ...) in dedicated hardware. This means routing at wire speed with essentially no CPU load, like on a switch. The CPU cycles are available for other stuff that can't be offloaded.
So it's like my old Router with MIPS SoC like I thought... Not a fan to be honest...
Can you at least mix both things without the need to disable any of the Offloading Features first ?
Or simply put don't have to deal with this kind of nonsense : https://old.reddit.com/r/Ubiquiti/comments/k5j2y4/why_would_i_enable_smart_queues_when_it_disables/
QuoteThe 64 MB NOR flash is for U-Boot and a small recovery Linux. The main OS (originally OpenWrt, now OPNsense) is installed on the 32 GB eMMC.
The OPNsense image we've just made available uses GPT and ZFS, but Gateway currently doesn't use FreeBSD's UEFI kernel loader. Instead, U-Boot loads the kernel directly.
Keep in mind that this is ongoing development and things may and probably will change in the final production version.
So with that kind of setup the issue with the FreeBSD Bootloader needing an upgrade from time to time can be ignored, right ?
And now something that I wanted to ask you for some time now since we are talking about this kind of hardware anyway :
Are there any AARCH64 Mainboards out there that can run FreeBSD or simply OPNsense without any big issues ?
Since you are our AARCH64 Releases guy and all :)
Quote from: Patrick M. Hausen on March 29, 2026, 10:51:26 PMQuote from: nero355 on March 29, 2026, 08:03:51 PMAnd why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!
Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces? [...]
Just to expand this a bit, there's a lot of odd, somewhat useless (in practice) capability built into PC network interfaces. If you're bored, look at DPDK's NIC overview (https://doc.dpdk.org/guides/nics/overview.html).
Example: the Intel E810 has a tiny (8000 entry, IIRC) TCAM, and it (the chip, not the TCAM) is used in several Deciso boxes. It's really too small for a useful flow cache, but it could make a decent partial policy offload (basically everything but address - which it can handle, but something like a geoip db would overflow it a bit).
Another example: the Chelsio T5/6/7 NICs have a truly tiny (~500 entry IIRC) TCAM, and some have onboard RAM that can support exact-match filters, potentially good for flow matching. They have some header rewrite capability, too (e.g. for NAT). The T6/7 also have a crypto engine (with a FreeBSD driver).
At any rate, the problems tend to arise when you globally enable features in a mixed-NIC system. The failure mode for unsupported features should be graceful...
Quote from: nero355 on March 30, 2026, 01:02:05 AMI like to avoid YouTube whenever I can when it comes to this kind of stuff : Reading about it is more my style :)
https://docs.mono.si/gateway-development-kit/hardware-description
https://docs.mono.si/tutorials/development-set-up (note the references at the end)
Quote from: nero355 on March 30, 2026, 01:02:05 AMSo it's like my old Router with MIPS SoC like I thought.
Pretty sure that didn't run OPNsense. :)
Quote from: nero355 on March 30, 2026, 01:02:05 AMCan you at least mix both things without the need to disable any of the Offloading Features first ?
Not sure what you mean by "mix both things". And it's all about the offloading, disabling it wouldn't make sense.
Quote from: nero355 on March 30, 2026, 01:02:05 AMSo with that kind of setup the issue with the FreeBSD Bootloader needing an upgrade from time to time can be ignored, right ?
Correct. That's not the reason why it was implemented this way, but I guess it could be considered a positive side effect.
Quote from: nero355 on March 30, 2026, 01:02:05 AMAre there any AARCH64 Mainboards out there that can run FreeBSD or simply OPNsense without any big issues ?
Yes, this one. :)
(But seriously, I don't have any other recommendations. Other aarch64 SBCs which can run OPNsense might be nice for hobby projects, but not for serious networking.)
Quote from: nero355 on March 30, 2026, 01:02:05 AMSince you are our AARCH64 Releases guy and all :)
I've only used it on VMs until recently. Gateway changed that.
If the packets never touch the cores doesn't it make just a glorified switch running at wire speed?
Is there still full state tracking and traffic accounting?
In other words, as long as there's a HW bypass how and where is OPNsense still in control?
Quote from: newsense on March 30, 2026, 07:55:51 AMIn other words, as long as there's a HW bypass how and where is OPNsense still in control?
This is mentioned in the video Maurice linked (timestamp 00:04:23).
There seems to be some coordination between several layers they programmed (PCB, CMM, CDX) and pf, but pf is only making decisions and establishing the state.
What I didn't quite understand is NAT offloading, but I'm not into the technical details of how this works even in normal pf. He mentioned that NAT gets done in the hardware flow in order to maintain wire speeds. Isn't NAT one of the responsibilities of pf itself? What other pf functions get taken over by the Mono microcode? How do things like shaping and packet tagging work?
That looks quite interesting, except for the price.
Quote from: newsense on March 30, 2026, 07:55:51 AMIf the packets never touch the cores doesn't it make just a glorified switch running at wire speed?[...]
He didn't say "all". Policy evaluation and flow setup is done by pf, and flows are passed to the engine. I'd have to dig into the coordination between CPU and engine to see if, for instance, expiry is updated, age is honored and teardown is correctly reflected in pf, and if counters are maintained. I doubt the last, but you never can tell. Thinking about it, I would expect logging to be normal, as pf logs are pretty limited anyway. Coordination failures could be tough to diagnose.
I'd expect the device to fall flat on its face if its flow limit is exceeded, either through eviction to the CPU or (more likely) discard. But I haven't exceeded ~10000 active flows, so its limit is likely OK for normal use. Most users will see far fewer, and are minimally vulnerable to external flow setup attacks. (I am, as I have servers with standard ports open to the Internet. On the other hand, I don't use proxies and such, so processing required on the firewall is... reduced.)