Mono Gateway (an NXP-based router thingie)

[pfry]

I didn't see this flying around, and someone had to start it, so I figured I might as well.

An NXP-based router that... should support OPNsense. I wonder about the hardware offloading, as I figure it would play hell with monitoring/logging.

It looks like the first development kit sale has closed, awaiting a second or production run. Hopefully they make it, but it's a tough market.

Maurice, do you have one?

[Maurice]

Yes, I have one and it does indeed run OPNsense. Hardware offloading is supported and really sets it apart from anything I've seen before. And yes, it can offload connections which are firewalled by pf. Pretty impressive.

I recommend watching Tomaž's latest video on YouTube.

Cheers
Maurice

Full disclosure: I've been contracted by Mono to maintain their OPNsense update server.

[OPNenthu]

Tomaž's videos would show up in my algorithm from time to time.  Awesome development w.r.t to OPNsense.  Very cool, @Maurice! 

I did cringe a bit when he mentioned Claude, but those are very appreciable gains that he discussed.  I guess the question I have is whether the AI produces fewer bugs and vulnerabilities than a team of humans would.

[Patrick M. Hausen]

I just placed a pre order :-)

[meyergru]

I followed Tomasz'sYT video series for a while now, and noted early that I would like to see OpnSense on it, instead of OpenWRT - this seemed infeasible at the time...

However, apart from the entertainment and enthusiast aspect of this effort, which in itself deserves praise, I see three problems:

1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.

2. The price is now 600€ (the finished product will even be more, AFAIU), which is more than I would pay for an x64-based appliance with 3x 2.5 GbE and 2x SFP+. AFAIK, the routing speed is 10 Gbps as well for those.

3. As it appears, there are some legal obstacles with conflicts of the GPL v2 and commercially licensed code, which Tomasz has acknowledged in a pinned comment under his video and now seeks legal counsel as to if and how it will be possible to do it they way he intended.

[nero355]

Yes, I have one and it does indeed run OPNsense. Hardware offloading is supported and really sets it apart from anything I've seen before. And yes, it can offload connections which are firewalled by pf. Pretty impressive.

But where does one put this SoC based on it's performance ?

Similar to ARM64 A3x/A5x/A7x or maybe even X1 and the likes ?
Similar to Intel Atom N97/100/150/305 ?

Of is this like the 500 MHz MIPS SoC was in my old Router that pretty much completely depended on it's Offloading Features ?!

And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!

This is also something I have no experience with :

64 MB NOR flash for Bootloader

Is that like :
- /boot ?
- UEFI Boot Partition ?

And how much do we currently need for OPNsense ?
How "futureproof" is it sizewise ?

I did cringe a bit when he mentioned Claude, but those are very appreciable gains that he discussed.

I guess the question I have is whether the AI produces fewer bugs and vulnerabilities than a team of humans would.

+1 :)

I like stuff actual humans actually thought about!

And especially when it costs € 600+ !!!

I would like to see OpnSense on it, instead of OpenWRT

I don't mind OpenWRT as long as there is active development and good community support for the port for a specific device.

But that's not always the case sadly...

1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.

2. The price is now 600€ (the finished product will even be more, AFAIU), which is more than I would pay for an x64-based appliance with 3x 2.5 GbE and 2x SFP+. AFAIK, the routing speed is 10 Gbps as well for those.

+1 :)

If you are asking € 600 for a device then just give me 4 x 10 Gbps RJ45 NICs that can work at 1/2,5/5/10 Gbps and be done with it!

_{(This is based on 4 x € 100 for the NICs and another € 200 for the AARCH64 SoC Mainboard which is IMHO more or less the average price for those parts sold seperately.)}

[Patrick M. Hausen]

And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!

Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces?

If we have an SOC with a switching fabric and vendor support for OPNsense - great!

There is nothing wrong with offloading, only that the features supported by generic server hardware don't buy you much in the first place in terms of forwarding speed and sometimes plain don't work in FreeBSD as soon as pf, NAT and friends come into play.

Kind regards,
Patrick

[Maurice]

I really recommend Tomaž's videos for the hardware offloading deep dive. He's the expert on this, I'm not.

But where does one put this SoC based on it's performance ?

It has 4 Cortex A72 cores. But most packets never touch these cores.

And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!

What you're probably thinking of is offloading basic packet processing like checksums to the NICs.

Gateway doesn't have NICs in the traditional sense. The PHYs connect directly to the SoC, which handles routing and other frame and packet processing (VLANs, NAT, PPP, ...) in dedicated hardware. This means routing at wire speed with essentially no CPU load, like on a switch. The CPU cycles are available for other stuff that can't be offloaded.

The 64 MB NOR flash is for U-Boot and a small recovery Linux. The main OS (originally OpenWrt, now OPNsense) is installed on the 32 GB eMMC.
The OPNsense image we've just made available uses GPT and ZFS, but Gateway currently doesn't use FreeBSD's UEFI kernel loader. Instead, U-Boot loads the kernel directly.

Keep in mind that this is ongoing development and things may and probably will change in the final production version.

Cheers
Maurice

[OPNenthu]

1. The box only has 2x SFP+ ports and 3x 1 GbE ones. I know why it was infeasible to do 2.5 GbE, however, I think that is something left to be desired.

It's kind of an odd spot for those with ISP plans above 1 Gbps and only have an RJ45 interface to the modem.

The largest terrestrial ISPs where I live have "multi-gigabit" plans now, which really just means 2+ Gbps.  A common one is Comcast/Xfinity with their XB7 and similar DOCSIS customer gateways that can be put into bridge mode.  Optionally customers can bring their down DOCSIS modem, like this one:

https://www.amazon.com/ARRIS-Surfboard-S33-Multi-Gigabit-Ethernet/dp/B08FMSC5WZ

Those only have 2.5GbE ports.  Maybe they got up to 10GbE now (I'm not sure).  The main concern is that the latency with DOCSIS is already quite high, then you add the RJ45 SFP+ transceiver into the mix.  How much latency does it add and is it negligible?

[Maurice]

The SFP+ cage is there, but how much latency does the RJ45 transceiver add?

Essentially none (less than a microsecond?), that's just a PHY.

[OPNenthu]

Cool, then I can stop avoiding them :)

[nero355]

And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!

Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces?

There is nothing wrong with offloading, only that the features supported by generic server hardware don't buy you much in the first place in terms of forwarding speed and sometimes plain don't work in FreeBSD as soon as pf, NAT and friends come into play.

So to avoid bugs just like I remember from a very long time ago :)

But who says this will always work correctly ?
Or not need "in-chip" fixing that can't be done via firmware updates ?

Tricky...

I really recommend Tomaž's videos for the hardware offloading deep dive. He's the expert on this, I'm not.

I like to avoid YouTube whenever I can when it comes to this kind of stuff : Reading about it is more my style :)

It has 4 Cortex A72 cores. But most packets never touch these cores.

Still old hardware then basically...

What you're probably thinking of is offloading basic packet processing like checksums to the NICs.

Gateway doesn't have NICs in the traditional sense. The PHYs connect directly to the SoC, which handles routing and other frame and packet processing (VLANs, NAT, PPP, ...) in dedicated hardware. This means routing at wire speed with essentially no CPU load, like on a switch. The CPU cycles are available for other stuff that can't be offloaded.

So it's like my old Router with MIPS SoC like I thought... Not a fan to be honest...

Can you at least mix both things without the need to disable any of the Offloading Features first ?

Or simply put don't have to deal with this kind of nonsense : https://old.reddit.com/r/Ubiquiti/comments/k5j2y4/why_would_i_enable_smart_queues_when_it_disables/

The 64 MB NOR flash is for U-Boot and a small recovery Linux. The main OS (originally OpenWrt, now OPNsense) is installed on the 32 GB eMMC.
The OPNsense image we've just made available uses GPT and ZFS, but Gateway currently doesn't use FreeBSD's UEFI kernel loader. Instead, U-Boot loads the kernel directly.

Keep in mind that this is ongoing development and things may and probably will change in the final production version.

So with that kind of setup the issue with the FreeBSD Bootloader needing an upgrade from time to time can be ignored, right ?



And now something that I wanted to ask you for some time now since we are talking about this kind of hardware anyway :

Are there any AARCH64 Mainboards out there that can run FreeBSD or simply OPNsense without any big issues ?

Since you are our AARCH64 Releases guy and all :)

[pfry]

And why suddenly use Offloading while it's always recommended to disable all of it for both OPNsense and pfSense ?!

Because it does not work reliably on generic AMD64 hardware and Intel/Broadcom/etc. network interfaces? [...]

Just to expand this a bit, there's a lot of odd, somewhat useless (in practice) capability built into PC network interfaces. If you're bored, look at DPDK's NIC overview.

Example: the Intel E810 has a tiny (8000 entry, IIRC) TCAM, and it (the chip, not the TCAM) is used in several Deciso boxes. It's really too small for a useful flow cache, but it could make a decent partial policy offload (basically everything but address - which it can handle, but something like a geoip db would overflow it a bit).

Another example: the Chelsio T5/6/7 NICs have a truly tiny (~500 entry IIRC) TCAM, and some have onboard RAM that can support exact-match filters, potentially good for flow matching. They have some header rewrite capability, too (e.g. for NAT). The T6/7 also have a crypto engine (with a FreeBSD driver).

At any rate, the problems tend to arise when you globally enable features in a mixed-NIC system. The failure mode for unsupported features should be graceful...

[Maurice]

I like to avoid YouTube whenever I can when it comes to this kind of stuff : Reading about it is more my style :)

https://docs.mono.si/gateway-development-kit/hardware-description
https://docs.mono.si/tutorials/development-set-up (note the references at the end)

So it's like my old Router with MIPS SoC like I thought.

Pretty sure that didn't run OPNsense. :)

Can you at least mix both things without the need to disable any of the Offloading Features first ?

Not sure what you mean by "mix both things". And it's all about the offloading, disabling it wouldn't make sense.

So with that kind of setup the issue with the FreeBSD Bootloader needing an upgrade from time to time can be ignored, right ?

Correct. That's not the reason why it was implemented this way, but I guess it could be considered a positive side effect.

Are there any AARCH64 Mainboards out there that can run FreeBSD or simply OPNsense without any big issues ?

Yes, this one. :)
(But seriously, I don't have any other recommendations. Other aarch64 SBCs which can run OPNsense might be nice for hobby projects, but not for serious networking.)

Since you are our AARCH64 Releases guy and all :)

I've only used it on VMs until recently. Gateway changed that.