Mono Gateway (an NXP-based router thingie)

[OPNenthu]

In other words, as long as there's a HW bypass how and where is OPNsense still in control?

This is mentioned in the video Maurice linked (timestamp 00:04:23).

There seems to be some coordination between several layers they programmed (PCB, CMM, CDX) and pf, but pf is only making decisions and establishing the state. 

What I didn't quite understand is NAT offloading, but I'm not into the technical details of how this works even in normal pf.  He mentioned that NAT gets done in the hardware flow in order to maintain wire speeds.  Isn't NAT one of the responsibilities of pf itself?  What other pf functions get taken over by the Mono microcode?  How do things like shaping and packet tagging work?

[bimbar]

That looks quite interesting, except for the price.

[pfry]

If the packets never touch the cores doesn't it make just a glorified switch running at wire speed?[...]

He didn't say "all". Policy evaluation and flow setup is done by pf, and flows are passed to the engine. I'd have to dig into the coordination between CPU and engine to see if, for instance, expiry is updated, age is honored and teardown is correctly reflected in pf, and if counters are maintained. I doubt the last, but you never can tell. Thinking about it, I would expect logging to be normal, as pf logs are pretty limited anyway. Coordination failures could be tough to diagnose.

I'd expect the device to fall flat on its face if its flow limit is exceeded, either through eviction to the CPU or (more likely) discard. But I haven't exceeded ~10000 active flows, so its limit is likely OK for normal use. Most users will see far fewer, and are minimally vulnerable to external flow setup attacks. (I am, as I have servers with standard ports open to the Internet. On the other hand, I don't use proxies and such, so processing required on the firewall is... reduced.)