Messages

We are experimenting with Chelsios as well - but we have XEON CPUs and not Atoms. This 6Gbps are in our case for single iperf streams - we don't know why this limit is at 6Gbps, it's more or less the same for intel X810 cards. If we enable RSS and switch to multiple iperf streams we are getting far more (>20Gbps). Cheers

That is interesting - are you able to share more details? Hardware specs, setup, config etc. I never got anywhere near those numbers despite trial-and-error my way through various undocumented snags (mostly documented here https://forum.opnsense.org/index.php?topic=25263). RSS at the time wasn't mature, but the load and interrupts seemingly looked well balanced across cores anyway, nevertheless never got close to 10Gb line rate even with multiple streams.

*sense isn't great at routing.

It's a FreeBSD thing rather than specific to *sense.  I forget the reasons why Linux performs better on the same hardware.

While this is probably true, like you also mentioned I also got much better results with TrueNas (on FreeBSD). So there seems to be something going on in additions which is specific to OpnSense. Haven't tried pfsense.

I looked at VyOS but wasn't for me. Only marginally more convenient than rolling your own nftables config file on top of a minimal Debian install, but comes with the downside of vendor lock-in and faff with getting the ISOs etc.

One of those threads were mine and no, afraid not. Moved to another solution for now.

For what it's worth but probably not what you want to hear. Had similar challenges, managed to get my Chelsio 10Gbit card to put through approx 5Gb/s after extensive Googling and experimenting and tuning - some of it is documented in previous threads here. At the same time a minimal Linux Debian environment on the same hardware with no tuning maxes very close to line rate in iPerf and NAT firewalling without breaking a sweat (CPU 85% idle). This experience had me abandon OpnSense for now in favour of Debian with a bunch of nftables rules and services configured through the shell. I might come back one day, but it seems to me that there is something quite crippling either in FreeBSD itself or in its configuration in OpnSense.

Thanks for demystifying that earlier comment.  ;)

No doubt FreeBSD as well as other Unixes as well as Linux is capable of producing great results. And those Netflix stats are impressive. However, note also that the use case is very specific. They stream static files from SSDs and have carefully optimized and tuned everything along the way, from software to o/s to hardware and drivers. In some cases they have found and removed bottlenecks and submitted back to FreeBSD (e.g. async sendfile). They have also worked closely with AMD and Mellanox and others.

I'm sure some of that has benefitted FreeBSD more broadly, but I don't know how relevant it is to most users on this forum who are trying to get good filtering, routing and forwarding performance out of a range of different hardware, from small appliances to enterprise. And in addition, on a relatively complex setup that involves netmap, automatically generated firewall rules, and additional software such as suricata layered on top. That's quite a different gig.. and the complexity of it, in combination with the range of available x86 hardware, is presumably why this forum is so full of people reporting such a wide range of experience from OpnSense in terms of performance.

Btw, if you're looking for other extreme examples, Linux reached a forwarding packet rate of 1 terabit back in 2017 ;)
https://www.fiercetelecom.com/telecom/linux-foundation-s-fd-io-virtual-switch-project-doubles-packet-throughput-to-terabit-speeds

For the regular user though, I'm not sure this is any more relevant than the Netflix example. And I notice this is going off-topic as well - my intention was not to start an o/s war. Sorry. Will stop there.

I guess it depends on what your end goal is. At the end of the day you'll want to measure the effectiveness of opnsense in the way that you intend to use it. Iperf could be directly relevant if you intended to host data-heavy services on opnsense itself. Which I'm presuming you're not. Granted a bit concerning that some bottleneck prevents it from pushing full line speed - but may actually turn out to be completely irrelevant in your real-world scenarios.

If you look back through previous threads you'll find examples both of people with good solid 10Gb performance and others who are struggling to get near line rate despite strong hardware. Fwiw, iperf against opnsense in itself seems to often disappoint but that may actually not be indicative of forwarding/firewall performance *through* the firewall. Have you tested this as well?

Mellanox ConnectX-3 10gb SFP dual port here, 1 to WAN and 1 to my LAN. No tunables set up.

That's interesting. I have Chelsio NICs, which are supposedly well supported, but I had to mess around with tunables and settings before I managed to get netmap to run in native mode and offer half decent performance. https://forum.opnsense.org/index.php?topic=25263.0

you seem not understand the BSD ecosystem. It's not your fault and that's OK.

Thank you for your thoughtful contribution to the topic.

Indeed, I like the product including its gui and plug-in ecosystem and its community. And I am raising the question whether in the long run it would be better off based off Linux than BSD. I understand it's a sensitive topic for some. Alas iX systems and Netgate both seem to be heading in that direction so its not like nobody ever thought of it before. But feel free to lol if that makes you feel better ;-). Or maybe add some actual thoughts on the topic.

Good to hear those speeds are achievable. What NICs do you guys use? Did you have to fiddle with tunables in order to get the performance?

Fwiw I looked at Vyatta also but didn't really see the point. Nftables in itself is straightforward enough so not so much gained vs a vanilla Debian - where you also get more flexibility. In both cases losing out vs OpnSense's awesome gui.

Internet speeds won't make much difference in how much RAM you need. Only marginally, in that with higher speeds you may want to tune larger buffers etc. But with the speed your mentioning you are at least an order of magnitude away, if not more, before any of that starts to matter.

You do however run an awful lot of stuff. I slightly lost track in your description and in any case I don't have experience of zenarmor, but I think you are saying you have already been forced to compromise, so clearly more RAM could help then. Can't give more specific answer than that, sorry.

Also, I think different philosophies at play here... Personally I look to achieve as much as possible with as little as possible, ie always looking for the minimal setup that will get the job done (well). Tends to be more stable, predictable, maintainable and performant over time. I even ditched IDS recently as I didn't think it was worth the performance hit (despite of plenty of hardware) given that almost everything goes over encrypted lines now anyway, so not sure how effective it really is. I seemed to be spending most of the time filtering false positives.

DNS blocking (pihole) and geoip blocking feels worthwhile though. Easy to see in both cases how it regularly blocks loads of stuff - which has to be a good thing.

After all my woes (https://forum.opnsense.org/index.php?topic=25263.15) I managed to get forwarding performance up to ~5 Gb/s through the Chelsio T520-SO-CR and Ryzen hardware so bit weird that your performance is is so low after having followed similar steps. Will be interesting to hear your results on Intel X710. And as mentioned on Linux also. NB that's a side project for me as well - setting up a minimal Debian 11 with routing and firewall through nftables, also unbound and dhcp server etc. Not got as far as live-testing it yet but curious how it will perform in comparison.

Ok so I'm opening up this thread again now, because I've done exactly this. The results are kind of interesting.

Note first and foremost that I'm a big fan of OpnSense. The admin GUI is superb and it has really served me well (and continues to do so) and helped me get up the curve on networking stuff.

All that said, I think the testing reveals some differences between Linux and FreeBSD, or at least FreeBSD as configured in OpnSense. My Linux setup is a minimal Debian 11 VM in Proxmox with nftables for firewall & routing and dnsmasq for dns & dhcp. Dnsmasq forwards dns to unbound as a resolver. A bunch of rules that controll traffic between the various internal networks, and a bunch of dnat forwarding to services on the dmz, with hairpin.

I applied minimal tuning - increased some tcp buffer etc. Don't know if that made any difference or not.

Out of the box, iPerf3 against 2 other servers on the internal network are a solid 9.4-9.8 Gb/s and Debian still runs 95% idle. NAT routing performance out on the 10Gb wan (using fast.com and speedtest.com) varies according to client and time of day between 6-8 Gb/s while the Debian VM idles 98% (!).

Note that I'm not running suricata or any fancy metrics or instrumentation (only nftables stats).

Still, this is quite some difference. I never managed to get much more than 5-6Gb/s through OpnSense on the same hardware, and the CPU had to work much harder too.

Maybe it partly comes down to kernel optimizations for Ryzen? In any case, I wasn't expecting quite such a difference.

Would OpnSense ever consider re-basing on Linux? I realize it would be a non-trivial exercise... The TrueNas folks did it though...

Looks sweet. I guess it's based on the Epyc 3201. What kind of performance do you get out of it?

I'm asking because 1U, powerful and quiet may be a tough combo. You could build something around SuperMicro SuperChassis 515M-R804 and for example X12SPZ-LN4F motherboard? Won't be quiet but plenty of performance.

Or alternatively a 3u or 4u white box around for example Inter-tech 4U-40248 and an ASRock Rack x570d4u with a modern Ryzen CPU. Will be power efficient, powerful, and can be made whisper-quiet particularly if you swap out the fans for Noctuas.

Or buy a secondhand Dell PowerEdge R730 which are supposed to be reasonably quiet (never seen one in the flesh).

Choices...

Edit: except out of those, only the first option front face i/o...