OPNsense Forum
English Forums => Hardware and Performance => Topic started by: ck42 on February 18, 2022, 05:12:49 pm
-
(I've been reading through a ton of threads and other sites, but can't seem to find anything definitive)
Started with m0n0Wall, now on pfSense, and looking to move to OPN. This is for home use.
As stated, I have a 1Gbps fiber connection that uses PPPoE. Trying to get a sense of what CPU horsepower I might need for a new box. Looking for CPU that can max out the 1Gb connection.
Need DHCP service and maybe a few other small services, but don't plan on running any sort of IPS or VPN or any other heavy services.
Need a solution that is 1U and is as low heat and low noise as possible but still gets the job done.
The search though seems to begin with the CPU.
-
If you read all the forums you'd know that GB and pppoe is a huge trouble in BSD world. You would need very high clock rate, no matter how many cores, since it's single threaded. And no guarantee you'll get full throughput.
-
Yep, I certainly understand that 'challenge' after reading through various posts....hence after searching and not coming up with any answers...is why I'm here asking.
Really all I'm wanting to know is...what minimum CPU (single core speed) would I need for this application?
Would a modern dual core @3GHz be sufficient? Higher? Lower? Would a high speed Celeron work? Or Pentium? Or Core i needed? AMD something?
-
No idea, in my area 100mbit PPPoE is max
-
Found this in a week old comment in an OPN subreddit:
I have near gigabit internet service (900Mbps down / 180Mbps up - FTTH - GPON terminated on ISP's ONT) and PPPOE on OPNsense connected to CAT5E on ONT. I get full download speed using a Dell Wyze 5070 Extended which has 8gb RAM and a Pentium Silver J5005 CPU (4 x 1.5 GHz with boost to 2.8 GHz). When downloading 900Mbps the total CPU usage is about 34%.
So here we have someone claiming near full gig throughput on a LOWLY Pentium J5005. Base freq 1.5GHz, burst 2.8GHz. 10W TDP which is really nice.
-
Get the highest speed for a single core you can basically.
There are a few tunables that help
net.isr.dispatch = deferred
net.isr.maxthreads = -1
net.isr.bindthreads = 1
I run OPNsense virtually on Proxmox 7
Hardware is a basic HP 8300 Elite SFF with i7-3770 (3.4GHz - 3.9GHz)
Basically a cheap 24/7 desktop but really quite powerful for my purposes.
Also using intel NIC's
With a standard install I get the full speed on my 900/400 plan over PPPoE. Speedtests usually 950
I then also run ntopng which drops it to about 750/400.
Edit - Just upgraded Proxmox 7.1-10 and a speedtest got 850 with everything running.
I run a few light load VM's so sharing will cause resource contention and affect speedtest results.
I've tried passing through the NIC's but this doesn't improve throughput so I prefer virtio and benefit from live migration to another HP desktop.
Hope this helps
-
Appreciate your feedback!
So, using a CPU benchmarking site, I look at the single core scoring and this gives me (hopefully) a way to compare CPUs. So if the 3770 work w/o problem, then that SHOULD give me a way to compare various CPUs (looking at their single core scoring).
One other thing I'm wondering about now - In light of the whole single threaded PPPoE thing, would more cores/threads in a CPU help any other part of the system run better? Or, for example, with a dual core CPU, with one core/thread doing all the PPPoE work, would the 2nd core/thread then be completely sufficient to run everything else? Or would the 'everything else' part possibly benefit from a 4 core CPU so that 'everything else' would then have 3 cores vs just 1.
Basically, wondering if I should even bother looking at 4 core CPUs, or since dealing with PPPoE, is just a dual core (with proper single threaded scoring) be enough.
-
A lot of it comes does to your personal requirements.
PPPoE will use CPU0 from my reading and understanding. This is the first CPU in the queue and can cause issues when running at 100% for other services.
I personally run Proxmox which shares the CPU cores at the hypervisor to the guests. I get some benefit from this as I want to run multiple VM's on the same hardware, but am still limited by that first core when under high utilisation. I hardly ever max out my Internet link, and if I am its usually under controlled circumstances and I can expect other VM's to under perform.
If you are dedicating the hardware to you OPNsense and expect it to be running at full gigabit often, then two cores plus I'd recommend. And of course at the fastest GHz possibe.
Like others have said though - there are really no guarantees here.
-
One data point:
I have an HP T620 Plus with a 4-port Intel Ethernet card.
The CPU is an AMD GA-420CA, which is a lowish power (20W) 4-core 2GHz CPU. This CPU has a single-threaded passmark score of 667.
I get ~720down 750up over my CenturyLink Fiber (Gigabit Fiber with PPPoE) using OPNsense 22.1.2_1-amd64.
net.isr.dispatch = deferred
net.isr.maxthreads = 4
If I add - net.isr.bindthreads = 1, download drops and upload improves (610d/850u), it doesn't seem to make a difference whether I set maxthreads to -1 or 4.
One CPU core is saturated during the tests.
I figure I need at least 50% better single core performance to max out my connection. The Pentium J5005 mentioned above is in that ballpark, as is the Celeron J4105. I'd feel better with something that has at least 2x the single threaded performance.
-
Yes, and systems like that are available even with passive cooling, with a J4125 and 4x I225-V NICs:
https://www.amazon.com/gp/product/B09PHHXN9V
Depends on if you need 1U for rack mounting, though, but that is likely to have active cooling.
-
Newer devices with 10nm Elkhart Lake chips are starting to appear as well e.g. J6412 (10W).
https://www.cpubenchmark.net/compare/Intel-Atom-x6211E-vs-Intel-Celeron-J6412-vs-Intel-Atom-x6425E/4347vs4474vs4753
-
Do you have some recommandation for Elkhart based devices?
I could only find this:
https://german.alibaba.com/p-detail/Maxtang-1600441734696.html
which seems ok, but has only 2 LAN Ports. I would prefer something with more ports.
-
The Compulab Fitlet3 (https://fit-iot.com/web/ (https://fit-iot.com/web/)), which is supposed to ship in April but can be ordered now, has up to 4 Intel LAN ports. I'm using a Fitlet2 with J3455 at the moment and I think there are probably a few other Fitlet2 users here.
-
In used fitlet2 with celeron CPU till recently, and it can do 1Gbps pppoe with ease. But i didn't run any additional plugins like suricata or zenarmor. For openvpn it can do 500mbps. Hope it helps, its a great little box just running a little hot.
-
Really all I'm wanting to know is...what minimum CPU (single core speed) would I need for this application?
Would a modern dual core @3GHz be sufficient? Higher? Lower? Would a high speed Celeron work? Or Pentium? Or Core i needed? AMD something?
I’m running a ppoe/1gbps connection on an i7-6700 esxi server here - with the increased efficiency of v22.x I’ve been able to drop the vCPU allocation down to just 2 cores, and even with zenarmor on the lan side I never max out those two cores
without zenarmor I could almost certainly get away with a single vCPU
As others have said single thread speed matters for PPPoE - so I’d go with whatever option you can find that has the best single-thread performance (core i3 maybe? or some of the new AMD options).
Also remember the NICs matter too - you’ll want to avoid anything with realtek NICs
-
In used fitlet2 with celeron CPU till recently, and it can do 1Gbps pppoe with ease. But i didn't run any additional plugins like suricata or zenarmor. For openvpn it can do 500mbps. Hope it helps, its a great little box just running a little hot.
I am running my Fitlet2 J3455 with Zenarmor. I am on a 300 Mbps connection. It's using about 2gb of the 8GB of RAM I have installed. The CPU isn't being taxed but I'm only using it on a home network. Every time I check the temp it's running around 46C to 47C.
The single core performance of all the new Elkhart Lake CPUs (x6211E, J6412 & x6425E) in the Fitlet3 are a lot faster than the J3455: https://www.cpubenchmark.net/compare/Intel-Celeron-J3455-vs-Intel-Atom-x6211E-vs-Intel-Celeron-J6412-vs-Intel-Atom-x6425E/2875vs4347vs4474vs4753
The Fitlet3 also supports DDR4 and NVMe drives. I believe the NICs are Intel I210.
-
To futur
Found this in a week old comment in an OPN subreddit:
So here we have someone claiming near full gig throughput on a LOWLY Pentium J5005. Base freq 1.5GHz, burst 2.8GHz. 10W TDP which is really nice.
This is interesting and I did a little research. I have an Atom E3845 and use PPPoE W/Vlans on gigabit, and can can confirm I get ~700mbps max. It’s a bummer. According to geekbench5, lowly the J5005 is over twice as fast as mine in single core performance. So that discrepancy makes sense. (I’ve switch to OpenWRT for now and it works full speed no sweat).
I think a lot of the folks running into pppoe gigabit limits are using similar old lower power systems. Think of the super popular APU2 and related PCS. Or the J1900 mini PC systems. These are all similarly slow compared to a “lowly” J5005.
Bottom line for those interested: when deciding, compare single core benchmarks to these systems to get an idea of how your prospective system will perform. Report back on how things pan out.
-
The problem is, there is no router-benchmark test, that could reliably tell how many megabits or gigabits/sec a certain CPU could do under Freebsd 12/13, if PPPoE is the WAN protocol.
So the only true option you have, is to buy something that is 2-3-4 gigahertz (translation: overpowered 2-3times, just to be safe), and is actually a very recent microprocessor. In recent I mean: from the past 3-4 years. And be careful! I said not the product itself should be maximum 3-4 years, but the building block CPU/SoC age should be max. 3-4 years. As "some" companies ehem..ehem.. pcengines..ehem are selling a 10 year old AMD Jaguar CPU in their APU2-3-4-5-6-7 product line. At the end of 2022. So the product, like APU7 may be new (because they are sort of hiding the details about APU5-6-7, the public may be in the dark), but the CPU on the board is a rusty p.o.s in terms of routing performance in 2022.
-
Hello all,
So in case it's still of interest: I can report that it is possible to utilize a Deutsche Telekom fiber connection (1 GBit/s down) via the provided fiber modem with an OPNsense DEC2750. So to say tested with the NETBOARD A10 Gen3 and the AMD Embedded Ryzen V1500B.
Best regards,
Mike
-
Hello all,
So in case it's still of interest: I can report that it is possible to utilize a Deutsche Telekom fiber connection (1 GBit/s down) via the provided fiber modem with an OPNsense DEC2750. So to say tested with the NETBOARD A10 Gen3 and the AMD Embedded Ryzen V1500B.
Best regards,
Mike
With or without special tuning?
-
For reference I have a MiniPC YL-J3160L4 which is an Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (4 cores, 4 threads) with 4GB of ram and 4x1GBit ethernet (Intel(R) I210 Flashless (Copper))
Recently I change my connection from vdsl2 200/30 to ftth 1000/300
Initially I have just swapped the "isp router" and everything was working fine with speedtest results up to 906/310
As the new isp modem is an huge white box which is very ugly and as all Italian (maybe europe?) FTTH ISP use PPPoE and OPNSense support it, I decided to remove the "isp router" and use directly the OPNSense box and here started my trouble with performance which dropped down to 430/310
after some reading here, pfsense and general freebsd forums I found a configuration that raised performance up to 902/310
Enabled powerd
Disabled Hardware CRC/TSO/LRO and VLAN Filtering
Adjusted this Tunables:
net.isr.dispatch=deferred
net.isr.maxthreads=-1
net.isr.bindthreads=1
net.inet.ip.intr_queue_maxlen=3000
net.inet.rss.enabled=1
dev.igb.0.iflib.override_qs_enable=1
only the last one is hw specific for my Intel I210 and igb0 is the interface connected to the Optical Network Terminal (Optical to Ethernet signal converter)
about VoIP in OPNSense I have assigned a static Lan IP to the Wan interface of the "isp modem" and added a Port Forward of WAN address port 5060/UDP to the previous static IP and I was able to move the isp modem away from my eyes :D
-
I am running a Protectli VP2420. Celeron J6412 Quad Core 2GHz. Getting full speed on my Fiber 1GB Down/940GB Up on PPPoE. Performance-wise I left everything default and it's fine.
I have been doing a lot of research too, because of anticipated PPPoE-related issues and it would seem the single core issue has been addressed for a few years. Since OPNsense uses MPD (https://mpd.sourceforge.net/) to establish PPPoE connections, it should utilise all core by default. All the configurables documented everywhere should have little to no effect. To be taken with a grain of salt, I am a fairly new user to OPNsense and FreeBSD in general.
Here is a thread discussing this specifically: https://forum.opnsense.org/index.php?topic=30925.0
-
@Prevok can you confirm this is for when your ISP provides your internet connection as PPPoE?
I'm thinking you might be talking about running a PPP server on OPNsense and this is when MPD5 is used.
Sorry, but just to confirm, could you confirm your ISP is PPPoE and run top -P or top -1 (can't remember which one it is) to see if your first core is used or is the load spread when running at full download bandwidth.
I have moved to an ISP that does not use PPPoE so cannot confirm its fixed sorry.
Thanks
-
I tried it (top -P) and saw even distribution over 4 threads. My DEC750 has 8, but only the first four got utilized at ~30% load for 600 MBit/s. On another N5105-based box, all 4 cores were being used at 1000 MBit/s and ~10% load.
After I changed the number of RX and TX queues from 4 to 8 on the DEC750, the load on individual cores became so low that the distribution was uneven and it seemed like only 3 thread were used more. Matter-of-fact, with those speeds, both boxes are far from their limits.
-
@Prevok that wasn't my case
my box with default settings cant reach 500Mbit, probably due the weak cpu
-
@Prevok can you confirm this is for when your ISP provides your internet connection as PPPoE?
I'm thinking you might be talking about running a PPP server on OPNsense and this is when MPD5 is used.
Sorry, but just to confirm, could you confirm your ISP is PPPoE and run top -P or top -1 (can't remember which one it is) to see if your first core is used or is the load spread when running at full download bandwidth.
I have moved to an ISP that does not use PPPoE so cannot confirm its fixed sorry.
Thanks
I am not running any PPP server. I need to establish a PPPoE connection, over a specific vLAN, with my ISP.
When running a speedtest, with 'top -P -H' you can see activity jumping to roughly 30% on CPU 1 and 2. The number of threads increases by 2-3 as well (although unsure how accurate this might be).
-
@Prevok that wasn't my case
my box with default settings cant reach 500Mbit, probably due the weak cpu
Yeah has to be it. My previous box was running a J3160 (Protectli FW4B) as well. On OpenWRT and Debian I was able to reach 500-600Mbit, slightly less with pfSense and OPNsense with default configs, I wasn barely reaching 200Mbit.
After various tunables, I don't remember even reaching 400Mbit :(
At some point, even with multiple core and multithreading support, the IPC on that CPU is too low (among other things).
Edit: Didn't realised you actually almost maxed the line on that box. Good job :) I gave up and replaced the box :D
I find the information around PPPoE on FreeBSD very confusing.
-
maybe i was luckly, after switching from the isp router to opnsense I noticed the poor performance and after few read I found a big difference with:
net.isr.dispatch=deferred
net.isr.maxthreads=-1
net.isr.bindthreads=1
the rest make some kind of difference but not big as this one
btw FW4B is a rebranded YL-J3160L4 (I'm running protectcli coreboot bios on it)
-
I agree that it's super confusing trying to understand what's actually going on here.
I 'believe' it was to do with multi queue network adapters working with multi core cpu's.
So it became worse if you had a multi queue adapter with a multi core cpu that didn't have a high GHZ.
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics
https://redmine.pfsense.org/issues/4821
So I'm really not sure if it has been resolved or not?
Possibly some additional testing from those impacted to confirm what their hardware is (including NIC and capabilities)
Or perhaps someone more knowledgeable than me to confirm :)
-
Found this great write up about it
https://eyegog.co.uk/posts/a-sad-slow-pppoe-story/
Its solved in this case by replacing NIC hardware and using current software.
-
Awesome find. Would explain the behavior I had on my previous hardware, because I don't remember the CPU being that busy. It had to be at least partially responsible, but combined with the NIC limitation, the picture becomes clearer.
-
I’m also running a J3160 (Protectli FW4B) and have just upgraded to FTTP after 2 1/2 years of 4G!
Anyway, it’s probably frowned upon here but after experimenting I’ve settled on using the ISP provided router for pppoe, switched off NAT and pointed the DMZ to my Protecli box. I don’t appear to be suffering any disadvantages except for the extra box, and handing off pppoe gets around the OpnSense issues.
-
What sort of latency might I expect with a current ‘cheap’ cpu like a N100 using pppoe at 1 Gbps?
I see a lot written about throughput, but latency concerns me also. With my current mass-market fritzbox I’m getting around 4-5ms to london, occasionally 3ms. I am 80km away. Local lan latency sits around 0.3ms. It all seems rather decent. I’m thinking of trying opnsense, but only if I can have similar latency.
Will probably go for an aliexpress box… with intel 226 adapters
-
If you read all the forums you'd know that GB and pppoe is a huge trouble in BSD world. You would need very high clock rate, no matter how many cores, since it's single threaded. And no guarantee you'll get full throughput.
It this is still true today, does this mean "don't use opnsense" if you want to get throughput around 1 Gbit/s? Is that the conclusion? Because in that case i could stop looking for a better hardware and rather start looking for another firewall software.
-
Has anyone tried this?
Quote:
As for performance, APU2 can NAT at full Gbit on Linux and 650-700 on pfSense (see benchmarks here https://teklager.se/en/knowledge-base/apu2c0-ipfire-throughput-test-much-faster-pfsense/ 38 and in the other article where they mention OpenWrt being at 1Gbit on single or multi streams, because Linux https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/ 22 ).
It also has AES-NI crypto acceleration.
They mention "with a new bios" for the pfsense performance because new BIOS enables CPU boost to 1.4 Ghz so the CPU can clock higher when only a few cores are loaded (as I said BSDs can only run the interrupts on a single core)
Source;
https://forum.openwrt.org/t/tips-for-getting-cheap-used-x86-based-firewall-with-full-gbit-nat-a-pc-engines-apu-if-you-are-in-the-us/104490
-
Okay, not worth trying it...
Quote:
Only 1 CPU core will receive the full PPPoE network traffic, regardless of how many different flows are inside the encapsulated stream. Because a single core running at 1.0-1.4Ghz is insufficient to process the full Gigabit network traffic, you will never be able to reach 1 Gbit routing traffic using an APU2/3/4 board with BSD operating system. 1 CPU core is 100% utilised, while the other 3 CPU cores are IDLE.
Source:
https://teklager.se/en/knowledge-base/apu2-1-gigabit-throughput-pfsense/
-
A DEC650 from Deciso easily manages to do 1G with PPPoE. Just saying. APU devices are old. Like in stone age old. I like their design, I also like Soekris Geode based devices. ;) You get the idea.
-
First, sorry, i meant APU4, not APU2. My bad.
I also get the difference on the price tags. Just checking the prices, haven't yet found the 650, but the 630 is around 650 EUR.
As you wrote: APU2 is old (as well is APU4). As cpu and electronics prices have dropped constantly (with the corona exception), in the same price segment of an APU4 i would expect a more powerful device today. Shouldn't be that hard, as most of the components specifications could stay the same, only thing needed is double the boost frequency, as obv. not much more is needed if the OS would be offering multi-processor support for these kind of network applications.
So, i understand you're saying there is no such replacement, which means, for a lot of home users who would like to benefit from > 300 Mbit/s their ISPs are offering, opnsense isn't the solution to go with?
-
I will give an J4125 a try.
4 cores, clock speed 2 GHz, boost 2.7 GHz, passmark only about 4% below a J5005 on single thread performance.
Price tag for a configuration with 16 GB RAM and 512 GB ssd currently around 160 EUR (pot. plus MwSt.)
10 Watt, fanless, however looks like some additional heatsinks might be suitable.
Will share the results, might take a while.
-
All these passively cooled appliance style mini systems are expensive, granted.
For the hobbyist short on cash, why not a used desktop board? Or virtualise on that proxmox system you have running for a homelab, anyway?
Yes, FreeBSD PPPoE performance could be improved. But it's not "abysmal". I wonder if Deciso and Netgate are the only ones offering those "netboard" designs with AMD SoCs.
The DEC650 cost us 600+something at the time. It's been replaced by the current models. For business use that's a no-brainer. All my former Sidewinder customers bought from Deciso without thinking twice. Migrated everyone to OPNsense. "1500€ one time for the rackmount system and no yearly fee? Shut up and take my money!" ;)
-
I will give an J4125 a try.
4 cores, clock speed 2 GHz, boost 2.7 GHz, passmark only about 4% below a J5005 on single thread performance.
Price tag for a configuration with 16 GB RAM and 512 GB ssd currently around 160 EUR (pot. plus MwSt.)
10 Watt, fanless, however looks like some additional heatsinks might be suitable.
Will share the results, might take a while.
I have two of these running at 1 Gbit/s no problem. One even has old I210 NICs.
-
with the setting from my previous post I was able to get 1Gbps PPPoE with j3160, J4125 (as reported here) handle 1Gbps PPPoE w/o tuning, most probably N100 will handle 2.5GBps PPPoE without issue
https://www.cpu-monkey.com/it/compare_cpu-intel_celeron_j3160-vs-intel_processor_n100
-
Yes, they will. Unless they run unstable because of buggy CPU microcode. Luckily, there now is an option to make them work (https://forum.opnsense.org/index.php?topic=36139.0).
-
I have tried both J4125 and J5005 - both work great with Zenarmour and 1gig symmetric PPPoE direct from OPNsense to the ISP ONT (gigabit RJ45)
Don't know if they were needed, but I used the tunables earlier in the thread as well...
For a cheap J5005 box, I had a Wyse 5070 Extended with Intel I350 card from eBay.
9w idle at the wall with a kill-a-watt, and virtually silent
-
Hello
Today I got a Gigabit Vodafone fiber optic connection, I get 850 Mbit/s down and 520 Mbit/s up stream.
Intel J4125 and 8GB RAM.
Zenarmour is actively testing.
https://www.vodafone.de/media/downloads/pdf/GigaZuhause-1000-Glasfaser-dg.pdf
Best regades
Markus
-
A DEC650 from Deciso easily manages to do 1G with PPPoE. Just saying. APU devices are old. Like in stone age old. I like their design, I also like Soekris Geode based devices. ;) You get the idea.
According to https://www.deciso.com/product-catalog/DEC630/ it has an AMD GX-416A 1.6 Ghz quad core while the apu4d4 has an AMD GX-412TC 1 GHz quad core.
So 1.6 GHz on a single core is enough, but 1 Ghz on a single core isn't enough for PPPoE (FreeBSD limitaton).
-
The PPPoE performance on 24.1 (APU2) seems to be quite better (between 350 and 400 instead of between 170 and 320 MBit/s) so fare :)