Eastpect only single core?

[donatom3]

I just upgraded to a Xeon D 2123-it and am maxing out eastpect on a single core and only pulling 200 down on my gig connection.

Running top -P I see eastpect running maxing out a single cpu.


On my old i5-7600 this was working a bit better but as you can so many of my cores are sitting there idle.

[mb]

Hi donatom3,

Yes and No.

Eastpect (packet engine) architecture is multi-process and it can scale to the number of CPU cores in the system.

Having said that, currently, we're running single core. Reason is we need Receive Side Scaling (RSS) kernel option be compiled in to the kernel. RSS is a technique for the ethernet drivers to distribute incoming packets into multiple RX/TX queues (thus CPU cores) to be able to make use of the multi-core capability of the hardware:

https://wiki.freebsd.org/NetworkRSS

I think I should bring this to OPNsense team's attention. Should they see a fit for the project, we can work on this after the netmap work.

One question: what does "ubench -cs" report for you?

A note about the Xeon-D series processors. Our experience with them has not been much promising so far. I read that these processor are meant for low-end market. Looks like they converted a single core to multiple cores having the same total processing power. Meant for systems where there are many jobs with less cpu Hz requirements.

For Sensei, a simple desktop/mini pc with i5 or i7 CPU is performing way better than low Hz many core server CPUs.

[donatom3]

Mb

Ubench Single CPU:   464101 (0.40s)

[mb]

@donatom3, this looks good. Can you reach out? Let's have a closer look.

[donatom3]

Sure I'll open a ticket from the firewall and have it upload the logs.

For now I've switched to passive so I can collect data and get full speed.

I did want to mention since moving to the supermicro with Xeon D I've had much less problems with suricata and sensei taking minutes  after a reboot to get connectivity since they each caused the wan to release IP's.

This could also be a side effect of me getting rid of all the custom  tunables I did on the qotom running an i5-7600 with igb drivers and going with just the stock opnsense tunables. Could also mean the qotom was using a clone intel chipset.

[DenverTech]

@donatom3

I've definitely gone through that testing myself and can vouch for what MB was saying. I've gotten FAR better scores from i3s and i5s than from anything in the D-series. I had a D-2123 and a D-1528, which had abysmal scores. They're amazing for normal firewall use, but can't handle the higher burden that Sensei and programs like it put on them.

It made me sad in a big way to set aside my own Supermicro D-2123 in favor of an old 4th gen i5, but the performance difference was amazing.

One trick you can use though is to move Elasticsearch to another system. That lowers the load on the firewall a lot. It won't get you 1gbit speeds, but it'll help.

[donatom3]

@denvertech I see what you're saying but on the same page as my post is another one where mb said a score of 250,000 should be enough for 1gb. The D-2123 pulled a 460,000 on single core. There may have been something to hyperthreading that support asked me to turn off. I think one of the tunables I set made my system unstable so I reverted for now but will try again with hyperthreading off tonight.

[donatom3]

@mb

I'm not sure what did it but I'm back to the performance I'm expecting. NOt sure if it's 1.6.1 or if it's because I changed powerd from adaptive to hiadapative.

Also still have hyperthreading on and while doing a speedtest I can't see any single core go above 35%

[DenverTech]

I would be curious if you do manage to get solid speeds with a 2123, as support and I never could. I ended up offloading to an old i5 box rather than keep fighting with it. Tbf, it worked great on the 2123 in OPNsense 20.1, but we never got it quite right on 20.7.

[donatom3]

So far the speeds are holding up. I need to bust out my old box that was doing gig fine with sensei but that was an i5-7600k in a case that can't handle it's tdp.

I got rid of all custom tunables except the igb tx and rx process limit tunable. I may even drop that. The interface that's connected to my internal network is using the ixl driver on 10gb sfp+ to my Aruba 1930.

[actionhenkt]

Is sensei ever going to be multithreaded ? My download speeds over WG vpn seems to max out at 400mbit (of 1gb) with eastpect running 100% cpu on 1 core (out of 8 ) with average load of 60% on the complete box.

[mb]

Hi @actionhect, yes. It might be sooner than we originally planned, since we have requests from school districts to handle 10 gig. Like netmap work, we need to work with OPNsense team on this. There are two kernel options (RSS, PCBGROUP) that need to be enabled in the kernel.

[donatom3]

@mb I realize my last message was a bit confusing.

I'm getting 1gbps now with the D-2123it. Again not sure what did it. Multithreading would be great since I imagine traffic between vlans would have to go through Sensei if I'm using opnsense as my router.

[mb]

Hi @donatom3, thanks for the update & clarification.

[alexroz]

posted by mistake