Planning a High-Core-Count OPNsense Build: Xeon 36 Core 2.4GHz - Overkill or Fut

[dahapo8728]

Hello OPNsense community,

I am in the planning stages for a new, powerful router/firewall build intended for a lab environment that will eventually handle significant traffic, including multiple site-to-site IPsec tunnels, IDS/IPS (Suricata), and potentially a high number of concurrent states.

I have the opportunity to use a decommissioned server board with a Xeon 36 Core 2.4GHz processor. While I understand this is extreme overkill for a typical home setup, I'm interested in the technical considerations for OPNsense and FreeBSD.

My specific questions are:

Core Utilization & Affinity: With a CPU of this scale (36 cores / 72 threads), how effectively can OPNsense/FreeBSD distribute workloads like Suricata inspection, IPsec encryption/decryption, and the kernel's packet forwarding across so many cores? Is manual tuning with sysctl and setting process affinity for services like Suricata absolutely essential to avoid thread contention and cache misses, or will the scheduler handle it reasonably well?

Power Efficiency vs. Idle States: I'm concerned about power consumption. A CPU with this many cores likely doesn't idle as efficiently as a modern, low-core-count Xeon E or Intel Core series. Has anyone run OPNsense on similar high-core-count server hardware and found success with aggressive C-state configuration in the BIOS to manage power draw during low-traffic periods?

Hardware Compatibility: Are there any known issues or special driver requirements for the integrated NICs or other components commonly found on server boards (e.g., from Supermicro or Dell) that I should verify before committing to this hardware? I plan to use a dedicated, supported Intel NIC.

Performance Ceiling: In practical terms, at what point (e.g., number of gigabit tunnels, Suricata ruleset size, or states per second) would a CPU with this core count but a moderate 2.4GHz clock speed actually begin to show a significant advantage over a modern 8-core CPU with a much higher clock speed for a router's primary duties?

Thank you for sharing your expertise and any experience with similarly oversized hardware for OPNsense.

[pfry]

If you have the opportunity to test it, I'd be curious as to your experience. Deciso uses embedded Epycs, which (mostly due to power) tend toward throughput; I use fewer, higher-clocked consumer cores in my home firewall, as they're less expensive and responsive with few threads. I do not use an IPS.

That being said, that Ice Lake isn't exactly a flock of chickens.

[BrandyWine]

How many threads would an OPNsense install use?

[Greg_E]

I don't think you will see much benefit until you get a bunch of VPN connected.

Suricata is supposed to be multi-threaded, so this should help. It would be interesting to see how many cores are in use when you have just a few users.

Your power/heat/noise will likely be fairly high. I'd also couple this with a fair amount of RAM, at least 128GB, since it's probably DDR3 it should be cheap (last I bought was 96GB ECC for around $40usd on the used market). Lots of RAM also increases power/heat/noise.