Recommendations for new hardware for opnsense 25.7?

Started by phanos, August 04, 2025, 07:07:47 PM

Previous topic - Next topic
Let's not get sidetracked and refocus on the original problem again.

Hardware requirements:
  • 6x RJ45 1Gb/s LAN ports
  • Quiet, active cooling (or retrofitting a fan is possible)
  • Hardware properly supported by FreeBSD
  • Low power draw

Corrections:
  • I need full hardware monitoring (temperatures, voltage, fan PWM), not just CPU/SSD temperatures
  • BIOS updates are not on my list, they are optional but nice to have
  • Cost is ideally ~500€ (like mentioned, can be stretched, e.g. to 700€), it is not "under 500$ (425€)"

Some clarifications for context, no need to debate:
  • SSD endurance does not matter, even a cheap WD Blue (150 TBW) would need 137 GB of writes per day to hit that limit in 3 years. OPNsense logs are in the range of a few GB per day and anything bigger needs to go into a proper log aggregation system like Loki to be of use anyway.
  • A single 80x80 mm intake 1W fan dropped the CPU temperature by 10°C on my test device, despite the CPU being installed on the opposite site of the casing. More headroom means less throttling, less power draw, higher performance and higher lifespan of nearly all components. And I like to have the option to use the performance I paid for.
  • Proper Super I/O sensors help with detecting fan issues and system cooling issues before they become a problem. They detect degraded VRMs and may even detect a dying PSU, which can show up as voltage deviations. CPU and SSD temperature sensors are not related to this in any way and can not detect most of these issues reliably.
  • Not monitoring a 24/7 edge device is the unusual position here, not the other way round. Every serious vendor exposes board sensors by default. Proper hardware monitoring helps to identify problems before they become actual hardware failures and allows shutdown options through self-monitoring where BIOS options are not available or reliable.


Questions that are still open:
  • Turning ASPM off - Does this have any negative side effects aside from an increase in PCIe power consumption and is this bug present on every hardware?
  • What is the minimum budget for the requirements I've set?
  • Which hardware does ship with supported full hardware monitoring?
  • Are there other vendors that offer OPNsense compatible hardware, aside from the ones I listed yet?

Quote from: UbiquitousWhite on September 06, 2025, 07:01:27 PMI need full hardware monitoring (temperatures, voltage, fan PWM), not just CPU/SSD temperatures

https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

Comes with IPMI, full monitoring of everything like voltages, fan speed, temperatures availably with e.g. Observium. Can easily drive a 1Gbit/s uplink, no experience with higher speeds.
Deciso DEC750
People who think they know everything are a great annoyance to those of us who do. (Isaac Asimov)

Quote from: UbiquitousWhite on September 06, 2025, 07:01:27 PMSome clarifications for context, no need to debate:
  • SSD endurance does not matter, even a cheap WD Blue (150 TBW) would need 137 GB of writes per day to hit that limit in 3 years. OPNsense logs are in the range of a few GB per day and anything bigger needs to go into a proper log aggregation system like Loki to be of use anyway.

O.K. - last bit of debate about just one error in your string beliefs. Here is the smartctl output of a lightly use home installation of OpnSense:

Quote# smartctl -a /dev/nvme0ns1
smartctl 7.5 2025-04-30 r5714 [FreeBSD 14.3-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-25, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       KIOXIA-EXCERIA G2 SSD
Serial Number:                      XXXXXXXXXX
Firmware Version:                   ECFA17.3
PCI Vendor/Subsystem ID:            0x1e0f
IEEE OUI Identifier:                0x8ce38e
Total NVM Capacity:                 500,107,862,016 [500 GB]
Unallocated NVM Capacity:           0
Controller ID:                      1
NVMe Version:                       1.3
Number of Namespaces:               1
Namespace 1 Size/Capacity:          500,107,862,016 [500 GB]
Namespace 1 Formatted LBA Size:     4096
Namespace 1 IEEE EUI-64:            8ce38e 0300993420
Local Time is:                      Sun Sep  7 12:49:56 2025 CEST
Firmware Updates (0x12):            1 Slot, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL Self_Test
Optional NVM Commands (0x005f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat Timestmp
Log Page Attributes (0x0a):         Cmd_Eff_Lg Telmtry_Lg
Maximum Data Transfer Size:         512 Pages
Warning  Comp. Temp. Threshold:     72 Celsius
Critical Comp. Temp. Threshold:     90 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.69W       -        -    0  0  0  0        1       1
 1 +     6.18W       -        -    1  1  1  1        1       1
 2 +     5.42W       -        -    2  2  2  2        1       1
 3 -   0.0500W       -        -    3  3  3  3     7000    5000
 4 -   0.0050W       -        -    4  4  4  4    13000   36000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         2
 1 +    4096       0         1

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        59 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    14%
Data Units Read:                    3,715,694 [1.90 TB]
Data Units Written:                 38,131,588 [19.5 TB]
Host Read Commands:                 82,686,672
Host Write Commands:                294,063,995
Controller Busy Time:               619
Power Cycles:                       13
Power On Hours:                     3,661
Unsafe Shutdowns:                   5
Media and Data Integrity Errors:    0
Error Information Log Entries:      86
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Thermal Temp. 1 Transition Count:   129
Thermal Temp. 1 Total Time:         22224

Error Information (NVMe Log 0x01, 16 of 63 entries)
No Errors Logged

Self-test Log (NVMe Log 0x06, NSID 0xffffffff)
Self-test status: No self-test in progress
No Self-tests Logged

Oh, just in case you do not have a calculator at hand, that amounts to 222 GByte/day, and BTW: those were not logs. You'll learn the hard way.

Good luck, I am out of here.
Intel N100, 4* I226-V, 2* 82559, 16 GByte, 500 GByte NVME, ZTE F6005

1100 down / 800 up, Bufferbloat A+

Quote from: Patrick M. Hausen on September 07, 2025, 12:06:54 AMhttps://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

Comes with IPMI, full monitoring of everything like voltages, fan speed, temperatures availably with e.g. Observium. Can easily drive a 1Gbit/s uplink, no experience with higher speeds.

Thanks for the recommendation! Indeed, it is starting to look like that I have to utilize IPMI and do a custom build. Not my favourite option but probably what I have to go for in the end.

Quote from: meyergru on September 07, 2025, 01:04:32 PMO.K. - last bit of debate about just one error in your string beliefs. Here is the smartctl output of a lightly use home installation of OpnSense:

[...]

Oh, just in case you do not have a calculator at hand, that amounts to 222 GByte/day, and BTW: those were not logs. You'll learn the hard way.

Good luck, I am out of here.

19.5 TB data written over 3,661 power on hours should amount to 128 GB per day - not 222 GB per day.

Regardless, SSD endurance is still not a problem:
  • A 500 GB WD Red SN700 NVMe SSD (advertised for use in NAS environments) has an endurance of 1000 TBW and costs only marginally more than a budget WD NVMe SSD. Even at a rate of 222 GB data written per day, it would take 12 years for the SSD to exceed the guaranteed endurance.
  • Plenty of people are running their OPNsense comfortably for many years on SSDs with much less endurance and without replacing them. It does not seem to be the limiting factor.

I'm not here to debate my requirements; I do not find it productive and they are not that outlandish either. Of course I'm open to questions and very thankful for security concerns I have overlooked, such as the vendor BIOS updates but primarily I'm here to find out what options exist and whether I need to increase my budget.

If TBW is of any concern, I would use device that you can install two NVMe SSD's where the 2nd one is same size but low budget, and then every so often (6mo, 1yr) boot device with a liveLinux usb and ddrescue from primary to bkup SSD. When the primary dies you can image back.
Mini-pc N150 i226-V, GOD BLESS CHARLIE KIRK

Another idea....

I have just bought a topton miniPC from aliexpress, N150 with 3x i226-v and 2x 82599ES.

I added a tplink switch with 4x 2.5Gbps PoE + 1x 2.5Gbps non PoE and 1x 10Gbps SFP.

8x 2.5Gbps ports in total.

Both cost me USD 272,95

With a 500€ budget you can buy that setup twice. And have one as spare/stand by in case of any issue.