Random hardware crashes (on Supermicro X10SDV-TP8F)

[mkromer]

Hi everyone,

I'm closely getting to the point of frustration. I operate a few opnsenses, and I'm about to install a new opnsense installation with 2 X10SDV-TP8F boards in place. The hardware is quite around for some time now (~9 months) but we just finally get to it to install and operate them (So I really hope I don't have to RMA). Functionality wise everything fine. System is HA and setup as a transparent bridge for DMZ with some inner and outer networks. So far all standard.

Problem is: Literally in like a daily fashion (not exactly to be nailed down time wise) the firewalls go down. Sometimes just one, sometimes both. I've seen this already on 19.1 happening, but didn't take to much care of it because of another project and this could wait. Now with 19.7: all the same. Recurring crashes. I get no traces, no nothing, but they are dead toast when freezing.

System has 2 10GbE (ix) and 6 1GbE (igb) interfaces, all soldered on:

sysctl -a | grep -E 'dev.(igb|ix|em).*.%desc:'
dev.igb.5.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.4.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.ix.1.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k

What have I tried:
- Upgraded to 19.7
- Upgraded latest BIOS for the board (2.0c)
- Upgraded even the latest intel microcode
- Updated loader.conf with various options and try-outs:
  - cpu_microcode_load="YES"
  - cpu_microcode_name="/boot/firmware/intel-ucode.bin"
  - kern.ipc.nmbclusters="1000000"
  - hw.igb.num_queues="1"
  - kern.ipc.nmbjumbop="524288"
  - hw.intr_storm_threshold="100000"
  - hw.ix.num_queues="1"

Looking desperately for help. Also willing to pay if someone can seriously help and not just waste time.

Thanks in advance!

- mike

[opnsenuser]

Hi,
maybe I can link a similar case https://forum.opnsense.org/index.php?topic=13402.msg61688#msg61688 like with similar system setup to this thread...

greetings
opnsenuser