Router crashes and reboots under load

[Rallaster]

Dell Optiplex 790
i5-2500
16GB RAM
Crucial MX200 250GB SSD
Intel PRO/1000 VT Quad Port Server Adapter LP PCI-E EXPI9404VT
OPNsense 23.1.5_4-amd64 on a clean install

It can sit idle for DAYS without any issues, but as soon as I put any kind of throughput on the NIC it crashes and reboots.  It's not showing anything in the logs, so I'm pretty sure it's hardware related, but I've ran all of the preboot tests on the 790 and it's coming back clean.  I can take the NIC and SSD out of the Dell 790 and put it in a rig with a 3570K and 16GB RAM and without changing anything boot into OpnSense and run the dog snot out of it without so much as a dropped packet. 

I was running PfSense for years without any issues but I have had nothing but trouble with OpnSense.  I tried to go back to PfSense and it has started doing the same thing, so that's just adding to the confusion.  The 3570K doesn't support VT-d so I can't just throw it in a VM and call it a day, otherwise I'd just do that.


Edit to add: I have 2 Dell 790's that are identical and both exhibit the same issue.  Idle for days, but crash/reboot when under load.

[axsdenied]

Given this is a hardware issue by the sounds of it and you've said so yourself...  what is the ask to this community?

[Rallaster]

Because I'm not 100% that it's a hardware issue.

More info: I moved the WAN from the discrete Intel PRO/1000 to the Integrated Intel® 82579LM and it worked longer than it has in the last 2 days.  It still crashed and rebooted. I have also thrown a Windows 10 SSD in the computer and ran the NIC into the ground transferring 100's of GB of files transferred across my internal network to stress test the NIC and it worked as expected.

I'm now pretty convinced it's not a hardware failure. (I have ran Dell Preboot diagnostics as well as several bootable diagnostics and everything comes back clean) Incompatibility? Probably.  If there is a configuration that I need to tweak to make it work, I would expect the community to have insight into this.  I was able to get logs from just before the crash, but it doesn't appear from the crash itself.

[axsdenied]

Did a little bit of research as it could be a driver issue.  Here's an excerpt from the following link:
https://forums.serverbuilds.net/t/demystifying-intel-pro-1000-quad-port-nics/2401

"...Note: there is a known issue with these cards and some older systems (primarily the PT/VT chips and Dell motherboards for Sandy Bridge (2nd gen) Intel processors and for AMD embedded R-series from 2014), among many others - manifesting in kernel panics on FreeBSD systems (i.e. pfSense), missing memory, onboard diagnostics failing, and more. It is due to the SMBus pins on your card. If you have a mentioned system and want a quad port NIC, try the i340-T4 or the i350-T4. I recommend going for the i340 in any case."

[styr0]

Have you tried the settings in the Interface/Settings menu?
Specifically; 
Disable hardware checksum offload
Disable hardware TCP segmentation offload
Disable hardware large receive offload
Don't know if it will help but worth a try.

[meyergru]

Did a little bit of research as it could be a driver issue.  Here's an excerpt from the following link:
https://forums.serverbuilds.net/t/demystifying-intel-pro-1000-quad-port-nics/2401

That is not what is said there. It seems to be an issue with some Dell mobos under FreeBSD which can be fixed by masking the SMBUS pins on the PCIe cards, effectively disabling that feature. You should try that.

[Rallaster]

Thank you everyone.  I went in a different direction for the moment and plugged the SSD into an old Latitude laptop with an i5-5200u and a USB ethernet adapter.  It's less than ideal, but in the 48 hours I've been running it it's only had 1 hiccup. 

Did a little bit of research as it could be a driver issue.  Here's an excerpt from the following link:
https://forums.serverbuilds.net/t/demystifying-intel-pro-1000-quad-port-nics/2401

Thanks for that link.  I wish my searching had pulled it up, maybe would have saved everyone a bit of a headache. :p  I'm going to look at getting an i340 in the future, I just threw this together out of spare parts I had laying around.

That is not what is said there. It seems to be an issue with some Dell mobos under FreeBSD which can be fixed by masking the SMBUS pins on the PCIe cards, effectively disabling that feature. You should try that.

I'm going to give it a try here in the next few days as my schedule allows.  We'll see how it turns out.

[axsdenied]

Sorry I could have been more clear.  My hunch was that this sounded liked a driver issue but my search found this link related to the issue.

Did a little bit of research as it could be a driver issue.  Here's an excerpt from the following link:
https://forums.serverbuilds.net/t/demystifying-intel-pro-1000-quad-port-nics/2401

That is not what is said there. It seems to be an issue with some Dell mobos under FreeBSD which can be fixed by masking the SMBUS pins on the PCIe cards, effectively disabling that feature. You should try that.