OPNsense services failing

[mezzfit]

Hello all,
I have a weird one here. I just started using OPNsense on a Optiplex3060 that I installed an additional NIC into. The problem I am running into is that the webpage, prom exporter, and some other services all stop working occasionally(a few hours to a few days). Routing and the firewall are still functional, since I don't lose internet, but I can't get to the mgmt page. It also stops sending data to Prometheus. I have tried 2 different NVMe drives, run badblocks, tried UFS and ZFS, reinstalled 3 times, and I've dug through a ton of logs collected in logstash. I can't figure out what is going on with this thing. I disabled all services other than the basic routing and FW, and that made it last about 5 days, but it still stops responding. It seems like the more additional services that are running, the shorter the period. Other than rebooting it every night, what are my other troubleshooting options?
I am a Linux sysadmin, but I don't have much BSD experience.

Thanks in advance  :-\

[FullyBorked]

Just a guess, maybe bad ram?  Try running a long memtest and see what results you get.  Maybe more services pushes some services into a bad part of ram fewer services it manages to dodge that particular bad sport? 

[mezzfit]

I ran a full system test, including extended RAM, during the last reinstall. I do work at a college, so I could borrow a replacement set of sticks from the workstation guys, and swap them out as another test I guess, but I would think a memtest would throw an error. Temps and cpu usage are fine as well. It ran fine from about 8am to 1pm today.

[FullyBorked]

When you say system test do you mean the build in dell test?  If so I'd run memtestx86 vs dell's test, plus different tests can sometimes yield different results. Ram swap would be quicker and easier though frankly and would be less down time.

This def feels like a hardware issue though.  Either something is bad or incompatible.  In my humble opinion.

[mezzfit]

Yeah I just did the builtin test. I remembered this was happening before I added the 2nd memory stick, so I pulled the original one to see if it happens again.
https://bsd-hardware.info/?probe=85256a78f6
Here's a hardware dump of the device. BIOS/UEFI is all the way up to date. This was an issue before I installed the RTL8125 interface.

[FullyBorked]

Yeah I just did the builtin test. I remembered this was happening before I added the 2nd memory stick, so I pulled the original one to see if it happens again.
https://bsd-hardware.info/?probe=85256a78f6
Here's a hardware dump of the device. BIOS/UEFI is all the way up to date. This was an issue before I installed the RTL8125 interface.

Hmm, assume the other Realtek interface is the onboard nic?

You could also try the Realtek driver plugin "os-realtek-re" to see if maybe you have some driver issue? 

[mezzfit]

Yes that other Realtek is the onboard. I used the 'realtek-re-kmod-198.00_1' package for the M.2 adapter that I added. I just looked and saw that other realtek plugin, so I'll add that(and update to 23.1.7_3 I guess haha).

[mezzfit]

Well, I tried the os-realtek-re plugin, and different RAM and it still bites the dust. Maybe it's the chipset, maybe it's the NIC drivers, but I'm not sure. It's still crashing. This exact hardware combo just might not be compatible.

[JohnBailey]

I ran a full system test, including extended RAM, during the last reinstall. I do work at a college, so I could borrow a replacement set of sticks from the workstation guys, and swap them out as another test I guess, but I would think a memtest would throw an error. Temps and cpu usage are fine as well. It ran fine from about 8am to 1pm today.

Thank you for sharing! Studying in college opens up a world of opportunities for personal and academic growth. It allows us to delve deep into our chosen fields, explore diverse subjects, and acquire valuable skills. While the journey may come with its fair share of challenges, the rewards are worth it. One of the challenges students often face is completing complex assignments, such as Java programming tasks. That's where Edubirdie's Java assignment assistance comes in handy. With Edubirdie's expert guidance, students can overcome difficulties, grasp intricate Java concepts, and excel in their coursework. Edubirdie's reliable support ensures that students receive top-notch assignments that demonstrate their understanding and proficiency. By combining the benefits of college education with Edubirdie's (https://essays.edubirdie.com/java-assignment) assistance, students can thrive in their academic journey and pave the way for a successful future in the field of Java programming.