WebGui/SSH dies randomly after a few days

[ProximusAl]

Afternoon all,

I have 2 identical hardware devices - The R86S for those that may wish to know.

Both devices are running OPNSense 23.7.1_3, and run together side by side as separate routers on separate ISP's.

Both devices are configured as routing only devices, in that it has a WAN/LAN and a management port, on separate leased lines.

On one of these devices only, after a certain period of time, both the WebGUI and SSH become inaccessible from the management LAN.
Traffic continues to flow through the device normally when this happens..(Thank goodness!)

I suspect this is something to do with certificates, because both port 443 and 22 are open, just wont talk.

On the problem device, I have even replaced it with 2 other spare R86S devices (restoring config) and the same thing happens after a while. (Talking days rather than hours)

I put a port watcher on 22 with PRTG, but it's always green because the port is always up, just no access to SSH/WebGUI.

Unfortunately, I cant get to the console, because OPNSense/FreeBSD doesn't like you adding an HDMI monitor after it's booted, so it's going to be a power pull

I've done this before (ZFS thankfully), and OPNSense boots back up nicely, and management is back.....for a while.

Both devices are using identical certificates (it's a wildcard, so identical on both)

For now, my plan is to get this back up and running, and change the WebGUI port to httprather than https, leaving SSH running.
(I know, I know...naughty...but WebGUI and SSH are only available on my management LAN)
In this configuration, I can try to prove this is some kind of SSL issue.

I was just wondering, if anyone had any insight as to why this may be happening.
I've Google'd to the most of my ability and found various posts on the WebGUI and SSH dying, but nothing concrete as to *why* it's happening.

Thanks for taking the time to read, and any insight you may provide.

[franco]

Perhaps https://docs.opnsense.org/manual/settingsmenu.html#listen-interfaces


Cheers,
Franco

[ProximusAl]

Hi Franco,

I did consider that, as I do specifically bind to a certain interface for management, but the sister device is the same and it has no issues.

In fact, I actually took the config from the sister device and just modified the IP's after restore, as config is identical between two devices.

If you are saying this is likely (even though the ports *are* open on the interface) then it is something I can try, but annoyingly, this is an out of hours job as it's in production.

When I SSH it, it gives me nothing meaningful, apart from "Connection reset by....."

[franco]

I don't see your config. I can't really help. The guide is very specific on what works and how it's implemented.

Logs also help, checking daemon running and inspecting the configuration for the obvious missing listen address.


Cheers,
Franco

[ProximusAl]

I appreciate your input

I can only get to the logs once I've given it a power cycle, so will do that indeed.

Ideally, I'm hoping by using http, when SSH goes down, that WebGUI is still accessible.

If it is, it's unlikely what you suggested, but if it *also* goes down, then yes, you may be on to something.

Looks like I'm in for the long run as the number of days it goes pop...is random.

Thanks again

[bebef]

That kind of sounds familiar.

Do you see core dumps in dmesg?

[ProximusAl]

Not really seeing anything untoward in the logs at all.

I've been a bit naughty, and pulled the plug....blamed it on the ISP....and moved on.

Got the web management back now, and changed it to http, so now got a PRTG monitor on the http to see when it goes.

Will have to manually check SSH every now and again to see when that breaks.

I've cleared the logs down now so starting from scratch so I hopefully know when it breaks.

[ProximusAl]

Well it's been a month and it is all still working fine.

All I did was change the webGUI from HTTPS to HTTP.

This appears to stop the whole WebGUI and SSH from not connecting issue I have been seeing for months.

SSH is still working fine.

If I could be bothered, I could try WebGUI HTTPS, and SSH off, but as it's working as is, I don't think I'm going to spend any more time diagnosing.  It's almost as if HTTPS and SSH share somehing that breaks over time (OpenSSL?), but anyway just posting in case anyone else has a similar issue to me with a potential workaround.