Can't access / reboot opnsense remotely

[Olli]

Hi there,
i've an opnsense at home but it seems, that it hung up. It's the first time this happend and i cant access via gui (503 Service Unavailable) and ssh (connection reset by peer).
I have access to the lokal network with a site2site vpn config, but i cannot connect to all devices in the network. It seems that routing / dns / network management in the opnsense hung up and its not possible for me to reboot the device until next week physically. Also no internet services got resolved by trying to ping it out of that network.
But i need to get the location working.

Has someone an idea to get access or reboot that opnsense remotely, like a tool, an inproper action to get that device make a reboot? The firmware Version was updated at 10th of july.
Zenarmor is not responding via zenconsole.

For the system check and the "why" of that behavior i have to analyse the logfiles, but now i want only that reboot. yesterday all was workin fine.

Thanks, Olli

[cookiemonster]

Remote access must be setup in advance for what I hope are obvious security reasons. So no, if a firewall is doing its job properly, nobody should be able to administer it remotely without a previous secure setup.

[Olli]

True, but I had a setup including vpn and remote access from the two separate vpn. And I have access to some network devices. But now something happend to the opnsense, so that I cannot connect to GUI and ssh since today and the opnvpn did not work, only my site2site vpn is still connected.

So, how I can restart the machine without physical access

[cookiemonster]

Even the VPN needs to be setup in advance to be able to reach the firewall itself, otherwise only the tunnel.
So if that's failed, you will need another failsafe.

[Olli]

But it was setup that I can reach the firewall via vpn, via site2site and locally.
And something must hang up or going wrong that today nothing work correctly anymore. And I'm looking for a solution, hoping the insect tunnel stay active. So how I access ssh when everything worked before today and nobody changed anything because I'm on vacation...
I did not understand that behavior, since three years I had not such problems.

[cookiemonster]

Sorry, I have no suggestions especially if no ssh access.

[Patrick M. Hausen]

Have someone power cycle it. Only option I can think of. In a multi-tennant rental house, someone could pull the fuse to your entire appartment if absolutely necessary.

[Olli]

@cookiemonster thanks for the try.

@patrick Thought of this, but the fuses are in the appartment and the usv will cover minimum half an hour. so the whole building without power is not an option for resetore my access

There seems to be no possibility, so I have to wait and see until i'm back at this location

[cookiemonster]

For something like this I've thought many times of getting one of those tinypilot units but they are so expensive (for me). I always end up leaving it in the cart. Maybe do a pikvm as a little project when I can spare the time.
Edit: it wouldn't help from outside but I have another WAN I'd be looking a making a separate vpn with.

[ProximusAl]

I've literally just received the TinyPilot Voyager 2a.

I agree, it is expensive, but I honestly dont know how I've gone without it.

I do a lot of headless systems and it is a breeze.

Ordered it from punkt.de

[Patrick M. Hausen]

Most firewall appliances have serial consoles and many server systems still have a serial port. If you have a two node HA cluster or two different uplinks with two different firewalls, cross-connect these. All you need to initiate a serial terminal session is in FreeBSD base and so on OPNsense:

Code: [Select]

cu -s 115200 -l /dev/<port device>

[cookiemonster]

thanks Patrick. My previous appliance did have a serial port. The new one doesn't.

[Olli]

So, i'm back with local access to the opnsense.
Thanks Patrick, i will think about it.

Elasticsearch had 270% CPU Usage
After reboot:
98301 root          1 103    0    58M    35M CPU3     3   7:06  99.59% python3.11
Java with 23 %
still running to hot.

In the system log on the crash day:
A lot of:
<13>1 2024-07-18T15:36:46+02:00 Sense.home kernel - - [meta sequenceId="35"] kern.maxfiles limit exceeded by uid 965, (java) please see tuning(7).

<13>1 2024-07-18T15:39:43+02:00 Sense.home kernel - - [meta sequenceId="214"] <7>sonewconn: pcb 0xfffff801d4c8f000 ([::7f00:1]:9200 (proto 6)): Listen
 queue overflow: 193 already in queue awaiting acceptance (81 occurrences

[Patrick M. Hausen]

Elasticsearch is an absolute ressource hog. Don't run on the firewall if not absolutely necessary. Use a separate host or VM.

[Olli]

but i had two years no problems with it and opnsense should have enough capacity on the system, i thought.
16GB Ram, Intel i5-7200U
Is there an better alternative to elastic? ho can i prevent such issues? I plan to reinstall the sense, becaue i think the system had the issue with the last 24.1 release.
Another error i found was in unbound logfiles:
[41432:1] error: can't create socket: Too many open files in system