Hello!
My OPNSense instance is half-crashed...? I noticed my HomeAssistant plugin stopped working, and when I visit the OPNSense webgui, only the Announcements Widget on the Lobby: Dashboard is working. Everything else spins/waits for data, then gives up with "Failed to load widget".
I am pretty sure I am on the latest firmware, though I dont see anywhere in the only-mostly-functional webgui to confirm that. (Home Assistant still shows OPNSense as 24.7.11_2 firmware even though the plugin is no longer working)
When I visit the plugins page, I see "os-ddclient (missing)" with a bunch of N/A's. If I try to install the missing plugin, I again jump to the Updates page where the circle spins but nothing really happens. If I try the "automatic resolver", or the "reset all conflicts" options, again I wind up at the Updates page where the circle spins but nothing really happens. I suspect this is for my use of duckdns, but dont know why it's suddenly missing. It's been working fine for years.
The only way to get anything interesting to happen is to hit the "Check for updates" button on the Status page. That bounces me to the Updates page, but I eventually get "No previous action log found" and nothing else happens.
The webgui works well enough for me to turn SSH back on (i dont keep it enabled), but ssh connection attempts time out.
I was able to pull a configuration backup and am wondering whether I should just restore from backup at this point? Any other ideas/suggestions?
Using your backup might as well be a fallback in case you can't fix your install.
I'd start with interface diagnostics (checking WAN, LAN, DNS...). Then look at the FW logs to see why SSH is blocked and fix that.
I am open to attempting diagnostics, but my GUI doesn't seem capable. Nothing under Interfaces/Diagnostics gives me any interesting information. They all seem to go to "No results found!" (I checked ARP Table, got no results. DNS Lookup doesn't do anything. NDP Table says no results. Netstat all 6 options are all blank. I tried a Trace Route and just got a blank response.) I am open to alternative suggestions for further diagnosis.
If I go with the 'fallback' restore from backup, what do I lose? I have Wireguard settings, DuckDNS tracking my public IP, and some static IP addresses. All of that would come back with a restore from backup, right?
Broken disk/SSD and consequently corrupt installation? If you can log in via SSH, does "dmesg" show anything helpful?
Per for to last paragraph in initial message, the OP has no SSH access.
I'm actually a bit confused about state, because the OP claims diagnostics are all broken, yet he seems to have enough connectivity to access the GUI...
Console access with screen and keyboard?
I haven't taken the time to connect a keyboard/mouse and screen. Should I do so? It's a mild pain to do it. I might sooner restore from backup if the group thinks that will get me functional again.
If you haven't done that, it means you actually accessed the web GUI from another machine.
You have some connectivity IP connectivity, likely enough to make forward progress (e.g. enable SSH, test WAN, fix DNS).
Yes, can confirm. I can log into webgui from a laptop by visiting the 192.168.x.x ip address for the device. But even after doing so, and ensuring that ssh is enabled (which I dont leave on by default), I still cant get into ssh. My attempts to do so just time out.
I relatively-recently used ssh to get into the router to install the home assistant plugin, so I know how to enable it. That plugin seemed to work well, but may well be the underlying cause of this problem. I dont know, that's why I am asking for diagnostic suggestions before doing a restore from backup.
Do the firewall rules on LAN permit SSH access to the OPNsense box? If the PC you are using for UI access is connected to a different interface then the rules on that one? Is SSH listening to the standard port (22)? What is the listen interface set to? Anything but "All (recommended)"?
I don't see anything in my Firewall rules that would prevent me from accessing the box. And the same pc that's reaching it by web is the one trying via ssh.
Beyond that, according to the webgui as of now, ssh is enabled. It's allowed over LAN. It's listening to 22. Password-based login is enabled.
All of which is semi-recently (a month ago or so) arranged so I could ssh in for that home assistant plugin. When that project concluded all I did was disable ssh entirely. Now re-enabling it isn't helping for some reason.
Other suggestions?
You ought to be able to locate the rule that enables SSH (nothing that would prevent it is not good enough when deny all is the default).
For that matter, as you attempt to ssh in, with logging enabled for default rules, you should see a pass or fail in the live view (filter on dst_port = 22 if too noisy).
As Patrick mentioned it, the interface that's relevant is the one the PC is connected to.
The PC is on LAN. Wouldn't the default "LAN to any" rule allow the ssh?
Quote from: EricPerl on January 08, 2025, 08:34:45 AMYou ought to be able to locate the rule that enables SSH (nothing that would prevent it is not good enough when deny all is the default).
For that matter, as you attempt to ssh in, with logging enabled for default rules, you should see a pass or fail in the live view (filter on dst_port = 22 if too noisy).
As Patrick mentioned it, the interface that's relevant is the one the PC is connected to.
Can you confirm where to view the live view?
Nevermind, I found Firewall / Log Files / Live View.
But nothing's happening. I have Auto refresh enabled, and I hit the refresh button. Even before applying the filter, I see ZERO activity.
I tried an SSH connection with this page up. The connection timed out, but the log still showed nothing.
I am not seeing anything online that talks about this kind of problem. I'm also not getting much from reddit or this forum. Should I just restore from backup? Anything I should know, like "definitely dont select this option that's going to ensure the problem isn't solved"?
You might need to enable logging for the default pass rules in FW settings.
Do you have a multi-WAN setup?
Did you disable the anti-lockout rules by any chance?
Any port forwarding?
Feel free to use your backup at any time... It's your call.
Do you have a multi-WAN setup? No.
Did you disable the anti-lockout rules by any chance? No.
Any port forwarding? Just one for Plex (not the standard port). I do have a rule for forwarding to nginx, but I keep that disabled. That's what I use the HA plugin for. I can remotely enable that port to use a service when I'm not home (which is rare).
For now my internet and WireGuard are still working, so I don't mind taking some time/effort at diagnostics. But if at a dead end, I am comforted knowing a restore from backup should resolve this.
Anything else you think I should try?
If it were me, I would enable logging on default pass & block rule and verify what happens in live view as you try to ssh.
Not seeing anything implies the request is not reaching the interface.
I only know a few things that cause this:
* port forwarding (precedence over FW, especially if the port forward rule is set to pass).
* gateway specified in FW rule (as in Multi WAN (https://docs.opnsense.org/manual/how-tos/multiwan.html))
If you don't mind me saying, you're going in a tangential direction to the problem.
The symptoms are of a hardware problem and those are hardly found on the GUI. The GUI will at best, show you symptoms. Looking at firewall rules and the like won't give you a hint on any underlying hardware problem. For that, you need to enable ssh for comfort, or attach a keyboard and monitor.
What to look for? That's the thing with PCs and *nix-like OSses. You need to start learning how to diagnose.
Start with dmesg (system buffer). Older logs too, as the buffer starts (latest.log) only from last boot, but previous are alongside it.
Then look around all other related logs.
Quote from: cookiemonster on January 08, 2025, 10:44:15 PMyou're going in a tangential direction to the problem
I tend to agree with this. It doesn't seem like there's some esoteric setting blocking SSH. I think I've had a glitch/failure of some kind. Someone on reddit mentioned the possibility of the SSD being filled up by logs, though none of the attempts to clear logs has helped. Maybe the plugin I installed lately caused the hard drive to fill, etc. Without SSH (or convenient console) access, I am in a severely limited state - to only what the webGUI can do.
Fortunately, it's still FUNCTIONAL, or this failure would have been immediately been met with a reinstall and restore from backup. Strange as it is...
At this point, though, I think it's time to restore from backup and see whether that brings back functionality. I may wait until after upcoming vacation though, in case the restore from backup only makes things worse somehow...
Well today I finally got around to restoring from backup. I had to walk away after clicking "go" but when I got back I saw a message about OPNSense is rebooting. Unfortunately, the dashboard was still broken.
I never actually saw it reboot, so later in the day I use the webGUI to force a reboot. Although I still didn't actually hear the reboot beep, so maybe it wasn't actually rebooting...
So I walked over and pulled the plug. When it came back online, the webGUI was feeling better. I had all my dashboards back. I was able to check for, and perform, available updates. When the updates finished, I did hear the device beep for reboot and got all my dashboards back afterwards.
Situation resolved, I guess. Thanks to everyone that offered suggestions. I will pull another backup now, just in case this is preparatory for catastrophic hardware failure.