Messages

2) Does reinstalling the firewall [...] change SSH keys?

Yes. This is expected. The keys are generated when a host first boots after installation. Remove the old key(s) from your ~/.ssh/known_hosts and acknowledge the new one(s).

Thanks for confirming. Would SSH Keys be retained from the config file if I imported it during the Firewall install itself rather than in Web GUI after the installation ? I read somewhere on this forum that apparently the config file should retain the old keys if you import it that way but not sure if something changed with OPNSense over the years and its different now.

Hi,

I have nuked old installation of my OPNSense, did a new install then proceeded to do the initial configuration via the web GUI followed by restoring the old config via the web browser in the said web gui. I then tried SSHing into my firewall and was greeted with this error:

Code Select Expand

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@  WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!  @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
ECDSA Key here
Please contact your system administrator.
Add correct key in system path here/known_hosts to get rid of this message.
Offending ECDSA key in system path here/known_hosts:1
remove with:
command to remove it
ECDSA host key for IP Here has changed and you have requested strict checking.
Host key verification failed.

So my questions are:

1) Does major upgrade of the Firewall from one version to another major one could cause to rotate SSH keys ?
2) Does reinstalling the firewall, doing the initial setup in Web GUI followed by restore of the configuration file via web gui would change SSH keys ?

I'm wondering if something nefarious is happening on my firewall as reading online, restoring config file to OPNSense should still retain the old SSH Keys but it this case this haven't happened.

Thanks

There are plenty of options :

- ZenArmor
- Suricata
- Pi-Hole

The first two are full IDS/IPS solutions and the last one is a DNS Blocklist based system which you can combine with this : https://forum.opnsense.org/index.php?topic=9245.0

I would say install a VM for each and have a look around in their webGUI :)

I took a look at the link you provided but the guide is broken as the images are not available anymore. So, to use Pi-Hole, I need to make additional changes within OPNSense while with the first two solutions (Zenarmor and Suricata) I don't need to make much of adjustments when it comes to DNS within OPNSense ?

Hi,

I'm looking to see what are the options to detect and block malicious IPs and Domains that made inbound \ outbound connections per individual device on the network. This is to establish if a device was compromised in some way and to detect and block malicious connection(s) taking place. When it comes to features, what I would like to see is:

• Log IPs \ Domains per device with a timestamp upon malicious connection and when it was blocked.
• Archive logs of IPs \ Domains within selected amount of time for detection and blocking in the future.
• Dashboard with data and metrics.
• Highlight connections to suspicious \ malicious domains in the dashboard.
• Search functionality where I can manually search if a specific IP \ Domain made a connection on my network.
• Automatically grab fresh feeds of data to keep database of malicious domains and IPs up to date.
• Store logs up to selected amount of months or years e.g 3 months or 1 year as an example.
• Automatically block connections to malicious domains \ IPs that are on the downloaded data feeds.

I'm thinking about buying external hard drive that I would connect via USB port to store logs so storage is not a problem. However, my hardware is relatively weak with 4 CPU cores and 8GB of DDR4 RAM. I'm looking for something more automated where I set it up and it just works or I can occasionally do maintenance on it to review blocked domains and ips. My initial plan was to just monitor if a malicious connection took place but automatically blocking it would make things much easier. I'm looking for a solution that is aimed at home usage. I don't mind paying a small monthly fee if the solution does what I need with all the required features and very up to date data feed, however, I would prefer something free.

Any suggestions how I can go about it and what are my options ?

1. Schedule regular ZFS pool scrubs.

2. Follow this procedure of mine to get SMART data into a tool named Scrutiny:

https://forum.opnsense.org/index.php?topic=48101.msg242617#msg242617

HTH,
Patrick

Thanks Patrick, I will look into this as it's well needed.

Hi,

Does anyone know if there is a plugin\built in utility that would continuously check SSDs\Hard Drives used in the firewall for S.M.A.R.T status or bad sectors, so once a malfunction is detected it would notify you in the web panel about it ? I'm asking because it would be hard to just power off the firewall just to check with a 3rd party tools to see if the drive\ssd is faulty or healthy.

Thanks

I have only ever seen these:

Code Select Expand

ahcicho 0: Timeout on slot 7 port 0
CAM Status: Command Timeout
Retrying command, 2 more tries remain
with dying devices. If I saw that in a new unit I would never put that into production before I had successfully eliminated the cause. Timeouts in the CAM subsystem must not happen. If they do, something is broken. Never ignore them.

What do you mean by "vanilla" and "backwards compatible"? Save the configuration from your current unit, fix the hardware, install the very same version, restore configuration ...

Thanks, good to know what the mentioned error codes mean, I guess you really do learn something new every day haha. Once I replace the drive, I will check dmesg again but hopefully it will be fine. I must have gotten a unit with a faulty SSD and haven't realised it, it's been running fine for a long time but it became more severe recently. Thanks for helping me with the diagnosis of the issue, it's very much appreciated.

The firmware health audit can probably confirm?

Not sure if there is one, I already ordered a replacement so will replace and see how it goes. Hopefully the reboot will be instant compared to the current one where it takes quite a while to reboot.

This thread can be closed now, thanks all for your help.

Your SSD/disk is dying. Save a configuration while you still can, replace the drive, reinstall, restore configuration.

Which part makes it obvious ? It's interesting because I always got the following errors (ahcicho 0: Timeout on slot 28 port 0, CAM Status: Command Timeout , Retrying command, 3 more tries remain) since I got the box and what it only caused was slower reboot time trying to do TRIMs I think which some SSDs don't support so I presumed my SSD didn't support that feature and only recently started causing issues, hopefully it's my SSD and nothing else.

Do you know if the vanilla configuration file is backwards compatible with older versions of OPNsense ?

You type D M E S G followed by the ENTER key after logging in to the firewall via SSH or connected to the console and selecting "8" for a shell. All lower case letters.

I tried doing that and it only shows me the logs after the reboot took place I think.

Here are the errors I get when I view dmesg:

Code Select Expand

WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0.

acpi0: Power Button (fixed)
Unknown: I/O range not supported
cpu0: <ACPI CPU> on acpi0

atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: Warning: Couldn't map I/O

ns8250: UART FCR is broken (This one appears 3 times)

ahcicho 0: Timeout on slot 28 port 0
CAM Status: Command Timeout
Retrying command, 3 more tries remain

ahcicho 0: Timeout on slot 7 port 0
CAM Status: Command Timeout
Retrying command, 2 more tries remain

ahcicho 0: Timeout on slot 12 port 0
CAM Status: Command Timeout
Retrying command, 1 more tries remain

ahcicho 0: Timeout on slot 17 port 0
CAM Status: Command Timeout
Retrying command, 0 more tries remain

ahcicho 0: Timeout on slot 22 port 0
CAM Status: Command Timeout
Error 5, Retries exhausted


You type D M E S G followed by the ENTER key after logging in to the firewall via SSH or connected to the console and selecting "8" for a shell. All lower case letters.

Is there a way to do this from the web interface ?

This screams "hardware problem". Either memory, CPU, overheating, ... or a dying SSD/disk.

What does "dmesg" output?

What makes it obvious it's a hardware problem ? Is it the "b'Bus error'" error ?
How can I check dmesg output ?

[General Logs:]

Hi,

My firewall not long ago started throwing minor erros from time to time when I have been updating it and it usually gave me the following red message on the screen: "Danger, unexpected error, check log for details." followed by the successful update with no errors even after I refreshed the page during the update. However, this time it got worse and I'm starting to worry about my firewall and its state to the point where I'm wondering if I should just re-install it alltogether, on the surface everything works fine as I checked a few things but still what happened is seriously worrying me.

When I started the update process this time around, I got the same red error message as before: "Danger, unexpected error, check log for details." with the following action still taking place for a very long time:

[19/93] Extracting liblz4-1.10.0_2,1:.

As the update was stuck on the above action taking place and the progress dot not moving at all for a long time, I then decided to refresh the web page in the browser, which at first gave me 403 Access Forbidden followed by 404 Not Found errors when I was refreshing the web page like a mad man to check if the update killed my firewall entirely. The mentioned error codes worried me that my internal devices got exposed to the internet due to the failure, also I'm not quite sure how OPNsense handles failures when something malfunctions. Once the firewall became online as I presume it rebooted, I checked a few things and everything runs fine so I don't see any issues.

During the time when the above issues with errors took place, I managed to note down the time frame of it taking place and upon checking logs on the firewall, I got the following errors, many of which were repeated in the logs at least a few times:

General Logs:

Code Select Expand


Notice kernel <6>[9338999] pid 62018 (sh), jid 0, uid 0: exited on signal 10 (no core dump - bad address)
Error opnsense /usr/local/sbin/pluginctl: The command `/sbin/ifconfig -Lmv ` failed to execute ()
Error php #0 /usr/local/opnsense/scripts/health/library/OPNsense/RRD/Factory.php(123): ReflectionClass - >_construct('\\OPNsense\\RRD\\T...')
Error php Stack trace:
Error php Error instatiating Traffic [ReflectionException: Class "\OPNsense\RRD\Types\Traffic" does not exist in /usr/local/opnsense/scripts/health/library/OPNsense/RRD/Factory.php:123
Error php #2 {main}]
Error php #1 /usr/local/opnsense/scripts/health/updaterrd.php(64): OPNsense\RRD\Factory->updateAll(false)

Backend Logs:

Code Select Expand

Error configd.py [redacted-id] Script action stderr returned "b'Bus error'"
Error configd.py [redacted-id] Script action failed with Command '/usr/local/opnsense/scripts/routes/gateway_status.php' died with <Signals.SIGBUS: 10>. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/actions/script_output.py", line 89, in execute subprocess.run(script_command, env=self.config_environment,shell=True, File "/usr/local/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args,subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/routes/gateway_status.php' died with <Signals.SIGBUS: 10>.

Is there anything in the above errors that I should worry about ? I don't really understand why my firewall is throwing all the errors, my setup is fairly basic and vanilla with no additional applications or modifications outside of what the web interface offers.