OPNsense Forum

English Forums => 24.7, 24.10 Series => Topic started by: gothbert on August 22, 2024, 01:21:04 PM

Title: [Solved] OPNsense suddenly unreachable - How to debug (other issue)
Post by: gothbert on August 22, 2024, 01:21:04 PM
Dear all,

it randomly occurs that my OPNSense is suddenly not reachable any more. I am currently (still) on OPNsense 24.7.1-amd64 but had these issues say once every quarter for long.

It just occured ten minutes ago. I had to power the device down and back on. I read the thread with the same subject from today but it did not help. There is only the latest log showing entries since the last boot at 13:03 in /var/log/system. What I need are the logs from the previous run. Where have they gone? System | Settings | Logging, Tab "Local" is set to keep 31 log files.

Any help would be appreciated.

Best regards,
Boris
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: doktornotor on August 22, 2024, 01:43:49 PM
They have gone nowhere. They are in /var/log. Rotated and retained according to your settings.


# ls -la /var/log/system/
total 405
drwx------   2 root wheel     34 Aug 22 13:01 .
drwxr-xr-x  27 root wheel     42 Aug 20 08:26 ..
lrwxr-x---   1 root wheel     35 Aug 22 13:01 latest.log -> /var/log/system/system_20240822.log
-rw-------   1 root wheel  16053 Jul 23 23:37 system_20240723.log
-rw-------   1 root wheel  15880 Jul 24 23:37 system_20240724.log
-rw-------   1 root wheel  16512 Jul 25 23:37 system_20240725.log
-rw-------   1 root wheel  17310 Jul 26 23:37 system_20240726.log
-rw-------   1 root wheel  18025 Jul 27 23:37 system_20240727.log
-rw-------   1 root wheel  19380 Jul 28 23:37 system_20240728.log
-rw-------   1 root wheel  23648 Jul 29 23:39 system_20240729.log
-rw-------   1 root wheel  17510 Jul 30 23:37 system_20240730.log
-rw-------   1 root wheel  15723 Jul 31 23:37 system_20240731.log
-rw-------   1 root wheel  16634 Aug  1 23:37 system_20240801.log
-rw-------   1 root wheel  16059 Aug  2 23:37 system_20240802.log
-rw-------   1 root wheel  19690 Aug  3 23:37 system_20240803.log
-rw-------   1 root wheel  20682 Aug  4 23:49 system_20240804.log
-rw-------   1 root wheel  19866 Aug  5 23:37 system_20240805.log
-rw-------   1 root wheel  15168 Aug  6 23:37 system_20240806.log
-rw-------   1 root wheel  16392 Aug  7 23:37 system_20240807.log
-rw-------   1 root wheel  19368 Aug  8 23:37 system_20240808.log
-rw-------   1 root wheel  17314 Aug  9 23:41 system_20240809.log
-rw-------   1 root wheel 397859 Aug 10 23:45 system_20240810.log
-rw-------   1 root wheel 131689 Aug 11 23:46 system_20240811.log
-rw-------   1 root wheel  19785 Aug 12 23:57 system_20240812.log
-rw-------   1 root wheel  15010 Aug 13 23:46 system_20240813.log
-rw-------   1 root wheel 190857 Aug 14 23:35 system_20240814.log
-rw-------   1 root wheel  15588 Aug 15 23:35 system_20240815.log
-rw-------   1 root wheel  14807 Aug 16 23:35 system_20240816.log
-rw-------   1 root wheel 133861 Aug 17 23:47 system_20240817.log
-rw-------   1 root wheel  92127 Aug 18 23:58 system_20240818.log
-rw-------   1 root wheel  18280 Aug 19 23:42 system_20240819.log
-rw-------   1 root wheel  15359 Aug 20 23:42 system_20240820.log
-rw-------   1 root wheel  15168 Aug 21 23:42 system_20240821.log
-rw-------   1 root wheel 173904 Aug 22 13:28 system_20240822.log



As for the various question on how to filter / parse them, really do not have time for this. See the grep manpage (https://man.freebsd.org/cgi/man.cgi?query=egrep&apropos=0&sektion=0&manpath=FreeBSD+14.1-RELEASE+and+Ports&arch=default&format=html).

Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: gothbert on August 22, 2024, 02:02:37 PM
Actually, they are not there. That's why I am asking. What you showed is what I expected but the directory only contains the current log file:

# ls -la /var/log/system/
total 61
drwx------   2 root wheel   128 Aug 22 14:01 .
drwxr-xr-x  12 root wheel   960 Aug 22 13:10 ..
lrwxr-x---   1 root wheel    35 Aug 22 14:01 latest.log -> /var/log/system/system_20240822.log
-rw-------   1 root wheel 61245 Aug 22 13:35 system_20240822.log

Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: gothbert on August 22, 2024, 02:04:23 PM
It just dawned to me: /var/log is on tmpfs and does not survive a power off/power on cycle.
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: doktornotor on August 22, 2024, 02:08:07 PM
Well, that won't work. You can either disable that feature (that should've been gone years ago IMO), or set up a syslog server somewhere to forward the logs to.
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: chemlud on August 22, 2024, 02:20:06 PM
System -> Settings -> Logging

Have you enabled local logging? You can choose the number of days to be saved...
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: doktornotor on August 22, 2024, 02:21:25 PM
Quote from: chemlud on August 22, 2024, 02:20:06 PM
Have you enabled local logging? You can choose the number of days to be saved...

He's got them on ramdisk (tmpfs). That is not very persistent on crash.
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: chemlud on August 22, 2024, 02:24:08 PM
That would be

System -> Settings -> Misc -> Section "Disk Memory Settings"

I think this is not enabled as tmpfs by default, or?
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: doktornotor on August 22, 2024, 02:34:06 PM
Quote from: chemlud on August 22, 2024, 02:24:08 PM
I think this is not enabled as tmpfs by default, or?

Definitely not.
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: gothbert on August 22, 2024, 03:26:29 PM
I must have enabled tmpfs for logging years ago to reduce the wearout on the SSD. I have disabled it under System | Settings | Miscellaneous. It requires a reboot, though, to become effective.
Title: Re: OPNsense suddenly unreachable - How to debug (other issue)
Post by: chemlud on August 22, 2024, 03:36:06 PM
Quote from: gothbert on August 22, 2024, 03:26:29 PM
I must have enabled tmpfs for logging years ago to reduce the wearout on the SSD. I have disabled it under System | Settings | Miscellaneous. It requires a reboot, though, to become effective.

...which makes perfectly sense to me, though...