Hi all,
System Info: OPNsense 18.7.1_3-amd64 on AMD GX-412TC SOC
Currently I'm facing an strange issue with my OPNSense box. A while ago I noticed that my firewall logging is not updating anymore (and so do other logs like System->Log Files->General).
For example if I go to Firewall>Log Files>Live View the last entries there are from March this year - and no current log events are displayed. The firewall itself seems to work just fine by the way.
I also recognized that the syslog service on the dashboard is stopped - if I try to restart the service it runs for a couple of minutes and dies again afterwards. Could this be related?
Any suggests? Can I provide you with additional information?
Cheers,
FillFeile
Are you able to log on to the console and see what's in the /var/log/syslog to see why syslogd is stopping?
Thanks for your fast answer Animosity022!
Actually there is on syslog file under /var/log...
root@OPNsense:/var/log # la
./ flowd.log.000006 pf.today suricata.log.2
../ flowd.log.000007 pf.yesterday suricata.log.3
bsdinstaller/ flowd.log.000008 portalauth.log suricata.log.4
configd.log flowd.log.000009 ppps.log suricata.log.5
dhcpd.log flowd.log.000010 relayd.log suricata.log.6
dmesg.today gateways.log resolver.log suricata.syslog.log
dmesg.yesterday installer/ routing.log system.log
dnsmasq.log ipsec.log setuid.today userlog
filter.log lastlog setuid.yesterday utx.lastlogin
flowd.log lighttpd/ squid/ utx.log
flowd.log.000001 lighttpd.log squid.syslog.log vpn.log
flowd.log.000002 mount.today suricata/ wireless.log
flowd.log.000003 ntp/ suricata.log
flowd.log.000004 ntpd.log suricata.log.0
flowd.log.000005 openvpn.log suricata.log.1
Any ideas?
You should see something probably in /var/log/system.log
I killed syslog and it generated an error:
root@phoenix:/var/log # tail system.log
Aug 28 11:48:59 phoenix syslogd: exiting on signal 15
Sadly I didn't find any message from my syslog deamon in my system.log. Here is the most current content and nothing was added after the daemon died again:
Sep 29 10:59:23 OPNsense configd.py: [32ea2b13-4821-48a8-9d0b-78a481c59ffa] request pfctl byte/packet counters
Sep 29 10:59:30 OPNsense configd.py: [d82f2168-41c7-4f26-b655-4543d180a8ef] request pfctl byte/packet counters
Sep 29 10:59:37 OPNsense configd.py: [7e116d12-1f8e-4377-9d59-a4ca92d14732] request pfctl byte/packet counters
I debugged a little more on my side and made an interesting discovery - after starting the syslog service via the web interface the syslogd process shows up in the process list. Observing the process list for a little while showed be this:
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
3494 root 1 25 0 1051M 602M bo_wwa 3 0:12 8.49% syslogd
3494 root 1 24 0 1051M 760M bo_wwa 3 0:15 9.63% syslogd
3494 root 1 25 0 1051M 843M bo_wwa 3 0:17 9.78% syslogd
3494 root 1 26 0 1051M 929M bo_wwa 0 0:20 11.04% syslogd
3494 root 1 27 0 1051M 1029M bo_wwa 2 0:23 7.54% syslogd
The reserved (I think this is what RES stands for?) memory grows over time and the services dies after hitting the 1051M border.
Any ideas?
RES is how much the process is currently using. From the FreeBSD man page for top:
QuoteBy default, top(1) also displays the amount of memory space taken by the process. This is split into two columns: one for total size and one for resident size. Total size is how much memory the application has needed and the resident size is how much it is actually using now.
Try this.... from the console, it'll delete all the existing log files, but there may be a silly lock on something.
# cd /var/log
find the syslog process ID, e.g.
#ps -auxw | grep syslog
Kill it
# kill -9 {process_ID}
You should now see all the logs being fed to the console.
Delete all the logs
# rm *
Anything still left?
# ls
if yes, then I would go to single user mode and try removing the log file that's left.
Go to the GUI interface and restart the daemon. See if it behaves any better.
BTW, my tops figures for syslog are the same as yours.
@marjohn56 you made my day! Clearing /var/log an restarting the daemon did the trick.
syslogd is not eating up the RAM anymore and the firewall live logging works again.
Thank you some much! ;)