Messages

I have 3 OPNsense installations. 2 of 3 updated from 21.1 to 21.7 without any problem. The last one I stumbled upon a problem with syslogd not starting. syslog-ng DOES start without any problems.

Logging does not work (no new logs in /var/log).
The daemon won't start using the GUI (no response).
I tried starting the daemon manually:

dmesg shows:

-> pid: 60070 ppid: 88710 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
pid 60070 (syslogd), jid 0, uid 0: exited on signal 11 (core dumped)

When starting syslogd manually:

# /usr/local/sbin/syslogd -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf   
syslogd: child pid 33106 exited on signal 11 (core dumped)

When starting syslogd in debug mode:

/usr/local/sbin/syslogd -d -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf

The following error is shown at the end:

# /usr/local/sbin/syslogd -d -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf
Trying peer: /var/run/legacy_log
new socket fd is 6
listening on socket
sending on socket
Trying peer: /var/run/legacy_logpriv
new socket fd is 7
listening on socket
sending on socket
off & running....
init
loading timezone data via tzset()
cfline("*.*                %/var/log/audit.log", f, "audit", "*")
cfline("*.*                %/var/log/configd.log", f, "configd.py", "*")
cfline("*.*                %/var/log/dhcpd.log", f, "dhcpd,dhcrelay", "*")
cfline("*.*                %/var/log/filter.log", f, "filterlog", "*")
cfline("*.*                %/var/log/gateways.log", f, "dpinger", "*")
cfline("*.*                %/var/log/lighttpd.log", f, "lighttpd", "*")
cfline("*.*                %/var/log/pkg.log", f, "pkg,pkg-static", "*")
cfline("*.*                %/var/log/portalauth.log", f, "captiveportal", "*")
cfline("*.*                %/var/log/ppps.log", f, "ppp", "*")
cfline("*.*                %/var/log/resolver.log", f, "unbound", "*")
cfline("*.*                %/var/log/routing.log", f, "radvd,routed,rtsold,olsrd,zebra,ospfd,bgpd,miniupnpd", "*")
cfline("*.*                %/var/log/wireless.log", f, "hostapd", "*")
cfline("*.*                %/var/log/dnsmasq.log", f, "dnsmasq", "*")
cfline("*.*                %/var/log/ipsec.log", f, "charon", "*")
cfline("*.*                %/var/log/ntpd.log", f, "ntp,ntpd,ntpdate", "*")
cfline("*.*                %/var/log/openvpn.log", f, "openvpn", "*")
cfline("*.*                %/var/log/squid.log", f, "(squid-1)", "*")
cfline("*.*                %/var/log/suricata.log", f, "suricata", "*")
cfline("local3.*                                                        %/var/log/vpn.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("local4.*                                                        %/var/log/portalauth.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("local7.*                                                        %/var/log/dhcpd.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("*.notice;kern.debug;lpr.info;mail.crit;daemon.none              %/var/log/system.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
Segmentation fault (core dumped)

The core is dumped to /syslogd.core
I've tried to analyse this with gdb:

(gdb) core syslogd.core
[New LWP 100190]
Core was generated by `/usr/local/sbin/syslogd -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000209f0f in ?? ()
(gdb) bt
#0  0x0000000000209f0f in ?? ()
#1  0x0000011893eab020 in ?? ()
#2  0x0000011893eab039 in ?? ()
#3  0x0000011893eab000 in ?? ()
#4  0x0000000000000000 in ?? ()

How to analyse this further?
Thanks.

Ok, that could be a problem to find the issue then. My first thought was a proxy blacklist or an IDS signature update using the full memory is responsible for taking a large amount of memory.

We don't use any blacklists or IDS functionality.

Has this to do with an action like a cron job?

We have no custom cron jobs configured. bsnmp is killed at different times (00:10, 07:00...).

We experience the same problem on an OPNsense HA pair. SNMP is crashing every couple of days on both nodes. Graphs show memory spike, then bsnmp daemon gets killed. Log shows:

Code Select Expand

pid 18770 (bsnmpd), uid 0, was killed: out of swap space

We already upped memory from 1GB to 2GB, but this does not help. There is no swap space avaialable:

Code Select Expand

root@opnsense:~ # swapinfo
Device          1K-blocks     Used    Avail Capacity
Any advice?

Yes, but how to update if the LAN is unplugged? If I replug the LAN it will switch back to being the active node.

Hi Bart, thanks for the reply. Do you know how to failover to the other device other than unplugging or shutting down the active node?

Thanks.

Does anyone have experience with this? Thanks.

Hi,

I currently run 2 OPNsense 16.7.0 instances in active/standby using CARP and pfSync. There are updates available. Is there any best practices method available for updating a HA setup of OPNsense?