Syslogd segmentation fault after upgrade to 21.7.1

Started by yomeyo, August 27, 2021, 05:28:55 PM

Previous topic - Next topic
I have 3 OPNsense installations. 2 of 3 updated from 21.1 to 21.7 without any problem. The last one I stumbled upon a problem with syslogd not starting. syslog-ng DOES start without any problems.

Logging does not work (no new logs in /var/log).
The daemon won't start using the GUI (no response).
I tried starting the daemon manually:

dmesg shows:
Quote-> pid: 60070 ppid: 88710 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
pid 60070 (syslogd), jid 0, uid 0: exited on signal 11 (core dumped)

When starting syslogd manually:
Quote# /usr/local/sbin/syslogd -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf   
syslogd: child pid 33106 exited on signal 11 (core dumped)

When starting syslogd in debug mode:

Quote/usr/local/sbin/syslogd -d -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf

The following error is shown at the end:

Quote# /usr/local/sbin/syslogd -d -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -S /var/run/legacy_logpriv -k -s -s -f /var/etc/syslog.conf
Trying peer: /var/run/legacy_log
new socket fd is 6
listening on socket
sending on socket
Trying peer: /var/run/legacy_logpriv
new socket fd is 7
listening on socket
sending on socket
off & running....
init
loading timezone data via tzset()
cfline("*.*                %/var/log/audit.log", f, "audit", "*")
cfline("*.*                %/var/log/configd.log", f, "configd.py", "*")
cfline("*.*                %/var/log/dhcpd.log", f, "dhcpd,dhcrelay", "*")
cfline("*.*                %/var/log/filter.log", f, "filterlog", "*")
cfline("*.*                %/var/log/gateways.log", f, "dpinger", "*")
cfline("*.*                %/var/log/lighttpd.log", f, "lighttpd", "*")
cfline("*.*                %/var/log/pkg.log", f, "pkg,pkg-static", "*")
cfline("*.*                %/var/log/portalauth.log", f, "captiveportal", "*")
cfline("*.*                %/var/log/ppps.log", f, "ppp", "*")
cfline("*.*                %/var/log/resolver.log", f, "unbound", "*")
cfline("*.*                %/var/log/routing.log", f, "radvd,routed,rtsold,olsrd,zebra,ospfd,bgpd,miniupnpd", "*")
cfline("*.*                %/var/log/wireless.log", f, "hostapd", "*")
cfline("*.*                %/var/log/dnsmasq.log", f, "dnsmasq", "*")
cfline("*.*                %/var/log/ipsec.log", f, "charon", "*")
cfline("*.*                %/var/log/ntpd.log", f, "ntp,ntpd,ntpdate", "*")
cfline("*.*                %/var/log/openvpn.log", f, "openvpn", "*")
cfline("*.*                %/var/log/squid.log", f, "(squid-1)", "*")
cfline("*.*                %/var/log/suricata.log", f, "suricata", "*")
cfline("local3.*                                                        %/var/log/vpn.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("local4.*                                                        %/var/log/portalauth.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("local7.*                                                        %/var/log/dhcpd.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
cfline("*.notice;kern.debug;lpr.info;mail.crit;daemon.none              %/var/log/system.log", f, "-(squid-1),audit,bgpd,captiveportal,charon,configd.py,dhcpd,dhcrelay,dnsmasq,dpinger,filterlog,hostapd,lighttpd,miniupnpd,ntp,ntpd,ntpdate,olsrd,openvpn,ospfd,pkg,pkg-static,ppp,radvd,routed,rtsold,suricata,unbound,zebra", "*")
Segmentation fault (core dumped)


The core is dumped to /syslogd.core
I've tried to analyse this with gdb:

Quote(gdb) core syslogd.core
[New LWP 100190]
Core was generated by `/usr/local/sbin/syslogd -s -c -c -P /var/run/syslog.pid -p /var/run/legacy_log -'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000209f0f in ?? ()
(gdb) bt
#0  0x0000000000209f0f in ?? ()
#1  0x0000011893eab020 in ?? ()
#2  0x0000011893eab039 in ?? ()
#3  0x0000011893eab000 in ?? ()
#4  0x0000000000000000 in ?? ()

How to analyse this further?
Thanks.

At first you should do a health check of the affected installation and additionally compare the checksums of the syslogd binaries (e.g. sha1).
OPNsense 24.7.11_2-amd64