OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: KernelKat on August 04, 2020, 10:30:16 pm

Title: Syslog-ng constantly crashing
Post by: KernelKat on August 04, 2020, 10:30:16 pm
This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.
Title: Re: Syslog-ng constantly crashing
Post by: mimugmail on August 05, 2020, 06:05:29 am
Can you Upload them somewhere?
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 05, 2020, 07:07:27 pm
How do I pull the core dumps? I feel so stupid I didn't even think about that.
Title: Re: Syslog-ng constantly crashing
Post by: gpb on August 05, 2020, 08:03:14 pm
Not 100% sure but I have a syslog-ng.core file in /usr/.  Guessing that's it.

EDIT: not "/usr"..."/var/db".
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 05, 2020, 08:44:41 pm
Can you Upload them somewhere?
Uploaded!

Not 100% sure but I have a syslog-ng.core file in /usr/.  Guessing that's it.
Found it. /var/db

https://filebin.net/ew8rz8m7gxkdcf4s




Title: Re: Syslog-ng constantly crashing
Post by: gpb on August 05, 2020, 09:28:06 pm
Sorry 'bout that...I updated my post to reflect that...why I put "usr" I have no idea.  :/
Title: Re: Syslog-ng constantly crashing
Post by: pilotboy72 on August 05, 2020, 09:49:58 pm
This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.


I'm seeing the same thing.  I can't seem to start the syslog-ng service.  This is what I see in the logs when I try to start the service:
kernel: pid 78934 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)


 I have a core file for this attempt to start syslog-ng.  Let me know if there is someplace I should upload it.
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 06, 2020, 02:23:04 am
This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.


I'm seeing the same thing.  I can't seem to start the syslog-ng service.  This is what I see in the logs when I try to start the service:
kernel: pid 78934 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)


 I have a core file for this attempt to start syslog-ng.  Let me know if there is someplace I should upload it.

You can  upload to any online fil host site and then post link here ;)
Title: Re: Syslog-ng constantly crashing
Post by: rackenthogg on August 06, 2020, 08:22:03 am
I have the same problem after upgrade to 20.7. Syslog-ng cannot start, restarts won't help, the error message is always the same:  kernel: pid xxxx (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)

Coredump attached:
https://filebin.net/b08o90rzoo9sfsxk
Title: Re: Syslog-ng constantly crashing
Post by: clopmz on August 06, 2020, 08:22:33 am
Same problem here. Syslog-ng always crashing when is configured to send remote logs.
Title: Re: Syslog-ng constantly crashing
Post by: mfpck on August 06, 2020, 02:36:09 pm
same here - but it also not always is able to even come up during booting...
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 06, 2020, 05:47:44 pm

Keep posting those core dumps all maybe we will find some solutions :)

Has anyone tried removing syslog-ng and then reinstalling the same or different version? I'm not well versed in BSD so I'm still learning.
Title: Re: Syslog-ng constantly crashing
Post by: Taomyn on August 07, 2020, 10:43:24 am
I'm seeing the same, did try to reinstall syslog-ng and it seemed to work:


Quote
2020-08-06T13:26:42   pkg-static[60590]: syslog-ng327 reinstalled: 3.27.1_1 -> 3.27.1_1



But I just needed to reboot the firewall and it syslog-ng crashes on start again:


Quote
2020-08-07T10:32:51   kernel: pid 93417 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)



I'd upload the core dump, but I need it to be secure as a brief look at the file I already see some information I'd rather not see on public servers e.g. FQDN of the firewall, so who knows what else is buried in the file
Title: Re: Syslog-ng constantly crashing
Post by: erickufrin on August 07, 2020, 07:43:59 pm
Same problem here. syslog-ng not working.
Title: Re: Syslog-ng constantly crashing
Post by: erickufrin on August 07, 2020, 07:51:33 pm
Once I disable "circular logging" syslog-ng is able to start and I am seeing log events now.

I dont know what the remifications of disabling that are? Can anyone shed light?

FWIW once disabling circular logging, syslogd now shows as being stopped.
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 07, 2020, 09:16:27 pm
Once I disable "circular logging" syslog-ng is able to start and I am seeing log events now.

I dont know what the remifications of disabling that are? Can anyone shed light?

FWIW once disabling circular logging, syslogd now shows as being stopped.
What are your other settings looking like? I have mine disabled but syslog-ng still won't start. Ar you using anything else like IPS/IDS or other 3rd party or is this a vanillia install? Trying to figure out what is breaking mine.
Title: Re: Syslog-ng constantly crashing
Post by: erickufrin on August 08, 2020, 02:55:40 am
Its holding steady right now since this morning.

I use a remote syslog server.... Most everything is pretty vanilla at the moment because I am battling a multi-wan issue and wanted to "start over".
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 10, 2020, 08:43:37 pm
Its holding steady right now since this morning.

I use a remote syslog server.... Most everything is pretty vanilla at the moment because I am battling a multi-wan issue and wanted to "start over".

I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.
Title: Re: Syslog-ng constantly crashing
Post by: gpb on August 10, 2020, 10:16:57 pm
I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.

Do you actually have a remote syslog server set up?  If so do you have anything being routed to it?  I have one running on an Rpi with one logging target for logged firewall rules.  Aside from that I too am mostly vanilla...I use NUT (UPS support), NTP server, Shaper, vnStat, and this shouldn't matter, but disabled all network acceleration (which should be default).
Title: Re: Syslog-ng constantly crashing
Post by: KernelKat on August 11, 2020, 05:19:04 am
I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.

Do you actually have a remote syslog server set up?  If so do you have anything being routed to it?  I have one running on an Rpi with one logging target for logged firewall rules.  Aside from that I too am mostly vanilla...I use NUT (UPS support), NTP server, Shaper, vnStat, and this shouldn't matter, but disabled all network acceleration (which should be default).

Yea, that I do. It's been getting all my logs from other devices before going to splunk so that aspect I know works. Think I might just go back to 20.1 or try rolling back some of my snapshots first see what results I get.
Title: Re: Syslog-ng constantly crashing
Post by: CraigS on August 12, 2020, 09:04:15 pm
Hi KernelKat,

Resetting the logfiles seemed to fix it.

I can now start syslog-ng again.

OPNSense 20.7

Title: Re: Syslog-ng constantly crashing
Post by: mimugmail on August 12, 2020, 09:58:51 pm
There is also a patch which comes with 20.7.1 tomorrow, maybe this will fix it
Title: Re: Syslog-ng constantly crashing
Post by: gpb on August 12, 2020, 09:59:16 pm
Hi KernelKat,

Resetting the logfiles seemed to fix it.

I can now start syslog-ng again.

OPNSense 20.7

That likely explains why I don't have the issue because I also did this (via command line deleted all log files) trying to debug the logging target issue in RC1 (fixed in 20.7).  Good find!
Title: Re: Syslog-ng constantly crashing
Post by: Taomyn on August 13, 2020, 08:35:04 am
This is strange, I re-enabled circular logging in order to test if clearing the logs fixed the problem and it didn't. The syslogd service started and the syslog-ng one remained stopped - it was the other way around when circular logging was disabled.


I was then reading the text for the log file size and decided to increase it - I changed it to 10240 as I have the room, and after saving then once again clearing the logs now both syslogd and syslogd-ng are running.