This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.
Can you Upload them somewhere?
How do I pull the core dumps? I feel so stupid I didn't even think about that.
Not 100% sure but I have a syslog-ng.core file in /usr/. Guessing that's it.
EDIT: not "/usr"..."/var/db".
Quote from: mimugmail on August 05, 2020, 06:05:29 AM
Can you Upload them somewhere?
Uploaded!
Quote from: gpb on August 05, 2020, 08:03:14 PM
Not 100% sure but I have a syslog-ng.core file in /usr/. Guessing that's it.
Found it. /var/db
https://filebin.net/ew8rz8m7gxkdcf4s
Sorry 'bout that...I updated my post to reflect that...why I put "usr" I have no idea. :/
Quote from: KernelKat on August 04, 2020, 10:30:16 PM
This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.
I'm seeing the same thing. I can't seem to start the syslog-ng service. This is what I see in the logs when I try to start the service:
kernel: pid 78934 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped) I have a core file for this attempt to start syslog-ng. Let me know if there is someplace I should upload it.
Quote from: pilotboy72 on August 05, 2020, 09:49:58 PM
Quote from: KernelKat on August 04, 2020, 10:30:16 PM
This afternoon I can't get syslog-ng to come back online to forward to splunk. I've tried disabling syslog, upgrading to the latest version. Any other ideas? All I am seeing is core dumps.
I'm seeing the same thing. I can't seem to start the syslog-ng service. This is what I see in the logs when I try to start the service:
kernel: pid 78934 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)
I have a core file for this attempt to start syslog-ng. Let me know if there is someplace I should upload it.
You can upload to any online fil host site and then post link here ;)
I have the same problem after upgrade to 20.7. Syslog-ng cannot start, restarts won't help, the error message is always the same: kernel: pid xxxx (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)
Coredump attached:
https://filebin.net/b08o90rzoo9sfsxk
Same problem here. Syslog-ng always crashing when is configured to send remote logs.
same here - but it also not always is able to even come up during booting...
Keep posting those core dumps all maybe we will find some solutions :)
Has anyone tried removing syslog-ng and then reinstalling the same or different version? I'm not well versed in BSD so I'm still learning.
I'm seeing the same, did try to reinstall syslog-ng and it seemed to work:
Quote2020-08-06T13:26:42 pkg-static[60590]: syslog-ng327 reinstalled: 3.27.1_1 -> 3.27.1_1
But I just needed to reboot the firewall and it syslog-ng crashes on start again:
Quote2020-08-07T10:32:51 kernel: pid 93417 (syslog-ng), jid 0, uid 0: exited on signal 11 (core dumped)
I'd upload the core dump, but I need it to be secure as a brief look at the file I already see some information I'd rather not see on public servers e.g. FQDN of the firewall, so who knows what else is buried in the file
Same problem here. syslog-ng not working.
Once I disable "circular logging" syslog-ng is able to start and I am seeing log events now.
I dont know what the remifications of disabling that are? Can anyone shed light?
FWIW once disabling circular logging, syslogd now shows as being stopped.
Quote from: erickufrin on August 07, 2020, 07:51:33 PM
Once I disable "circular logging" syslog-ng is able to start and I am seeing log events now.
I dont know what the remifications of disabling that are? Can anyone shed light?
FWIW once disabling circular logging, syslogd now shows as being stopped.
What are your other settings looking like? I have mine disabled but syslog-ng still won't start. Ar you using anything else like IPS/IDS or other 3rd party or is this a vanillia install? Trying to figure out what is breaking mine.
Its holding steady right now since this morning.
I use a remote syslog server.... Most everything is pretty vanilla at the moment because I am battling a multi-wan issue and wanted to "start over".
Quote from: erickufrin on August 08, 2020, 02:55:40 AM
Its holding steady right now since this morning.
I use a remote syslog server.... Most everything is pretty vanilla at the moment because I am battling a multi-wan issue and wanted to "start over".
I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.
Quote from: KernelKat on August 10, 2020, 08:43:37 PM
I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.
Do you actually have a remote syslog server set up? If so do you have anything being routed to it? I have one running on an Rpi with one logging target for logged firewall rules. Aside from that I too am mostly vanilla...I use NUT (UPS support), NTP server, Shaper, vnStat, and this shouldn't matter, but disabled all network acceleration (which should be default).
Quote from: gpb on August 10, 2020, 10:16:57 PM
Quote from: KernelKat on August 10, 2020, 08:43:37 PM
I may just have to reinstall vanilla and go from there. shurg. idk what else to try and do at this point.
Do you actually have a remote syslog server set up? If so do you have anything being routed to it? I have one running on an Rpi with one logging target for logged firewall rules. Aside from that I too am mostly vanilla...I use NUT (UPS support), NTP server, Shaper, vnStat, and this shouldn't matter, but disabled all network acceleration (which should be default).
Yea, that I do. It's been getting all my logs from other devices before going to splunk so that aspect I know works. Think I might just go back to 20.1 or try rolling back some of my snapshots first see what results I get.
Hi KernelKat,
Resetting the logfiles seemed to fix it.
I can now start syslog-ng again.
OPNSense 20.7
There is also a patch which comes with 20.7.1 tomorrow, maybe this will fix it
Quote from: CraigS on August 12, 2020, 09:04:15 PM
Hi KernelKat,
Resetting the logfiles seemed to fix it.
I can now start syslog-ng again.
OPNSense 20.7
That likely explains why I don't have the issue because I also did this (via command line deleted all log files) trying to debug the logging target issue in RC1 (fixed in 20.7). Good find!
This is strange, I re-enabled circular logging in order to test if clearing the logs fixed the problem and it didn't. The syslogd service started and the syslog-ng one remained stopped - it was the other way around when circular logging was disabled.
I was then reading the text for the log file size and decided to increase it - I changed it to 10240 as I have the room, and after saving then once again clearing the logs now both syslogd and syslogd-ng are running.