(Tentatively Solved) syslog-ng spamming general log once per minute

[Koldnitz]

Good afternoon,

I upgraded to 20.7 and everything is fine.

I noticed that syslogd was going to be deprecated (eventually), and I had previously read that circular logging was on it s way out so I decided to turn off this feature.

Everything seems to be logging correctly but now I am receiving messages like this once per minute to the general (gui) / system (console) log:

2020-08-01T12:29:33   syslog-ng[68466]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-01T12:28:33   syslog-ng[68466]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-01T12:27:33   syslog-ng[68466]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-01T12:26:33   syslog-ng[68466]: Destination timeout has elapsed, closing connection; fd='29'

This fd bit changes: I have 30, 31, 26 ...

I have googled this and found a post about changing the filter to only show warnings and above

https://serverfault.com/questions/1020432/syslog-ng-set-loglevel-priority-to-warning-or-more-to-be-less-verbose

but I figured I should ask if there is something else going on before I did that.

Looking through my logs I could not find anything else firing once per minute, and no one on these forums seems to have had this issue yet (or they did not post if they did)

Any help will be much appreciated.

Cheers,

[gpb]

Same issue with syslog-ng messages, also disabled circular logging then noticed this...did not reboot though (if that matters...assuming it doesn't since there was no warning/message to do so).

Code: [Select]

2020-08-02T21:11:40 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-02T21:10:40 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='28'
2020-08-02T21:10:05 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='27'
2020-08-02T21:09:47 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-02T21:08:47 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-02T21:07:47 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-02T21:06:47 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-02T21:05:47 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='28'
2020-08-02T21:05:46 syslog-ng[70248]: Destination timeout has elapsed, closing connection; fd='27'

[bunchofreeds]

I'm experiencing this also.

2020-08-03T18:25:44   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-03T18:24:44   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='28'
2020-08-03T18:24:36   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-03T18:23:36   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-03T18:22:36   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-03T18:21:36   syslog-ng[88320]: Destination timeout has elapsed, closing connection; fd='7'
2020-08-03T18:20:36   syslog-ng[88320]: syslog-ng starting up; version='3.27.1'
2020-08-03T18:20:36   syslogd: exiting on signal 15

Taken from System>Log Files>General

Also my syslog-ng service does not seem to run reliably. Sorry I have not completed any testing to confirm or deny this completely.

I do know that syslog-ng was not running, I then disabled circular logging within System>Settings>Logging and syslog-ng is now running (appears to have thrown the above 'timeout' log entries). But now syslogd service is not running.

Does anyone have any remote logging setup within Intrusion Detection?
Just wondering if its related?

[gpb]

My understanding is syslogd is being replaced with syslog-ng and once circular logging is turned off, syslogd is no longer needed so opnsense disables it properly.  Syslog-ng seems fine for me, aside from the log messages...not using IDS, but am using external syslog server.

[ivwang]

Right, functionality wise 20.7 syslog-ng seems doing its job. Albeit the log spamming and seemingly tendency to coredump.

Found several times syslog-ng coredumped at boot (found it before I disable clog) and have to start it manually. Since the direction is moving away from clog I opt to disable it in my 20.7 box too.

By the way I recall seeing Franco saying in another thread that disabling clog makes syslog to be stopped as expected.

[retatefw]

I have disabled circular logging on 20.7 and I am seeing the once a minute syslog-ng destination timeout messages.  Services shows an error for syslogd.  I assume that syslogd not running is expected behavior.  I haven't looked to see what needs to be done to remove it from services monitoring.

[annoniempjuh]

i have this exact same problem, started since 20.7. every minute a logbook registration from syslog-ng:

Code: [Select]

2020-09-16T20:06:08 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:05:08 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='27'
2020-09-16T20:04:49 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T20:04:01 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='25'
2020-09-16T20:03:10 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T20:02:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:01:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T20:00:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T19:59:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='5'
2020-09-16T19:58:35 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='26'
2020-09-16T19:58:01 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='23'
2020-09-16T19:57:34 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='27'
2020-09-16T19:56:41 syslog-ng[41410] Destination timeout has elapsed, closing connection; fd='23'
it's pretty annoying while trouble shouting problem with rondom WAN drop offline isseu...

[MTR]

Same here, pretty annoying. Since these are info level messages from syslog-ng i figured to edit /usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-local.conf with a conditional filter that will send syslog-ng messages to system.log only when they are level warn or above and simply discard anything else. But i failed. Not enough knowledge on syslog-ng configuration. 
So then i created a .conf file for syslog-ng in /usr/local/opnsense/service/templates/OPNsense/Syslog/local/. Now all facility(syslog) logging is send to /var/log/syslog-ng/ and the main system.log is free of this syslog-ng spam.

So for now i'm happy, but it would be nice if somebody could implement some form of conditional filtering -as in only warn and above for facility(syslog)- in syslog-ng-local.conf.

[mimugmail]

Can you give a bit more details please?

[Koldnitz]

What kind of details are you looking for? i.e. please tell me where you want me to look or what logs you need to be shown.

This has been happening to me since 20.7 (I removed the circular logs)

It does not seem to be detrimental to anything but it is really annoying if you need to look for something in the logs.

Cheers,

[MTR]

Can you give a bit more details please?

Well, in /usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-local.conf there is this section:

Code: [Select]

################################################################################
# not captured elsewhere, but relevant, send to system[__].log
################################################################################
filter f_local_system {
    not filter({{ all_filters|join(') and not filter(') }})
    and level(notice..emerg)
};

destination d_local_system {
    file(
        "/var/log/system/system_${YEAR}${MONTH}${DAY}.log"
        create-dirs(yes)
    );
};

log {
    source(s_all);
    filter(f_local_system);
    destination(d_local_system);
};
I tried to edit this to something like this:

Code: [Select]

################################################################################
# not captured elsewhere, but relevant, send to system[__].log
################################################################################
filter f_local_system {
    not filter({{ all_filters|join(') and not filter(') }})
    and level(notice..emerg)
};

filter f_warn { severity(warning..emergency) };

destination d_local_system {
    file(
        "/var/log/system/system_${YEAR}${MONTH}${DAY}.log"
        create-dirs(yes)
    );
};

log {
    source(s_all);
    if (facility(syslog) {
        filter { filter(f_warn); };
    } else {
        filter { filter(f_local_system); };
    };
    destination(d_local_system);
};
This is not working, getting syntax error on restarting syslog-ng. I tried some other things but couldn't make it work.

So i choose to create /usr/local/opnsense/service/templates/OPNsense/Syslog/local/syslog-ng.conf:

Code: [Select]

###################################################################
# Local syslog-ng configuration filter definition [syslog-ng].
###################################################################
filter f_local_syslog-ng {
    facility(syslog);
};
This will send all facility(syslog) messages to its own destination.

I really don't know what is the correct course of action here. Obviously the "Destination timeout has elapsed, closing connection; fd='xx'" spam messages are annoying to see in the system.log every minute, so my latter option is okay-ish; it just filters all syslog messages into its own log destination. Downside is you don't see any syslog messages in the GUI's System->Log Files->General.

Also i don't even know if these spam messages are the result of something in my own OPNsense config, OPNsense's syslog-ng config or it's something that should be fixed in syslog-ng itself. They started to appear after 20.7.2 upgrade. First i had no logging at all until i checked 'Disable circular logs' in System->Settings->Logging. This made logging started to work again, but with these spam messages.

[gpb]

Thanks @MTR for posting this...working well so far. 

[MTR]

Right, face-palm moment...

Remember i said i couldn't make it work? That i got a syntax error on restarting syslog-ng? Well, I found the error in the syntax:

Code: [Select]

log {
    source(s_all);
    if (facility(syslog) {
should be:

Code: [Select]

log {
    source(s_all);
    if (facility(syslog)) {
I forgot a closing bracket after syslog). 

So with this fixed my first option actually works! However... it might be nice to see some of syslog-ng's info level messages in the system.log. So i read up in syslog-ng's docs and found there is a 'message' option to use in filters. Fiddled a bit more and now i made syslog-ng log everything to system.log except the dreaded timeout messages. Yay.

So now we have three options to deal with these messages in system.log:

1) Let syslog-ng filter facility(syslog) into its own log destination:
Create /usr/local/opnsense/service/templates/OPNsense/Syslog/local/syslog-ng.conf:

Code: [Select]

###################################################################
# Local syslog-ng configuration filter definition [syslog-ng].
###################################################################
filter f_local_syslog-ng {
    facility(syslog);
};
2) Log only facility(syslog) level warn and above to system.log:
Edit /usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-local.conf:

Code: [Select]

################################################################################
# not captured elsewhere, but relevant, send to system[__].log
################################################################################
filter f_local_system {
    not filter({{ all_filters|join(') and not filter(') }})
    and level(notice..emerg)
};

filter f_warn { severity(warning..emergency) };

destination d_local_system {
    file(
        "/var/log/system/system_${YEAR}${MONTH}${DAY}.log"
        create-dirs(yes)
    );
};

log {
    source(s_all);
    if (facility(syslog)) {
        filter { filter(f_warn); };
    } else {
        filter { filter(f_local_system); };
    };
    destination(d_local_system);
};

3) Log everything from facility(syslog) but the spam messages (and level debug too) to system.log:
Edit /usr/local/opnsense/service/templates/OPNsense/Syslog/syslog-ng-local.conf:

Code: [Select]

################################################################################
# not captured elsewhere, but relevant, send to system[__].log
################################################################################
filter f_local_system {
    not filter({{ all_filters|join(') and not filter(') }})
    and level(notice..emerg)
};

filter f_not_message {
    not message('Destination timeout has elapsed, closing connection; fd=')
    and level(notice..emerg)
};

destination d_local_system {
    file(
        "/var/log/system/system_${YEAR}${MONTH}${DAY}.log"
        create-dirs(yes)
    );
};

log {
    source(s_all);
    if (facility(syslog)) {
        filter { filter(f_not_message); };
    } else {
        filter { filter(f_local_system); };
    };
    destination(d_local_system);
};
Obviously option 3) will break when syslog-ng devs change the actual text of the message, but for now this will work.

Oh, after removing the syslog-ng.conf i had to do 'service syslog-ng reload' for it to pick up the change. For some reason 'service syslog-ng restart' did not work; it kept logging into /var/log/syslog-ng/.

[mimugmail]

I think you can also just add "message('Destination timeout has elapsed, closing connection; fd=')" at the end of the exclude line.


I will discuss with devs ...

[MTR]

Thanks for following up on this @mimugmail. There are probably more ways to omit these messages from syslog-ng logs, but i think the real question is why these messages are here. Can something be done to avoid them instead of hiding them; what triggers syslog-ng to spam these messages?