External syslog stops working on connection loss

[guest34985]

Hi,

I have a syslog server running as a docker container and noticed that after every update using docker-compose syslog messages from all systems comes in again like before with the exception of OPNsense. After a connection loss to the syslog server I have to manually restart syslog-ng service in order to have OPNsense sending syslog again.

Is that a known problem, and do you maybe have a hint on this?

Thanks in advance!

[meyergru]

How did you define the log target?

I use UDP on port 514 and for all I know, this is stateless and per RFC 5426 unreliable, so what would a connection loss even mean?

My graylog instance runs on a VM that is restarted more often than my OpnSense. Once it is up, it receives log messages from my OpnSense without restarting anything else.

That may be different with other, stateful protocols like TCP, TLS or their likes, however.

[guest34985]

My target definition is pretty simple: IP, UDP(4) + Port. My syslog server is also Graylog btw. The thing is that it seems to be more than a coincidence. Without a Graylog update OPNsense logs come in just fine. After the container update no more OPNsense come in until I restart syslog-ng. Found a similar report for pfsense here.

[CJ]

I'm running Graylog in Docker and I've restarted it multiple times, including a version update with no issues.  I haven't updated to 5.2 yet as it just dropped, but I don't expect to see an issue there either.

I am using Portainer stacks and not bare docker-compose, though.

[meyergru]

I just upgraded to graylog 5.2, during this, the service was restarted. After it came up, OpnSense continued to log to this server. I do not run graylog in a container, though.

[guest34985]

Thanks for all that feedback. I will keep an eye on this issue and try the narrow it down. It does not seem to be a principal problem, but maybe I can find out the root cause of my setup or configuration.

[guest34985]

I think I've found the problem. It's the source interface that OPNsense sends syslog from. In my case the syslog is sent from my WAN interface as source not the LAN interface which is on the same subnet as the syslog server.

Would it be possible to add a feature that allows to specify the source ip for the syslog? Or is there another way to fix or work around this problem?

[meyergru]

That sounds strange. Essentially, this should never happen. I would look at routing and / or NAT.

[guest34985]

That sounds strange. Essentially, this should never happen. I would look at routing and / or NAT.

Yes, it is strange. No static routing table entries, NAT disabled (done on another device).

But I found several bits (on this forum and elsewhere) where this issue was mentioned, e.g. here.

[meyergru]

It could happen if your LAN adapter is unavailable at the time that syslog-ng gets (re)started.

Unless the UDP API of syslog-ng gets updated, this problem will persist.

But you could try if switching to TCP transport works for you. You can easily launch a new "Syslog TCP input" to graylog and use TCP(4) transport type from OpnSense. As TCP is connection-oriented and answering back to the wrong IP will not work anyway, the TCP connection should either be established correctly or not at all, in which case a retry should use the LAN interface IP at some point.

[guest34985]

Yes, that would make perfect sense. Fun fact is that syslog-ng even uses the WAN interface after a restart while the LAN interfaces are definitely all up and running.

I will try the TCP-based approach you suggested.

Thanks for taking the time to help!