Why are my syslog messages using the WAN interface address?

[eponymous]

Hi,

I've got OPNsense set up to send logs over TCP to a syslog server (ELK running on Linux).

Normally the logs show up on the syslog server as coming from "_gateway" (my OPNsense router) which should resolve to the RFC1918 address assigned on the LAN interface. However just recently it changed and the host field was now showing the public IP assigned on my WAN interface which I found very odd! The syslog server shouldn't be able to see that public IP at all right?

Right when the change happened from "host:_gateway" to "host:<my public WAN IP>" there was a pause of around 1 minute and there was a "Syslog connection closed" error message. There was an "EOF occurred while idle" message immediately afterwards as well. So possibly either the syslog server crashed and/or the syslog-ng daemon on OPNsense crashed.

I've since restarted ELK and it's gone back to "host:_gateway". I also confirmed using Wireshark that the messages themselves don't actually contain any IP so I presume the syslog server is pulling this from the packet headers.

Nothing else has changed in my setup that could explain this.

Has anyone else seen this before or have any ideas how this can happen?

[eponymous]

Update: This is still happening every few days. Looking at wireshark traces, the actual address of the router in the IP header is my external public IP (pppoe interface) instead of the LAN IP. Since the traffic is being passed out of my LAN interface I don’t know how this can be happening.

Restarting either the syslog server or the router temporarily solves the issue which might suggest it’s a connection setup related problem.

[eponymous]

So I know now what is causing this issue but not why.

I have a separate issue in that my LAN switch reboots every now and then and hence brings down all the VLAN interfaces. The only interface left up is the WAN.

When this happens the TCP link to the LAN syslog server expectedly goes down and when the switch comes back up and hence the syslog link comes back up, it has the WAN interface address as the source IP.

If I restart the syslog server it fixes it.

So something about the LAN links going down is causing some odd behaviour.

It’s almost as though the traffic destined for the syslog server is actually going part way through NAT but I can’t understand why that would be since since the syslog server address is RFC1918 (private) and it shouldn’t be trying to route it through the WAN interface  - I mean is that even possible?

Why would opnsense try and route traffic destined to 192.168.0.2 through the WAN?

Does anyone have any suggestions on how to debug this further?

Best.

[Patrick M. Hausen]

Why would opnsense try and route traffic destined to 192.168.0.2 through the WAN?

Because all other interfaces are down at that precise moment. When an interfaces is down, the interface address and all directly connected routes are not present, anymore. So it tries to reach the syslog server via the only interface that is left.

[Fright]

https://github.com/syslog-ng/syslog-ng/issues/3177#issuecomment-599016847

[eponymous]

Why would opnsense try and route traffic destined to 192.168.0.2 through the WAN?

Because all other interfaces are down at that precise moment. When an interfaces is down, the interface address and all directly connected routes are not present, anymore. So it tries to reach the syslog server via the only interface that is left.

So does that imply the logs are actually going out of the interface onto the public Internet? If so isn’t that a security flaw?

https://github.com/syslog-ng/syslog-ng/issues/3177#issuecomment-599016847

Thanks - that does seem to explain what I’m seeing.

[bartjsmit]

So does that imply the logs are actually going out of the interface onto the public Internet? If so isn’t that a security flaw?

No, the ISP will drop the packet since it is not internet routable

[eponymous]

So does that imply the logs are actually going out of the interface onto the public Internet? If so isn’t that a security flaw?

No, the ISP will drop the packet since it is not internet routable

Would it be advisable to add a rule to the OUT side of the WAN interface to block this in my case? For me, no packets with a destination of RFC1918 should need go out that interface. In fact, the WAN interface is ultimately a PPPoE connection.

[schnipp]

So does that imply the logs are actually going out of the interface onto the public Internet? If so isn’t that a security flaw?

No, the ISP will drop the packet since it is not internet routable

Would it be advisable to add a rule to the OUT side of the WAN interface to block this in my case? For me, no packets with a destination of RFC1918 should need go out that interface. In fact, the WAN interface is ultimately a PPPoE connection.

Yes, I recommend to do this. The blocking rule prevents that private network packets will leave a public interface when a route goes down for whatever reason. Because of possible data leak, it can be a security issue.

[meyergru]

Would it be advisable to add a rule to the OUT side of the WAN interface to block this in my case? For me, no packets with a destination of RFC1918 should need go out that interface. In fact, the WAN interface is ultimately a PPPoE connection.

Yes, I recommend to do this. The blocking rule prevents that private network packets will leave a public interface when a route goes down for whatever reason. Because of possible data leak, it can be a security issue.

How would you do this when an automatically generated rule exists for WAN which essentially "lets out anything from firewall host itself" and which is put before any manually created rules?

[schnipp]

How would you do this when an automatically generated rule exists for WAN which essentially "lets out anything from firewall host itself" and which is put before any manually created rules?

Create a floating rule which has a higher priority than interface rules.

[bartjsmit]

Do you have block private networks ticked on your WAN interface?

[meyergru]

I had that toggle active. However, after I created a floating rule with logging enabled, I found some leaking IPs because of remnants of old configurations. The WAN interface usually has the default route, so that is really no surprise.

The toggles handle only incoming RFC1918 traffic, not outgoing. You can see that in the automatic interface rule that handles only "in" packets for private/bogon IPs.

[eponymous]

Do you have block private networks ticked on your WAN interface?

I specifically ensured this was set along with block bogons when I first set up the firewall.

However I thought that only blocked inbound as in from the Internet in my case?

[Patrick M. Hausen]

If you are using TCP with syslog, no information will be leaked, because the initial 3-way handshake always fails. UDP datagrams on the other hand might leak logged information to your ISP.