Syslog, Graylog, not human readable logs?

[Tomj]

Side note and back to the non-readable. I disabled sending syslog and tried to send Netflow. These logs are not readable. 

[aimdev]

This is the rule I use in the pipe for trapping firewall blocks

rule "Reverse Dns: Firewall src_ip"
when

has_field("src_ip") && is_not_null("src_ip")

then
// Declare a variable that will pull the value from
// the source address field for use in the lookup.
let ts = to_string($message.src_ip);

// Declare a variable that looks up the value stored in the
// lookup table specified matching the value found in the previous variable.
let lv = lookup_value(lookup_table:"reverse_dns", key:ts);

// Set field with name specified and value returned from the previous lookup.
set_field("src_hostname", lv);

You will need a lookup table to do the reverse dns from the ip address.
let lv = lookup_value(lookup_table:"reverse_dns", key:ts);

BTW, you may eventually be using GeoIP databases (MaxMind and others are available!!) to get geospatial data,
the source ip (src_ip) used is confusing, and really should be the search_ip (ie could be a source or destination ip depending on what you are doing)

Also in your post there is a reference to influxdb, there is an graylog output to influxdb V1, which is not the latest, its now V2, its access and query mechanism has changed considerably. It is possible to use a V1 connection with the old query mechanism, it is documented on the influxdb page, it does work, though one expects it to disappear in the future, so all my influxdb's are V2

[Tomj]

Thanks.
I’m using Maxmind for the outside world. It works fine except sometimes not populating the city for dst_ip (seems random).

The objective was more for internal ips, like 192.168.30.4 = computerxxx. I was wondering if there was a way to pass the information in the logs send from opnsense. If not, yes maybe lookup table can help.

[aimdev]

Re netflow, I am using V5 sent from opnsense using graylog netflow input, port 2055.
Using the show received message, they are readable

Re the hostname of internal ip addresses, I don't know if they are included in the syslog message any where, I have never seen them.
I use the syslog messages, sent on a different port, (see earlier post). To get the reverse dns I use the graylog reverse dns input then in the pipeline the rule I showed earlier. I assume it uses the local dns server to get the internal host names, I use unbound, and the internal hostnames are set up there.

RE maxmind (other files are available), they are never going to be 100% accurate or complete.

Re Reverse dns for external addresses, some organisations do not provide the hostname

[Tomj]

Here comes the "limit" of dockers approach. I added the Netflow plugin set the port but nothing was coming in.
Quick check the port was not open, which I put on the fact I didn't declare it during the docker setup.

Anyway, I stopped the Syslog input and moved Netflow to the 1514 port. It receives well the data and as you said in clear readable format. So having the right plugin help

@cditty not sure if that relates to your initial issue.

Now just need to learn Grafana and how to build the proper reports.

Thanks and sorry to have kind of Hijacked the initial thread.

[cditty]

Thanks Tomj,
I will verify if Netflow is coming in. Can you tell me when it was unreadable (before the plugin) was the reported source something like 172.17.0.1? That is what I am seeing and that is the Docker bridge gateway.

[Tomj]

Hello

All messages were showing the firewall ip in Source. so 192.168....
Nothing else was readable except Timestamp.

[amarek]

OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple).

I have installed Graylog 4.3 + Mongo 4.2 + Elasticsearch 7.17. I have setup inputs (and extractors), indices, and streams in GrayLog, I have this on port 1514 and then created a logging target in OpnSense UDP(4) everything left as default except the hostname and port. I see ingress and I can see the logs and messages, communication seems to be working.

My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out. This is what a log looks like in GrayLog:

Code: [Select]

2022-05-25 16:52:25.651 172.17.0.1
�>�b�^�
J�
\���l��PJS0


G�0�0��5�@"P���JS0


M�0�05�)

k
Any ideas?

Thanks!

*** UPDATE ***

I configured a Unifi Controller to send syslogs and in GrayLog they ARE human readable. So, it appears that it is something with OpnSense.

i had the same problem.
it was solved after disabling "SYSTEM: SETTINGS: LOGGING / TARGETS" and restarting the telegraf service and enabling my object in "SYSTEM: SETTINGS: LOGGING / TARGETS" again.

[thatso]

OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple). [...]
My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out.

I've stumbled upon this problem recently and this thread was pretty much the only mention I could find. However, it was never really solved. So for future readers, the answer is as simple as somewhat unexpectedly logical:

If you go to System > Settings > Logging/targets and enter Graylog as a new destination, the logging format will be Syslog.
For the Telegraf addon, Enable Graylog Output will use GELF (Graylog Extended Log Format).
Simply create a corresponding input in Graylog and the logs will be very much human readable.