Syslog, Graylog, not human readable logs?

[cditty]

OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple).

I have installed Graylog 4.3 + Mongo 4.2 + Elasticsearch 7.17. I have setup inputs (and extractors), indices, and streams in GrayLog, I have this on port 1514 and then created a logging target in OpnSense UDP(4) everything left as default except the hostname and port. I see ingress and I can see the logs and messages, communication seems to be working.

My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out. This is what a log looks like in GrayLog:

Code: [Select]

2022-05-25 16:52:25.651 172.17.0.1
�>�b�^�
J�
\���l��PJS0


G�0�0��5�@"P���JS0


M�0�05�)

k
Any ideas?

Thanks!

*** UPDATE ***

I configured a Unifi Controller to send syslogs and in GrayLog they ARE human readable. So, it appears that it is something with OpnSense.

[cditty]

OK, looking further into this, apparently there are logs that are coming over along with a bunch of "garbage?". Since I do not know what it is I will just call it that. I added a rule to throw out anything with a facility_num <= 0. That seems to have cleaned up the logs.

[Tomj]

Hello,

I've started to work on the same topic this weekend, first looking at Telegraf/Influxdb/Grafana. But I don't collect enough information as I'm mostly interested in monitoring:
- User/device (IP)  traffic usage and destinations
- Suricata monitoring
- Firewall
- Bandwith

So started to look at Graylog and inspiration from this https://github.com/lephisto/pfsense-analytics.
Now the issue is to convert the extractor to OpnSense but not being able to read the Syslog message sent to Graylog is not helping.

Would you mind to share, what you have done so far? Did you create your own Extractor ?
Do you collect all logs or just some in Graylog ?
"I added a rule to throw out anything with a facility_num <= 0": did you do that in Graylog Stream ?

Thanks

[aimdev]

I have opnsense sending logs, trapped for the firewall monitor (using grafana table & map)
In addition I have netflow V5 feeding flows to graylog for monitoring (using grafana table & map)
Suricata is still on the naughty step for causing issues, maybe with the wan interface.
Graylog is a bit of a learning curve.
Elastic search is the database (boo hoo as I prefer TIKS but couldnt find a working solution).
The impact on opnsense is minimal, all the processing performed on a VM, so opnsense is just the sender.
I did have an issue with opensense's  netflow aggregator process, racking lots of cpu, so I managed to disable it, its not needed as I have my pretty grafana graphs.

[Tomj]

Indeed learning curve.
Thanks to your message I figured out why I was receiving no message (sending to TCP which was not active instead of UDP).
So now I managed to send Syslog to a dedicated stream in Graylog, progress...
Still can't figure out why the GeoIP is not yet working but have some ideas (just looking at src_ip).

Funny how I didn't even think about sending Netflow and was looking at ntopng.

And here comes which is log is best for what, before even starting yet to create dashboard in Grafana.

[aimdev]

The src_ip is a bit misleading.
Here is my rule in the pipeline


rule "GeoIP City: src_ip"
when
has_field("src_ip") && is_not_null("src_ip")
then
let geo = lookup("mm-city", to_string($message."src_ip"));
set_field("src_ip_geo_location", geo["coordinates"]);
set_field("src_ip_geo_country", geo["country"].iso_code);
set_field("src_ip_geo_city", geo["city"].names.en);
end


I used lookup tables to resolve the MaxMind (and other mmdb's) to get the geospatial info, not sure if its correct with grafana's map though (it appears to have its own lookup based on country codes) , and the new grafana map needs geohashes (not available from graylog afaik), however the grafana table is getting the data from the elasticsearch indices.

I am unsure if I have got it all efficient and correct (graylog), but its a better solution than logstash.

[Tomj]

I duplicated the default one with dst_ip. I like to know where things connect
It resolving only country probably becasue the line is
let geo = lookup("geoip", to_string($message."src_ip"));

Trying yours now, but being in a docker might require some adjustments for the file locations.
This guy was able to make it work in Grafana for the country part.
https://github.com/bsmithio/OPNsense-Dashboard
But that might be thanks to a change of format through Content pack or extractor. I'll dig into that later.

[cditty]

I have just started and I have not progressed very far, but I do have things mostly working. Here are the things that I have done:

I have 3 Dockers installed, I used Portainer to install Mongo 4.2 and ElasticSearch 7.17.3. Their setup was simple enough:

Mongo:
port 27017:27017
volume data-mongo:/data/db
volume data-mongo:/data/configdb
restart-policy: unless-stopped

ElasticSearch:
volume data-eleasticsearch:/usr/share/elaticsearch/data
restart-policy: unless-stopped

Then for GrayLog I installed via cli, Portainer does not support linking:

docker run -d --name=graylog --link mongo --link elasticsearch -p 12201:12201 -p 1514:1514 -p 9001:9000 -p 5555:5555 -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" -e GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 -e GRAYLOG_PASSWORD_SECRET=16characterstarterstring -v data-graylog:/usr/share/graylog/data --restart unless-stopped graylog/graylog:4.3.0

I later edited the container via Portainer to extend the listening port range and to turn on UDP as required. I wanted to use docker-compose as a stack, but I was having issues and instead of debugging I approached each app separately. Not as clean maybe, but easier to get it going.

After logging into GrayLog, changing to DarkMode (perhaps the most import step ), updating the user/pass/timezone, I started setting up the input, extractor, indices, and stream/stream rules.

Input: syslog udp -> Set Title, Port, check "Save full message"
Extractor: Import from https://github.com/IRQ10/Graylog-OPNsense_Extractors
Indices: -> Set (Title = Description =  Prefix = "opnsense"), Rotation Strategy -> Index Size, Max Size = 524288000, Max Number of Indices = 10
Stream: -> Set Title, Index Set -> opnsense
Stream Rules: gl2_source_input must match exactly 628e665caaa5017cfbc3f1ab, facility_num must be greater than 0

For me, once I had messages to look at, I could get "628e665caaa5017cfbc3f1ab" from "Show Received Messages" when looking at the Input.

I configured OpnSense to send the syslog.

System->Settings->Logging/Targets->New
Transport->UDP(4), Applications->Filter, Set Host/Port, do NOT check rfc5424

Checking rfc5424 (Syslog) format seemed like a good idea, but it will not work with the extractor.

At this point you should have basic FW logs making their way into GrayLog with all headers defined and searchable. This is as far as I have made it. Hopefully it is a starting point for someone else.

*** EDIT ***
Performance -

I am aggregating roughly 500MBs of logs per day
I have my dockers set up in a ProxMox LXC with 12 CPU cores / 10 GB memory on a DL360
Currently have 7 containers spun up including the 3 for logging
Metrics: averaging < 2% CPU usage and 5GB memory for entire LXC


 

[cditty]

So, this is working for me, but I am still seeing "garbage" coming across. Is anyone else seeing this? Most of my logs are clean, but there is still 10% that is not human readable. It is not affecting my logging, but it bothers me that I don't understand what it is, and I worry that maybe it is a sign of something that needs tweaking.

[Tomj]

Same, installation in Docker, still need to figure out why it didn’t take the proper time sone in the config and how to change.

I set the stream rule on “source”=myopensense hostname.
Sending all syslog for now to the stream, i’ll probably split in different stream later. No garbage log, everything is properly populating the fields. I’m using the extractor provided by bsmithio but will probably convert the one from pfsense and change.
The grafana dashboard provided by bsmithio displays the map properly too.

I tried Graylog dashboard, but it seems pretty limited with free version. I couldn’t figure out how to create a map.

Next step will be to send ntopng to Graylog and see what i can get. Then slowly work on the dashboards.

[cditty]

For the timezone, I created a new user, assigned the role "admin" and set the timezone for that user. That seemed to do what I needed.

[Tomj]

Thanks
Eventually I modified the graylog.conf file in xxxxx/docker/volumes/monitoring_graylog_data/_data/config, that fixed it.

I can't test ntopng, it requires to upgrade to 22.1.8 to install the plugin, which I will avoid looking at the other thread

[cditty]

I just tried this too and it seems to work: ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime I restarted the container and it was sticky.

[aimdev]

I never used the Docker approach, to me its another level of complexity, I just installed graylog, elastic on a vm.
In opensense I filtered the firewall, see attached. Note the port used, this allows for simple routing to graylog, so the input must use the same port/protocol
Inputs do not use any extractors, Streams have a simple filter , for firewall its

message must contain block

Then its on to pipelines, more fun 

[Tomj]

I tried docker, just to "learn" about it a bit and for testing. It should help to keep things isolated, not polluting too much the server when I finally decide what to use between Graylo, Influxdb, Victoriametrics, Grafana.

Does anyone know how to send local hostname along with the ip? it would be easier to track which wonderful iOTs device tries to talk too much.