OPNsense Forum
Archive => 22.1 Legacy Series => Topic started by: cditty on May 25, 2022, 06:55:53 pm
-
OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple).
I have installed Graylog 4.3 + Mongo 4.2 + Elasticsearch 7.17. I have setup inputs (and extractors), indices, and streams in GrayLog, I have this on port 1514 and then created a logging target in OpnSense UDP(4) everything left as default except the hostname and port. I see ingress and I can see the logs and messages, communication seems to be working.
My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out. This is what a log looks like in GrayLog:
2022-05-25 16:52:25.651 172.17.0.1
�>�b�^�J�\���l��PJS0G�0�0��5�@"P���JS0M�0�05�)
k
Any ideas?
Thanks!
*** UPDATE ***
I configured a Unifi Controller to send syslogs and in GrayLog they ARE human readable. So, it appears that it is something with OpnSense.
-
OK, looking further into this, apparently there are logs that are coming over along with a bunch of "garbage?". Since I do not know what it is I will just call it that. I added a rule to throw out anything with a facility_num <= 0. That seems to have cleaned up the logs.
-
Hello,
I've started to work on the same topic this weekend, first looking at Telegraf/Influxdb/Grafana. But I don't collect enough information as I'm mostly interested in monitoring:
- User/device (IP) traffic usage and destinations
- Suricata monitoring
- Firewall
- Bandwith
So started to look at Graylog and inspiration from this https://github.com/lephisto/pfsense-analytics.
Now the issue is to convert the extractor to OpnSense but not being able to read the Syslog message sent to Graylog is not helping.
Would you mind to share, what you have done so far? Did you create your own Extractor ?
Do you collect all logs or just some in Graylog ?
"I added a rule to throw out anything with a facility_num <= 0": did you do that in Graylog Stream ?
Thanks
-
I have opnsense sending logs, trapped for the firewall monitor (using grafana table & map)
In addition I have netflow V5 feeding flows to graylog for monitoring (using grafana table & map)
Suricata is still on the naughty step for causing issues, maybe with the wan interface.
Graylog is a bit of a learning curve.
Elastic search is the database (boo hoo as I prefer TIKS but couldnt find a working solution).
The impact on opnsense is minimal, all the processing performed on a VM, so opnsense is just the sender.
I did have an issue with opensense's netflow aggregator process, racking lots of cpu, so I managed to disable it, its not needed as I have my pretty grafana graphs.
-
Indeed learning curve.
Thanks to your message I figured out why I was receiving no message (sending to TCP which was not active instead of UDP).
So now I managed to send Syslog to a dedicated stream in Graylog, progress...
Still can't figure out why the GeoIP is not yet working but have some ideas (just looking at src_ip).
Funny how I didn't even think about sending Netflow and was looking at ntopng.
And here comes which is log is best for what, before even starting yet to create dashboard in Grafana.
-
The src_ip is a bit misleading.
Here is my rule in the pipeline
rule "GeoIP City: src_ip"
when
has_field("src_ip") && is_not_null("src_ip")
then
let geo = lookup("mm-city", to_string($message."src_ip"));
set_field("src_ip_geo_location", geo["coordinates"]);
set_field("src_ip_geo_country", geo["country"].iso_code);
set_field("src_ip_geo_city", geo["city"].names.en);
end
I used lookup tables to resolve the MaxMind (and other mmdb's) to get the geospatial info, not sure if its correct with grafana's map though (it appears to have its own lookup based on country codes) , and the new grafana map needs geohashes (not available from graylog afaik), however the grafana table is getting the data from the elasticsearch indices.
I am unsure if I have got it all efficient and correct (graylog), but its a better solution than logstash.
-
I duplicated the default one with dst_ip. I like to know where things connect :)
It resolving only country probably becasue the line is
let geo = lookup("geoip", to_string($message."src_ip"));
Trying yours now, but being in a docker might require some adjustments for the file locations.
This guy was able to make it work in Grafana for the country part.
https://github.com/bsmithio/OPNsense-Dashboard
But that might be thanks to a change of format through Content pack or extractor. I'll dig into that later.
-
I have just started and I have not progressed very far, but I do have things mostly working. Here are the things that I have done:
I have 3 Dockers installed, I used Portainer to install Mongo 4.2 and ElasticSearch 7.17.3. Their setup was simple enough:
Mongo:
port 27017:27017
volume data-mongo:/data/db
volume data-mongo:/data/configdb
restart-policy: unless-stopped
ElasticSearch:
volume data-eleasticsearch:/usr/share/elaticsearch/data
restart-policy: unless-stopped
Then for GrayLog I installed via cli, Portainer does not support linking:
docker run -d --name=graylog --link mongo --link elasticsearch -p 12201:12201 -p 1514:1514 -p 9001:9000 -p 5555:5555 -e GRAYLOG_HTTP_EXTERNAL_URI="http://127.0.0.1:9000/" -e GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 -e GRAYLOG_PASSWORD_SECRET=16characterstarterstring -v data-graylog:/usr/share/graylog/data --restart unless-stopped graylog/graylog:4.3.0
I later edited the container via Portainer to extend the listening port range and to turn on UDP as required. I wanted to use docker-compose as a stack, but I was having issues and instead of debugging I approached each app separately. Not as clean maybe, but easier to get it going.
After logging into GrayLog, changing to DarkMode (perhaps the most import step ;)), updating the user/pass/timezone, I started setting up the input, extractor, indices, and stream/stream rules.
Input: syslog udp -> Set Title, Port, check "Save full message"
Extractor: Import from https://github.com/IRQ10/Graylog-OPNsense_Extractors
Indices: -> Set (Title = Description = Prefix = "opnsense"), Rotation Strategy -> Index Size, Max Size = 524288000, Max Number of Indices = 10
Stream: -> Set Title, Index Set -> opnsense
Stream Rules: gl2_source_input must match exactly 628e665caaa5017cfbc3f1ab, facility_num must be greater than 0
For me, once I had messages to look at, I could get "628e665caaa5017cfbc3f1ab" from "Show Received Messages" when looking at the Input.
I configured OpnSense to send the syslog.
System->Settings->Logging/Targets->New
Transport->UDP(4), Applications->Filter, Set Host/Port, do NOT check rfc5424
Checking rfc5424 (Syslog) format seemed like a good idea, but it will not work with the extractor.
At this point you should have basic FW logs making their way into GrayLog with all headers defined and searchable. This is as far as I have made it. Hopefully it is a starting point for someone else.
*** EDIT ***
Performance -
I am aggregating roughly 500MBs of logs per day
I have my dockers set up in a ProxMox LXC with 12 CPU cores / 10 GB memory on a DL360
Currently have 7 containers spun up including the 3 for logging
Metrics: averaging < 2% CPU usage and 5GB memory for entire LXC
-
So, this is working for me, but I am still seeing "garbage" coming across. Is anyone else seeing this? Most of my logs are clean, but there is still 10% that is not human readable. It is not affecting my logging, but it bothers me that I don't understand what it is, and I worry that maybe it is a sign of something that needs tweaking.
-
Same, installation in Docker, still need to figure out why it didn’t take the proper time sone in the config and how to change.
I set the stream rule on “source”=myopensense hostname.
Sending all syslog for now to the stream, i’ll probably split in different stream later. No garbage log, everything is properly populating the fields. I’m using the extractor provided by bsmithio but will probably convert the one from pfsense and change.
The grafana dashboard provided by bsmithio displays the map properly too.
I tried Graylog dashboard, but it seems pretty limited with free version. I couldn’t figure out how to create a map.
Next step will be to send ntopng to Graylog and see what i can get. Then slowly work on the dashboards.
-
For the timezone, I created a new user, assigned the role "admin" and set the timezone for that user. That seemed to do what I needed.
-
Thanks
Eventually I modified the graylog.conf file in xxxxx/docker/volumes/monitoring_graylog_data/_data/config, that fixed it.
I can't test ntopng, it requires to upgrade to 22.1.8 to install the plugin, which I will avoid looking at the other thread :)
-
I just tried this too and it seems to work: ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime I restarted the container and it was sticky.
-
I never used the Docker approach, to me its another level of complexity, I just installed graylog, elastic on a vm.
In opensense I filtered the firewall, see attached. Note the port used, this allows for simple routing to graylog, so the input must use the same port/protocol
Inputs do not use any extractors, Streams have a simple filter , for firewall its
message must contain block
Then its on to pipelines, more fun :)
-
I tried docker, just to "learn" about it a bit and for testing. It should help to keep things isolated, not polluting too much the server when I finally decide what to use between Graylo, Influxdb, Victoriametrics, Grafana.
Does anyone know how to send local hostname along with the ip? it would be easier to track which wonderful iOTs device tries to talk too much.
-
Side note and back to the non-readable. I disabled sending syslog and tried to send Netflow. These logs are not readable.
-
This is the rule I use in the pipe for trapping firewall blocks
rule "Reverse Dns: Firewall src_ip"
when
has_field("src_ip") && is_not_null("src_ip")
then
// Declare a variable that will pull the value from
// the source address field for use in the lookup.
let ts = to_string($message.src_ip);
// Declare a variable that looks up the value stored in the
// lookup table specified matching the value found in the previous variable.
let lv = lookup_value(lookup_table:"reverse_dns", key:ts);
// Set field with name specified and value returned from the previous lookup.
set_field("src_hostname", lv);
You will need a lookup table to do the reverse dns from the ip address.
let lv = lookup_value(lookup_table:"reverse_dns", key:ts);
BTW, you may eventually be using GeoIP databases (MaxMind and others are available!!) to get geospatial data,
the source ip (src_ip) used is confusing, and really should be the search_ip (ie could be a source or destination ip depending on what you are doing)
Also in your post there is a reference to influxdb, there is an graylog output to influxdb V1, which is not the latest, its now V2, its access and query mechanism has changed considerably. It is possible to use a V1 connection with the old query mechanism, it is documented on the influxdb page, it does work, though one expects it to disappear in the future, so all my influxdb's are V2
-
Thanks.
I’m using Maxmind for the outside world. It works fine except sometimes not populating the city for dst_ip (seems random).
The objective was more for internal ips, like 192.168.30.4 = computerxxx. I was wondering if there was a way to pass the information in the logs send from opnsense. If not, yes maybe lookup table can help.
-
Re netflow, I am using V5 sent from opnsense using graylog netflow input, port 2055.
Using the show received message, they are readable
Re the hostname of internal ip addresses, I don't know if they are included in the syslog message any where, I have never seen them.
I use the syslog messages, sent on a different port, (see earlier post). To get the reverse dns I use the graylog reverse dns input then in the pipeline the rule I showed earlier. I assume it uses the local dns server to get the internal host names, I use unbound, and the internal hostnames are set up there.
RE maxmind (other files are available), they are never going to be 100% accurate or complete.
Re Reverse dns for external addresses, some organisations do not provide the hostname
-
Here comes the "limit" of dockers approach. I added the Netflow plugin set the port but nothing was coming in.
Quick check the port was not open, which I put on the fact I didn't declare it during the docker setup.
Anyway, I stopped the Syslog input and moved Netflow to the 1514 port. It receives well the data and as you said in clear readable format. So having the right plugin help :)
@cditty not sure if that relates to your initial issue.
Now just need to learn Grafana and how to build the proper reports.
Thanks and sorry to have kind of Hijacked the initial thread.
-
Thanks Tomj,
I will verify if Netflow is coming in. Can you tell me when it was unreadable (before the plugin) was the reported source something like 172.17.0.1? That is what I am seeing and that is the Docker bridge gateway.
-
Hello
All messages were showing the firewall ip in Source. so 192.168....
Nothing else was readable except Timestamp.
-
OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple).
I have installed Graylog 4.3 + Mongo 4.2 + Elasticsearch 7.17. I have setup inputs (and extractors), indices, and streams in GrayLog, I have this on port 1514 and then created a logging target in OpnSense UDP(4) everything left as default except the hostname and port. I see ingress and I can see the logs and messages, communication seems to be working.
My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out. This is what a log looks like in GrayLog:
2022-05-25 16:52:25.651 172.17.0.1
�>�b�^�J�\���l��PJS0G�0�0��5�@"P���JS0M�0�05�)
k
Any ideas?
Thanks!
*** UPDATE ***
I configured a Unifi Controller to send syslogs and in GrayLog they ARE human readable. So, it appears that it is something with OpnSense.
i had the same problem.
it was solved after disabling "SYSTEM: SETTINGS: LOGGING / TARGETS" and restarting the telegraf service and enabling my object in "SYSTEM: SETTINGS: LOGGING / TARGETS" again.
-
OK, I have searched, and I have not seen this issue, I am sure that I am overlooking something (hopefully simple). [...]
My problem is that the logs are not human readable. It seems like there is encoding that is happening and I am not sure how to work it out.
I've stumbled upon this problem recently and this thread was pretty much the only mention I could find. However, it was never really solved. So for future readers, the answer is as simple as somewhat unexpectedly logical:
If you go to System > Settings > Logging/targets and enter Graylog as a new destination, the logging format will be Syslog.
For the Telegraf addon, Enable Graylog Output will use GELF (Graylog Extended Log Format).
Simply create a corresponding input in Graylog and the logs will be very much human readable. ;)