OPNsense Forum
English Forums => Development and Code Review => Topic started by: CJ on October 29, 2023, 03:56:07 pm
-
I'm still digging through the code, but I may be looking at the wrong place so I thought I'd post here. I'm centralizing my logging with Graylog and was attempting to determine if a domain was blocked or not.
Did the DNSBL implementation change when the reporting was added? I thought it was done through Unbound configuration but now it seems like there's python code handling it and logging the query and result for the reporting. I assume that's how it knows which blocklist is responsible?
-
Hi,
Yes, the DNSBL implementation changed. The Unbound configuration method had the drawback of having a significant reload time, regardless of whether this was done dynamically or statically. The blocklists themselves are still being preprocessed by https://github.com/opnsense/core/blob/master/src/opnsense/scripts/unbound/blocklists.py
You can check out the following actions: https://github.com/opnsense/core/blob/master/src/opnsense/service/conf/actions.d/actions_unbound.conf#L19-L47 which handle the reporting end, but can also be utilized to query some details. DuckDb is used to store queries up to a maximum of 7 days, so as to not needlessly fill up disk space.
For logging purposes you can also consider enabling logging in Unbound -> Advanced, we've done a bit of work there to better differentiate between query/reply if you want to run the data through syslog.
Cheers,
Stephan
-
Yes, the DNSBL implementation changed. The Unbound configuration method had the drawback of having a significant reload time, regardless of whether this was done dynamically or statically. The blocklists themselves are still being preprocessed by https://github.com/opnsense/core/blob/master/src/opnsense/scripts/unbound/blocklists.py
Ah, interesting. I didn't realize that was the reason, but I do recall there being an issue with that. After I posted I ended up looking at the dev documentation and that helped a lot. That and learning that Unbound could execute python scripts. :D
You can check out the following actions: https://github.com/opnsense/core/blob/master/src/opnsense/service/conf/actions.d/actions_unbound.conf#L19-L47 which handle the reporting end, but can also be utilized to query some details. DuckDb is used to store queries up to a maximum of 7 days, so as to not needlessly fill up disk space.
I knew about the actions and API and had already looked at that but I'd prefer to have the info inside the logs themselves instead of having to correlate from an additional source.
For logging purposes you can also consider enabling logging in Unbound -> Advanced, we've done a bit of work there to better differentiate between query/reply if you want to run the data through syslog.
Can you provide some more detail on what you mean? I've tested with every single different logging option on that page and they all seem to be just passthroughs to the Unbound config options. I have query and reply logging turned on but it doesn't provide me any information about the blocklists. Unbound also doesn't seem to have a way to log the result of a lookup either, unfortunately.
You can see that both the allowed and blocked domains were resolved in 0ms and not pulled from the cache.
[66717:1] query: 192.168.1.42 blocked.example.com. A IN
[66717:1] reply: 192.168.1.42 blocked.example.com. A IN NOERROR 0.000000 0 54
[66717:3] query: 192.168.1.42 allowed.example.com. A IN
[66717:3] reply: 192.168.1.42 allowed.example.com. A IN NOERROR 0.000000 0 84
Even setting Unbound to the max verbosity doesn't seem to include any blocklist information, and I'm not sure how reliable the recursive response is, plus it's a lot to log for such little information gleaned.
What I'd like to see is the dnsbl script logging to the logs as well as to the logger pipe buffer.
https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Unbound/core/dnsbl_module.py#L210
Although it may be more performant to log during the process from the logger pipe buffer.
https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Unbound/core/dnsbl_module.py#L227
-
I knew about the actions and API and had already looked at that but I'd prefer to have the info inside the logs themselves instead of having to correlate from an additional source.
This one is going to be rather difficult with the plumbing that's available.
I have query and reply logging turned on but it doesn't provide me any information about the blocklists.
This information can be used to determine if a domain was blocked, which I believe to be your original question, but it indeed does not correlate anything to configured blocklists. You'd still have to build your own source for that.
Perhaps it's an option for you to see if you can send some bulk data to syslog using the actual logger: https://github.com/opnsense/core/blob/master/src/opnsense/scripts/unbound/logger.py. All the information you require is in there.
-
I'm not sure I follow. What gets sent to the logger includes the blocklist that caused the action. That's why I said that logging what gets pushed to the logger provides me what I need.
The logging could happen in the logger instead of the dnsbl module but then it would be subject to a delay. I would also think that it would lose the Unbound tagging of id and thread but I haven't seen anything logged from the module to confirm.
-
I'm not sure I follow either, I assume you're customizing the plumbing to fit your needs? Did you try to send the necessary info straight to syslog from the dnsbl_module? A simple log_info() call sends it straight to the resolver log.
My original point is that there's no hook available for you to inject custom code which can survive upgrades for your use case. You're welcome to submit a PR if you think this is a useful addition. In this case make sure the feature is included as an option with a warning that this impacts performance.
I'm happy to provide feedback if you plan on implementing something like this.
-
I have it working locally. Now I just need to figure out the config setup to add a UI toggle.
What do you suggest putting as a prefix for the log statement besides just "dnsbl_module:"? I'm currently using Result but not sure if there's anything better.
-
What do you suggest putting as a prefix for the log statement besides just "dnsbl_module:"? I'm currently using Result but not sure if there's anything better.
Perhaps "logger:" would suffice, it should only be unique within the scope of Unbound. I need to think a bit on this, but this can always be reviewed in the PR.
I have it working locally. Now I just need to figure out the config setup to add a UI toggle.
Most of the steps required to build this are documented: https://docs.opnsense.org/development/frontend.html. Luckily the general page was already migrated to MVC, so this should be plug&play.
What might not be obvious is the communication to the dnsbl_module itself. There's a concrete example here: https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Unbound/core/dnsbl_module.py#L167, in combination with https://github.com/opnsense/core/blob/master/src/etc/inc/plugins.inc.d/unbound.inc#L413
-
Perhaps "logger:" would suffice, it should only be unique within the scope of Unbound. I need to think a bit on this, but this can always be reviewed in the PR.
It's definitely fun to figure out. Are you suggesting "loggger:" instead of "dnsbl_module:" or in addition to it?
Most of the steps required to build this are documented: https://docs.opnsense.org/development/frontend.html. Luckily the general page was already migrated to MVC, so this should be plug&play.
What might not be obvious is the communication to the dnsbl_module itself. There's a concrete example here: https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Unbound/core/dnsbl_module.py#L167, in combination with https://github.com/opnsense/core/blob/master/src/etc/inc/plugins.inc.d/unbound.inc#L413
Thanks. I'll take a look.