Unbound DNS blacklist

[Gary7]

Upgrade to 20.7 went relatively smoothly. Now, I'm spending too much time optimizing loader.conf.local and sysctl.conf.local.
Previously, I was using some scripts to generate the blacklist and put it in a conf file.
Now, with DNS Blacklist integrated into Unbound DNS, I'm using the predefined lists and will probably add a few additional lists.

I reviewed the list of predefined blacklists in the python code, dnsbl.py, and there are a couple of details that could be updated.

Ransomware Tracker is no longer available. Go to the link and you get the text "# Ransomware Tracker has been discontinued on Dec 8th, 2019"

AdAway List is set to https://adaway.org/hosts.txt. In the comments of this file, it says
# Fetch the latest version of this file:
# https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt

Maybe change AdAway to use https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt

Blocklist.site Ads, Fraud, Phishing get re-directed to https://raw.githubusercontent.com
Maybe change:
Blocklist.site   Ads      https://raw.githubusercontent.com/blocklistproject/Lists/master/ads.txt
Blocklist.site   Fraud   https://raw.githubusercontent.com/blocklistproject/Lists/master/fraud.txt
Blocklist.site   Phishing   https://raw.githubusercontent.com/blocklistproject/Lists/master/phishing.txt

My home firewall is an APU2D4 and I use a RAM disk for /var and /tmp.
So, the /var directory is recreated at boot and many of the Unbound files are under /var.
I've added System:Settings:Cron "Download Unbound DNSBLs and restart" to run each morning.
Do I need to add another Cron "Download Unbound DNSBLs and restart" to run at boot time?

It doesn't appear that the blacklists are enabled after a reboot. Since the combined blacklist file is /var/unbound/etc/dnsbl.conf, how does it get created at boot?

Maybe, I just don't know where the documentation for Unbound blacklist is located. Maybe, github.com ?

Thank you all.

[Gary7]

I did some more testing of the default lists in Unbound DNS: Blacklist and I'm convinced that Blocklist.site Ads/Fraud/Phishing lists are not getting retrieved.

Testing:
Enable Unbound DNS: Blacklist, don't select any additional DNSBL lists, and save. A default /var/unbound/etc/dnsbl.conf is created.
Select only Blocklist.site Ads/Fraud/Phishing lists and save. The generated /var/unbound/etc/dnsbl.conf is the same size.
Modify /usr/local/opnsense/scripts/unbound/dnsbl.py
    "bla": "https://raw.githubusercontent.com/blocklistproject/Lists/master/ads.txt",
    "blf": "https://raw.githubusercontent.com/blocklistproject/Lists/master/fraud.txt",
    "blp": "https://raw.githubusercontent.com/blocklistproject/Lists/master/phishing.txt",
Then save. The generated /var/unbound/etc/dnsbl.conf file is about 50% bigger.

I've convinced myself that Unbound DNS: Blacklist doesn't work with redirects to lists.

In dnsbl.py, the Blocklist.site Ads/Fraud/Phishing URLs (i.e. https://blocklist.site/app/dl/ads) will redirect to https://raw.githubusercontent.com/blocklistproject/Lists/master/

Could somebody verify my findings and modify the URLs in dnsbl.py ?

I suspect that it will be a lot easier to modify the lists instead of modifying the scripts to handle redirects to lists.

Thank you.

[AhnHEL]

Love OPNsense, but I personally got frustrated with the whole Unbound DNSBL experience and have recently just set up a PiHole on my network with OPNsense and have been extremely satisfied with its performance.  Sorry I don't have a proper response to your issue but food for thought.

[hushcoden]

It'actually works for me: https://forum.opnsense.org/index.php?topic=18997.msg86922#msg86922

[Gary7]

In OPNsense 20.7.2, /usr/local/opnsense/scripts/unbound/dnsbl.py was changed. The links for the Blockist.site lists were updated. Previously, the Ads link was https://blocklist.site/app/dl/ads which does a re-direct. In 20.7.2, the Ads link is https://blocklistproject.github.io/Lists/ads.txt. Now, OPNsense correctly retrieves the Blocklist.site lists.

During my earlier testing, when I enabled Blacklist but didn't select any additional lists and saved, the generated dnsbl.conf was about 17 MB. A default blacklist was enabled. When I selected the 3 updated Blocklist.site lists and saved, the dnsbl.conf file was about 25 MB. That shows me that the links for Blocklist.site are working.

[hushcoden]

One thing is not clear: I cannot creat a cron job (see attachment), so when/how does the update happen ??

[Gary7]

When I was testing, I just used the GUI Services: Unbound DNS: Blacklist and hit the Save button to manually update DNSBL.
I'm logging-in to the GUI using root and I have no problem creating cron entries in System: Settings: Cron
I selected Download Unbound DNSBLs and restart. I run it early each morning.
After you Save and Apply, login to the shell and look at /var/cron/tabs/nobody. See if there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl" (or maybe unboundplus depending on which "Download Unbound DNSBLs and restart" you selected in the GUI.

[CJ]

When I was testing, I just used the GUI Services: Unbound DNS: Blacklist and hit the Save button to manually update DNSBL.
I'm logging-in to the GUI using root and I have no problem creating cron entries in System: Settings: Cron
I selected Download Unbound DNSBLs and restart. I run it early each morning.
After you Save and Apply, login to the shell and look at /var/cron/tabs/nobody. See if there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl" (or maybe unboundplus depending on which "Download Unbound DNSBLs and restart" you selected in the GUI.

Do you have two Download Unbound entries?  When I look at what they actually enter into cron one is for Unbound and one is for UnboundPlus.

[hushcoden]

Do you have two Download Unbound entries?  When I look at what they actually enter into cron one is for Unbound and one is for UnboundPlus.

I have two entries, and does it make any difference on which one I select ?

I selected the first one and if I look at "/var/cron/tabs/nobody" I see there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl"

[Gary7]

The first one is preferred. Both currently work.
https://forum.opnsense.org/index.php?topic=18951.msg86657#msg86657

[hushcoden]

Great, thanks !

[Vaultboy]

I have the strangest issue with DNSBL on 20.7.2. Seems that the "enable" checkbox under Services: Unbound DNS: Blacklist is reversed.

When the "enable" checkbox is checked:

1.  /var/unbound/etc/dnsbl.conf is empty
2. /var/unbound/etc/lists.inc has my list source from https://block.energized.pro/ultimate/formats/domains.txt


When the "enable" checkbox is UNchecked:

1.  /var/unbound/etc/dnsbl.conf is populated with entires from https://block.energized.pro/ultimate/formats/domains.txt
2. /var/unbound/etc/lists.inc is blank


Seems that entries from https://block.energized.pro/ultimate/formats/domains.txt resolve to localhost only when the "enable" checkbox is UNchecked. However, when it is checked we resolve names to actual addresses.

Is this expected? Something seems amiss.

Thank you folks!

[mimugmail]

Can you tick enable and apply several times?

[Vaultboy]

Can you tick enable and apply several times?

Oh yes, I did. Even with a reboot between ticking and unticking. It takes a lot longer to "save" changes when unticking the enable box as it is downloading the list.

What I ended up doing is backing up populated /var/unbound/etc/dnsbl.conf, ticking the box and restoring /var/unbound/etc/dnsbl.conf. This seems to hold after reboot, but probably not a long term fix.

[mfpck]

This behaviour is a bit weird and the logic behind is still unclear for me, but yes, if u disable/enable and hit apply a bunch of times in an eclectic style somewhen it works  8)