OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: Gary7 on August 01, 2020, 08:25:32 pm
-
Upgrade to 20.7 went relatively smoothly. Now, I'm spending too much time optimizing loader.conf.local and sysctl.conf.local.
Previously, I was using some scripts to generate the blacklist and put it in a conf file.
Now, with DNS Blacklist integrated into Unbound DNS, I'm using the predefined lists and will probably add a few additional lists.
I reviewed the list of predefined blacklists in the python code, dnsbl.py, and there are a couple of details that could be updated.
Ransomware Tracker is no longer available. Go to the link and you get the text "# Ransomware Tracker has been discontinued on Dec 8th, 2019"
AdAway List is set to https://adaway.org/hosts.txt. In the comments of this file, it says
# Fetch the latest version of this file:
# https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Maybe change AdAway to use https://raw.githubusercontent.com/AdAway/adaway.github.io/master/hosts.txt
Blocklist.site Ads, Fraud, Phishing get re-directed to https://raw.githubusercontent.com
Maybe change:
Blocklist.site Ads https://raw.githubusercontent.com/blocklistproject/Lists/master/ads.txt
Blocklist.site Fraud https://raw.githubusercontent.com/blocklistproject/Lists/master/fraud.txt
Blocklist.site Phishing https://raw.githubusercontent.com/blocklistproject/Lists/master/phishing.txt
My home firewall is an APU2D4 and I use a RAM disk for /var and /tmp.
So, the /var directory is recreated at boot and many of the Unbound files are under /var.
I've added System:Settings:Cron "Download Unbound DNSBLs and restart" to run each morning.
Do I need to add another Cron "Download Unbound DNSBLs and restart" to run at boot time?
It doesn't appear that the blacklists are enabled after a reboot. Since the combined blacklist file is /var/unbound/etc/dnsbl.conf, how does it get created at boot?
Maybe, I just don't know where the documentation for Unbound blacklist is located. Maybe, github.com ?
Thank you all.
-
I did some more testing of the default lists in Unbound DNS: Blacklist and I'm convinced that Blocklist.site Ads/Fraud/Phishing lists are not getting retrieved.
Testing:
Enable Unbound DNS: Blacklist, don't select any additional DNSBL lists, and save. A default /var/unbound/etc/dnsbl.conf is created.
Select only Blocklist.site Ads/Fraud/Phishing lists and save. The generated /var/unbound/etc/dnsbl.conf is the same size.
Modify /usr/local/opnsense/scripts/unbound/dnsbl.py
"bla": "https://raw.githubusercontent.com/blocklistproject/Lists/master/ads.txt",
"blf": "https://raw.githubusercontent.com/blocklistproject/Lists/master/fraud.txt",
"blp": "https://raw.githubusercontent.com/blocklistproject/Lists/master/phishing.txt",
Then save. The generated /var/unbound/etc/dnsbl.conf file is about 50% bigger.
I've convinced myself that Unbound DNS: Blacklist doesn't work with redirects to lists.
In dnsbl.py, the Blocklist.site Ads/Fraud/Phishing URLs (i.e. https://blocklist.site/app/dl/ads) will redirect to https://raw.githubusercontent.com/blocklistproject/Lists/master/
Could somebody verify my findings and modify the URLs in dnsbl.py ?
I suspect that it will be a lot easier to modify the lists instead of modifying the scripts to handle redirects to lists.
Thank you.
-
Love OPNsense, but I personally got frustrated with the whole Unbound DNSBL experience and have recently just set up a PiHole on my network with OPNsense and have been extremely satisfied with its performance. Sorry I don't have a proper response to your issue but food for thought.
-
It'actually works for me: https://forum.opnsense.org/index.php?topic=18997.msg86922#msg86922
-
In OPNsense 20.7.2, /usr/local/opnsense/scripts/unbound/dnsbl.py was changed. The links for the Blockist.site lists were updated. Previously, the Ads link was https://blocklist.site/app/dl/ads which does a re-direct. In 20.7.2, the Ads link is https://blocklistproject.github.io/Lists/ads.txt. Now, OPNsense correctly retrieves the Blocklist.site lists.
During my earlier testing, when I enabled Blacklist but didn't select any additional lists and saved, the generated dnsbl.conf was about 17 MB. A default blacklist was enabled. When I selected the 3 updated Blocklist.site lists and saved, the dnsbl.conf file was about 25 MB. That shows me that the links for Blocklist.site are working.
-
One thing is not clear: I cannot creat a cron job (see attachment), so when/how does the update happen ??
-
When I was testing, I just used the GUI Services: Unbound DNS: Blacklist and hit the Save button to manually update DNSBL.
I'm logging-in to the GUI using root and I have no problem creating cron entries in System: Settings: Cron
I selected Download Unbound DNSBLs and restart. I run it early each morning.
After you Save and Apply, login to the shell and look at /var/cron/tabs/nobody. See if there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl" (or maybe unboundplus depending on which "Download Unbound DNSBLs and restart" you selected in the GUI.
-
When I was testing, I just used the GUI Services: Unbound DNS: Blacklist and hit the Save button to manually update DNSBL.
I'm logging-in to the GUI using root and I have no problem creating cron entries in System: Settings: Cron
I selected Download Unbound DNSBLs and restart. I run it early each morning.
After you Save and Apply, login to the shell and look at /var/cron/tabs/nobody. See if there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl" (or maybe unboundplus depending on which "Download Unbound DNSBLs and restart" you selected in the GUI.
Do you have two Download Unbound entries? When I look at what they actually enter into cron one is for Unbound and one is for UnboundPlus.
-
Do you have two Download Unbound entries? When I look at what they actually enter into cron one is for Unbound and one is for UnboundPlus.
I have two entries, and does it make any difference on which one I select ?
I selected the first one and if I look at "/var/cron/tabs/nobody" I see there's an entry for executing "/usr/local/sbin/configctl unbound dnsbl"
-
The first one is preferred. Both currently work.
https://forum.opnsense.org/index.php?topic=18951.msg86657#msg86657 (https://forum.opnsense.org/index.php?topic=18951.msg86657#msg86657)
-
Great, thanks !
-
I have the strangest issue with DNSBL on 20.7.2. Seems that the "enable" checkbox under Services: Unbound DNS: Blacklist is reversed.
When the "enable" checkbox is checked:
1. /var/unbound/etc/dnsbl.conf is empty
2. /var/unbound/etc/lists.inc has my list source from https://block.energized.pro/ultimate/formats/domains.txt
When the "enable" checkbox is UNchecked:
1. /var/unbound/etc/dnsbl.conf is populated with entires from https://block.energized.pro/ultimate/formats/domains.txt
2. /var/unbound/etc/lists.inc is blank
Seems that entries from https://block.energized.pro/ultimate/formats/domains.txt resolve to localhost only when the "enable" checkbox is UNchecked. However, when it is checked we resolve names to actual addresses.
Is this expected? Something seems amiss.
Thank you folks!
-
Can you tick enable and apply several times?
-
Can you tick enable and apply several times?
Oh yes, I did. Even with a reboot between ticking and unticking. It takes a lot longer to "save" changes when unticking the enable box as it is downloading the list.
What I ended up doing is backing up populated /var/unbound/etc/dnsbl.conf, ticking the box and restoring /var/unbound/etc/dnsbl.conf. This seems to hold after reboot, but probably not a long term fix.
-
This behaviour is a bit weird and the logic behind is still unclear for me, but yes, if u disable/enable and hit apply a bunch of times in an eclectic style somewhen it works 8)
-
I have the strangest issue with DNSBL on 20.7.2. Seems that the "enable" checkbox under Services: Unbound DNS: Blacklist is reversed.
When the "enable" checkbox is checked:
1. /var/unbound/etc/dnsbl.conf is empty
2. /var/unbound/etc/lists.inc has my list source from https://block.energized.pro/ultimate/formats/domains.txt
When the "enable" checkbox is UNchecked:
1. /var/unbound/etc/dnsbl.conf is populated with entires from https://block.energized.pro/ultimate/formats/domains.txt
2. /var/unbound/etc/lists.inc is blank
Seems that entries from https://block.energized.pro/ultimate/formats/domains.txt resolve to localhost only when the "enable" checkbox is UNchecked. However, when it is checked we resolve names to actual addresses.
Is this expected? Something seems amiss.
Thank you folks!
Seeing the exact same issue on 20.1.9_1
-
I'm seeing similar inconsistencies when applying changes to blacklists. It's unclear if saving on the blacklist page reloads the lists or if you need to restart the unbound service from the top page.
I'm also unable to get whitelists to work using a github-hosted text file that contains a list of hosts. Has anyone been able to get that to work?
-
Whitelist only works with domains, not URL where to retrieve them
-
Ok. got it- the trick is simple untick and hit safe a couple of times and optionally watch the size of dsbl.conf to get sure.
Now I am trying the opposite way, to disable blocking without success:
I disabled all lists and disabled blocking feature as well the size of dsb.conf is back to zero ok.
To get sure I restarted unbound but the client was still unable to resolve one domain I picked from the dsbl.conf...hmm. I also tried to reboot opnsense but the client request was still resolved to 0.0.0.0 ?
Is it a server or client caching issue somehow or does the following unbound options maybe speed that up:
Prefetch Support
and
Prefetch DNS Key Support
What I did not tried is to reboot the client after disabling the blocking feature maybe it's the way to go ?
-
If the conf is empty and unbound restartet it should block anything.
-
It should ;-)
-
not 8)
-
@mimugmail are you able to enter more than 1 domain or regex in the whitelist field and get it to work?
-
I am having some delays with blacklist within unbound and was hoping for some clarification. When I select a new list, hitting save doesn't seem to pull an immediate download and update. Do I also need to restart unbound? What would trigger an immediate download?
-
Hi all, just one questions. I switched from manual blacklist to the Unbound blacklist feature.
The blacklist entries should be stored in the /var/unbound/etc/dnsbl.conf or I am wrong here?
In my case the file is still empty after enabling the feature, even so all files in /var/unbound/etc/
Where are the entries stored?
br
-
Hi Mks,
yes, the entries be stored in the /var/unbound/etc/dnsbl.conf.
The download or the update is controlled by the file:
/usr/local/opnsense/scripts/unbound/download_blacklists.py
root@xxx:/var/unbound/etc # ls -lsa
total 39416
4 drwxr-x--- 2 unbound unbound 512 Oct 23 07:16 .
4 drwxr-xr-x 9 unbound unbound 512 Oct 25 04:18 ..
4 -rw-r----- 1 unbound unbound 1295 Oct 25 04:17 blacklists.ini
39392 -rw-r----- 1 unbound unbound 40285646 Oct 25 13:21 dnsbl.conf
4 -rw-r----- 1 unbound unbound 42 Oct 23 06:21 dnsbl.inc
0 -rw-r----- 1 unbound unbound 0 Oct 25 04:17 dot.conf
4 -rw-r----- 1 unbound unbound 78 Oct 23 06:21 lists.inc
0 -rw-r----- 1 unbound unbound 0 Oct 25 04:17 miscellaneous.conf
4 -rw-r----- 1 unbound unbound 106 Oct 23 06:21 whitelist.inc
Greetings meditux
-
Ok thanks!
Just doublechecked, it works.
br
-
I am running v20.7.4 and I have a fundamental question/issue with Unbound blacklist.
In Services:Unbound DNS:Blacklist, I enable lists, select lists, and Save. /var/unbound/etc/dnsbl.conf is created/updated
In System:Settings:Cron, add "Download Unbound DNSBLs and restart" which will execute "/usr/local/sbin/configctl unbound dnsbl". /var/unbound/etc/dnsbl.conf is created/updated
However, Unbound is not updated. New blacklist is not loaded into Unbound. Either adding or removing blacklist entries.
I have to stop & restart Unbound in Lobby:Dashboard. This is not desirable.
Or
In a shell, execute "/usr/local/sbin/configctl unbound reload"
If /var/unbound/etc/dnsbl.conf is null, "/usr/local/sbin/configctl unbound reload" reloads immediately. If /var/unbound/etc/dnsbl.conf has a lot of entries, Unbound takes a while to reload the new dnsbl.conf as expected.
Am I missing something?
A related very minor detail in Services:Unbound DNS:Blacklistl: With no lists enabled, hit "Save" and it returns immediately. With multiple lists enabled, hit "Save" and the animation of the Save button never ends even after dnsbl.conf is created.
On a somewhat related item:
I am using a PCEngines apu2 and using memory file system for /var & /tmp.
At bootup, is there a recommended way to execute "/usr/local/sbin/configctl unbound dnsbl" or equivalent and create dnsbl.conf? Then, execute "/usr/local/sbin/configctl unbound reload"?
In /etc/crontab @reboot, I could simply run a little script that sleeps for a while to let everything start and then execute the 2 commands.
Thanks
-
I found this blacklist, it looks interesting. There are duplicate domains but a lot are new compare to the Unbound's list (~700.000+)
https://dbl.oisd.nl/ (https://dbl.oisd.nl/)
Ciao
-
I am experiencing this problem.
However, it seems that when I hit apply, the little animated ellipse shows up. If I leave it alone in this state for a while until it returns, then it works. If I select away to a different menu item it breaks it.