I managed to get ad blocking done using only Unbound. Our command line friends,
curl and
awk do the heavy lifting.
This post is a little messy right now, but I'll get it cleaned up a little later.
HOUSEKEEPING INFO:You will need to be able to log into the machine using either the console or SSH to get this accomplished.
The list I'm using is Steven Black's "Unified+Gambling+Fake News". You can choose from any of his other lists at https://github.com/StevenBlack/hosts/blob/master/readme.md (https://github.com/StevenBlack/hosts/blob/master/readme.md). Rather than using one of the "Raw Hosts" URLS, choose one of the "Non Github Mirror" URLS. Curl seems to have an issue with the "Raw Hosts" URLS.
GETTING STARTED:
- Make sure you've got a fully working Unbound setup.
- Then, log in via the command line, choose #8 and create a directory to house your working files.
- If you want to use the script as-is, then use "/usr/share/blocklist"
- Place the following script in the directory and name it "getlist.sh" (also as attachment)
If you've opted to use a different filter list, you'll have to change the address that curl is using in the following script.
#!/bin/sh
#
# First, let's get the list
curl http://sbc.io/hosts/alternates/fakenews-gambling/hosts -o blockhosts.txt
# Too bad it's got a lot of stuff at the beginning we don't need.
# Time to remove the cruft
awk 'NR>=35{ print }' blockhosts.txt > hosts-fixed.txt
# Time to make it usable for the unbound service
echo -n "server:" > block.conf # MUST HAVE THIS AT THE TOP OF THE FILE
cat hosts-fixed.txt | grep '^0\.0\.0\.0' | awk '{ print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\"" }' >> block.conf
# Present it to unbound
chown unbound:unbound block.conf
mv block.conf /var/unbound/block.conf
unbound-control -c /var/unbound/unbound.conf reload
- Make the script executable: chmod u+x /usr/share/blocklist/getlist.sh
- You must run the script ONCE from the command line or this will fail: /usr/share/blocklist/getlist.sh
- ** Hint: don't log out just yet....
Here's where you add the block list to Unbound.
- Open your browser and log into the opnsense page.
- Navigate to Services -> Unbound DNS -> General (https://<your_machine>/services_unbound.php (https://<your_machine>/services_unbound.php))
- Scroll down the "General" page until you see the button for Advanced and click on it.
- In the "Custom Options" box, enter: include: /var/unbound/block.conf
- Save your changes and then apply them.
Provided everything was entered correctly, the list should now be applied.
Since the "Cron" opnsense page doesn't allow for custom actions, you'll have to head back to the command line.
I've opted to update the list every night at 23:00 local.
If you're still logged in, keep going. Otherwise, log back into the command line and choose option #8.
Then:
- Type crontab -e, press Enter and go to the end of the file (you may have to hit the "End" key on the last line)
- Press a
- Press ENTER
- Type: 0 23 * * * (/usr/share/blocklist/getlist.sh) > /dev/null
- Press ESC, then : and finally wq!
Your system should now update the list, mangle it into a format that Unbound can use and reload the list every night at 2300 (local).
I'm working on making a command line install script to make the process a little easier for inexperienced people. Please be patient.
This is great, thanks for sharing! I was waiting for UnboundBL to materialize, but this will work nicely instead..
I'll be keeping an eye on this--would love to get a blacklist feature working
Thanks!
https://github.com/opnsense/plugins/pull/1495
Hey buddy I saw your post of blocking ad by unbound I applied same but not working ....plzz tell me how to set up the unbound ...moreover by this I can block my own define websites or not??
Which post do you mean?
Leave that..sir....just tell me that how to block domain aur websites in opn sense....I tried your app detection method in IPS but by that I can only block limited sites...if I want to block any other perticular site like Wikipedia...Amazon so how I can do it..
Transparent Proxy and blacklisted sites.
Sir I want to know how to use black listed sites I tried my best but can't able to do show...plz can u tell me step by step
Please
https://docs.opnsense.org/manual/how-tos/proxytransparent.html
With option to only log SNI, then you can use the blacklist feature in access control
Ok thanku sir I will try this....by the way how enable this SNI option
Just follow the guide, you will see it :)
Thanku so much...I will try it ...but I want your support also...I want to learn this firewall ...kepp supporting sir
Hello again...I tried this method and I was successful ...but still I need one help ...first of all tell me why we are using transperant proxy in this web filtering what is the role of that...means I am able to block the domain by web filter without use of transparent proxy...so what is the use of transparent proxy please tell me
Hello there :)
first i'm sorry for my poor english it is not my native language and i'm better in reading it then writing ::)
I'm using a similar solution and i'm writing here to show you the problems you can have with your script.
And i don't want create another DNS-Block thread.
Quoteunbound-control -c /var/unbound/unbound.conf reload
Is a very bad Solution ... it will fail if the list is to big
QuoteThen:
Type crontab -e, press Enter and go to the end of the file (you may have to hit the "End" key on the last line)
Press a
Press ENTER
Type: 0 23 * * * (/usr/share/blocklist/getlist.sh) > /dev/null
Press ESC, then : and finally wq!
It is maybe better to create a action-file in /usr/local/opnsense/service/conf/actions.d to configure cron via web-interface
I will now show the script iam currently using you have to install wget and bash via pkg to use it.
Feel free to edit it to your needs any hints to make the script better is very much appreciated.
ee /root/adblockscriptand insert
#!/usr/local/bin/bash
#Erstelle Temp Datein
tmp1="$(mktemp)"
tmp2="$(mktemp)"
tmp3="$(mktemp)"
tmp4="$(mktemp)"
tmp5="$(mktemp)"
tmp6="$(mktemp)"
tmp7="$(mktemp)"
file="/var/unbound/adblocklist.conf"
# Download Blocklist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/ultimate/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/xtreme/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/regional/formats/domains.txt; \
} > $tmp1
# Download Whitelist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt; \
cat /root/whitelist; \
} > $tmp4
# Saeubere die Blocklisten
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp1
cat $tmp1 | tr -d '\r' >> $tmp2
sed -i '' -e 's/ *$//' $tmp2 && sort -uf $tmp2 |tee |uniq -i > $tmp3
sed -i '' -e '/^$/d' $tmp3
rm $tmp1 $tmp2
# Saeubere die Whitelist
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp4
cat $tmp4 | tr -d '\r' >> $tmp5
sed -i '' -e 's/ *$//' $tmp5 && sort -uf $tmp5 |tee |uniq -i > $tmp6
sed -i '' -e '/^$/d' $tmp6
rm $tmp4 $tmp5
# Fuege Blocklist und Whitelist zusammen
comm -23 $tmp3 $tmp6 > $tmp7
rm $tmp3 $tmp6
sed -i '' -e '/^$/d' $tmp7
awk '$0="local-zone: \""$0"\" static"' $tmp7 > $file
rm $tmp7
if [ "$1" == info ]; then
domains=$(awk '!/^#/ && !/^$/{c++}END{print c}' $file | awk '{ len=length($0); res=""; for (i=0;i<=len;i++) { res=substr($0,len-i+1,1) res; if (i > 0 && i < len && i % 3 == 0) { res = "," res } }; print res }')
echo "Anzahl zu Blockender Domains = $domains"
fi
# Aendere Benutzer und Gruppe
chown unbound:unbound $file
# Starte Unbound neu
pluginctl dns
exit 0
To create the Whitelist-file:
echo '# Domains nach dieser Zeile einfuegen. Diese Zeile nichr loeschen!' > /root/whitelistIn the Whitelist-File you can add domains after the first line that shouldn't be blocked.
Create the action-File for cron in "/usr/local/opnsense/service/conf/actions.d"
ee /usr/local/opnsense/service/conf/actions.d/actions_AdBlock.confand insert
[reload]
command:/root/adblockscript
parameter:
type:script
message:Update AdBlocklist and load them
description:Update AdBlocklist and load them
then
configctl configd restartnow you can configure the Cron Job via Web Interface
to install bash and wget
pkg lock -y pkg
pkg install bash wget
pkg unlock -y pkg
make the script executable:
chmod +x /root/adblockscriptrun the script via
./adblockscript infoit shows how many domains are blocked and create the blockfile for unbound
I'm using the root folder as working environment i know it is kind of laziness ::)
For the blacklist i am using: https://github.com/EnergizedProtection/block (https://github.com/EnergizedProtection/block)
Output-File is:
/var/unbound/adblocklist.conf <= insert this path in your Unbound config
Thanks for your patience and maybe this help someone ;D
Sinnce https://energized.pro started to publish Unbound-readable blocklists, we should simplify our tutorial to something like this:
1. add
include: /var/unbound/ad-blacklist.conf into Custom options of Unbound
2. Create
Ad-blacklist-refresh.sh in /var/unbound with:
curl https://raw.githubusercontent.com/EnergizedProtection/block/master/blu/formats/unbound.conf -o /var/unbound/ad-blacklist.conf
3. insert a regular execution of this script into crontab
There are multiple levels of compiled lists by energized.pro team - just pick the right strength, let Unbound use it and enjoy ad-free browsing.
Quote from: mihak on January 18, 2020, 09:15:04 PM
Sinnce https://energized.pro started to publish Unbound-readable blocklists, we should simplify our tutorial to something like this:
1. add include: /var/unbound/ad-blacklist.conf into Custom options of Unbound
2. Create Ad-blacklist-refresh.sh in /var/unbound with:
curl https://raw.githubusercontent.com/EnergizedProtection/block/master/blu/formats/unbound.conf -o /var/unbound/ad-blacklist.conf
3. insert a regular execution of this script into crontab
There are multiple levels of compiled lists by energized.pro team - just pick the right strength, let Unbound use it and enjoy ad-free browsing.
Hello there ;D
yes your are right you can do that but using a script has also his benefits.
- you can combine more then one list
- you can use a whitlist and a blocklist created by your own
- you are fail safe if the original list has character like ( / \ ) see: https://github.com/EnergizedProtection/block/issues/326 (https://github.com/EnergizedProtection/block/issues/326)
- you can optimize the list for Unbound while create a case insensitive list this will shrink the size
and two personal reason
- i dont like to mess with the crontable im not internally shure but think it is possible that the crontable will be overwrite at update/upgrade
- maybe you learn a bit in writing scripts
If you like it realy easy and wont do anything it is better use a addon for Opnsense like: https://forum.opnsense.org/index.php?topic=14116.0 (https://forum.opnsense.org/index.php?topic=14116.0)
Thanks Itow
-edit
I have add a function to check the config file of unbound to be more fail safe
#!/usr/local/bin/bash
#Erstelle Temp Datein
tmp1="$(mktemp)"
tmp2="$(mktemp)"
tmp3="$(mktemp)"
tmp4="$(mktemp)"
tmp5="$(mktemp)"
tmp6="$(mktemp)"
tmp7="$(mktemp)"
file="/var/unbound/adblocklist.conf"
filebackup="/var/unbound/adblocklist.bck"
# Download Blocklist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/ultimate/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/xtreme/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/regional/formats/domains.txt; \
} > $tmp1
# Download Whitelist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt; \
cat /root/whitelist; \
} > $tmp4
# Saeubere die Blocklisten
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp1
cat $tmp1 | tr -d '\r' >> $tmp2
sed -i '' -e 's/ *$//' $tmp2 && sort -uf $tmp2 |tee |uniq -i > $tmp3
sed -i '' -e '/^$/d' $tmp3
rm $tmp1 $tmp2
# Saeubere die Whitelist
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp4
cat $tmp4 | tr -d '\r' >> $tmp5
sed -i '' -e 's/ *$//' $tmp5 && sort -uf $tmp5 |tee |uniq -i > $tmp6
sed -i '' -e '/^$/d' $tmp6
rm $tmp4 $tmp5
# Backup der alten Blocklist
if [ -f $file ]; then
mv $file $filebackup
fi
# Fuege Blocklist und Whitelist zusammen
comm -23 $tmp3 $tmp6 > $tmp7
rm $tmp3 $tmp6
sed -i '' -e '/^$/d' $tmp7
awk '$0="local-zone: \""$0"\" static"' $tmp7 > $file
# Check Unbound Config
if ! unbound-checkconf /var/unbound/unbound.conf; then
rm $file
echo Nutze alte Blockliste
mv $filebackup $file
else
# echo Config ist Okey
rm $filebackup
fi
if [ "$1" == info ]; then
domains=$(awk '!/^#/ && !/^$/{c++}END{print c}' $file | awk '{ len=length($0); res=""; for (i=0;i<=len;i++) { res=substr($0,len-i+1,1) res; if (i > 0 && i < len && i % 3 == 0) { res = "," res } }; print res }')
echo "Anzahl zu Blockender Domains = $domains"
fi
# Aendere Benutzer und Gruppe
chown unbound:unbound $file
# Starte Unbound neu
pluginctl dns
exit 0
if you do not add server: to the start of the first line of .conf file unbound will not start.
I only could start Unbound with that:
example:
server:local-zone: "0--ass-cinema-newsp.da.ru" static
local-zone: "0--bondage.dk" static
local-zone: "0--fightingshaving.da.ru" static
local-zone: "0--foodwarez.da.ru" static
Cheers and thanks for your work and scripts, Working like a charm.
Quote from: eprom on January 23, 2020, 04:51:41 PM
if you do not add server: to the start of the first line of .conf file unbound will not start.
I only could start Unbound with that:
example:
server:local-zone: "0--ass-cinema-newsp.da.ru" static
local-zone: "0--bondage.dk" static
local-zone: "0--fightingshaving.da.ru" static
local-zone: "0--foodwarez.da.ru" static
Cheers and thanks for your work and scripts, Working like a charm.
Hi,
usually it is not nethethery to add anything to the created Blocklist.
Please check if you have add
include: /var/unbound/adblocklist.conf in your Unbound configuration under Custom options in the Web interface. And try remove any addition to the custom options to check if Unbound will function normally without it.
-
I have added a check if the crucial line exist in the unbound config.
Also added checks if Custom Whitelist and Blacklist exist.
#!/usr/local/bin/bash
#Erstelle Temp Datein
tmp1="$(mktemp)"
tmp2="$(mktemp)"
tmp3="$(mktemp)"
tmp4="$(mktemp)"
tmp5="$(mktemp)"
tmp6="$(mktemp)"
tmp7="$(mktemp)"
file="/var/unbound/adblocklist.conf"
filebackup="/var/unbound/adblocklist.bck"
whitelist="/root/whitelist"
blacklist="/root/blacklist"
# Backup der alten Blocklist
if [ -f $file ]; then
mv $file $filebackup
fi
if ! [ -f $whitelist ]; then
touch $whitelist
echo '# Domains nach dieser Zeile einfuegen. Diese Zeile nichr loeschen!' > $whitelist
fi
if ! [ -f $blacklist ]; then
touch $blacklist
echo '# Domains nach dieser Zeile einfuegen. Diese Zeile nichr loeschen!' > $blacklist
fi
# Pruefe Custom White and Blacklist
# Download Blocklist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/ultimate/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/xtreme/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/regional/formats/domains.txt; \
cat $blacklist; \
} > $tmp1
# Download Whitelist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt; \
cat $whitelist; \
} > $tmp4
# Saeubere die Blocklisten
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp1
cat $tmp1 | tr -d '\r' >> $tmp2
sed -i '' -e 's/ *$//' $tmp2 && sort -uf $tmp2 |tee |uniq -i > $tmp3
sed -i '' -e '/^$/d' $tmp3
rm $tmp1 $tmp2
# Saeubere die Whitelist
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp4
cat $tmp4 | tr -d '\r' >> $tmp5
sed -i '' -e 's/ *$//' $tmp5 && sort -uf $tmp5 |tee |uniq -i > $tmp6
sed -i '' -e '/^$/d' $tmp6
rm $tmp4 $tmp5
# Fuege Blocklist und Whitelist zusammen
comm -23 $tmp3 $tmp6 > $tmp7
rm $tmp3 $tmp6
sed -i '' -e '/^$/d' $tmp7
awk '$0="local-zone: \""$0"\" static"' $tmp7 > $file
# Check Unbound Config
if ! unbound-checkconf /var/unbound/unbound.conf; then
rm $file
echo Nutze alte Blockliste
mv $filebackup $file
exit 1
else
# echo Config ist Okey
if ! grep -cq "include: $file" /var/unbound/unbound.conf; then
echo Please add "include: $file" to your custom Unbound settings.
rm $filebackup
fi
fi
if [ "$1" == info ]; then
domains=$(awk '!/^#/ && !/^$/{c++}END{print c}' $file | awk '{ len=length($0); res=""; for (i=0;i<=len;i++) { res=substr($0,len-i+1,1) res; if (i > 0 && i < len && i % 3 == 0) { res = "," res } }; print res }')
echo "Anzahl zu Blockender Domains = $domains"
fi
# Aendere Benutzer und Gruppe
chown unbound:unbound $file
# Starte Unbound neu
pluginctl dns
Thanks Itow
Hello there :)
yesterday i got following error:
[1580222648] unbound-checkconf[27733:0] error: cannot parse name queda212..duckdns.org
[1580222648] unbound-checkconf[27733:0] error: bad zone name queda212..duckdns.org static
[1580222648] unbound-checkconf[27733:0] fatal error: failed local-zone, local-data configuration
So i update the script to replace more then one dot with a single dot.
#!/usr/local/bin/bash
oldtmp="$(find /tmp -type f -name 'tmp.*' | wc -l)"
if [ $oldtmp -gt 0 ]; then
echo Found $oldtmp old tmp-files.
echo Delete the old Files
find /tmp -type f -name 'tmp.*' -exec rm -f {} \;
fi
#Erstelle Temp Datein
tmp1="$(mktemp)"
tmp2="$(mktemp)"
tmp3="$(mktemp)"
tmp4="$(mktemp)"
tmp5="$(mktemp)"
tmp6="$(mktemp)"
tmp7="$(mktemp)"
file="/var/unbound/adblocklist.conf"
filebackup="/var/unbound/adblocklist.bck"
whitelist="/root/whitelist"
blacklist="/root/blacklist"
actionfile="/usr/local/opnsense/service/conf/actions.d/actions_AdBlock.conf"
# Backup der alten Blocklist
if [ -f $file ]; then
mv $file $filebackup
else
touch $filebackup
echo '# Empty File' > $filebackup
fi
# Pruefe Custom White and Blacklist
if ! [ -f $whitelist ]; then
touch $whitelist
echo '# Domains nach dieser Zeile einfuegen. Diese Zeile nicht loeschen!' > $whitelist
fi
if ! [ -f $blacklist ]; then
touch $blacklist
echo '# Domains nach dieser Zeile einfuegen. Diese Zeile nicht loeschen!' > $blacklist
fi
# Pruefe und Erstelle Actionfile fuer Cron
if ! [ -f $actionfile ]; then
touch $actionfile
printf "[reload]\ncommand:/root/adblockscript\nparameter:\ntype:script\nmessage:Update AdBlocklist and load them\ndescription:Update AdBlocklist and load them" >> $actionfile
service configd restart
echo 'Now you can configure the Cron Job via Web Interface'
fi
# Download Blocklist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/ultimate/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/xtreme/formats/domains.txt; \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/block/master/extensions/regional/formats/domains.txt; \
cat $blacklist; \
} > $tmp1
# Download Whitelist
{ \
wget -qO- https://raw.githubusercontent.com/EnergizedProtection/unblock/master/basic/formats/domains.txt; \
cat $whitelist; \
} > $tmp4
# Saeubere die Blocklisten
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp1
cat $tmp1 | sed -r 's/\.+/\./' | tr -d '\r' >> $tmp2
sed -i '' -e 's/ *$//' $tmp2 && sort -uf $tmp2 |tee |uniq -i > $tmp3
sed -i '' -e '/^$/d' $tmp3
rm $tmp1 $tmp2
# Saeubere die Whitelist
sed -i '' -e '/\//d;/:/d;/(/d;/|/d;/\[/d;/\]/d;/#/d;/^$/d;/[\]/d' $tmp4
cat $tmp4 | sed -r 's/\.+/\./' | tr -d '\r' >> $tmp5
sed -i '' -e 's/ *$//' $tmp5 && sort -uf $tmp5 |tee |uniq -i > $tmp6
sed -i '' -e '/^$/d' $tmp6
rm $tmp4 $tmp5
# Fuege Blocklist und Whitelist zusammen
comm -23 $tmp3 $tmp6 > $tmp7
rm $tmp3 $tmp6
sed -i '' -e '/^$/d' $tmp7
awk '$0="local-zone: \""$0"\" static"' $tmp7 > $file
rm $tmp7
# Check Unbound Config
if ! unbound-checkconf /var/unbound/unbound.conf; then
rm $file
echo Nutze alte Blockliste
mv $filebackup $file
exit 1
else
# echo Config ist Okey
if ! grep -cq "include: $file" /var/unbound/unbound.conf; then
echo Please add "include: $file" to your custom Unbound settings.
fi
rm $filebackup
fi
if [ "$1" == info ]; then
domains=$(awk '!/^#/ && !/^$/{c++}END{print c}' $file | awk '{ len=length($0); res=""; for (i=0;i<=len;i++) { res=substr($0,len-i+1,1) res; if (i > 0 && i < len && i % 3 == 0) { res = "," res } }; print res }')
echo "Anzahl zu Blockender Domains = $domains"
fi
# Aendere Benutzer und Gruppe
chown unbound:unbound $file
# Starte Unbound neu
pluginctl dns
exit 0
Thanks Itow
--Edit 30.01.20
fix Typos and few Issues add new checks
Thanks for the tutorial.
BTW as of today the github link to the host list works fine for me while the non-github link is timing out.
Is this tutorial sill valid?
I've just installed the unbound-plus plugin and selected which block list use. Did I go wrong?
Quote from: Jul1991 on July 11, 2020, 09:52:53 AM
Is this tutorial sill valid?
I've just installed the unbound-plus plugin and selected which block list use. Did I go wrong?
No, unbound-plus is enough
Is there a way to have a dashboard and a detailed log to have an idea on what's been blocked?
Quote from: Jul1991 on July 11, 2020, 09:59:49 AM
Is there a way to have a dashboard and a detailed log to have an idea on what's been blocked?
+1000 to this request
We have web security when blocking URLs 8) but we don't have the possibility to check who is blocking them :-[.