Firewall Alias for adblocking

[Curly060]

Hi,

I migrated from OpenWRT to OPNSense in the past couple of weeks and can't believe that all these years I wasn't aware of pfSense/OPNSense! I am very happy with pretty much everything, except for the adblocking situation (and perhaps the google rank situation  ).

What I would really like to see is DNS based adblocking. I have searched the forum, but the resolution always seems to be transparent proxy (really not an option for me) or some firewall rule with a set of IPs (alias). The latter I am trying to implement:
So I have created a very big list of IP addresses and Domains from various sources (PI-Hole, OpenWRT adblock plugin etc.). The list contains about 120000 entries. Then I created an alias like this:

• Name: Adblock
• Type: URL Table (IPs)
• URL: URL to my list (not public)

Now I do have some questions about this:

• Is it ok to have a mixture of IPs and domain names in my list? I would say yes, OPNSense seems to resolve domains in the background and creates a text file in /var/db/aliastables/Adblock.txt which contains only IPs.
• Is there a limit as to how many IPs I can have for an alias?
• What is a healthy amount of IPs inside an alias? Would it be
• When I look in /var/db/aliastables/Adblock.txt file I notice a lot of duplicate IPs. Should this maybe optimized or does it not matter?
• What is OPNSense's strategy when the file changes? Will it only look at differences or parse the file fully each time?
• My Alias does not show up under Firewall: Diagnostics: pfTables. However, if I create the alias with a non capital first letter, then it will appear there. Bug?
• Firewall: Diagnostics: pfTables tells me that there are no entries in my alias. However, when I make my list smaller, suddenly it will show the IP addresses. I have experimented a little: 5000 entries were fine, 10000 already not. So it looks there is indeed a limit somewhere.
• Are there any plans to integrate DNS based adblocking? Sure I can set up a Pi next to my OPNSense with PI-Hole on it, but I'd prefer the all-in-one solution. Would the DNS based adblocking not be far, far more efficient? I mean, only when a resource is actually requested, the DNS resolver would have to check against the black list (might be costly, but can be cached). As firewall rule with, say 5000 entries, this needs to be checked for pretty much every packet, no? Transparent proxy adblock is IMHO far too complex (CA certificate on every client, maintaing no-ssl-bump list,...)

Sorry for the sheer volume of the questions, I hope someone takes time to answer them. I have been googling a lot (which is painful because google always returns results for pfSense. OPNSense must get more popular to drive pfSense off the first ranks ) and the forum has only like 2-3 posts about adblocking...


Cheers, Curly060 =;->

[bartjsmit]

There is a third option - set your DNS to use OpenDNS and create an account there with adware as part of the Web Content Filtering.

Bart...

[paramedic233]

Fourth option, and what I have running.
https://docs.opnsense.org/manual/how-tos/cachingproxy.html
Simple and works for me.

[Curly060]

Hi,

thanks for the answers.
OpenDNS is not an option for me. The whole point of my OpnSense installation is that I do not need to rely on external services (esp. when their owners are located in the States...). I prefer having as much under my direct control as possible.

Using a proxy is fine, but every client needs to be configured. That will end up in configuration hell and whenever a client bypasses the proxy, I get ads again. So it would have to be a transparent proxy, which I didn't want in the first place.

Well, I have solved the problem in the meantime like this:

• a custom script on a Raspberry Pi creates a hosts file from various sources every week (similar to what Pi Hole does) and uploads it to OpnSense
• Dnsmasq is configured to use this file as additional hosts file

=> Works perfectly and for all clients in my LAN

I was hoping I do not have to rely on the Pi to do this, but well, that's how it is now.

Cheers, Curly060 =;->

[bartjsmit]

OPNsense has a full OS and it may be possible to run your RPi script on the firewall. The best way to slot it in is through a plugin, if you're up for a bit of scripting.

Bart...

[Curly060]

Well, that would for sure be the cleanest and best solution. My scripts use bash, curl, grep and sed, so they should definitely run directly on the FW. However, I did not have the time yet to look into writing my own plugin.

So yes, I am definitely up for a bit of scripting, but where to start? Here? https://docs.opnsense.org/development/examples/helloworld.html

Cheers, Curly060 =;->

[bartjsmit]

Yes, that is definitely the start of your journey.

Send postcards ;-)

Bart...

[Nnyan]

Would you be willing to share your scripts?

[mimino]

Hello crowd,
Newbee in opnSense here, and I'm trying to accomplish something similar to OP.
I run a script on the same box that aggregates hosts from multiple sources, then I simply copy the resulting file to /usr/local/www. My adblock alias is set up to grab the URL Table from local web server. Not sure if there's a better way to do it, but I figured this would be simple enough and should work.
Now, the questions OP have asked are still very relevant. Could somebody from local gurus step in and try to answer them? Especially these:
 

• Is there a limit as to how many IPs I can have for an alias?
• What is a healthy amount of IPs inside an alias?

Any input will be greatly appreciated.

[NightShade]

Not a firewall setup but should work pretty well for anyone who can setup the DNS Resolver and set their clients to use the OpnSense box as their DNS server.

https://devinstechblog.com/block-ads-with-dns-in-opnsense/

I was able to get it up and running and it does seem to mostly work so far but it may need some tweaking.  I do have to say that the end of each line in the file linked needs to be removed since it was likely saved on a windows machine and has "^M" at the end of each line but that is a simple fix and it should be looked over before using anyway.

The script itself is based off of https://wuffleton.com/code/unbound-blacklist/

You need to add the line "include: /var/unbound/ad-blacklist.conf" to the custom options in the DNS resolver.

If anyone with more experience than I have wants to look it over and make modifications and/or adjustments please feel free to do so.  I just figured this would be helpful to others who do not want to use a service for ad blocking when they can manage something in house.

[Curly060]

Would you be willing to share your scripts?

Sorry, only saw this now...

I should probably clean it up and parameterize some stuff, but well, here it is, unrevised, as is, use at your own risk:

Code: [Select]

#!/bin/bash
filter() {
  # filter out lines with the following criteria:
  # - comment lines (starts with optional blank followed by hash)
  # - empty lines (only blank or completely empty)
  # - lines without any dot
  # - lines with ' localhost' inside
  grep -v -e '^[[:space:]]*#' -e '^[[:space:]]*$' -e '^[^.]*$' -e ' localhost' | tr ' ' ' '| sed -e 's#\s\+$##'
}
adblockListTemp=/tmp/adblocklist.temp
adblockListFinal=/tmp/adblock.list
webserverIp=${1:-192.168.1.201}
:> $adblockListTemp
# Adblock
curl -s https://adaway.org/hosts.txt | filter | cut -f2 -d' ' >> $adblockListTemp
# Shallalist
curl -s http://www.shallalist.de/Downloads/shallalist.tar.gz | tar -C /tmp -xz
cat /tmp/BL/{adv,costtraps,spyware,tracker,warez}/domains >> $adblockListTemp
rm -rf /tmp/BL/
# Yoyo
curl -s 'https://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext' >> $adblockListTemp
### PI-Hole
curl -s https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s http://mirror1.malwaredomains.com/files/justdomains >> $adblockListTemp
curl -s http://sysctl.org/cameleon/hosts | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist' >> $adblockListTemp
curl -s https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt >> $adblockListTemp
curl -s https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt >> $adblockListTemp
curl -s https://hosts-file.net/ad_servers.txt | tr -d '\r' | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt | tr -d '\r' | cut -f2 -d' ' >> $adblockListTemp
### Clean
cat $adblockListTemp | filter | grep -vP '^(\d+\.){3}\d+$' | sed -e "s#^#$webserverIp #" | sort -u > $adblockListFinal
scp -i adblock.key -o UserKnownHostsFile=adblock.known_hosts $adblockListFinal adblock@opnsense:./adblock.list
rm $adblockListTemp $adblockListFinal

nginx snippet:

Code: [Select]

server {
  listen 80 default_server;
  listen 443 ssl http2 default_server;
  server_name 192.168.1.201;

  error_log /var/log/nginx/opnsense-hole.error.log info;
  access_log /var/log/nginx/opnsense-hole.access.log;

  ssl_session_cache shared:SSL_opnsense-hole:10m;
  ssl_certificate /etc/ssl/opnsense-hole.cert.pem;
  ssl_certificate_key /etc/ssl/opnsense-hole.key.pem;
  location / {
    expires max;
    return 204;
  }
}

dnsmasq advanced setting:

Code: [Select]

addn-hosts=/home/adblock/adblock.list

A couple of notes to make this work:

• in the script: webserverIp: 192.168.1.201 is the IP that all ad domains resolve to. This is the IP of my Raspi where I have nginx running returning 204 responses (no content) for every request
• Setup a user called "adblock" on OpnSense and configure it for non-interactive ssh access

Limitations:

• You get security warnings for blocked https content, because the SSL certificate in nginx is of course wrong (but it's rare since usually no ads are loaded in the first place)
• no support to whitelist domains (but should be easy to add this)
• probably more,but can't think of them this late.

Well, this is my own version of Pi-Hole. You would also have to schedule this script.

Any suggestions are of course welcome!


Cheers, Curly060 =;->

[Nnyan]

No worries I just saw your reply!  I'm on 17.7 so I'll see if i can get this to work there too.