Messages

1

17.7 Legacy Series / Re: dnsmasq: cannot resolve external hosts

« on: August 13, 2017, 09:50:26 am »

Surely that should always be specified if you're running a DNS server on the firewall?

Why? The DNS servers come from my ISP and that's why I had enabled the setting "Allow DNS server list to be overridden by DHCP/PPP on WAN " in System: Settings: General.
In 16.7 and 17.1 this worked perfectly. Since I haven't changed anything during the upgrade from 17.1 to 17.7 I guess something changed in 17.7.

Although I use my own DNS servers inside the LAN and not dnsmasq I should also ask (just in case), I assume that dnsmasq is not listening on the WAN interface as well is it?

Indeed it is not listening on the WAN interface:
Services: Dnsmasq DNS: Settings: Interfaces: DMZ, LAN, Localhost, OpenVPN

Cheers, Curly060 =;->

2

17.7 Legacy Series / Re: dnsmasq: cannot resolve external hosts

« on: August 12, 2017, 01:55:17 am »

I spoke too soon. Over night DNS resolving stopped to work, so I guess I am having the same problem as others already reported...

Only way to make it work is to manually specify DNS servers and disable " Allow DNS server list to be overridden by DHCP/PPP on WAN"

Cheers, Curly060 =;->

3

16.7 Legacy Series / Re: Firewall Alias for adblocking

« on: August 12, 2017, 01:51:12 am »

Would you be willing to share your scripts?

Sorry, only saw this now...

I should probably clean it up and parameterize some stuff, but well, here it is, unrevised, as is, use at your own risk:

Code: [Select]

#!/bin/bash
filter() {
  # filter out lines with the following criteria:
  # - comment lines (starts with optional blank followed by hash)
  # - empty lines (only blank or completely empty)
  # - lines without any dot
  # - lines with ' localhost' inside
  grep -v -e '^[[:space:]]*#' -e '^[[:space:]]*$' -e '^[^.]*$' -e ' localhost' | tr ' ' ' '| sed -e 's#\s\+$##'
}
adblockListTemp=/tmp/adblocklist.temp
adblockListFinal=/tmp/adblock.list
webserverIp=${1:-192.168.1.201}
:> $adblockListTemp
# Adblock
curl -s https://adaway.org/hosts.txt | filter | cut -f2 -d' ' >> $adblockListTemp
# Shallalist
curl -s http://www.shallalist.de/Downloads/shallalist.tar.gz | tar -C /tmp -xz
cat /tmp/BL/{adv,costtraps,spyware,tracker,warez}/domains >> $adblockListTemp
rm -rf /tmp/BL/
# Yoyo
curl -s 'https://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&showintro=0&mimetype=plaintext' >> $adblockListTemp
### PI-Hole
curl -s https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s http://mirror1.malwaredomains.com/files/justdomains >> $adblockListTemp
curl -s http://sysctl.org/cameleon/hosts | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s 'https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist' >> $adblockListTemp
curl -s https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt >> $adblockListTemp
curl -s https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt >> $adblockListTemp
curl -s https://hosts-file.net/ad_servers.txt | tr -d '\r' | filter | cut -f2 -d' ' >> $adblockListTemp
curl -s https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt | tr -d '\r' | cut -f2 -d' ' >> $adblockListTemp
### Clean
cat $adblockListTemp | filter | grep -vP '^(\d+\.){3}\d+$' | sed -e "s#^#$webserverIp #" | sort -u > $adblockListFinal
scp -i adblock.key -o UserKnownHostsFile=adblock.known_hosts $adblockListFinal adblock@opnsense:./adblock.list
rm $adblockListTemp $adblockListFinal

nginx snippet:

Code: [Select]

server {
  listen 80 default_server;
  listen 443 ssl http2 default_server;
  server_name 192.168.1.201;

  error_log /var/log/nginx/opnsense-hole.error.log info;
  access_log /var/log/nginx/opnsense-hole.access.log;

  ssl_session_cache shared:SSL_opnsense-hole:10m;
  ssl_certificate /etc/ssl/opnsense-hole.cert.pem;
  ssl_certificate_key /etc/ssl/opnsense-hole.key.pem;
  location / {
    expires max;
    return 204;
  }
}

dnsmasq advanced setting:

Code: [Select]

addn-hosts=/home/adblock/adblock.list

A couple of notes to make this work:

• in the script: webserverIp: 192.168.1.201 is the IP that all ad domains resolve to. This is the IP of my Raspi where I have nginx running returning 204 responses (no content) for every request
• Setup a user called "adblock" on OpnSense and configure it for non-interactive ssh access

Limitations:

• You get security warnings for blocked https content, because the SSL certificate in nginx is of course wrong (but it's rare since usually no ads are loaded in the first place)
• no support to whitelist domains (but should be easy to add this)
• probably more,but can't think of them this late.

Well, this is my own version of Pi-Hole. You would also have to schedule this script.

Any suggestions are of course welcome!


Cheers, Curly060 =;->

4

17.7 Legacy Series / dnsmasq: cannot resolve external hosts

« on: August 11, 2017, 12:48:19 am »

Hi,

first of all thanks a lot for the new release. Everything works like a charm, except DNS resolving of external hosts. I am using dnsmasq DNS. My settings are as follows:

• System: Settings: General: no manual DNS server entries
• System: Settings: General:  [X] Allow DNS server list to be overridden by DHCP/PPP on WAN

Now I make a query to an external host:

Code: [Select]

ingo@router:~ % drill google.de
;; ->>HEADER<<- opcode: QUERY, rcode: REFUSED, id: 36706
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; google.de. IN A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.1
;; WHEN: Fri Aug 11 00:35:47 2017
;; MSG SIZE  rcvd: 27

Why am I getting rcode: REFUSED?

For hosts in the LAN everything works as expected.

If I manually add DNS servers in "System: Settings: General" then it also works, however, I did not have to do this in the 17.1 version.

Any suggestions (apart from switching to Unbound which currently is not yet an option for me).

Cheers, Ingo =;->

5

16.7 Legacy Series / Re: Firewall Alias for adblocking

« on: February 03, 2017, 11:10:58 am »

Well, that would for sure be the cleanest and best solution. My scripts use bash, curl, grep and sed, so they should definitely run directly on the FW. However, I did not have the time yet to look into writing my own plugin.

So yes, I am definitely up for a bit of scripting, but where to start? Here? https://docs.opnsense.org/development/examples/helloworld.html

Cheers, Curly060 =;->

6

16.7 Legacy Series / Re: Firewall Alias for adblocking

« on: February 02, 2017, 12:18:51 pm »

Hi,

thanks for the answers.
OpenDNS is not an option for me. The whole point of my OpnSense installation is that I do not need to rely on external services (esp. when their owners are located in the States...). I prefer having as much under my direct control as possible.

Using a proxy is fine, but every client needs to be configured. That will end up in configuration hell and whenever a client bypasses the proxy, I get ads again. So it would have to be a transparent proxy, which I didn't want in the first place.

Well, I have solved the problem in the meantime like this:

• a custom script on a Raspberry Pi creates a hosts file from various sources every week (similar to what Pi Hole does) and uploads it to OpnSense
• Dnsmasq is configured to use this file as additional hosts file

=> Works perfectly and for all clients in my LAN

I was hoping I do not have to rely on the Pi to do this, but well, that's how it is now.

Cheers, Curly060 =;->

7

16.7 Legacy Series / Firewall Alias for adblocking

« on: January 24, 2017, 04:43:25 pm »

Hi,

I migrated from OpenWRT to OPNSense in the past couple of weeks and can't believe that all these years I wasn't aware of pfSense/OPNSense! I am very happy with pretty much everything, except for the adblocking situation (and perhaps the google rank situation  ).

What I would really like to see is DNS based adblocking. I have searched the forum, but the resolution always seems to be transparent proxy (really not an option for me) or some firewall rule with a set of IPs (alias). The latter I am trying to implement:
So I have created a very big list of IP addresses and Domains from various sources (PI-Hole, OpenWRT adblock plugin etc.). The list contains about 120000 entries. Then I created an alias like this:

• Name: Adblock
• Type: URL Table (IPs)
• URL: URL to my list (not public)

Now I do have some questions about this:

• Is it ok to have a mixture of IPs and domain names in my list? I would say yes, OPNSense seems to resolve domains in the background and creates a text file in /var/db/aliastables/Adblock.txt which contains only IPs.
• Is there a limit as to how many IPs I can have for an alias?
• What is a healthy amount of IPs inside an alias? Would it be
• When I look in /var/db/aliastables/Adblock.txt file I notice a lot of duplicate IPs. Should this maybe optimized or does it not matter?
• What is OPNSense's strategy when the file changes? Will it only look at differences or parse the file fully each time?
• My Alias does not show up under Firewall: Diagnostics: pfTables. However, if I create the alias with a non capital first letter, then it will appear there. Bug?
• Firewall: Diagnostics: pfTables tells me that there are no entries in my alias. However, when I make my list smaller, suddenly it will show the IP addresses. I have experimented a little: 5000 entries were fine, 10000 already not. So it looks there is indeed a limit somewhere.
• Are there any plans to integrate DNS based adblocking? Sure I can set up a Pi next to my OPNSense with PI-Hole on it, but I'd prefer the all-in-one solution. Would the DNS based adblocking not be far, far more efficient? I mean, only when a resource is actually requested, the DNS resolver would have to check against the black list (might be costly, but can be cached). As firewall rule with, say 5000 entries, this needs to be checked for pretty much every packet, no? Transparent proxy adblock is IMHO far too complex (CA certificate on every client, maintaing no-ssl-bump list,...)

Sorry for the sheer volume of the questions, I hope someone takes time to answer them. I have been googling a lot (which is painful because google always returns results for pfSense. OPNSense must get more popular to drive pfSense off the first ranks ) and the forum has only like 2-3 posts about adblocking...


Cheers, Curly060 =;->