OPNsense Forum

English Forums => General Discussion => Topic started by: Kiwifruta on February 27, 2020, 11:29:03 AM

Title: How do I add hundreds of domain overrides
Post by: Kiwifruta on February 27, 2020, 11:29:03 AM
Hi.
I'm new to OPNsense, having moved from OpenWRT and Ubiquiti.

I have a list of several hundred domains that need to use custom DNS, i.e. not those configured under the System - Settings - General.

With OpenWRT and Ubiquiti I ran a script that downloaded the list of domain overrides (e.g. server=/tvnz.co.nz/8.8.8.8) and populated a configuration file read by dnsmasq. Can I do something similar in OPNsense?

DDGing for information about configuring OPNsense by CLI/Shell has been fruitless, so I'm wondering if the only way to configure OPNsense is via the GUI. I found the /conf/config.xml file. Is this the file to edit the configuration via the shell?
Will putting dnsmasq configuration files in /etc/dnsmasq.d dnd edit /usr/loca/etc/dnsmasq.conf to point to that directory work, or will it get overriden by /conf/config.xml?

I'd prefer not to enter hundreds of domain overrides by hand in to Services-Dnsmasq DNS-Settings-Domain Overrides, as it'll take a long time plus it's also subject to change. With the other firmware I ran the script on a cron job to provide updates.


Thanks

Title: Re: How do I add hundreds of domain overrides
Post by: siga75 on February 27, 2020, 03:35:16 PM
best way is probably using API, I did something for nginx, probably something similar is available for dnsmasq

root@myfw:~ # cat nginx-ban-purge
#!/bin/sh
key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
secret="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
sed -n -e '/<Nginx/,/<\/Nginx>/p' /conf/config.xml | sed -n -e '/<ban uuid/,/<\/ban>/p' | awk -F'\"|<|>' -v TSTAMP=`date +%s` '/ban uuid/ {uuid=$3; getline; getline; if($3<TSTAMP-90*24*60*60) print uuid }' | while read UUID ; do curl --ssl --insecure -X POST --data "{}" -H "Content-Type: application/json" --user "$key":"$secret" https://127.0.0.1:8443/api/nginx/bans/delban/$UUID ; done
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on February 27, 2020, 07:36:22 PM
Thank you.
Unfortunately there isn't an API for dnsmasq, although there is for Unbound.

I have lots of CNAME entries (to map google domains to forcesafesearch.google.com) so need dnsmasq for this.

So it looks like I'd need to run dnsmasq and unbound together.
Dnsmasq to doing the CNAME records, and Unbound for external address.

Thanks for your direction.
Title: Re: How do I add hundreds of domain overrides
Post by: mimugmail on February 27, 2020, 08:04:07 PM
Just edit config.xml by hand
Title: Re: How do I add hundreds of domain overrides
Post by: stefanpf on February 27, 2020, 08:56:33 PM
You could simply add this Server entries
into the advanced Options field or add an extra config File e.g. with
conf-file=/usr/local/etc/dnsstuff.conf
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 02, 2020, 07:32:44 AM
Thanks for your replies @mimugmail and @stefanpf.
mimugmail, I have hundreds to do and these could change slightly at any time, and so an automatic process is much more preferable. This is a home set up.

stefanpf, I tried several times to add conf-file=/usr/local/etc/myfile.conf to the advanced options field but without success. This by far my preferred way to do it.


Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 02, 2020, 10:41:11 AM
What is the syntax to edit config.xml by hand to set under dnsmasq conf-dir=/etc/dnsmasq.d?
Title: Re: How do I add hundreds of domain overrides
Post by: mimugmail on March 02, 2020, 11:14:36 AM
Just add one override, grep the xml container in config.xml and add as many as you like in similar syntax
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 03, 2020, 08:22:58 AM
Quote from: mimugmail on March 02, 2020, 11:14:36 AM
Just add one override, grep the xml container in config.xml and add as many as you like in similar syntax
Okay, I'll give that a shot.


Sent from my iPhone using Tapatalk
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 08, 2020, 03:53:45 AM
[UPDATE - below did NOT work]

Found the solution
1) put an additional hosts/dnsmasq file at /etc/hosts.additional

2) In /conf/config.xml, under <dnsmasq> enter this line
<hostsdir>/etc/hosts.additional</hostsdir>

3) restart dnsmasq
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 08, 2020, 04:40:13 AM
In the end instead of using domain overrides I used a custom hosts file which I was appended to /etc/hosts

first create a back up of /etc/hosts
touch /etc/hosts.bak
cp /etc/hosts /etc/hosts.bak

then insert the current hosts file to the hosts file
cat /etc/hosts.custom >> /etc/hosts

It doesn't survive firmware upgrades, but it does survive dnsmaq restarts.

Now I need to figure out how to run the automatic checking for updates to the custom hosts file.

Title: Re: How do I add hundreds of domain overrides
Post by: franco on March 08, 2020, 08:40:03 AM
Didn't Dnsmasq support multiple host files via command line arguments? You can use the advanced configuration field to configure it...


Cheers,
Franco
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 08, 2020, 09:45:21 AM
Franco, yes you are right, and that does work, thank you.

The GUI says the advanced field is about to be deprecated, so I'm looking for a method that won't disappear when the firmware is upgraded.


Sent from my iPhone using Tapatalk
Title: Re: How do I add hundreds of domain overrides
Post by: franco on March 08, 2020, 01:07:29 PM
It's deprecated for security reasons and thus only accessible for administrators, but a (safer) replacement has not yet been decided on.

Usually we favour file-based approaches to plugin-capable core capabilities which require shell access (which should only be given to administrators) or a properly built OPNsense plugin. These file-based solutions will the be picked up by the system without further GUI interaction.


Cheers,
Franco
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 10, 2020, 09:18:37 AM
Quote from: stefanpf on February 27, 2020, 08:56:33 PM
You could simply add this Server entries
into the advanced Options field or add an extra config File e.g. with
conf-file=/usr/local/etc/dnsstuff.conf

I followed your approach but set up a directory and put it in /etc, so
conf-dir=/etc/dnsmasq.d
It worked perfectly, thanks.
Title: Re: How do I add hundreds of domain overrides
Post by: franco on March 10, 2020, 03:13:51 PM
Thanks, it seems we have our replacement strategy here... let me record this in a ticket for 20.7 :)


Cheers,
Franco
Title: Re: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 10, 2020, 06:33:36 PM
Franco,
being able to point to the files and/or directories where dnsmasq configuration file, host files and server files are located will be great.

My next step is writing a script to automatically populate the above files.

I'm really appreciating the active and helpful nature of this OPNsense forum, especially to have a core developer such as yourself responding. Cheers.


Sent from my iPhone using Tapatalk
Title: How do I add hundreds of domain overrides
Post by: Kiwifruta on March 10, 2020, 07:13:24 PM
Quote from: Kiwifruta on March 10, 2020, 09:18:37 AM
Quote from: stefanpf on February 27, 2020, 08:56:33 PM
You could simply add this Server entries
into the advanced Options field or add an extra config File e.g. with
conf-file=/usr/local/etc/dnsstuff.conf

I followed your approach but set up a directory and put it in /etc, so
conf-dir=/etc/dnsmasq.d
It worked perfectly, thanks.
I need to clarify that the CNAME records are in the above directory.

These are my current advanced settings

addn-hosts=/etc/hosts.custom
servers-file=/etc/hosts.dnsmasq
conf-file=/etc/dnsmasq.d/safe.conf


Sent from my iPhone using Tapatalk