unbound blocklists not downloading in 22.7.11

[jaydub]

My production OPNsense ver 22.7.11 no longer is blocking porn and other things because the blocklists are not downloading. From the error logs:
2023-01-24T16:19:40-07:00   Error   unbound   blocklist download : unable to download file from https://raw.githubusercontent.com/chadmayfield/pihole-blocklists/master/lists/pi_blocklist_porn_top1m.list (error : HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /chadmayfield/pihole-blocklists/master/lists/pi_blocklist_porn_top1m.list (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x802615b20>: Failed to establish a new connection: [Errno 8] Name does not resolve')))

[cookiemonster]

As per the error, have you been able to verify the name resolves from your firewall?

[jaydub]

Hi Cookie,
Yes, when I put that address in a browser it brings up the list of site names so the dns is working. It had been working before 22.7 but I don't know the exact version that broke it.

[jaydub]

I also should note that it is just not that one list, it is every list I have selected that gives that same error. I can go to any of these addresses in a browser and it shows me the text lists. Here are the ones I have selected and failed:
https://raw.githubusercontent.com/chadmayfield/pihole-blocklists/master/lists/pi_blocklist_porn_top1m.list

https://blocklistproject.github.io/Lists/alt-version/torrent-nl.txt

https://blocklistproject.github.io/Lists/alt-version/scam-nl.txt 

https://blocklistproject.github.io/Lists/alt-version/redirect-nl.txt

https://blocklistproject.github.io/Lists/alt-version/ransomware-nl.txt

https://blocklistproject.github.io/Lists/alt-version/porn-nl.txt

https://blocklistproject.github.io/Lists/alt-version/piracy-nl.txt 

https://blocklistproject.github.io/Lists/alt-version/malware-nl.txt

https://blocklistproject.github.io/Lists/alt-version/gambling-nl.txt

[franco]

> Name does not resolve

That clearly tells us the firewall cannot look up any IP so all would naturally fail... but perhaps all work from your client without an issue.

You can test quite easily via Interfaces: Diagnostics: DNS Lookup.


Cheers,
Franco

[cookiemonster]

Hi Cookie,
Yes, when I put that address in a browser it brings up the list of site names so the dns is working. It had been working before 22.7 but I don't know the exact version that broke it.

As I tried to point out when I wrote "from your firewall?", and franco now, your browser is not your firewall, where the name resolution seems to be failing.
The diagnostic suggested shall confirm no names resolve and from there we'll need to see where the dns is misconfigured for your firewall. ie. not for the clients in your network.

[jaydub]

I upgraded to 23.1_6 today but now unbound doesn't work at all.  See thread https://forum.opnsense.org/index.php?topic=32352.msg156382#msg156382

[harison]

@franco what is a resolution? in the log to unbound:
blocklist download : unable to download file from https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/porn/hosts (error : HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /StevenBlack/hosts/master/alternates/porn/hosts (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x80261cc10>: Failed to establish a new connection: [Errno 8] Name does not resolve'))).

Thanks

[cookiemonster]

@franco what is a resolution? in the log to unbound:
blocklist download : unable to download file from https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/porn/hosts (error : HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /StevenBlack/hosts/master/alternates/porn/hosts (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x80261cc10>: Failed to establish a new connection: [Errno 8] Name does not resolve'))).

Thanks

Aren't we at the same place? Unbound can't download the blocklist because it can't resolve the hostname.

[harison]

what list are you use?

[cookiemonster]

Me? None with Unbound.
I think there is a better place for these. I use the AdGuard plugin and that is what pulls the blocklists.

[Bunch]

@franco what is a resolution? in the log to unbound:
blocklist download : unable to download file from https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/porn/hosts (error : HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /StevenBlack/hosts/master/alternates/porn/hosts (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x80261cc10>: Failed to establish a new connection: [Errno 8] Name does not resolve'))).

Thanks

It doesn't matter what list others using, when you can't even resolve those listed in stock.
The possible cause is there is no upstream server set for your OPNSense.

You need to set
either
using ISP DNS server as upstream server of OPNSense (System: Settings: General, Allow DNS server list to be overridden by DHCP/PPP on WAN. Services: Unbound DNS: Query Forwarding, Use System Nameservers)
or
using other servers as upstream server (System: Settings: General, add 1.1.1.1 if you want to use cloudflare. Services: Unbound DNS: Query Forwarding, Use System Nameservers)

If you confirmed you have set either one of it.
Check Interfaces: Overview: WAN interface to see what DNS servers are pushed by your ISP and try
Interfaces: Diagnostics: DNS Lookup
Hostname or IP: raw.githubusercontent.com
Server: one of the listed DNS server you found in Overview

If the name cannot be resolved by your ISP, it can be blocked by your ISP and your ISP redirected all DNS traffic to her own server. DOT would be solution for that case

[Chaskel]

Hello,

I too have experienced this issue in more recent versions. Unfortunately I am unable to say when I started noticing the change, but here is some information in case it helps determine what could be going on...

1. Reboot of OPNSense at 2 locations I have running OPNsense 23.1.5_4-amd64 yields the following each time:

Notice   unbound   blocklist: https://adaway.org/hosts.txt (exclude: 0 block: 0)   
Notice   unbound   blocklist download: 0 total lines downloaded for https://adaway.org/hosts.txt   
Error   unbound   blocklist download : unable to download file from https://adaway.org/hosts.txt (error : HTTPSConnectionPool(host='adaway.org', port=443): Max retries exceeded with url: /hosts.txt (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x8027cf640>: Failed to establish a new connection: [Errno 8] Name does not resolve')))

2. Manual restarting of Unbound service (e.g. restart service button on Blocklist page) does not appear to initiate download of list (based on not seeing messages such as those listed above).

3. If I disable Blocklist/Apply, then Enable Blocklist/Apply, it appears to trigger getting data:

Notice   unbound   blocklist parsing done in 0.58 seconds (7355 records)   
Notice   unbound   blocklist: https://adaway.org/hosts.txt (exclude: 2 block: 7355)   
Notice   unbound   blocklist download: 11782 total lines downloaded for https://adaway.org/hosts.txt   
Notice   unbound   blocklist download : exclude domains matching ^(?![a-zA-Z_\d]).*|.*localhost$   

NOTE: Even though the data seems to be retrieved, it appears it is not active until I then restart the service* (e.g. restart service button on Blocklist page).

*It also seems as though I need to go through the disable/enable steps then restart service an additional time to have everything fully work. I am not sure if it is always just one time, but I do know that doing the entire process once does not usually get everything working.

DNS config information that may be of interest:
1. Services->Unbound DNS->Blocklist - "AdAway List" selected and all other fields empty.
2. Services->Unbound DNS->DNS over TLS - 2 IPv4 and 2 IPv6 servers defined. All 4 using port 853.
3. Services->Unbound DNS->General - DNSSEC support enabled.
4. System->Settings->General - No DNS servers manually defined.
5. System->Settings->General - Allow DNS server list to be overridden by DHCP/PPP on WAN is enabled.

If it is not a setting issue, I am wondering if perhaps the following may relate to what I am seeing:

1. For bootup situation (DNS resolution error), perhaps a service dependency needs to be made if the blocklist process is launching before DNS resolution services are fully up and running (if that is what is actually happening).

2. For the manual service restart item, perhaps there are additional processes that need to be restarted behind the scenes as part of the service restart to trigger getting the URL to process the data.

I hope the above is helpful.

Thank you

[Darkfella]

I think you need Allow DNS server list to be overridden by DHCP/PPP on WAN to be disabled, else you are using the dns server provided by DHCP on your WAN port.

[Chaskel]

Thank you for your response. I had actually tried toggling that setting, but it did not appear to make a difference. I also did a packet capture and saw the 853 traffic relating to the DNS servers defined in the Unbound section.