White Listed Domains not working in Unbound DNS: Blocklist

[MVerBerkmoes]

I  have a pretty upset wife because she cannot complete her 'surveys' where she earns her mad money for shopping.  Hopefully, somene can help 'save a marriage' by helping me fix this.   

Running v23.7.8.1 and have implemented Services->UnboundDNS->Blocklists.  I have enabled 'Steven Black List' and 'YoYo List' among several others within the 'Type of DNSBL' list.

When she first complained about NXDOMAIN errors after enabling the Block Lists, I was able to see the blocking occur in the Reporting->Unbound DNS->Details view.  I noted that 2 of the domains she desired to access were surveyjunkie.com and surveytakingjunkie.com.  Both are included in 'Steven Black List' and 'YoYo List' lists.  So I manually added them to the 'Whitelist Domains' in Services->UnboundDNS->Blocklists as shown.

Now, they still show up as blocked in 'Steven Black List', but the option to add them to the 'Whitelist Domains' is no longer available in the far left column.

All of that to ask, if they're white listed, why are they still being blocked?

[doktornotor]

...

[CJ]

What domains are you attempting to allow?  There's a bug in the current implementation that if the allowed domain is a CNAME to a blocked domain it'll still get blocked until you allow the additional domain.

https://github.com/opnsense/core/issues/6722

[MVerBerkmoes]

Thank you CJ!

THAT, is exactly the issue, need surveytakingjunkie.com to be passed.  click.surveytakingjunkie.com as an A record is passed, but track.surveytakingjunkie.com as a CNAME is blocked.

Thanks for the heads up about the bug ..

[doktornotor]

Code: [Select]

(.*)?(\.)?surveytakingjunkie.com
should work.

[CJ]

Code: [Select]

(.*)?(\.)?surveytakingjunkie.com
should work.

The default DNSBL doesn't work with Regex.

[doktornotor]

Well, it definitely does. See the first (super-long) line.

Code: [Select]

Unbound 95214 - [meta sequenceId="1"] blocklist download : exclude domains matching download.ccleaner.com|img.blesk.cz|srtb.msn.com|s3-website.ca-central-1.amazonaws.com|s3-website.ap-south-1.amazonaws.com|(.*)?(\.)?googledrive.com|s3-website-ap-southeast-2.amazonaws.com|(.*)?(\.)?rghost.net|beacons-handoff.gcp.gvt2.com|(.*)?(\.)?blogspot.fr|www.msn.com|i.imgur.com|www.googleadservices.com|cn-northwest-1.eb.amazonaws.com.cn|www.facebook.com|downloads.dell.com-v2-dd.edgekey.net.globalredir.akadns.net|(.*)?(\.)?blogspot.vn|(.*)?(\.)?p.typekit.net-v3.edgekey.net|www.script.crazyegg.com|(.*)?(\.)?blogspot.co.id|(.*)?(\.)?blogspot.al|www.dslreports.com|www.c.bing.com|www.g.msn.com|upload.wikimedia.org|(.*)?(\.)?wildcard2.cdn.responsys.net.edgekey.net|(.*)?(\.)?downloads.hpe.com|www.kdukvh.com|(.*)?(\.)?twimg.com|downloads.dell.com|compute.amazonaws.com.cn|track.cj.akadns.net|(.*)?(\.)?blogspot.ru|(.*)?(\.)?e5439.x.akamaiedge.net|www.cookie-cdn.cookiepro.com|s3-sa-east-1.amazonaws.com|s3-website-ap-southeast-1.amazonaws.com|(.*)?(\.)?blogspot.jp|(.*)?(\.)?blogspot.ae|www.upload.ee|(.*)?(\.)?blogspot.com.ar|api.segment.io|us-west-2.compute.amazonaws.com|www.nirsoft.net|(.*)?(\.)?blogspot.com.mt|(.*)?(\.)?ssi-elb.go2cloud.org|(.*)?(\.)?blogspot.ba|(.*)?(\.)?blogspot.my|(.*)?(\.)?blogspot.de|ssl.google-analytics.com|download.bleepingcomputer.com|(.*)?(\.)?edgekey.net|(.*)?(\.)?tinyurl.com|www.exploit-db.com|www.me-client.eservice.emarsys.net|(.*)?(\.)?blogspot.lu|www.api.segment.io|g.msn.com|(.*)?(\.)?dropbox.com|(.*)?(\.)?blogspot.co.za|(.*)?(\.)?mozilla.org|(.*)?(\.)?persona.ly|g-msn-com-nsatc.trafficmanager.net|s3.dualstack.eu-west-1.amazonaws.com|s3-website.us-east-2.amazonaws.com|data.emb-api.com|s3.dualstack.ca-central-1.amazonaws.com|(.*)?(\.)?qps.cint.com|(.*)?(\.)?dl.sourceforge.net|dl.dropboxusercontent.com|s3-ap-southeast-2.amazonaws.com|(.*)?(\.)?blogspot.com.ng|s3.ap-south-1.amazonaws.com|s3.dualstack.ap-southeast-1.amazonaws.com|app.adjust.com|node1.upload.ee|a-0003.a-msedge.net|s3-eu-west-2.amazonaws.com|download.mozilla.org|www.odorik.cz|script.crazyegg.com|(.*)?(\.)?cdburnerxp.se|www.image.ibb.co|www.duckdns.org|(.*)?(\.)?blogspot.nl|(.*)?(\.)?blogspot.re|s3.dualstack.eu-west-2.amazonaws.com|www-alv.google-analytics.com|s3.dualstack.ap-northeast-2.amazonaws.com|(.*)?(\.)?microsoft.com|www.app.adjust.com|(.*)?(\.)?theoremreach.com|pastebin.com|(.*)?(\.)?aukro.cz|s3.dualstack.us-east-1.amazonaws.com|(.*)?(\.)?blogspot.td|web.archive.org|(.*)?(\.)?tracking.surveycheck.com|www.maxmind.com|(.*)?(\.)?gslb-downloads-hpe-com.glb1.hpe.com|(.*)?(\.)?gitlab.com|(.*)?(\.)?clarity.ms|(.*)?(\.)?blogspot.ie|(.*)?(\.)?blogspot.ca|(.*)?(\.)?microsoft.com.akadns.net|(.*)?(\.)?eicar.org|www.openwall.com|(.*)?(\.)?rghost.ru|elb.amazonaws.com.cn|(.*)?(\.)?blogspot.qa|(.*)?(\.)?blogspot.in|ap-northeast-2.compute.amazonaws.com|(.*)?(\.)?adbx.io|(.*)?(\.)?cint-collector-noe.azurewebsites.net|(.*)?(\.)?blogspot.ug|(.*)?(\.)?google.com|s3-website-us-west-1.amazonaws.com|e28.dsce4.akamaiedge.net|dqcev5ui4x43j.cloudfront.net|s3-eu-west-1.amazonaws.com|lists.alioth.debian.org|prd-snap-broker-alb-1914988209.eu-west-1.elb.amazonaws.com|g.live.com|s3-external-1.amazonaws.com|sstats.adobe.com|s3.ap-northeast-2.amazonaws.com|kqzyfj.com|script.crazyegg.com.cdn.cloudflare.net|(.*)?(\.)?msdn.com|(.*)?(\.)?blogspot.hr|eu-central-1.compute.amazonaws.com|(.*)?(\.)?githubusercontent.com|as.wkcr.cz|us-east-1.amazonaws.com|www.s.click.aliexpress.com|s3-ca-central-1.amazonaws.com|s3-website-ap-northeast-1.amazonaws.com|(.*)?(\.)?e6653.dscf.akamaiedge.net|login.live.com|s3-website-us-east-1.amazonaws.com|azurewebsites.net|cn-north-1.compute.amazonaws.com.cn|ap-southeast-1.compute.amazonaws.com|cj.dotomi.com|(.*)?(\.)?blogspot.mr|me-client-api-glb.gservice.emarsys.net|(.*)?(\.)?blogspot.ch|(.*)?(\.)?blogspot.com.co|(.*)?(\.)?blogspot.bg|www.downloads.dell.com|us-west-1.compute.amazonaws.com|s.click.aliexpress.com|ap-southeast-2.compute.amazonaws.com|(.*)?(\.)?p.typekit.net|(.*)?(\.)?github.com|(.*)?(\.)?blogspot.kr|(.*)?(\.)?samsung-firmware.org|(.*)?(\.)?blogspot.com.eg|www.dropbox.com|s3-eu-west-3.amazonaws.com|www.ssl.google-analytics.com|prod.python.map.fastly.net|(.*)?(\.)?c.cintnetworks.com|(.*)?(\.)?blogspot.lt|me-client.eservice.emarsys.net|s3-ap-southeast-1.amazonaws.com|s3.amazonaws.com|eu-west-1.compute.amazonaws.com|(.*)?(\.)?blogspot.se|c-bing-com.a-0001.a-msedge.net|(.*)?(\.)?archive.org|us-gov-west-1.compute.amazonaws.com|s3-eu-central-1.amazonaws.com|(.*)?(\.)?ytimg.com|(.*)?(\.)?bit.ly|duckdns.org|(.*)?(\.)?blogspot.cz|(.*)?(\.)?blogspot.fi|(.*)?(\.)?blogspot.hk|(.*)?(\.)?blogspot.pe|s3-website-sa-east-1.amazonaws.com|www.srtb.msn.com|(.*)?(\.)?pingomatic.com|(.*)?(\.)?blogspot.com.uy|(.*)?(\.)?sourceforge.net|z-1.compute-1.amazonaws.com|s3.dualstack.eu-central-1.amazonaws.com|(.*)?(\.)?blogspot.com.by|www.grc.com|www.kcsoftwares.com|config.emb-api.com|compute-1.amazonaws.com|s3-us-gov-west-1.amazonaws.com|www.dpm.demdex.net|(.*)?(\.)?mail.ru|(.*)?(\.)?blogspot.com.au|(.*)?(\.)?blogspot.mx|ap-northeast-1.compute.amazonaws.com|s3.eu-west-3.amazonaws.com|s3.dualstack.ap-northeast-1.amazonaws.com|(.*)?(\.)?blogspot.sk|(.*)?(\.)?blogspot.si|www.beacons.gcp.gvt2.com|iframe.sponsorpay.com|www.google-analytics.com|(.*)?(\.)?cdn.onesignal.com|(.*)?(\.)?nmap.org|s3-fips-us-gov-west-1.amazonaws.com|(.*)?(\.)?google.cz|(.*)?(\.)?static.cdn.responsys.net|s3-ap-northeast-1.amazonaws.com|c.bing.com|(.*)?(\.)?blogspot.rs|s3.dualstack.eu-west-3.amazonaws.com|adobetarget.data.adobedc.net|(.*)?(\.)?blogspot.be|(.*)?(\.)?blogspot.tw|(.*)?(\.)?api2.branch.io|dpm.demdex.net|google-analytics.com|image.ibb.co|cookie-cdn.cookiepro.com|s3.dualstack.us-east-2.amazonaws.com|(.*)?(\.)?dl.osdn.jp|cn-north-1.eb.amazonaws.com.cn|s3.dualstack.ap-south-1.amazonaws.com|www.config.emb-api.com|s3-website-us-west-2.amazonaws.com|z-2.compute-1.amazonaws.com|s3.dualstack.ap-southeast-2.amazonaws.com|s3-us-west-1.amazonaws.com|www.lcprd1.samsungcloudsolution.net|(.*)?(\.)?blogspot.bj|(.*)?(\.)?akamaiedge.net|(.*)?(\.)?gstatic.com|s3.dualstack.sa-east-1.amazonaws.com|(.*)?(\.)?blogspot.am|kdukvh.com|(.*)?(\.)?affiliateclub.go2cloud.org|sa-east-1.compute.amazonaws.com|(.*)?(\.)?blogspot.com|(.*)?(\.)?blogspot.dk|s3-website.eu-west-3.amazonaws.com|www.t.co|downloads-regions.dell-cidr.akadns.net|(.*)?(\.)?msftncsi.com|www.sstats.adobe.com|(.*)?(\.)?blogspot.co.nz|(.*)?(\.)?blogspot.it|(.*)?(\.)?blogspot.gr|(.*)?(\.)?blogspot.hu|(.*)?(\.)?goo.gl|(.*)?(\.)?blogspot.co.ke|s3.us-east-2.amazonaws.com|(.*)?(\.)?consensu.org|beacons.gcp.gvt2.com|(.*)?(\.)?akamai.net|www.download.ccleaner.com|www-msn-com.a-0003.a-msedge.net|(.*)?(\.)?blogspot.sg|(.*)?(\.)?gslb-downloads-hpe-com.ext.hpe.com|(.*)?(\.)?pointclicktrack.com|www.kqzyfj.com|(.*)?(\.)?blogspot.co.uk|t.co|(.*)?(\.)?blogspot.cf|adobe.tt.omtrdc.net|(.*)?(\.)?blogspot.ro|pypi.python.org|s3-us-west-2.amazonaws.com|.*localhost$|dual-a-0001.a-msedge.net|(.*)?(\.)?collector-main.trafficmanager.net|s3-website-eu-west-1.amazonaws.com|s3-website.eu-west-2.amazonaws.com|(.*)?(\.)?w3.org|uswildcard.alicdn.com.edgekey.net|www.data.emb-api.com|(.*)?(\.)?download.teamviewer.com|(.*)?(\.)?blogspot.md|(.*)?(\.)?defcon.org|(.*)?(\.)?router.cint.com|(.*)?(\.)?bitbucket.org|(.*)?(\.)?waws-prod-osl-001.cloudapp.net|azure-mobile.net|s3.cn-north-1.amazonaws.com.cn|(.*)?(\.)?microsoft.akadns.net|(.*)?(\.)?blogspot.is|elb.amazonaws.com|www.adobe.tt.omtrdc.net|s3-website.eu-central-1.amazonaws.com|(.*)?(\.)?cdn.branch.io|(.*)?(\.)?blogspot.com.br|lcprd1.samsungcloudsolution.net|e1429.x.akamaiedge.net|(.*)?(\.)?twitter.com|(.*)?(\.)?blogspot.com.es|grc.com|s3-website.ap-northeast-2.amazonaws.com|(.*)?(\.)?blogspot.li|(.*)?(\.)?owasp.org|downloads.dell-cidr.akadns.net|googleadservices.com|www.beacons.gvt2.com|(.*)?(\.)?googleapis.com|(.*)?(\.)?blogspot.sn|(.*)?(\.)?blogspot.cv|downloads.dell.com-v2-dd.edgekey.net|(.*)?(\.)?blogspot.com.ee|(.*)?(\.)?notepad-plus-plus.org|(.*)?(\.)?blogspot.co.il|(.*)?(\.)?blogspot.mk|(.*)?(\.)?blogspot.cl|www.pastebin.com|(.*)?(\.)?blogspot.com.tr|beacons.gvt2.com|elasticbeanstalk.cn-north-1.amazonaws.com.
<165>1 2023-12-02T00:53:06+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 25959 total lines from cache for https://threatfox.abuse.ch/downloads/hostfile
<165>1 2023-12-02T00:53:06+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://threatfox.abuse.ch/downloads/hostfile (exclude: 31 block: 25919 wildcard: 0)
<165>1 2023-12-02T00:53:37+01:00 gw.example.com unbound 95214 - [meta sequenceId="3"] blocklist download: 10607 total lines from cache for https://adaway.org/hosts.txt
<165>1 2023-12-02T00:53:37+01:00 gw.example.com unbound 95214 - [meta sequenceId="4"] blocklist: https://adaway.org/hosts.txt (exclude: 149 block: 6393 wildcard: 0)
<165>1 2023-12-02T00:55:01+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 22066 total lines from cache for https://justdomains.github.io/blocklists/lists/easylist-justdomains.txt
<165>1 2023-12-02T00:55:01+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://justdomains.github.io/blocklists/lists/easylist-justdomains.txt (exclude: 13 block: 22053 wildcard: 0)
<165>1 2023-12-02T00:56:39+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 20270 total lines from cache for https://justdomains.github.io/blocklists/lists/easyprivacy-justdomains.txt
<165>1 2023-12-02T00:56:39+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://justdomains.github.io/blocklists/lists/easyprivacy-justdomains.txt (exclude: 32 block: 20238 wildcard: 0)
<165>1 2023-12-02T00:56:41+01:00 gw.example.com unbound 95214 - [meta sequenceId="3"] blocklist download: 420 total lines from cache for https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt
<165>1 2023-12-02T00:56:41+01:00 gw.example.com unbound 95214 - [meta sequenceId="4"] blocklist: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt (exclude: 0 block: 409 wildcard: 0)
<165>1 2023-12-02T00:56:55+01:00 gw.example.com unbound 95214 - [meta sequenceId="5"] blocklist download: 3784 total lines from cache for http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&mimetype=plaintext
<165>1 2023-12-02T00:56:55+01:00 gw.example.com unbound 95214 - [meta sequenceId="6"] blocklist: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&mimetype=plaintext (exclude: 28 block: 3756 wildcard: 0)
<165>1 2023-12-02T00:56:59+01:00 gw.example.com unbound 95214 - [meta sequenceId="7"] blocklist download: 946 total lines from cache for https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-domains.txt
<165>1 2023-12-02T00:56:59+01:00 gw.example.com unbound 95214 - [meta sequenceId="8"] blocklist: https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-domains.txt (exclude: 3 block: 943 wildcard: 0)
<165>1 2023-12-02T00:57:00+01:00 gw.example.com unbound 95214 - [meta sequenceId="9"] blocklist parsing done in 357.18 seconds (75962 records)
<30>1 2023-12-02T00:57:57+01:00 gw.example.com unbound 38511 - [meta sequenceId="10"] [38511:0] info: dnsbl_module: updating blocklist.
<30>1 2023-12-02T00:57:58+01:00 gw.example.com unbound 38511 - [meta sequenceId="11"] [38511:0] info: dnsbl_module: blocklist loaded. length is 75962

When you put an invalid regex there, you'll get something like the following in /var/log/resolver/latest.log

Code: [Select]

<163>1 2023-11-26T13:53:54+01:00 gw.example.com unbound 18391 - [meta sequenceId="405"] blocklist download : skip invalid whitelist exclude pattern "custom_pattern_1" (*\.example.net)


[CJ]

Well, it definitely does. See the first (super-long) line.

Where did you put the regex?  The help text mentions that whitelist supports regex but that's it.

[doktornotor]

Hmmm? In the "Whitelist domains" textarea field, obviously...

[CJ]

ah.  I thought you were saying that the blocklists themselves supported regex.