Messages

631

23.7 Legacy Series / Re: Saving "System - Settings - General" is super slow

« on: December 12, 2023, 10:28:56 am »

Adding some "benchmark" here:

Code: [Select]

# time certctl rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
58.523u 7.417s 1:06.24 99.5%    407+219k 0+0io 0pf+0w

# time /usr/local/opnsense/scripts/system/certctl.py rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
1.511u 0.314s 0:02.16 84.2%     5+168k 0+0io 0pf+0w

So it's about 30x faster. 

632

23.7 Legacy Series / Re: White Listed Domains not working in Unbound DNS: Blocklist

« on: December 06, 2023, 11:21:41 am »

Hmmm? In the "Whitelist domains" textarea field, obviously...

633

Development and Code Review / Cron GUI validation

« on: December 06, 2023, 09:54:30 am »

Before I start messing with the code, is there a particularly good reason why it is impossible to use @reboot or other special strings in cron jobs due to validation?

634

23.7 Legacy Series / Re: Saving "System - Settings - General" is super slow

« on: December 05, 2023, 01:28:10 pm »

The upstream "improvement" keeps driving me nuts! Switching another box to OPNsense... Franco, how much work would it take to add an "Apply" button to System: Trust: Authorities? Spent 40 minutes importing the existing  CAs, out of that half an hour rewriting the trust store - after adding every single CA.

Aaaaaaaaaargh! 

635

23.7 Legacy Series / Re: [CALL FOR TESTING] Unbound DNS over TLS without explicit CA bundle

« on: December 02, 2023, 12:10:38 pm »

Works here as well.

636

23.7 Legacy Series / Re: White Listed Domains not working in Unbound DNS: Blocklist

« on: December 02, 2023, 11:40:44 am »

Well, it definitely does. See the first (super-long) line.

Code: [Select]

Unbound 95214 - [meta sequenceId="1"] blocklist download : exclude domains matching download.ccleaner.com|img.blesk.cz|srtb.msn.com|s3-website.ca-central-1.amazonaws.com|s3-website.ap-south-1.amazonaws.com|(.*)?(\.)?googledrive.com|s3-website-ap-southeast-2.amazonaws.com|(.*)?(\.)?rghost.net|beacons-handoff.gcp.gvt2.com|(.*)?(\.)?blogspot.fr|www.msn.com|i.imgur.com|www.googleadservices.com|cn-northwest-1.eb.amazonaws.com.cn|www.facebook.com|downloads.dell.com-v2-dd.edgekey.net.globalredir.akadns.net|(.*)?(\.)?blogspot.vn|(.*)?(\.)?p.typekit.net-v3.edgekey.net|www.script.crazyegg.com|(.*)?(\.)?blogspot.co.id|(.*)?(\.)?blogspot.al|www.dslreports.com|www.c.bing.com|www.g.msn.com|upload.wikimedia.org|(.*)?(\.)?wildcard2.cdn.responsys.net.edgekey.net|(.*)?(\.)?downloads.hpe.com|www.kdukvh.com|(.*)?(\.)?twimg.com|downloads.dell.com|compute.amazonaws.com.cn|track.cj.akadns.net|(.*)?(\.)?blogspot.ru|(.*)?(\.)?e5439.x.akamaiedge.net|www.cookie-cdn.cookiepro.com|s3-sa-east-1.amazonaws.com|s3-website-ap-southeast-1.amazonaws.com|(.*)?(\.)?blogspot.jp|(.*)?(\.)?blogspot.ae|www.upload.ee|(.*)?(\.)?blogspot.com.ar|api.segment.io|us-west-2.compute.amazonaws.com|www.nirsoft.net|(.*)?(\.)?blogspot.com.mt|(.*)?(\.)?ssi-elb.go2cloud.org|(.*)?(\.)?blogspot.ba|(.*)?(\.)?blogspot.my|(.*)?(\.)?blogspot.de|ssl.google-analytics.com|download.bleepingcomputer.com|(.*)?(\.)?edgekey.net|(.*)?(\.)?tinyurl.com|www.exploit-db.com|www.me-client.eservice.emarsys.net|(.*)?(\.)?blogspot.lu|www.api.segment.io|g.msn.com|(.*)?(\.)?dropbox.com|(.*)?(\.)?blogspot.co.za|(.*)?(\.)?mozilla.org|(.*)?(\.)?persona.ly|g-msn-com-nsatc.trafficmanager.net|s3.dualstack.eu-west-1.amazonaws.com|s3-website.us-east-2.amazonaws.com|data.emb-api.com|s3.dualstack.ca-central-1.amazonaws.com|(.*)?(\.)?qps.cint.com|(.*)?(\.)?dl.sourceforge.net|dl.dropboxusercontent.com|s3-ap-southeast-2.amazonaws.com|(.*)?(\.)?blogspot.com.ng|s3.ap-south-1.amazonaws.com|s3.dualstack.ap-southeast-1.amazonaws.com|app.adjust.com|node1.upload.ee|a-0003.a-msedge.net|s3-eu-west-2.amazonaws.com|download.mozilla.org|www.odorik.cz|script.crazyegg.com|(.*)?(\.)?cdburnerxp.se|www.image.ibb.co|www.duckdns.org|(.*)?(\.)?blogspot.nl|(.*)?(\.)?blogspot.re|s3.dualstack.eu-west-2.amazonaws.com|www-alv.google-analytics.com|s3.dualstack.ap-northeast-2.amazonaws.com|(.*)?(\.)?microsoft.com|www.app.adjust.com|(.*)?(\.)?theoremreach.com|pastebin.com|(.*)?(\.)?aukro.cz|s3.dualstack.us-east-1.amazonaws.com|(.*)?(\.)?blogspot.td|web.archive.org|(.*)?(\.)?tracking.surveycheck.com|www.maxmind.com|(.*)?(\.)?gslb-downloads-hpe-com.glb1.hpe.com|(.*)?(\.)?gitlab.com|(.*)?(\.)?clarity.ms|(.*)?(\.)?blogspot.ie|(.*)?(\.)?blogspot.ca|(.*)?(\.)?microsoft.com.akadns.net|(.*)?(\.)?eicar.org|www.openwall.com|(.*)?(\.)?rghost.ru|elb.amazonaws.com.cn|(.*)?(\.)?blogspot.qa|(.*)?(\.)?blogspot.in|ap-northeast-2.compute.amazonaws.com|(.*)?(\.)?adbx.io|(.*)?(\.)?cint-collector-noe.azurewebsites.net|(.*)?(\.)?blogspot.ug|(.*)?(\.)?google.com|s3-website-us-west-1.amazonaws.com|e28.dsce4.akamaiedge.net|dqcev5ui4x43j.cloudfront.net|s3-eu-west-1.amazonaws.com|lists.alioth.debian.org|prd-snap-broker-alb-1914988209.eu-west-1.elb.amazonaws.com|g.live.com|s3-external-1.amazonaws.com|sstats.adobe.com|s3.ap-northeast-2.amazonaws.com|kqzyfj.com|script.crazyegg.com.cdn.cloudflare.net|(.*)?(\.)?msdn.com|(.*)?(\.)?blogspot.hr|eu-central-1.compute.amazonaws.com|(.*)?(\.)?githubusercontent.com|as.wkcr.cz|us-east-1.amazonaws.com|www.s.click.aliexpress.com|s3-ca-central-1.amazonaws.com|s3-website-ap-northeast-1.amazonaws.com|(.*)?(\.)?e6653.dscf.akamaiedge.net|login.live.com|s3-website-us-east-1.amazonaws.com|azurewebsites.net|cn-north-1.compute.amazonaws.com.cn|ap-southeast-1.compute.amazonaws.com|cj.dotomi.com|(.*)?(\.)?blogspot.mr|me-client-api-glb.gservice.emarsys.net|(.*)?(\.)?blogspot.ch|(.*)?(\.)?blogspot.com.co|(.*)?(\.)?blogspot.bg|www.downloads.dell.com|us-west-1.compute.amazonaws.com|s.click.aliexpress.com|ap-southeast-2.compute.amazonaws.com|(.*)?(\.)?p.typekit.net|(.*)?(\.)?github.com|(.*)?(\.)?blogspot.kr|(.*)?(\.)?samsung-firmware.org|(.*)?(\.)?blogspot.com.eg|www.dropbox.com|s3-eu-west-3.amazonaws.com|www.ssl.google-analytics.com|prod.python.map.fastly.net|(.*)?(\.)?c.cintnetworks.com|(.*)?(\.)?blogspot.lt|me-client.eservice.emarsys.net|s3-ap-southeast-1.amazonaws.com|s3.amazonaws.com|eu-west-1.compute.amazonaws.com|(.*)?(\.)?blogspot.se|c-bing-com.a-0001.a-msedge.net|(.*)?(\.)?archive.org|us-gov-west-1.compute.amazonaws.com|s3-eu-central-1.amazonaws.com|(.*)?(\.)?ytimg.com|(.*)?(\.)?bit.ly|duckdns.org|(.*)?(\.)?blogspot.cz|(.*)?(\.)?blogspot.fi|(.*)?(\.)?blogspot.hk|(.*)?(\.)?blogspot.pe|s3-website-sa-east-1.amazonaws.com|www.srtb.msn.com|(.*)?(\.)?pingomatic.com|(.*)?(\.)?blogspot.com.uy|(.*)?(\.)?sourceforge.net|z-1.compute-1.amazonaws.com|s3.dualstack.eu-central-1.amazonaws.com|(.*)?(\.)?blogspot.com.by|www.grc.com|www.kcsoftwares.com|config.emb-api.com|compute-1.amazonaws.com|s3-us-gov-west-1.amazonaws.com|www.dpm.demdex.net|(.*)?(\.)?mail.ru|(.*)?(\.)?blogspot.com.au|(.*)?(\.)?blogspot.mx|ap-northeast-1.compute.amazonaws.com|s3.eu-west-3.amazonaws.com|s3.dualstack.ap-northeast-1.amazonaws.com|(.*)?(\.)?blogspot.sk|(.*)?(\.)?blogspot.si|www.beacons.gcp.gvt2.com|iframe.sponsorpay.com|www.google-analytics.com|(.*)?(\.)?cdn.onesignal.com|(.*)?(\.)?nmap.org|s3-fips-us-gov-west-1.amazonaws.com|(.*)?(\.)?google.cz|(.*)?(\.)?static.cdn.responsys.net|s3-ap-northeast-1.amazonaws.com|c.bing.com|(.*)?(\.)?blogspot.rs|s3.dualstack.eu-west-3.amazonaws.com|adobetarget.data.adobedc.net|(.*)?(\.)?blogspot.be|(.*)?(\.)?blogspot.tw|(.*)?(\.)?api2.branch.io|dpm.demdex.net|google-analytics.com|image.ibb.co|cookie-cdn.cookiepro.com|s3.dualstack.us-east-2.amazonaws.com|(.*)?(\.)?dl.osdn.jp|cn-north-1.eb.amazonaws.com.cn|s3.dualstack.ap-south-1.amazonaws.com|www.config.emb-api.com|s3-website-us-west-2.amazonaws.com|z-2.compute-1.amazonaws.com|s3.dualstack.ap-southeast-2.amazonaws.com|s3-us-west-1.amazonaws.com|www.lcprd1.samsungcloudsolution.net|(.*)?(\.)?blogspot.bj|(.*)?(\.)?akamaiedge.net|(.*)?(\.)?gstatic.com|s3.dualstack.sa-east-1.amazonaws.com|(.*)?(\.)?blogspot.am|kdukvh.com|(.*)?(\.)?affiliateclub.go2cloud.org|sa-east-1.compute.amazonaws.com|(.*)?(\.)?blogspot.com|(.*)?(\.)?blogspot.dk|s3-website.eu-west-3.amazonaws.com|www.t.co|downloads-regions.dell-cidr.akadns.net|(.*)?(\.)?msftncsi.com|www.sstats.adobe.com|(.*)?(\.)?blogspot.co.nz|(.*)?(\.)?blogspot.it|(.*)?(\.)?blogspot.gr|(.*)?(\.)?blogspot.hu|(.*)?(\.)?goo.gl|(.*)?(\.)?blogspot.co.ke|s3.us-east-2.amazonaws.com|(.*)?(\.)?consensu.org|beacons.gcp.gvt2.com|(.*)?(\.)?akamai.net|www.download.ccleaner.com|www-msn-com.a-0003.a-msedge.net|(.*)?(\.)?blogspot.sg|(.*)?(\.)?gslb-downloads-hpe-com.ext.hpe.com|(.*)?(\.)?pointclicktrack.com|www.kqzyfj.com|(.*)?(\.)?blogspot.co.uk|t.co|(.*)?(\.)?blogspot.cf|adobe.tt.omtrdc.net|(.*)?(\.)?blogspot.ro|pypi.python.org|s3-us-west-2.amazonaws.com|.*localhost$|dual-a-0001.a-msedge.net|(.*)?(\.)?collector-main.trafficmanager.net|s3-website-eu-west-1.amazonaws.com|s3-website.eu-west-2.amazonaws.com|(.*)?(\.)?w3.org|uswildcard.alicdn.com.edgekey.net|www.data.emb-api.com|(.*)?(\.)?download.teamviewer.com|(.*)?(\.)?blogspot.md|(.*)?(\.)?defcon.org|(.*)?(\.)?router.cint.com|(.*)?(\.)?bitbucket.org|(.*)?(\.)?waws-prod-osl-001.cloudapp.net|azure-mobile.net|s3.cn-north-1.amazonaws.com.cn|(.*)?(\.)?microsoft.akadns.net|(.*)?(\.)?blogspot.is|elb.amazonaws.com|www.adobe.tt.omtrdc.net|s3-website.eu-central-1.amazonaws.com|(.*)?(\.)?cdn.branch.io|(.*)?(\.)?blogspot.com.br|lcprd1.samsungcloudsolution.net|e1429.x.akamaiedge.net|(.*)?(\.)?twitter.com|(.*)?(\.)?blogspot.com.es|grc.com|s3-website.ap-northeast-2.amazonaws.com|(.*)?(\.)?blogspot.li|(.*)?(\.)?owasp.org|downloads.dell-cidr.akadns.net|googleadservices.com|www.beacons.gvt2.com|(.*)?(\.)?googleapis.com|(.*)?(\.)?blogspot.sn|(.*)?(\.)?blogspot.cv|downloads.dell.com-v2-dd.edgekey.net|(.*)?(\.)?blogspot.com.ee|(.*)?(\.)?notepad-plus-plus.org|(.*)?(\.)?blogspot.co.il|(.*)?(\.)?blogspot.mk|(.*)?(\.)?blogspot.cl|www.pastebin.com|(.*)?(\.)?blogspot.com.tr|beacons.gvt2.com|elasticbeanstalk.cn-north-1.amazonaws.com.
<165>1 2023-12-02T00:53:06+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 25959 total lines from cache for https://threatfox.abuse.ch/downloads/hostfile
<165>1 2023-12-02T00:53:06+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://threatfox.abuse.ch/downloads/hostfile (exclude: 31 block: 25919 wildcard: 0)
<165>1 2023-12-02T00:53:37+01:00 gw.example.com unbound 95214 - [meta sequenceId="3"] blocklist download: 10607 total lines from cache for https://adaway.org/hosts.txt
<165>1 2023-12-02T00:53:37+01:00 gw.example.com unbound 95214 - [meta sequenceId="4"] blocklist: https://adaway.org/hosts.txt (exclude: 149 block: 6393 wildcard: 0)
<165>1 2023-12-02T00:55:01+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 22066 total lines from cache for https://justdomains.github.io/blocklists/lists/easylist-justdomains.txt
<165>1 2023-12-02T00:55:01+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://justdomains.github.io/blocklists/lists/easylist-justdomains.txt (exclude: 13 block: 22053 wildcard: 0)
<165>1 2023-12-02T00:56:39+01:00 gw.example.com unbound 95214 - [meta sequenceId="1"] blocklist download: 20270 total lines from cache for https://justdomains.github.io/blocklists/lists/easyprivacy-justdomains.txt
<165>1 2023-12-02T00:56:39+01:00 gw.example.com unbound 95214 - [meta sequenceId="2"] blocklist: https://justdomains.github.io/blocklists/lists/easyprivacy-justdomains.txt (exclude: 32 block: 20238 wildcard: 0)
<165>1 2023-12-02T00:56:41+01:00 gw.example.com unbound 95214 - [meta sequenceId="3"] blocklist download: 420 total lines from cache for https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt
<165>1 2023-12-02T00:56:41+01:00 gw.example.com unbound 95214 - [meta sequenceId="4"] blocklist: https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt (exclude: 0 block: 409 wildcard: 0)
<165>1 2023-12-02T00:56:55+01:00 gw.example.com unbound 95214 - [meta sequenceId="5"] blocklist download: 3784 total lines from cache for http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&mimetype=plaintext
<165>1 2023-12-02T00:56:55+01:00 gw.example.com unbound 95214 - [meta sequenceId="6"] blocklist: http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml&mimetype=plaintext (exclude: 28 block: 3756 wildcard: 0)
<165>1 2023-12-02T00:56:59+01:00 gw.example.com unbound 95214 - [meta sequenceId="7"] blocklist download: 946 total lines from cache for https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-domains.txt
<165>1 2023-12-02T00:56:59+01:00 gw.example.com unbound 95214 - [meta sequenceId="8"] blocklist: https://raw.githubusercontent.com/dibdot/DoH-IP-blocklists/master/doh-domains.txt (exclude: 3 block: 943 wildcard: 0)
<165>1 2023-12-02T00:57:00+01:00 gw.example.com unbound 95214 - [meta sequenceId="9"] blocklist parsing done in 357.18 seconds (75962 records)
<30>1 2023-12-02T00:57:57+01:00 gw.example.com unbound 38511 - [meta sequenceId="10"] [38511:0] info: dnsbl_module: updating blocklist.
<30>1 2023-12-02T00:57:58+01:00 gw.example.com unbound 38511 - [meta sequenceId="11"] [38511:0] info: dnsbl_module: blocklist loaded. length is 75962

When you put an invalid regex there, you'll get something like the following in /var/log/resolver/latest.log

Code: [Select]

<163>1 2023-11-26T13:53:54+01:00 gw.example.com unbound 18391 - [meta sequenceId="405"] blocklist download : skip invalid whitelist exclude pattern "custom_pattern_1" (*\.example.net)


637

23.7 Legacy Series / Re: unbound Enable AAAA-only mode and squid dns_v4_first

« on: December 02, 2023, 11:34:49 am »

Unbound is not even compiled with FILTER_AAAA. Use BIND.

Code: [Select]

# pkg info unbound
unbound-1.19.0
Name           : unbound
Version        : 1.19.0
Installed on   : Sat Nov 25 18:41:29 2023 CET
Origin         : dns/unbound
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : dns
Licenses       : BSD3CLAUSE
Maintainer     : jaap@NLnetLabs.nl
WWW            : https://www.nlnetlabs.nl/projects/unbound
Comment        : Validating, recursive, and caching DNS resolver
Options        :
        DEP-RSA1024    : off
        DNSCRYPT       : on
        DNSTAP         : off
        DOCS           : off
        DOH            : on
        DYNLIB         : on
        ECDSA          : on
        EVAPI          : off
        FILTER_AAAA    : off
        GOST           : on
        HIREDIS        : off
        LIBEVENT       : on
        MUNIN_PLUGIN   : off
        PYTHON         : on
        SUBNET         : off
        TFOCL          : off
        TFOSE          : off
        THREADS        : on


638

23.7 Legacy Series / Re: White Listed Domains not working in Unbound DNS: Blocklist

« on: November 26, 2023, 02:57:41 pm »

Code: [Select]

(.*)?(\.)?surveytakingjunkie.com
should work.

639

23.7 Legacy Series / Re: High CPU usage when downloading

« on: November 25, 2023, 07:07:53 pm »

Get some better HW, or move on.

640

23.7 Legacy Series / Re: Web proxy and problems to download updates for antyvirus WithSecure Elements EPP

« on: November 19, 2023, 05:21:41 pm »

Do not proxy antivirus updates. Problem solved.

641

23.7 Legacy Series / Re: White Listed Domains not working in Unbound DNS: Blocklist

« on: November 19, 2023, 09:41:39 am »

...

642

23.7 Legacy Series / Re: DNS Priority, IPv4, IPv6

« on: November 19, 2023, 03:16:54 am »

No, it's never been solved properly. Multicast is randomly broken with various FW versions both on APs and switches.

643

23.7 Legacy Series / Re: OPNsense-23.7-serial-amd64.img seems broken

« on: November 18, 2023, 02:10:43 pm »

Created a (completely untested) pull request here: https://github.com/opnsense/tools/pull/386, subscribe there if you want to discuss.

644

23.7 Legacy Series / Re: OPNsense-23.7-serial-amd64.img seems broken

« on: November 18, 2023, 11:36:25 am »

Thanks for the follow up. I'd say it'd make more sense to publish the checksums (also) for the unpacked images, exactly for cases like this.

645

23.7 Legacy Series / Re: OPNsense-23.7-serial-amd64.img seems broken

« on: November 18, 2023, 10:00:00 am »

Do you mean this image?

Code: [Select]

SHA256 (OPNsense-23.7-serial-amd64.img.bz2) = 03c774f53520414c73cdcaa4fe3b34c4165395963bef74c533c3878a07b80138

Well, that's what I used for installing OPNsense on multiple APUx boxes... Perhaps try with something else than Ubuntu 23. Works and boots just fine when flashed with Etcher: https://etcher.balena.io/

Or, something like:

Code: [Select]

dd if=OPNsense-23.7-serial-amd64.img of=/dev/sdb bs=16k status=progress conv=fsync