A few squid questions and issues

[agrumpyhermit]

New opnsense user here as of this past weekend and I am overall extremely happy with this project. Almost everything has just worked, and almost every bump along the way has been quickly solved by researching either the docs or forums. Excellent work. Of course, the downside to it going so well is I might not be much help yet to others here.

Most of the issues I haven't been able to resolve are related to squid. I've set up a transparent cache with ssl_bump and Linux package cache enabled. Here's what isn't working and/or I haven't been able to locate instructions for:
1. Remote ACLs will not download. The UT1 list in the instructions time out even with wget on my desktop system, but MESD, Shallalist, and yoyo (adblock) lists won't download either. I can download each one through my browser just fine.
2. Once I can download lists, I cannot locate instructions to separate filtering in any manner so that my wife and I can access certain sites while blocking access for the kids. On a related note, is there a method for blocking certain youtube channels without blocking the whole site?
3. Caching linux updates worked great for Arch linux and really sped up the process. For Fedora (and probably CentOS), I had to put fedoraproject.org in the SSL no bump list for it to be able to update at all. It looks like the rpms were cached, but then squid replaces them on the next update. I don't think this is an opnsense issue, but does anyone know a workaround for this?
4. I haven't been able to find instructions to make squid cache work offline. I live in a very remote area and when I have connectivity my bandwidth is good, but connectivity isn't the most reliable. Since we homeschool our kids there are a lot of static information sites we'd like to serve offline when necessary.

For the last two issues, I can setup another VM as a LAN webserver to host a repo and httrack mirror, but I don't know how to make squid redirect the urls to the lan host. Ideally I'd like to keep that a transparent process, and I'd rather use squid so we're only caching what's necessary.

[fabian]

3. Caching linux updates worked great for Arch linux and really sped up the process. For Fedora (and probably CentOS), I had to put fedoraproject.org in the SSL no bump list for it to be able to update at all. It looks like the rpms were cached, but then squid replaces them on the next update. I don't think this is an opnsense issue, but does anyone know a workaround for this?

Thanks, I am the one who implemented it (also Arch Linux user).

For Fedora, I made a regex matching files ending with rpm and drpm:

https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Proxy/squid.conf#L321-L330

I hope this helps you.

4. I haven't been able to find instructions to make squid cache work offline. I live in a very remote area and when I have connectivity my bandwidth is good, but connectivity isn't the most reliable. Since we homeschool our kids there are a lot of static information sites we'd like to serve offline when necessary.

You can do this with DNS, since the Browser should try the other server if the first one fails.

For the last two issues, I can setup another VM as a LAN webserver to host a repo and httrack mirror, but I don't know how to make squid redirect the urls to the lan host. Ideally I'd like to keep that a transparent process, and I'd rather use squid so we're only caching what's necessary.

You can host static content directly on OPNsense using the nginx plugin, which can act as a load balancer and web server.

[agrumpyhermit]

Thanks, I am the one who implemented it (also Arch Linux user).

For Fedora, I made a regex matching files ending with rpm and drpm:

https://github.com/opnsense/core/blob/master/src/opnsense/service/templates/OPNsense/Proxy/squid.conf#L321-L330

I hope this helps you.

I checked your history on that file, and the related changes from Oct of last year, against the same files on my system and I don't any difference. If I remove fedoraproject.org from the SSL no bump list, the fedora client machine gives a "Failed to synchronize cache for repo 'fedora-modular'" error any time yum/dnf is run. I'm no dev so I'm probably not understanding what you're trying to show me.

The easy way out would be to just use Arch only, and I'm kinda going that direction anyways, but I'd like to learn how this works.

You can do this with DNS, since the Browser should try the other server if the first one fails.

I just spent a few hours learning my way through BSD and your file structure (long time linux user, but never used bsd before now) and studying the linux package cache part. I'd bet I can find docs on using DNSmasq to do this later tonight or tomorrow. I have all DNS running through OPNsense for DNSCrypt-Proxy.

You can host static content directly on OPNsense using the nginx plugin, which can act as a load balancer and web server.

I saw nginx in the plugin list so I figured that would be an option. I don't see httrack or any other packages for site mirroring with it though. I know openbsd has httrack, but I don't know the bsd ports system well enough yet to know how to install it on OPNsense. It showed as unavilable when I tried to use the pkg command.

Thanks for taking some time to help me out on this.

[mimugmail]

1. Remote ACLs will not download. The UT1 list in the instructions time out even with wget on my desktop system, but MESD, Shallalist, and yoyo (adblock) lists won't download either. I can download each one through my browser just fine.

File format for remote acl's should be domain only. Shalla should work fine, UT1 could be offline, no idea if yoyo fits this format.

[agrumpyhermit]

File format for remote acl's should be domain only. Shalla should work fine, UT1 could be offline, no idea if yoyo fits this format.

I'm using "http://www.shallalist.de/Downloads/shallalist.tar.gz" and "http://squidguard.mesd.k12.or.us/blacklist.tgz." I don't get any categories to sort through and apply. Tried logging into the shell to use "find" to see if maybe the lists downloaded but didn't uncompress, but nothing.

yoyo is working now with "http://pgl.yoyo.org/adservers/serverlist.php?hostformat=nohtml". Maybe my connection was being fickle. Got the "access denied" page when I tried to purposely visit an ad site from the list. Shalla and MESD, still nothing. Only difference I see is that their lists are tar compressed while yoyo's is just a text file. I can download, uncompress, and view the category folders & lists just fine on my desktop. I suppose I could copy lists over, but that would make keeping them up to date a bit less convenient.

I'm guessing UT1 is offline. I went to their page to download the categories individually through my browser and it still timed out.

[agrumpyhermit]

Been focused on the issue with remote ACLs not downloading this morning. I'll work on the offline cache next.

UT1 is working again. I can download every list on my desktop and by logging into OPNsense via ssh and using wget. I ran

Code: [Select]

time curl -C - -O '(acl list link) and they all downloaded in seconds.

Per another thread I ran [/code] /usr/local/opnsense/scripts/proxy/fetchACLs.py[/code] I got nothing.

I've tried running the download from webgui and the above command with the lists enabled and disabled. No difference either way.

On the webgui logs (system, firewall, and proxy logs) I see nothing that looks related to these ACLs.

I just checked /usr/local/etc/squid/externalACLs.conf (both with them enabled and disabled in the webgui) and the yoyoads acl is the only one listed. I'm guessing the webgui isn't passing along the remainder of the list. I haven't figured out yet where that externalACLs.conf is generated from to check it.

I'm sure this is unrelated to this issue at all, but it is a change I made from my previously stated setup so thought I'd mention it. After learning more about unbound DNS from fabian on some other posts I switched from DNSmasq to Unbound. No issues with the switch.

[agrumpyhermit]

After my last post (2 days ago) I set a cron job to download remote acls every hour. Woke up this morning and the webgui was the only thing I could access. Every website I tried returned "access denied." The opnsense dashboard showed ram maxed out, swap nearly maxed out, and cpu usage at 97%. Couldn't pull up log files and couldn't ssh into the machine so I did a hard reboot.

Squid wouldn't start after reboot. Logs showed it couldn't load ftp port 2121, so I disabled that feature. It started fine after that, but still couldn't pull up a single website. Turns out the remote acls finally downloaded and every category of every list was enabled and applied. Couldn't disable any individual categories so had to disable the lists. Applying the change stopped the squid service and it wouldn't restart through the webgui. SSH into opnsense to start it at the cli and watch for errors. No errors shown, but squid wouldn't start. Another reboot, and squid loaded without the acls enabled and I obviously have access to regular websites again.

Maybe after a few hours of normal life I'll come back and play with the acl categories.