Hello everyone,
I noticed in the last months some issues regarding my Opnsense mirror dns-root.de which is routed and cached via Cloudflare.
Error's like
Package checksum mismatch or missing packages which are stored on the disk but not cached by Cloudflare. :'(
Unfortunatly out of personal reasons I was not able to deep dive into this problems in the last year, but now I want finally fix this issues and provide you a stable mirror.My setup:
- Nginx docker container as webserver
- Rsync container triggered by a systemd timer
- Cloudflare to cache everything except defined files
(https://forum.opnsense.org/index.php?action=dlattach;topic=31907.0;attach=25197)
The Cloudflare config is configured to
- Bypass changelog.txz, bogons.txz, meta.txz & packagesite.tgz
- Cache everything in the /releases folder for one month (installation iso's, so fine)
- Cache everything else for one month; but do not force this
(https://forum.opnsense.org/index.php?action=dlattach;topic=31907.0;attach=25199)
From nginx side there are additional rules:
- Add "no-cache" header to changelog.txz, bogons.txz, meta.txz & packagesite.tgz
- Everything else cache 30 days
So basically both sides are configured nearly the same; only the iso's are forced to be cached for one month (which make sense because they are static).
My theory why the checksum error is happening is the following:
If a package is updated without a version change in the filename, the new checksum is part of the uncached system files, but as the file changed, the error appears.
When looking through the mirror folder I saw that the folder "Latest" in the package structure contains unversioned pkg-Files, only the "All"-folder contains all versioned pkg-files.
My ideas to solve this until now:
- Delete the Cloudflare cache when there are files changed by rsync (not so easy to implement with rsync)
- Not caching the "Latest"-folder (less caching rate, more traffic for server...)
Maybe some of you have a idea to solve this on a different way, or I'm missing something out of missing FreeBSD packaging knowledge ;)
Thanks a lot,
Alex
PS.
Thanks for 2 TB traffic per month; it is great to support you and the great project on this way :)
(https://forum.opnsense.org/index.php?action=dlattach;topic=31907.0;attach=25201)