Help / Ideas needed to fix mirror config

[computeralex92]

Hello everyone,

I noticed in the last months some issues regarding my Opnsense mirror dns-root.de which is routed and cached via Cloudflare.
Error's like

Code: [Select]

Package checksum mismatch or missing packages which are stored on the disk but not cached by Cloudflare.  :'(
Unfortunatly out of personal reasons I was not able to deep dive into this problems in the last year, but now I want finally fix this issues and provide you a stable mirror.

My setup:

• Nginx docker container as webserver
• Rsync container triggered by a systemd timer
• Cloudflare to cache everything except defined files

The Cloudflare config is configured to

• Bypass changelog.txz, bogons.txz, meta.txz & packagesite.tgz
• Cache everything in the /releases folder for one month (installation iso's, so fine)
• Cache everything else for one month; but do not force this

From nginx side there are additional rules:

• Add "no-cache" header to changelog.txz, bogons.txz, meta.txz & packagesite.tgz
• Everything else cache 30 days

So basically both sides are configured nearly the same; only the iso's are forced to be cached for one month (which make sense because they are static).

My theory why the checksum error is happening is the following:

If a package is updated without a version change in the filename, the new checksum is part of the uncached system files, but as the file changed, the error appears.
When looking through the mirror folder I saw that the folder "Latest" in the package structure contains unversioned pkg-Files, only the "All"-folder contains all versioned pkg-files.

My ideas to solve this until now:

• Delete the Cloudflare cache when there are files changed by rsync (not so easy to implement with rsync)
• Not caching the "Latest"-folder (less caching rate, more traffic for server...)

Maybe some of you have a idea to solve this on a different way, or I'm missing something out of missing FreeBSD packaging knowledge 

Thanks a lot,
Alex



PS.
Thanks for 2 TB traffic per month; it is great to support you and the great project on this way