Howto setup rules for linux repository updates

[deajan]

Hello,

I have a couple of linux VM (Almalinux/CentOS/Debian/SLES) behind an OPNSense and would like to limit internet acces from those VMs for security reasons.
The problem is that I'd still need those machines to fetch their security updates from various mirrors.

If I setup firewall rules to allow things like mirrors.distro.tld, nothing works great since mirrors.distro.tld will redirect to some mirror, and I cannot just add all various mirrors to fqdn aliases.

So far I've investigated:
- Using squid proxy with whitelists (doesn't work well because for the same reasons above)
- Setting up local repositories... Way too much work and space used
- Making maintenance windows where internet is available via schedules (doesn't right right in terms of security)

Has anyone ever found an elegant solution for this scenario ?
Thanks.

[netnut]

Has anyone ever found an elegant solution for this scenario ?

If you're using an "Host Alias" this shouldn't be that difficult. As you said it depends which mirror you use and if/how the CDN behind it is operating. But with a little try & error you should setup your alias in 5min.

This is an example of an Host Alias that's working for years here, it's the official Debian Mirror which uses Fastly, but everything is behind a single CDN hostname which is automaticly resolved by OPNsense for IPv4 & IPv6. Took me 3 or 4 apt-get runs to define.

If you still have issues look for some other, more 'static' mirror which doesn't use a CDN or redirects. But the Host Alias is the way to go IMHO.

Code: [Select]

"3e976b85-d2a7-40d7-ac2b-b39059c37953": {
        "enabled": "1",
        "name": "PUB_DEBIAN_MIRROR",
        "type": "host",
        "proto": "",
        "interface": "",
        "counters": "0",
        "updatefreq": "",
        "content": "security.debian.org\ndeb.debian.org\ndebian.map.fastly.net\nmetadata.ftp-master.debian.org\n",
        "categories": "",
        "description": "Public Debian Mirror"
      },

[deajan]

Thanks, but that's more or less the point.
Using Almalinux for instance, it will connect to mirrors.almalinux.org and from there, select a random mirror of currently 342 mirrors, which I obviously don't want to enter manually.

I can setup a couple of those, but then updating will be painfully slow until a "good" mirror is found.
Not even speaking of other repos (epel, grafana...).

I could also force the package manager to select a particular mirror, but then I don't have resiliency anymore.

I thought maybe there was a "automagic" solution like lists of IPs one can download that have "all the nice mirror IPs" or something alike

Or any other solution I didn't think about, other than a regex that allows anything containing "mirror" ^^

[netnut]

Or any other solution I didn't think about, other than a regex that allows anything containing "mirror" ^^

Instead of some generic entry point which automagicly decides which is the nearest / fastest every time, pick one from this list which works for you.

https://mirrors.almalinux.org/


Until AI is at the next level: If you want a "restricted" (firewall) policy for a dynamic resource you should configure _something_.

[deajan]

You're probably right ^^
I just thought maybe someone maintains IP lists for such services, which would be convenient.

Thanks.

[netnut]

You can configure multiple DNS names in a Host Alias, if "resiliency" is an issue, configure multiple in your (neighbour) country. Looking at the hostnames in the mirror overview, they are pretty random, so a regex wouldn't work.

*.archive.ubuntu.com could work with a dns wildcard or regex, they standardized their mirror naming. For Debian fastly redirects the right mirror through debian.map.fastly.net.

You might want to write a HTML scrapper and run that one daily to fill the list.

[deajan]

Hmm... Can't achieve resiliency when I have to enter only one mirror as the repo url, but resiliency is icing on the cake, as long as I monitor that updates work.

Writing a HTML scrapper... I've not played with beautifulsoup for long, and don't intent to ^^
That's way to hackish IMO.

Thanks for your time.

[CJ]

I would recommend that you just set up a local mirror instead.  That mirror can have internet access to sync and then everything else on your network can just pull their packages from it.  That way none of them require any internet access.  It will also reduce your bandwidth needs and provide faster updates.

[netnut]

I would recommend that you just set up a local mirror instead.  That mirror can have internet access to sync and then everything else on your network can just pull their packages from it.  That way none of them require any internet access.  It will also reduce your bandwidth needs and provide faster updates.

You're now moving the OP's challenge to, the outbound firewall rule, of this mirror box ;-), it still needs the same "Alias" magic (with resiliency aka connectivity to potential _all_ mirrors).

...

That's way to hackish IMO.

Even better, their complete mirror list is available as JSON: https://github.com/AlmaLinux/mirrors/

Forget the DIY scrapper , create a cron job with a small script that fetches the mirror list, parse the JSON and fill an Alias. Voila

[deajan]

@netnut Now that's something I can deal with. Thank you. ^^

Didn't find the JSON file on the link you provided, but at least there is a list of YAML files in https://github.com/AlmaLinux/mirrors/tree/master/mirrors.d which I can download and parse

I'll create a quick python parser that downloads the mirrors git zip file, and parses all .yml files in mirrors.d. Shouln't be too horrible ^^

Next step would be to import aliases into OPNSense...
Any idea how to create / update aliases via CLI ?
I've checked the forum but only found https://forum.opnsense.org/index.php?topic=5581.0 and https://forum.opnsense.org/index.php?topic=20653.0

I've seen this post (https://forum.opnsense.org/index.php?topic=36683.msg179305#msg179305) where franco suggests a way to tamper the config.xml file and re-import it via cli... I'm sure this works, but isn't there an API do handle this scenario ?

PS: I've also found the EPEL mirror list at https://mirrors.fedoraproject.org/mirrorlist?repo=epel-9&arch=x86_64
Toghether will the almalinux mirror list, this should suffice for most updates, in a resilient and automatic way.

I'll probably have to add some other repos (grafana for example), but since most of them don't use mirrors, it's just adding one alias and monitoring that updates work an a reliable fashion (basically a sh script that creates a text file for prometheus' node_exporter to pickup).

[CJ]

You're now moving the OP's challenge to, the outbound firewall rule, of this mirror box ;-), it still needs the same "Alias" magic (with resiliency aka connectivity to potential _all_ mirrors).

Yes, but depending on the requirements it might not to be limited as heavily.  And it provides a centralized place of management.

[deajan]

Yes, but depending on the requirements it might not to be limited as heavily.  And it provides a centralized place of management.

To be fair, I played with the idea of making local repo mirrors too, but setting those up for various RHEL flavors, Debian an Suse is quite a maintenance burden. Especially when I still need 3rd party remote repositories (epel, grafana...). I am not managing enough VMs to make this scenario happen.

Anyway, since I can find the mirror lists, I now need to find a way to create aliases via api / cli.

[CJ]

Yes, but depending on the requirements it might not to be limited as heavily.  And it provides a centralized place of management.

To be fair, I played with the idea of making local repo mirrors too, but setting those up for various RHEL flavors, Debian an Suse is quite a maintenance burden. Especially when I still need 3rd party remote repositories (epel, grafana...). I am not managing enough VMs to make this scenario happen.

Anyway, since I can find the mirror lists, I now need to find a way to create aliases via api / cli.

Why so many different distros?

I've been meaning to set up a local mirror but haven't gotten to it yet.

[deajan]

Many distros because some applications only work on RHEL, others on SLES, others on Debian...
Would love to reduce too, but here I am