Firewall Aliases - Force reload on restored backups

[tracerrx]

I have noticed a couple of times that when restoring a backup that contains URL TABLE aliases, the aliases show with their last updated date as the date when the config was backed up.  If this frequency is daily, hourly etc then when you restore the aliases even though the tables are EMPTY, they will not update for a full day etc.  I understand the logic here is comparing last sync date to current date, but since the tables are actually empty maybe we could force an update?

Shouldn't ALL aliases URL tables update on a restore since they will all be empty?  If not can we add a force update button to the Firewall->Diagnostics->Aliases screen?

[Fright]

i thought that the alias update script runs every minute. if the table is empty it will try to update it

[tracerrx]

It will only update it if the "Last Update" time has expired.  Since I took a backup off a production box, it had that as the "last sync time"... Minutes later restored it to a new device.  The new table is empty, and will not re-populate for 1 day (since I have this particular list set to update daily).

[Fright]

sorry, do you think so or have you checked?

[ar]

Yeah, URL aliases seem to be fixed to the update interval with no direct way to force update them. Same goes for simple changes I need to deploy outside the update interval, I have to set hour to 0 and minute to 0.1 to get it applied, no way to force it, it seems.

[Fright]

@ar
if i understand @tracerrx correctly, hes talking about restoring the alias settings on a system where it did not exist (in which case the update script should update it as soon as possible).
if you change the url-table alias and it needs to be updated out of schedule, then you can simply delete the corresponding .md5.txt file in /var/db/aliastables imho

[senser]

I have the same issue. The url table aliases don’t seem to get populated correctly (I also see empty aliases). To populate them I need to disable and enable the alias.
Update interval is set to about a week.
The cron job to update aliases runs once per day (i think that is default behavior).

[Fright]

@senser
https://github.com/opnsense/core/blob/c0467fb54ae1a9f4d07a36249bcc28a39f9c2dd4/src/etc/inc/plugins.inc.d/pf.inc#L79

[senser]

@Fright i wish I could do something with that. :k
I have one urltable alias. It was empty for unknown reasons. Disabling and enabling it, populated the alias table. Can we make sure that it gets populated automatically?
In the cron setup I have a job for „Update and reload firewall aliases“ to run every day at 4 oclock in the morning. The alias itself also has a „Refresh frequency“ which is set to 5 days +1h.

So, as @tracerrx said…somehow is the table is not always populated. I also did a reinstall (from usb stick) and imported the old configuration (3 days ago). And somehow the alias never updated itself since then…until yesterday when I noticed that it is empty.

[senser]

I believe it is some sort of race condition.
When restoring a config (urltable)aliases may initially fail when no WAN is available (in my case a pppoe connection).
But maybe that is somehow dealt with.
Another issue may be dns. Maybe unbound is not ready at the time the aliases are populated?

I don’t know how opnsense works but maybe there is some kind of event for when WAN and DNS is working which then could be used to check (urltable)aliases?

[senser]

Thinking about unbound and blocklists: How do we initially resolve the blocklists used by unbound when unbound itself is the resolver? But that is another topic.

[Fright]

i wish I could do something with that

just pointing that system already have "5 asterisks" job for alias update. script checks alias ttl and updates it if needed.

Can we make sure that it gets populated automatically?

There can always be reasons why it won't work imho. so we can look at the update time and the number of records in the Firewall: Aliases table, as well as the content of the alias on the Firewall: Diagnostics: Aliases page

as I said, if there is a feeling that the alias is not updated and there are no error messages in the logs (problems with time sync on vm or some?), you can try deleting the corresponding corresponding .md5.txt file in /var/db/aliastable. within a minute the system will try to update the alias

[tracerrx]

sorry, do you think so or have you checked?

I have tested multiple times.  On both 21 and 22 series.  To replicate, create a url table alias that updated daily, let it populate.  Export the config to back, and restore the config on a new device.  The last check date will be equal to the backup date/time, and as it's set to only check daily, the url table alias will be empty unless you disable and then re-enable the alias.

[senser]

A possible fix could be to set the date for the „last update time“ in the backup to 0 seconds. Then the aliases should be updated as soon as an alias update is triggered (cron, whatever mechanism/event might trigger it). Maybe?

[tracerrx]

A possible fix could be to set the date for the „last update time“ in the backup to 0 seconds. Then the aliases should be updated as soon as an alias update is triggered (cron, whatever mechanism/event might trigger it). Maybe?

Yes, that would work... but then it would constantly be updating no?  Maybe a better solution would be to refresh all URL table aliases x minutes after a restore (to ensure wans up etc).  Or possible to have a second option on these aliases for refreshing when entries exist, and when entries don't exist?