I have noticed a couple of times that when restoring a backup that contains URL TABLE aliases, the aliases show with their last updated date as the date when the config was backed up. If this frequency is daily, hourly etc then when you restore the aliases even though the tables are EMPTY, they will not update for a full day etc. I understand the logic here is comparing last sync date to current date, but since the tables are actually empty maybe we could force an update?
Shouldn't ALL aliases URL tables update on a restore since they will all be empty? If not can we add a force update button to the Firewall->Diagnostics->Aliases screen?
i thought that the alias update script runs every minute. if the table is empty it will try to update it
It will only update it if the "Last Update" time has expired. Since I took a backup off a production box, it had that as the "last sync time"... Minutes later restored it to a new device. The new table is empty, and will not re-populate for 1 day (since I have this particular list set to update daily).
sorry, do you think so or have you checked?
Yeah, URL aliases seem to be fixed to the update interval with no direct way to force update them. Same goes for simple changes I need to deploy outside the update interval, I have to set hour to 0 and minute to 0.1 to get it applied, no way to force it, it seems.
@ar
if i understand @tracerrx correctly, hes talking about restoring the alias settings on a system where it did not exist (in which case the update script should update it as soon as possible).
if you change the url-table alias and it needs to be updated out of schedule, then you can simply delete the corresponding .md5.txt file in /var/db/aliastables imho
I have the same issue. The url table aliases don't seem to get populated correctly (I also see empty aliases). To populate them I need to disable and enable the alias.
Update interval is set to about a week.
The cron job to update aliases runs once per day (i think that is default behavior).
@senser
https://github.com/opnsense/core/blob/c0467fb54ae1a9f4d07a36249bcc28a39f9c2dd4/src/etc/inc/plugins.inc.d/pf.inc#L79
@Fright i wish I could do something with that. :k
I have one urltable alias. It was empty for unknown reasons. Disabling and enabling it, populated the alias table. Can we make sure that it gets populated automatically?
In the cron setup I have a job for ,,Update and reload firewall aliases" to run every day at 4 oclock in the morning. The alias itself also has a ,,Refresh frequency" which is set to 5 days +1h.
So, as @tracerrx said...somehow is the table is not always populated. I also did a reinstall (from usb stick) and imported the old configuration (3 days ago). And somehow the alias never updated itself since then...until yesterday when I noticed that it is empty.
I believe it is some sort of race condition.
When restoring a config (urltable)aliases may initially fail when no WAN is available (in my case a pppoe connection).
But maybe that is somehow dealt with.
Another issue may be dns. Maybe unbound is not ready at the time the aliases are populated?
I don't know how opnsense works but maybe there is some kind of event for when WAN and DNS is working which then could be used to check (urltable)aliases?
Thinking about unbound and blocklists: How do we initially resolve the blocklists used by unbound when unbound itself is the resolver? But that is another topic.
Quotei wish I could do something with that
just pointing that system already have "5 asterisks" job for alias update. script checks alias ttl and updates it if needed.
QuoteCan we make sure that it gets populated automatically?
There can always be reasons why it won't work imho. so we can look at the update time and the number of records in the Firewall: Aliases table, as well as the content of the alias on the Firewall: Diagnostics: Aliases page
as I said, if there is a feeling that the alias is not updated and there are no error messages in the logs (problems with time sync on vm or some?), you can try deleting the corresponding corresponding .md5.txt file in /var/db/aliastable. within a minute the system will try to update the alias
Quote from: Fright on February 18, 2022, 03:50:39 PM
sorry, do you think so or have you checked?
I have tested multiple times. On both 21 and 22 series. To replicate, create a url table alias that updated daily, let it populate. Export the config to back, and restore the config on a new device. The last check date will be equal to the backup date/time, and as it's set to only check daily, the url table alias will be empty unless you disable and then re-enable the alias.
A possible fix could be to set the date for the ,,last update time" in the backup to 0 seconds. Then the aliases should be updated as soon as an alias update is triggered (cron, whatever mechanism/event might trigger it). Maybe? :)
Quote from: senser on February 20, 2022, 09:51:51 PM
A possible fix could be to set the date for the ,,last update time" in the backup to 0 seconds. Then the aliases should be updated as soon as an alias update is triggered (cron, whatever mechanism/event might trigger it). Maybe? :)
Yes, that would work... but then it would constantly be updating no? Maybe a better solution would be to refresh all URL table aliases x minutes after a restore (to ensure wans up etc). Or possible to have a second option on these aliases for refreshing when entries exist, and when entries don't exist?
I don't mean zhe update interval....But the timestamp of the last update of the alias in the backup. So its initially outdated when restoring a backup.
Quote from: senser on February 21, 2022, 06:51:39 AM
I don't mean zhe update interval....But the timestamp of the last update of the alias in the backup. So its initially outdated when restoring a backup.
I understand, I believe that would fix the problem. When performing backups/exports of settings, set all last updated timestamps to == unix epoch.
QuoteTo replicate, create a url table alias that updated daily, let it populate. Export the config to back, and restore the config on a new device. The last check date will be equal to the backup date/time, and as it's set to only check daily, the url table alias will be empty unless you disable and then re-enable the alias.
hm. I do not understand how in this case the value of the last update time can appear. it is not stored in the config (so can not be "restored"). its a 'dynamic' field generated only in the model and is taken from the file "edit time".
have tested by this steps:
- create "URL Table (IPs)" Alias (FireHOL_L1) with 1 hour ttl.
- wait for Alias download.
- make config backup
- delete this alias, hit Apply
- restore backup from step 3 without restart(!)
everything correct up to this step?
'old' FireHOL_L1 alias appears in Firewall: Aliases page. with
empty Loaded# and Last Update values.
FireHOL_L1 alias is not updated automatically (because filter_tables template was not updated - restart after restore was not performed)
- hit Apply on Firewall: Aliases page
filter_tables template updated, FireHOL_L1 alias downloaded immediately
Quote from: Fright on February 21, 2022, 06:25:39 PM
have tested by this steps:
- create "URL Table (IPs)" Alias (FireHOL_L1) with 1 hour ttl.
- wait for Alias download.
- make config backup
- delete this alias, hit Apply
- restore backup from step 3 without restart(!)
everything correct up to this step?
Two differences
- My TTL is set to 1 day
- While hardware has identical specs (Protectli FW6B's), I am exporting on device #1 and importing on device #2. Device #2 is a clean install that has never had the alias defined.