[Solved] Configuration import verification

[EricPerl]

I've been running a virtualized proxmox for a few weeks now (very nice upgrade coming from TP-link).

I saw a recent post from @meyergru about recommended settings and thought I'd check them out.
If anything, that would be a good practice for future recovery...

I used scp to download the entire /conf folder (got some errors on ssh keys that I ignored).
I searched/replaced the passthrough NICs with bridge equivalents in the config.xml only (i.e. not the files in /conf/backup).
I made an iso out of the updated downloaded content that I made available to a new VM.
Creating the bridges over the NICs that used to be passed through was not fun (stopping the original VM was not sufficient. I had to reboot proxmox).

Anyway, I reran a full install and imported the configuration.
I'm reasonably confident that my core config was imported (VLANs, users, ISC DHCP, FW aliases and rules, Unbound block lists...).

But I was under the impression that:
* DHCP leases should have been imported too. By now most (if not all) devices have renewed with their existing IP but I checked right after the first boot and the list appeared small. Machines that I haven't booted since the install don't show up.
* The full history of configuration changes should be available as well. A bunch of files have been imported (confiXXX.xml naming scheme) but the only history that shows in the GUI is just 2 "new" files (config-1732751930.5814.xml and another).

So I now wonder how much I'm missing.
Is there a log of what the config importer did?

[meyergru]

Your impression was wrong. DHCP leases are not being reflected in config,xml, they are local to your DHCP daemon. Only reservations are kept in the configuration.

AFAIK, configuration changes are being constructed from diffs the configuration backups, at least they are not kept within config.xml itself.

There may be other things that may pose problems when you migrate an OpnSense installation:

* When you use configuration backups via Git, it will not work any more if you switch back and forth, because central repo and your local installation will get out of sync. You will have to scratch the repo and restart.


AFAIK, all the config import does is copy over config.xml, plus: you can control which sections will be transferred.
There are parts of the configuration that are not preserved, because they are outside of config.xml, like SSH host keys, root SSH keys and such.

[EricPerl]

There's plenty of posts indicating otherwise. For example:
https://forum.opnsense.org/index.php?topic=9442.0
https://forum.opnsense.org/index.php?topic=22307.0
https://forum.opnsense.org/index.php?topic=28020.0
And the posts are from @franco himself...

I found them looking for the procedure to use the importer in a virtualized scenario.
The importer ran silently during the install. I wonder if there's a leftover log...

[Patrick M. Hausen]

If you manually restore the entire /conf directory - which you also manually backed up, first - you probably get "everything". If you export and import from the UI you get the single XML file and optionally RRD data.

As far as I understand.

[meyergru]

There's plenty of posts indicating otherwise. For example:
https://forum.opnsense.org/index.php?topic=9442.0
https://forum.opnsense.org/index.php?topic=22307.0
https://forum.opnsense.org/index.php?topic=28020.0
And the posts are from @franco himself...

I found them looking for the procedure to use the importer in a virtualized scenario.
The importer ran silently during the install. I wonder if there's a leftover log...

IDK what Franco is talking about when he tells about "DHCP leases", but I think he really means DHCP reservations. Without even looking at the code, I doubt that anything beyond the static leases (aka reservations) can be meant, because they are kept by the DHCP daemon itself in a chroot directory (/var/dhcpd), which lives outside of /conf. So, in order to keep these dynamic database in the config, you would have to stop the daemon and re-import any dynamic leases into /conf/config.xml.

Just look at that file and you will find an XML section <dhcpd></dhcpd> that does not contain something like dynamic leases. If you can prove otherwise, please correct me.

I stand only partly corrected by the host SSH keys, however. They at least live in /conf, but not in /conf/config.xml, so they still get lost when you just import the latter - which you apparently did.

Patrick ist right - RRD data can be exported. There is a checkbox for that.

[EricPerl]

I did NOT just restore from the GUI. It's obvious in that case that only the content of the config.xml will be restored.

As indicated in the OP, I used scp to download the entire /conf folder, made an iso from that, mounted the iso to verify its structure and uploaded the iso to proxmox (made available to the VM as a 2nd image).
During install, I triggered the configuration importer (first interactive prompt).

My expectation was indeed that "everything" would be restored (apart from the ssh keys that scp failed to copy for lack of permissions).
My experience is different:
* At least some DHCP leases missing (some machines not powered since the re-install, even with inactive leases checked). The corresponding archive file is present, but I suspect it was recreated by ISC.
Arguably, I just went by the name of the file (/conf/dhcpleas.tgz)...

* Despite a whole bunch of files in /conf/backup, the only history visible in the GUI is 2 entries with files having a different naming scheme.
Arguably scp didn't perserve dates (might have been an option). I don't know if that matters.
The "old" install was running 24.7.7 (IIRC, possibly 24.7.8).
The ISO used for re-install was 24.7. It was upgraded to 24.7.9 shortly afterwards.

[EricPerl]

I just checked the content of that archive.
It contains the following structure:

Code Select Expand

./var/dhcpd/var/db/
./var/dhcpd/var/db/dhcpd.leases~
./var/dhcpd/var/db/dhcpd6.leases
./var/dhcpd/var/db/dhcpd.leases
There are clearly more than reservations in there.
I don't really have the patience to do a diff between this content and the now running content (mostly caught up).
But I think I should see expired leases for machines not powered since the reinstall. I do not (all visible leases are more recent).

I'm more concerned about the loss of history.
I personally don't expect to revert back at this point, but it looks like there might be a bug here.

[EricPerl]

And to be more precise on the content of the /conf/backup folder:

Code Select Expand

xxx@OPNsense:/conf/backup $ ls
confi000.xml                    confi00y.xml                    confi01w.xml
confi001.xml                    confi00z.xml                    confi01x.xml
...
confi00u.xml                    confi01s.xml                    confi02q.xml
confi00v.xml                    confi01t.xml                    config-1732751930.5814.xml
confi00w.xml                    confi01u.xml                    config-1732753883.9391.xml
confi00x.xml                    confi01v.xml                    config_1.xml

The naming scheme used to be sequential. All these files are present in the ISO image (& the config_1.xml).
The last 2 files appear to use time in file name. These are the only 2 files I can compare (or revert to) in System > Configuration > History.

I just noticed that the first one has the following note: "Root user reset from console".
It could explain why the earlier content is ignored.
This said, I never explicitly did this beyond the steps outlined earlier.
The last change appears to come from the upgrade I ran ~30 minutes later.

[meyergru]

Most disturbing for me is that, obviously, the "intended" approach always was "save the /conf directory".

Never knew that. Basically, all os-backup-* plugins are way off, then, as they only save /conf/config.xml (as I did manually).

[Patrick M. Hausen]

Same for me  ::)

[bimbar]

I am not sure DHCP leases should be in the backup, personally.

[franco]

Historically: different approaches for different goals.

A config.xml export (+ modify) + import is to get a new box up and running.

An installation config import is a way to retain the /conf directory plus some arcane old things like ISC DHCP leases , Netflow, Catpive Portal database. You can get a new system running with this as well, but it's tailored for replacement. This is mainly the domain of the opnsense-importer tool handling /conf imports.

Logs were never part of retention.

Eventually retention of service data is going away which leaves backups and SSH keys as the main motivator for opnsense-importer. RRD data injected into config.xml is another idiosyncrasy added to the mix.

There is more (and weird) history, but it's besides the main point.



Cheers,
Franco

[meyergru]

Ah, thanks, I did not get how this was supposed to work. If at all, it is not officially documented.

All you can do via "System: Configuration: Backups" is to download one XML file, supposedly /conf/config.xml. This does not comprise any of the other files located beneath /conf.

None of the os-backup plugins use these additional files or folders:

dhcpleases.tgz
rrd.tgz (although this is also contained in config.xml, if selected)
netflow.tfz
sshd/
backup/ (keeping the history of config.xml)
dhcp6c_duid
event_config_changed.json (this has something to do with track keeping for git, but it is also poorly documented, so causes more problems than it solves, IMHO)

I second that there is little use in migrating DHCP leases. SSH keys might be nice touch, as well as dhcp6c_duid.

[franco]

> None of the os-backup plugins use these additional files or folders:

Yes, as I said this is merely a convenience approach for when the installer is involved.

This is partially documented here:

https://docs.opnsense.org/manual/install.html#opnsense-importer

There is also a manual page for the utility itself which doesn't go into details other than /conf/config.xml:

https://github.com/opnsense/core/blob/master/src/man/man8/opnsense-importer.8

System: Settings: Misc: Periodic Backups has the other parts, but they need to be enabled in order to be eligible for use in the importer after a crash or a clean shutdown depending on the setting used there. These days they default to off as far as I remember for other reasons. The main reason for these types of backups was UFS being unreliable BTW. It's quite a rabbit hole to fall into but less relevant these days with current defaults and ZFS.

I'll bring the topic up internally to remove some of these things in future releases.


Cheers,
Franco

[franco]

PS: This is the actual code if anyone is interested... https://github.com/opnsense/core/blob/e4d452b37b/src/sbin/opnsense-importer#L340-L378