I've been running a virtualized proxmox for a few weeks now (very nice upgrade coming from TP-link).
I saw a recent post from @meyergru about recommended settings and thought I'd check them out.
If anything, that would be a good practice for future recovery...
I used scp to download the entire /conf folder (got some errors on ssh keys that I ignored).
I searched/replaced the passthrough NICs with bridge equivalents in the config.xml only (i.e. not the files in /conf/backup).
I made an iso out of the updated downloaded content that I made available to a new VM.
Creating the bridges over the NICs that used to be passed through was not fun (stopping the original VM was not sufficient. I had to reboot proxmox).
Anyway, I reran a full install and imported the configuration.
I'm reasonably confident that my core config was imported (VLANs, users, ISC DHCP, FW aliases and rules, Unbound block lists...).
But I was under the impression that:
* DHCP leases should have been imported too. By now most (if not all) devices have renewed with their existing IP but I checked right after the first boot and the list appeared small. Machines that I haven't booted since the install don't show up.
* The full history of configuration changes should be available as well. A bunch of files have been imported (confiXXX.xml naming scheme) but the only history that shows in the GUI is just 2 "new" files (config-1732751930.5814.xml and another).
So I now wonder how much I'm missing.
Is there a log of what the config importer did?
Your impression was wrong. DHCP leases are not being reflected in config,xml, they are local to your DHCP daemon. Only reservations are kept in the configuration.
AFAIK, configuration changes are being constructed from diffs the configuration backups, at least they are not kept within config.xml itself.
There may be other things that may pose problems when you migrate an OpnSense installation:
* When you use configuration backups via Git, it will not work any more if you switch back and forth, because central repo and your local installation will get out of sync. You will have to scratch the repo and restart.
AFAIK, all the config import does is copy over config.xml, plus: you can control which sections will be transferred.
There are parts of the configuration that are not preserved, because they are outside of config.xml, like SSH host keys, root SSH keys and such.
There's plenty of posts indicating otherwise. For example:
https://forum.opnsense.org/index.php?topic=9442.0 (https://forum.opnsense.org/index.php?topic=9442.0)
https://forum.opnsense.org/index.php?topic=22307.0 (https://forum.opnsense.org/index.php?topic=22307.0)
https://forum.opnsense.org/index.php?topic=28020.0 (https://forum.opnsense.org/index.php?topic=28020.0)
And the posts are from @franco himself...
I found them looking for the procedure to use the importer in a virtualized scenario.
The importer ran silently during the install. I wonder if there's a leftover log...
If you manually restore the entire /conf directory - which you also manually backed up, first - you probably get "everything". If you export and import from the UI you get the single XML file and optionally RRD data.
As far as I understand.
Quote from: EricPerl on November 28, 2024, 07:31:27 PM
There's plenty of posts indicating otherwise. For example:
https://forum.opnsense.org/index.php?topic=9442.0 (https://forum.opnsense.org/index.php?topic=9442.0)
https://forum.opnsense.org/index.php?topic=22307.0 (https://forum.opnsense.org/index.php?topic=22307.0)
https://forum.opnsense.org/index.php?topic=28020.0 (https://forum.opnsense.org/index.php?topic=28020.0)
And the posts are from @franco himself...
I found them looking for the procedure to use the importer in a virtualized scenario.
The importer ran silently during the install. I wonder if there's a leftover log...
IDK what Franco is talking about when he tells about "DHCP leases", but I think he really means DHCP reservations. Without even looking at the code, I doubt that anything beyond the static leases (aka reservations) can be meant, because they are kept by the DHCP daemon itself in a chroot directory (/var/dhcpd), which lives outside of /conf. So, in order to keep these dynamic database in the config, you would have to stop the daemon and re-import any dynamic leases into /conf/config.xml.
Just look at that file and you will find an XML section <dhcpd></dhcpd> that does
not contain something like dynamic leases. If you can prove otherwise, please correct me.
I stand only partly corrected by the host SSH keys, however. They at least live in /conf, but not in /conf/config.xml, so they still get lost when you just import the latter - which you apparently did.
Patrick ist right - RRD data
can be exported. There is a checkbox for that.
I did NOT just restore from the GUI. It's obvious in that case that only the content of the config.xml will be restored.
As indicated in the OP, I used scp to download the entire /conf folder, made an iso from that, mounted the iso to verify its structure and uploaded the iso to proxmox (made available to the VM as a 2nd image).
During install, I triggered the configuration importer (first interactive prompt).
My expectation was indeed that "everything" would be restored (apart from the ssh keys that scp failed to copy for lack of permissions).
My experience is different:
* At least some DHCP leases missing (some machines not powered since the re-install, even with inactive leases checked). The corresponding archive file is present, but I suspect it was recreated by ISC.
Arguably, I just went by the name of the file (/conf/dhcpleas.tgz)...
* Despite a whole bunch of files in /conf/backup, the only history visible in the GUI is 2 entries with files having a different naming scheme.
Arguably scp didn't perserve dates (might have been an option). I don't know if that matters.
The "old" install was running 24.7.7 (IIRC, possibly 24.7.8).
The ISO used for re-install was 24.7. It was upgraded to 24.7.9 shortly afterwards.
I just checked the content of that archive.
It contains the following structure:
./var/dhcpd/var/db/
./var/dhcpd/var/db/dhcpd.leases~
./var/dhcpd/var/db/dhcpd6.leases
./var/dhcpd/var/db/dhcpd.leases
There are clearly more than reservations in there.
I don't really have the patience to do a diff between this content and the now running content (mostly caught up).
But I think I should see expired leases for machines not powered since the reinstall. I do not (all visible leases are more recent).
I'm more concerned about the loss of history.
I personally don't expect to revert back at this point, but it looks like there might be a bug here.
And to be more precise on the content of the /conf/backup folder:
xxx@OPNsense:/conf/backup $ ls
confi000.xml confi00y.xml confi01w.xml
confi001.xml confi00z.xml confi01x.xml
...
confi00u.xml confi01s.xml confi02q.xml
confi00v.xml confi01t.xml config-1732751930.5814.xml
confi00w.xml confi01u.xml config-1732753883.9391.xml
confi00x.xml confi01v.xml config_1.xml
The naming scheme used to be sequential. All these files are present in the ISO image (& the config_1.xml).
The last 2 files appear to use time in file name. These are the only 2 files I can compare (or revert to) in System > Configuration > History.
I just noticed that the first one has the following note: "Root user reset from console".
It could explain why the earlier content is ignored.
This said, I never explicitly did this beyond the steps outlined earlier.
The last change appears to come from the upgrade I ran ~30 minutes later.
Most disturbing for me is that, obviously, the "intended" approach always was "save the /conf directory".
Never knew that. Basically, all os-backup-* plugins are way off, then, as they only save /conf/config.xml (as I did manually).
Same for me ::)
I am not sure DHCP leases should be in the backup, personally.
Historically: different approaches for different goals.
A config.xml export (+ modify) + import is to get a new box up and running.
An installation config import is a way to retain the /conf directory plus some arcane old things like ISC DHCP leases , Netflow, Catpive Portal database. You can get a new system running with this as well, but it's tailored for replacement. This is mainly the domain of the opnsense-importer tool handling /conf imports.
Logs were never part of retention.
Eventually retention of service data is going away which leaves backups and SSH keys as the main motivator for opnsense-importer. RRD data injected into config.xml is another idiosyncrasy added to the mix.
There is more (and weird) history, but it's besides the main point.
Cheers,
Franco
Ah, thanks, I did not get how this was supposed to work. If at all, it is not officially documented.
All you can do via "System: Configuration: Backups" is to download one XML file, supposedly /conf/config.xml. This does not comprise any of the other files located beneath /conf.
None of the os-backup plugins use these additional files or folders:
dhcpleases.tgz
rrd.tgz (although this is also contained in config.xml, if selected)
netflow.tfz
sshd/
backup/ (keeping the history of config.xml)
dhcp6c_duid
event_config_changed.json (this has something to do with track keeping for git, but it is also poorly documented, so causes more problems than it solves, IMHO)
I second that there is little use in migrating DHCP leases. SSH keys might be nice touch, as well as dhcp6c_duid.
> None of the os-backup plugins use these additional files or folders:
Yes, as I said this is merely a convenience approach for when the installer is involved.
This is partially documented here:
https://docs.opnsense.org/manual/install.html#opnsense-importer
There is also a manual page for the utility itself which doesn't go into details other than /conf/config.xml:
https://github.com/opnsense/core/blob/master/src/man/man8/opnsense-importer.8
System: Settings: Misc: Periodic Backups has the other parts, but they need to be enabled in order to be eligible for use in the importer after a crash or a clean shutdown depending on the setting used there. These days they default to off as far as I remember for other reasons. The main reason for these types of backups was UFS being unreliable BTW. It's quite a rabbit hole to fall into but less relevant these days with current defaults and ZFS.
I'll bring the topic up internally to remove some of these things in future releases.
Cheers,
Franco
PS: This is the actual code if anyone is interested... https://github.com/opnsense/core/blob/e4d452b37b/src/sbin/opnsense-importer#L340-L378
So I'm not entirely sure where this leaves me.
I was merely trying to comply with recommendations from @meyergru, hence "converting" from UFS+passthrough to ZFS+bridges.
It's clearly not the previewing/testing upgrade path.
It would fall in the migration path.
I'm a little confused because all the backup files (config history) were copied over the new install.
I suspect the dhcp leases archive was too.
Are you saying they were actually ignored later because the migration code path only cares about the config.xml?
It's not apparent to me how the code is aware of the 2 separate paths (the source drive FS?)...
In any case, given what I am trying to achieve, is there a path forward that preserves history?
Still baffled by the change in naming scheme...
Secondary question:
The previewing/testing upgrade path implies the existence of install media.
This is only available for major releases, right? IOW, the procedure is only useful for major upgrades?
If the backups were set to back up they are restored in the config-import-install case. It depends on how old the installation was WRT default backup settings and if they were manually altered.
If you copied the /conf folder from an older install to a newer everything is retained but not used depending on periodic backup settings (again).
Cheers,
Franco
The source machine was setup a few weeks back (24.7 install media, upgrade to 24.7.7, bunch of changes as I migrated my VLANs over from physical Tp-link router, then a few more as I performed the actual replacement).
All on UFS+passthrough based on online recos.
A few days back, I read a post from @meyergru with different recos and decided to give them a shot (ZFS + bridges to the same NICs).
scp /conf to mgmt PC, replaced igc -> vtnet in config.xml, made iso, uploaded ISO and installed with import.
Same install media, upgraded to 24.7.9 (latest).
I haven't touched the backup settings yet. They are all default.
On the old install, I had full history access.
On the new one, it's not available in the GUI (I realize I would have to replace igc -> vtnet in the files in /conf/backup too for reverts to succeed. Looks doable). The only history that shows is the result on the upgrade.
Ok, the JSON file may be interfering, but it's not my area of expertise. Just to be sure the old backups are in /conf/backup ?
Cheers,
Franco
Doh!
As I was replying, I doublechecked my steps again and realized that I compared file names against the generated iso, not the original scp'd copy...
And I screwed up generating the iso image, because it didn't preserve the file names (forcing them to 8.3).
That explains the naming scheme difference!
And you're probably referring to opn/conf/event_config_changed.json which points back to the last file.
Given the mangled file names, it probably got quite confused...
But now that I understand this better (I think), I can rerun the entire install.
I'll do a full (conf/backup files) igc->vtnet replacement beforehand, then regen the ISO (with -allow-multidot -l this time).
The safe thing to do would be to run this on a brand new VM, but I could even do it on the current one (as I understand it). I still have the old one to fall back too.
And this time, I'll do that from a machine in the same VLAN as the proxmox machine.
I realized that mistake quickly when I shutdown the VM ;D
This is all good practice!
Thx all. I'll report back when I do this tomorrow (at the earliest).
That was quicker than expected. I have practice now...
Moving a machine in the same VLAN as the proxmox host helped because I never lost connectivity.
My latest install is up and running. I have not upgraded it just yet.
Before regenerating the ISO, I processed all the backup configs (sed -i s/igc/vtnet/g) and got a fresh copy of the sshd directory (as root so no errors).
All my core settings were imported like last time.
Since I used the old dhcp leases archive, it was obvious that it had been imported properly (some expired leases from a few days ago).
I didn't get an error when I ssh'd back into OPN so the ssh keys were imported fine too.
All the old configs are present in /conf/backup, with proper names this time!
I still have no access to the history in the GUI though.
I noticed the /conf/event_config_changed.json was missing (not handled by the importer code, so that makes sense).
I copied it over and rebooted. No change.
I also checked its content references an existing file in the backup directory.
xxx@OPNsense:~ $ cat /conf/event_config_changed.json
{"last_processed_stamp":1732740233.7202}
xxx@OPNsense:~ $ ls /conf/backup
config_1731183347.3.xml config_1731269020.4566.xml config_1731531750.1009.xml
...
config_1731207470.4087.xml config_1731531563.4327.xml config_1732740233.7202.xml
config_1731207518.8078.xml config_1731531600.2763.xml
config_1731264024.3773.xml config_1731531637.3672.xml
System > Configuration > History indicates "no backups available".
System > Configuration > Backups has an empty count and indicates that 2.8M is currently used.
Looks like I'm close...
And I had made another mistake...
As can be seen in my previous post, the filenames were still mangled, yet more subtly...
The expected pattern is config-{time}.xml. A dash, not an underscore.
I must have been missing another option when using mkisofs.
I noticed the mismatch when I began to look at the code.
I renamed all the files in place and the GUI now shows the full history.
Here's the command line that seems to preserve the backup files names:
mkisofs -allow-multidot -l -relaxed-filenames -o somefile.iso dirwithslashconf
Thanks, that's very useful. Actually a bit of a tricky case the whole import from ISO case.
Cheers,
Franco
@EricPerl why did you use an ISO in the first place? It's a virtual machine, right? You could have attached a second virtual hard disk with UFS mimicking a USB drive ...
Mostly because my search for a way to use the config importer under virtualization lead me to this thread
https://forum.opnsense.org/index.php?topic=28020.0 (https://forum.opnsense.org/index.php?topic=28020.0)
where the OP was trying to do exactly what I was looking for.
My experiments my proxmox predate my experiments with OPNsense by about 1h...
I did not think about adding a 2nd drive. I would also have had to figure out how to copy the config files over.
I just looked at that and it seems reasonable.
I knew how to deal with prebuilt ISOs. Generating one looked easy (until it wasn't because of the entire file name mangling aspect). I learned one thing though: Compare with tools, not visually... And then you have to learn the tools but that's OK.
Looking back, mkisofs would have generated "warnings" in verbose mode during my initial attempt (conversion to 8.3). But it's not generating any without the relaxed-filenames options. That's not nice.
Importing from a disk was simpler in some regards.
I tried as a learning opportunity. Here's what I did:
* Created a new VM (2 bridges for LAN & WAN, one 16GB drive)
Proxmox is using ZFS so I ended up with a /dev/zd0 zvol
fdisk /dev/zd0
o # to create a MBR partition table
n with defaults # to create a partition
t e1 # to change the partition type to DOS access
w # to write the changes and exit
Back at the prompt:
lsblk -f # to verify what was done
...
zd0
└─zd0p1
mkfs -t msdos /dev/zd0p1 # to format the partition as MSDOS
mount -t auto /dev/zd0p1 /tmp/vmdisk # to mount the FS
scp -r root@opnsense.lan:/conf /tmp/vmdisk/ # to copy a running config
umount /tmp/vmdisk # Unmount the FS
Then I started the install, triggered the import from da0, and installed over the disk.
I know Patrick had suggested a UFS FS but partitioning was harder...
I didn't fully vet the new install (not hooked to the network to avoid conflicts) but verified all the files were in place and the opnsense shell indicated that all my VLANs had been imported.