[Solved] Configuration import verification

[EricPerl]

So I'm not entirely sure where this leaves me.

I was merely trying to comply with recommendations from @meyergru, hence "converting" from UFS+passthrough to ZFS+bridges.
It's clearly not the previewing/testing upgrade path.
It would fall in the migration path.

I'm a little confused because all the backup files (config history) were copied over the new install.
I suspect the dhcp leases archive was too.
Are you saying they were actually ignored later because the migration code path only cares about the config.xml?
It's not apparent to me how the code is aware of the 2 separate paths (the source drive FS?)...

In any case, given what I am trying to achieve, is there a path forward that preserves history?
Still baffled by the change in naming scheme...

Secondary question:
The previewing/testing upgrade path implies the existence of install media.
This is only available for major releases, right? IOW, the procedure is only useful for major upgrades?

[franco]

If the backups were set to back up they are restored in the config-import-install case. It depends on how old the installation was WRT default backup settings and if they were manually altered.

If you copied the /conf folder from an older install to a newer everything is retained but not used depending on periodic backup settings (again).


Cheers,
Franco

[EricPerl]

The source machine was setup a few weeks back (24.7 install media, upgrade to 24.7.7, bunch of changes as I migrated my VLANs over from physical Tp-link router, then a few more as I performed the actual replacement).
All on UFS+passthrough based on online recos.

A few days back, I read a post from @meyergru with different recos and decided to give them a shot (ZFS + bridges to the same NICs).
scp /conf to mgmt PC, replaced igc -> vtnet in config.xml, made iso, uploaded ISO and installed with import.
Same install media, upgraded to 24.7.9 (latest).

I haven't touched the backup settings yet. They are all default.
On the old install, I had full history access.
On the new one, it's not available in the GUI (I realize I would have to replace igc -> vtnet in the files in /conf/backup too for reverts to succeed. Looks doable). The only history that shows is the result on the upgrade.

[franco]

Ok, the JSON file may be interfering, but it's not my area of expertise. Just to be sure the old backups are in /conf/backup ?


Cheers,
Franco

[EricPerl]

Doh!
As I was replying, I doublechecked my steps again and realized that I compared file names against the generated iso, not the original scp'd copy...
And I screwed up generating the iso image, because it didn't preserve the file names (forcing them to 8.3).
That explains the naming scheme difference!
And you're probably referring to opn/conf/event_config_changed.json which points back to the last file.
Given the mangled file names, it probably got quite confused...

But now that I understand this better (I think), I can rerun the entire install.
I'll do a full (conf/backup files) igc->vtnet replacement beforehand, then regen the ISO (with -allow-multidot -l this time).
The safe thing to do would be to run this on a brand new VM, but I could even do it on the current one (as I understand it). I still have the old one to fall back too.

And this time, I'll do that from a machine in the same VLAN as the proxmox machine.
I realized that mistake quickly when I shutdown the VM  ;D
This is all good practice!
Thx all. I'll report back when I do this tomorrow (at the earliest).

[EricPerl]

That was quicker than expected. I have practice now...
Moving a machine in the same VLAN as the proxmox host helped because I never lost connectivity.

My latest install is up and running. I have not upgraded it just yet.
Before regenerating the ISO, I processed all the backup configs (sed -i s/igc/vtnet/g) and got a fresh copy of the sshd directory (as root so no errors).

All my core settings were imported like last time.
Since I used the old dhcp leases archive, it was obvious that it had been imported properly (some expired leases from a few days ago).
I didn't get an error when I ssh'd back into OPN so the ssh keys were imported fine too.

All the old configs are present in /conf/backup, with proper names this time!
I still have no access to the history in the GUI though.

I noticed the /conf/event_config_changed.json was missing (not handled by the importer code, so that makes sense).
I copied it over and rebooted. No change.
I also checked its content references an existing file in the backup directory.

Code Select Expand

xxx@OPNsense:~ $ cat /conf/event_config_changed.json
{"last_processed_stamp":1732740233.7202}

xxx@OPNsense:~ $ ls /conf/backup
config_1731183347.3.xml         config_1731269020.4566.xml      config_1731531750.1009.xml
...
config_1731207470.4087.xml      config_1731531563.4327.xml      config_1732740233.7202.xml
config_1731207518.8078.xml      config_1731531600.2763.xml
config_1731264024.3773.xml      config_1731531637.3672.xml


System > Configuration > History indicates "no backups available".
System > Configuration > Backups has an empty count and indicates that 2.8M is currently used.

Looks like I'm close...

[EricPerl]

And I had made another mistake...
As can be seen in my previous post, the filenames were still mangled, yet more subtly...
The expected pattern is config-{time}.xml. A dash, not an underscore.
I must have been missing another option when using mkisofs.
I noticed the mismatch when I began to look at the code.

I renamed all the files in place and the GUI now shows the full history.

[EricPerl]

Here's the command line that seems to preserve the backup files names:

Code Select Expand

mkisofs -allow-multidot -l -relaxed-filenames -o somefile.iso dirwithslashconf

[franco]

Thanks, that's very useful. Actually a bit of a tricky case the whole import from ISO case.


Cheers,
Franco

[Patrick M. Hausen]

@EricPerl why did you use an ISO in the first place? It's a virtual machine, right? You could have attached a second virtual hard disk with UFS mimicking a USB drive ...

[EricPerl]

Mostly because my search for a way to use the config importer under virtualization lead me to this thread
https://forum.opnsense.org/index.php?topic=28020.0
where the OP was trying to do exactly what I was looking for.

My experiments my proxmox predate my experiments with OPNsense by about 1h...
I did not think about adding a 2nd drive. I would also have had to figure out how to copy the config files over.
I just looked at that and it seems reasonable.

I knew how to deal with prebuilt ISOs. Generating one looked easy (until it wasn't because of the entire file name mangling aspect). I learned one thing though: Compare with tools, not visually... And then you have to learn the tools but that's OK.
Looking back, mkisofs would have generated "warnings" in verbose mode during my initial attempt (conversion to 8.3). But it's not generating any without the relaxed-filenames options. That's not nice.

[EricPerl]

Importing from a disk was simpler in some regards.
I tried as a learning opportunity. Here's what I did:
* Created a new VM (2 bridges for LAN & WAN, one 16GB drive)
Proxmox is using ZFS so I ended up with a /dev/zd0 zvol

Code Select Expand

fdisk /dev/zd0
o                    # to create a MBR partition table
n with defaults      # to create a partition
t e1                 # to change the partition type to DOS access
w                    # to write the changes and exit

Back at the prompt:

Code Select Expand

lsblk -f                                    # to verify what was done
...
zd0
└─zd0p1

mkfs -t msdos /dev/zd0p1                    # to format the partition as MSDOS
mount -t auto /dev/zd0p1 /tmp/vmdisk        # to mount the FS
scp -r root@opnsense.lan:/conf /tmp/vmdisk/ # to copy a running config
umount /tmp/vmdisk                          # Unmount the FS

Then I started the install, triggered the import from da0, and installed over the disk.

I know Patrick had suggested a UFS FS but partitioning was harder...
I didn't fully vet the new install (not hooked to the network to avoid conflicts) but verified all the files were in place and the opnsense shell indicated that all my VLANs had been imported.