Saving "System - Settings - General" is super slow

[doktornotor]

I can see this in the release announcement thread:

A hotfix release was issued as 23.7.7_1:

o firmware: speed up saving the firmware settings by avoiding the newly extended trust store rewrite

Perhaps System - Settings - General needs the same? Or a separate button for the rewrite? Takes well over a minute to save while I only want to change the theme. 

[franco]

Yes but no but ... meh.

The "certctl rehash" thing from FreeBSD is pretty slow. I've raised a discussion in FreeBSD but response has been almost nonexistent. And they also broke the curl port requiring the "certctl rehash" so for us there is no alternative.

The actual slowness comes from this addition:

https://github.com/opnsense/core/commit/5b9d7baccba979b5b41858b572145ee826dc389e
https://github.com/opnsense/core/commit/56e66ec809e138e95bb3391f1e5d77d06f062ecc

But just looking at it I guess I know what to do in the short-term here. BRB.


Cheers,
Franco

[franco]

https://github.com/opnsense/core/commit/cb05c53e0

# opnsense-patch cb05c53e0

[doktornotor]

Well, that is a whole LOT better. Takes "just" 20 secs or so now, not minutes. 👍

(P.S. Yes, I am well aware that it's basically an upstream issue with the new trust store mechanism, not something you invented.)

[doktornotor]

This upstream "improvement" is driving me nuts... The entire upgrade would easily be cut by 1/3 at least without this nonsense.

Franco, is there a way to NOT run the darned thing on FW upgrade -during  the "before reboot" phase? It runs yet again directly after boot, no need to run it directly before shutdown?

Arrrgh.

[franco]

Hmm, there are option here. Due to lack of ANY feedback from upstream regarding this progression towards certctl, however, it's a little hard to guess what approach would make the most sense and prevents foot-shooting later.

We could background it like pfSense, but I'd rather not do that unless we add proper locking to it (and that would include random runs of "certctl rehash" from ports as done in ca_root_nss at the moment upstream). We could hash over the input files to try and detect if we even have to rehash... but the ability to self-heal must be retained in order to ship an update that fixes problems with the certificate stash.

I also thought about just injecting our files into /etc/ssl/certs like certctl rehash would. It would avoid the processing time, but a "certctl rehash" would destroy the cert cache as the user would expect it... and ca_root_nss still needs to be dealt with. The nicest thing would be a pre and post-processing hook for certctl and I've made the suggestion but no response.


Cheers,
Franco

[doktornotor]

The upstream "improvement" keeps driving me nuts! Switching another box to OPNsense... Franco, how much work would it take to add an "Apply" button to System: Trust: Authorities? Spent 40 minutes importing the existing  CAs, out of that half an hour rewriting the trust store - after adding every single CA.

Aaaaaaaaaargh! 

[franco]

For everyone else: a permanent workaround was proposed here: https://github.com/opnsense/core/issues/7063#issuecomment-1851528527


Cheers,
Franco

[doktornotor]

Adding some "benchmark" here:

Code: [Select]

# time certctl rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
58.523u 7.417s 1:06.24 99.5%    407+219k 0+0io 0pf+0w

# time /usr/local/opnsense/scripts/system/certctl.py rehash
Scanning /usr/share/certs/blacklisted for certificates...
Scanning /usr/share/certs/trusted for certificates...
Scanning /usr/local/share/certs for certificates...
1.511u 0.314s 0:02.16 84.2%     5+168k 0+0io 0pf+0w

So it's about 30x faster.