Upgrade to 22.7 corrupted nginx service

[jd7]

After upgrading from 22.1.10_4-amd64 to 22.7.3_2-amd64 the nginx update broke the current setup. After restarting the nginx server, I continously got the error:

Code Select Expand

"BIO_new_file("/usr/local/etc/dh-parameters.4096") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/usr/local/etc/dh-parameters.4096','r') error:2006D080:BIO routines:BIO_new_file:no such file)"

After looking via shell to the nginx.conf file and the file dh-parameters.4096, I found out that all dh-parameters.<keysize> files are missing.

After generating these files with:

Code Select Expand

/usr/bin/openssl dhparam -dsaparam -out /usr/local/etc/dh-parameters.1024 1024
/usr/bin/openssl dhparam -dsaparam -out /usr/local/etc/dh-parameters.2048 2048
/usr/bin/openssl dhparam -dsaparam -out /usr/local/etc/dh-parameters.4096 4096

and restarting nginx, I seems to work.

Any ideas why this have to be done manually after upgrading?

[Fright]

are you sure you didn't lock the package?
nginx plugin uses own dh-params file since 1.28.2
https://github.com/opnsense/plugins/commit/abf01bff99e019b31792cf3489372f07e0c33653

[franco]

Locks are removed on upgrades. As far as I know nginx only used 4096 DH file.

I'd suggest doing a health audit first, then checking for the "bad" file that still looks for old dh-parameters.4096 file... it may be in one if your custom configs added into include directory. The new file is:

/usr/local/opnsense/data/OPNsense/Nginx/dh-parameters.4096.rfc7919


Cheers,
Franco

[jd7]

I think I started with a clean installation of 21.1. Since them I always used the update mechanism from the webui. It worked since the upgrade to 22.7. Here is the output of the health check.

***GOT REQUEST TO AUDIT HEALTH***
Currently running OPNsense 22.7.3_2 (amd64/OpenSSL) at Tue Sep  6 11:05:03 CEST 2022
>>> Check installed kernel version
Version 22.7.3 is correct.
>>> Check for missing or altered kernel files
No problems detected.
>>> Check installed base version
Version 22.7.3 is correct.
>>> Check for missing or altered base files
No problems detected.
>>> Check installed repositories
OPNsense
>>> Check installed plugins
os-clamav 1.7_1
os-dmidecode 1.1_1
os-git-backup 1.0_3
os-haproxy 3.11
os-iperf 1.0_1
os-nginx 1.29_1
os-postfix 1.23_2
os-redis 1.1_1
os-rspamd 1.12
os-smart 2.2
os-theme-cicada 1.29
os-wireguard 1.12
>>> Check locked packages
No locks found.
>>> Check for missing package dependencies
Checking all packages: .......... done
>>> Check for missing or altered package files
Checking all packages: .......... done
>>> Check for core packages consistency
Core package "opnsense" has 63 dependencies to check.
Checking packages: ................................................................. done
***DONE***

[franco]

Looks normal, let's try to find the file:

# grep -nr "/usr/local/etc/dh-parameters.4096" /usr/local/etc


Cheers,
Franco

[Fright]

@franco

Locks are removed on upgrades

didn't pay attention. thanks

jd7

can you try to just hit Apply on Services: Nginx: Configuration ?
(nginx templates not reloading on restart. on Apply only)

[jd7]

First, before I generated manually the files with dhparam, I tried to reinstall the plugin, but it doesn't worked.

After your proposal to press "apply", the nginx.conf changed:

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_dhparam /usr/local/opnsense/data/OPNsense/Nginx/dh-parameters.4096.rfc79
19;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    ssl_prefer_server_ciphers on;
    ssl_stapling off;

and now I can find dh-parameters.4096.rfc79 in the path above and it still works ;-)

My approach also worked, but I tried to understand whats going wrong. As I know see, that after an update I have to press "apply" to every service to reload changed templates?

[franco]

A reboot should apply all changes to configuration files, but for some reason here it did not?

The core DH parameter files were removed in 22.7 since they are no longer being used by most services. Only OpenVPN, Postfix and Nginx remain and for those we opted for a simple RFC 7919 approach of a fixed file instead of providing our own parameters file and ways to regenerate them by cron.

Plus it upgraded security for some users who have not chosen the largest bit parameter file yet.


Cheers,
Franco

[jd7]

Thanks for your explanation! Nothing is perfect. I try to better understand opnsense and hopefully help to improve it in future.

[Fright]

@franco

A reboot should apply all changes to configuration files, but for some reason here it did not?

sorry, could you give me a hint what exactly should trigger the plugins templates reload at system startup (some call of configd template reload?)?
i quick tested with postfix, haproxy and nginx - add control lines to config templates and restart OPN - none of them updated config files on system reboot

[franco]

Strange... this is supposed to be it: https://github.com/opnsense/core/blob/master/src/etc/rc.syshook.d/early/15-templates


Cheers,
Franco

[Fright]

Oh. Thanks! (I wonder where I was looking). I'll try again, knowing where to look)

[franco]

Only idea I have is configd is not ready so it's never executed. I've seen the "error" during boot but never investigated further.


Cheers,
Franco

[Fright]

Please ignore my findings for now. I definitely need to try again, but only when I sleep) (I definitely could have missed something)

[Fright]

@franco
yes, you are right. configd may not be ready so quickly ("configctl template reload \*" returns "configd socket missing" on my test vm on boot). giving a little time (changed to "configctl -w 5 template reload \*") did the trick. thanks!