Opnsense 22.7 stuck after upgrade

[schnipp]

Hi all,

yesterday I started to upgrade my Opnsense from 21.1.10 to 22.7.2. The upgrade process to 22.7.0 went fine till reboot. After rebooting the sense never came up and the console showed that booting was stuck at "Configuring dynamic DNS clients..."

I further updated to version 27.7.2, but the same issue still exists.

After the system was stuck I got the console after pressing Ctrl-C, so I could manually update to the latest version and start the text based opnsense-shell. Using the latter, restarting all services did not change anything. I also tried disabling DDNS by modifying the config.xml, but no chance.

Has anybody noticed a similar issue or has an idea how to debug?

[schnipp]

Does nobody has an idea how I can debug this issue? Currently, I am not able to keep the firewall up to date and patch any security issues :'(.

Maybe one of the developers can give hints how to execute the boot scripts separately or step-by-step. Further hints besides the ones mentioned are appreciated.

Thanks.

[lilsense]

have you tried uninstalling the DDNS client service?

[schnipp]

I tried in that direction by disabling the service via command line. Unfortunately, there is no command line reference existing how control the firewall (e.g. install/uninstall/enable/disable services). Instead, I tried editing the config file using "vi" and restarting all services. No luck.

In the stuck state the web gui is not accessible because either not started or blocked by the incomplete setup of firewall rules.

Do you know about a command line reference of CLI based opnsense tools?

Thanks

[lilsense]

I highly recommend rebuilding it from scratch and restoring your config.

[schnipp]

Thanks for your reply. In my eyes a complete reinstallation should only be the last resort. Before the update all files were intact (health audit). So, it looks like there is bug in version 22.7 which lead to enter a dead lock state during the boot process. To improve overall software quality such hints on bugs should be investigated.

[cookiemonster]

Without trying to sound like a completely unsympathetic person, and I ask with respect, how do you propose that anyone, including the devs to replicate your specific hardware/configuration mix?
I've never seen my OPN installations trying to "call home", i.e. to collect metrics, so there might (or not) be numbers available of the number of upgrades, but going by the forums posts, I make a guess that there are hundreds or more successful upgrades of those versions; so it doesn't seem (to me) like a widespread problem and more pertaining to a specific configuration.
That on top of the testing in-house that I'm sure is done prior to shipping a new version.
Back to the current issue. Are you able to ctrl+c to finish booting? If so, then get to the UI to disable the dynamic dns as a test, and console to gather system logs? I can see if you're remoting in and only ssh is available, this would be an unlikely scenario.

[WN1X]

Have you tried disabling ddclient? I believe editing the /etc/rc.conf.d/ddclient file will allow you to disable the service.

[schnipp]

Without trying to sound like a completely unsympathetic person, and I ask with respect, how do you propose that anyone, including the devs to replicate your specific hardware/configuration mix?

I think we misunderstood each other. It's almost impossible that any other can replicate my problem on the current level of information. But that's not my intention. I'd like to debug the issue myself and asked for hints how to control certain parameters via the CLI (e.g. interactive standalone execution of ddns scripts (line-by-line), disabling services, reloading specific services etc.).

I've never seen my OPN installations trying to "call home", i.e. to collect metrics, so there might (or not) be numbers available of the number of upgrades, but going by the forums posts, I make a guess that there are hundreds or more successful upgrades of those versions; so it doesn't seem (to me) like a widespread problem and more pertaining to a specific configuration.

It's in the nature of things. The fewer of these have a problem, the more help is required from the person concerned. BTW, nowadays there is a lot of software calling home or having some kind of telemetry included.This is often a plague and can also induce risks. So, I am happy that Opnsense does not have such things included.

Back to the current issue. Are you able to ctrl+c to finish booting? If so, then get to the UI to disable the dynamic dns as a test, and console to gather system logs? I can see if you're remoting in and only ssh is available, this would be an unlikely scenario.

Unfortunately, ctrl+c only terminates the script and drops me to the shell. This means Opnsense is incompletely started and does not offer the gui. But internet access via shell is available after adding a public DNS to /etc/resolv.conf. I guess I can debug and investigate the issue by temporarily disabling DDNS services and execute them standalone. But I don't know how to achieve this on the command line. I'll try the first hint given by @WN1X

[schnipp]

Have you tried disabling ddclient? I believe editing the /etc/rc.conf.d/ddclient file will allow you to disable the service.

I tried disabling DDNS by modifying the config during runtime. I know, this is not the best idea but followed this way because I did not know how to disable the service on the CLI. So, I'll give your hint a try. Thanks.

[schnipp]

Today, I tried to upgrade the Opnsense again. Before starting I proactively disabled the new dd-client in the web gui. Everything went fine including the latest update to version 22.7.2.

Now, I need to check how to investigate the dd-client script in conjunction with my configuration without breaking the installation again.