unbound didn't start after power failure

[RainerR]

Hi all.

I want to drop a short note how I found the root cause and how I fixed the problem.

First my system description:

• OPNsense 18.7.10-amd64
• 2-node HA Cluster
• HW: Zotac Zbox Nano

The Problem:
After a power failure of the backup firewall the unbound service didn't start anymore.

When I try to start the service I got the following entry in the 'General' log file:
opnsense: /usr/local/etc/rc.reload_all: The command '/usr/local/sbin/unbound -c '/var/unbound/unbound.conf'' returned exit code '1', the output was '[1547833339] unbound[85235:0] error: Error for server-cert-file: /var/unbound/unbound_server.pem [1547833339] unbound[85235:0] error: Error in SSL_CTX use_certificate_chain_file crypto error:02001002:system library:fopen:No such file or directory [1547833339] unbound[85235:0] error: and additionally crypto error:20074002:BIO routines:FILE_CTRL:system lib [1547833339] unbound[85235:0] error: and additionally crypto error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib [1547833339] unbound[85235:0] fatal error: could not set up remote-control'

After opening an ssh session on both host (master and backup firewall) I compared the content of the /var/unbound/ folder and I found out that the following files were missing on the backup firewall:

• unbound_control.key
• unbound_control.pem
• unbound_server.key
• unbound_server.pem

So I tried to run the following command to solve the issue: sudo -u unbound unbound-control-setup -d /var/unbound/

As result I got the following error:
setup in directory /var/unbound/
generating unbound_server.key
/usr/bin/openssl: Undefined symbol "[SL_set_jio"
/usr/local/sbin/unbound-control-setup fatal error: could not genrsa

In the 'General' log file I got this:
opnsense: /usr/local/etc/rc.reload_all: The command 'chroot -u unbound -g unbound / '/usr/local/sbin/unbound-control-setup' -d '/var/unbound'' returned exit code '1', the output was 'setup in directory /var/unbound generating unbound_server.key /usr/bin/openssl: Undefined symbol "[SL_set_jio" /usr/local/sbin/unbound-control-setup fatal error: could not genrsa'


I tried to get the openssl version (/usr/bin/openssl) by using the following command on the backup firewall: openssl version

I got the following error message:
/usr/bin/openssl: Undefined symbol "[SL_set_jio"

At this point I found the root cause because the openssl file was corrupted.
So as workaround I've copied the openssl file from the master to the backup firewall by using scp.

After this I was able to run the command sudo -u unbound unbound-control-setup -d /var/unbound/ successfully and also to start the unbound service on the backup firewall again.

On long term I will re-install the backup firewall because I don't know if other files are corrupted too.

Best regards,
Rainer.