OPNsense Forum
English Forums => 23.7 Legacy Series => Topic started by: anicoletti on August 01, 2023, 05:50:34 am
-
Ran into some issues upgrading to 23.7 and Unbound not starting. Figured I'd share this information as I did not see anyone else post this specific issue yet.
I upgraded from 23.1.11_1 to 23.7 on one of our client firewalls this evening. Upon completion, DNS services failed to start on the firewall. I was able to remote into another system and connect into the firewall and noticed Unbound was not running. Attempting to start it spun from 10-15 seconds then returned with it still offline. Connected to the firewall via SSH and ran the following command to check the status on starting the service:
Command:
unbound -c /var/unbound/unbound.conf
Results:
/var/unbound/etc/domainoverrides.conf:1: error: syntax error
read /var/unbound/unbound.conf failed: 1 errors in configuration file
[1690860690] unbound[25940:0] fatal error: Could not read config file: /var/unbound/unbound.conf. Maybe try unbound -dd, it stays on the commandline to see more errors, or unbound-checkconf
We still have some clients where domain overrides are under the Overrides section in the GUI and not moved over to the Query Forwarding yet. Upon removing the entries from the Overrides section and adding them back in under Query Forwarding, I was able to successfully start the Unbound services and query the internal domain overrides.
-
I realize you moved the entries, but is it possible to go back and get verbose output to see the exact breakage? e.g.
unbound -ddvv -c /var/unbound/unbound.conf
after stopping Unbound from the GUI.
Cheers,
Stephan
-
I have a bunch of domain overrides and didn't encounter this issue, so it must be something more specific. Would indeed be interesting if you could reproduce it.
Cheers
Maurice
-
Interesting. So two issues. First, after adding the items back under the Overrides GUI, I'm able to restart Unbound without issue. Second, when I attempt to run the command with -ddvv, I get the following error:
root@opnsense:/var/unbound/etc # unbound -ddvv -c /var/unbound/unbound.conf
[1690891368] unbound[46649:0] notice: Start of unbound 1.17.1.
[1690891368] unbound[46649:0] debug: chdir to /var/unbound
[1690891368] unbound[46649:0] debug: chroot to /var/unbound
[1690891368] unbound[46649:0] debug: drop user privileges, run as unbound
[1690891368] unbound[46649:0] debug: switching log to stderr
[1690891368] unbound[46649:0] debug: module config: "python iterator"
[1690891368] unbound[46649:0] notice: init module 0: python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
PYTHONHOME = (not set)
PYTHONPATH = (not set)
program name = 'unbound'
isolated = 0
environment = 1
user site = 1
import site = 0
sys._base_executable = ''
sys.base_prefix = '/usr/local'
sys.base_exec_prefix = '/usr/local'
sys.platlibdir = 'lib'
sys.executable = ''
sys.prefix = '/usr/local'
sys.exec_prefix = '/usr/local'
sys.path = [
'/usr/local/lib/python39.zip',
'/usr/local/lib/python3.9',
'/usr/local/lib/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'
Current thread 0x0000000829022000 (most recent call first):
<no Python frame>
I can see about pulling a backup of the configuration from prior to the upgrade to see if there is anything odd in how it generates the domainoverrides.conf file. I also have quite a few other units to upgrade so I can monitor those and post additional details if I run into it again.
-
I'm in much the same boat, trying to set up access control views, but running into the same inability to start unbound, along with the same output from `unbound -ddvv -c /var/unbound/unbound.conf`...
-
Can you guys run this and see what it reports?
# /usr/local/opnsense/mvc/script/run_migrations.php
Cheers,
Franco
-
I have the same problem.
I get this message. Where can I find the log?
*** OPNsense\Unbound\Unbound Migration failed, check log for details
-
I found these lines in System/general log:
2023-07-31T21:23:50 Notice kernel <118>You may need to manually remove /usr/local/etc/unbound/unbound.conf if it is no longer needed.
2023-07-31T21:23:42 Notice kernel <118>*** OPNsense\Unbound\Unbound Migration failed, check log for details
2023-07-31T21:23:42 Error config Model OPNsense\Unbound\Unbound can't be saved, skip ( OPNsense\Phalcon\Filter\Validation\Exception: [OPNsense\Unbound\Unbound:general.active_interface] option not in list{}
2023-07-31T21:23:42 Error config [OPNsense\Unbound\Unbound:general.active_interface] option not in list{}
2023-07-31T21:22:50 Notice kernel <118>[87/214] Extracting unbound-1.17.1_3: .......... done
2023-07-31T21:22:50 Notice kernel <118>Using existing user 'unbound'.
2023-07-31T21:22:50 Notice kernel <118>Using existing group 'unbound'.
2023-07-31T21:22:50 Notice kernel <118>[87/214] Upgrading unbound from 1.17.1_2 to 1.17.1_3...
2023-07-31T21:22:50 Notice kernel <118> unbound: 1.17.1_2 -> 1.17.1_3
2023-07-31T21:22:50 Notice kernel <118>unbound-1.17.1_2: already unlocked
-
Ok what does this return?
# pluginctl -g unbound.active_interface
Cheers,
Franco
-
Nothing - a blank line
-
Ok let's try differently:
# grep active_interface /conf/config.xml
Cheers,
Franco
-
<active_interface/>
<active_interface/>
-
Same commands on a router that is still running 23.1.11:
root@husabyvagen:~ # grep active_interface /conf/config.xml
<active_interface/>
root@husabyvagen:~ # pluginctl -g unbound.active_interface
root@husabyvagen:~ #
-
I have a specific interface selected in Unbound, maybe that's why I didn't encounter the issue?
# pluginctl -g unbound.active_interface
# grep active_interface /conf/config.xml
<active_interface>opt9</active_interface>
-
No, this is going to be pretty silly...
Can you remove that line "<active_interface/>" from /conf/config.xml and run the migration again?
# /usr/local/opnsense/mvc/script/run_migrations.php
Cheers,
Franco
-
For me, running the migration showed no output, and stopping and starting unbound with my custom conf file in place failed, with only this in the general log:
2023-08-01T11:32:08-04:00 Error opnsense /usr/local/sbin/pluginctl: The command '/bin/kill -'TERM' '65031'' returned exit code '1', the output was 'kill: 65031: No such process'
2023-08-01T11:32:08-04:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure unbound_start (execute task : unbound_configure_do(1))
2023-08-01T11:32:08-04:00 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure unbound_start (1)
EDIT: note that I removed the <active_interface/> line as well before running the migration.
-
Here's something else that I didn't notice before in the error reporter:
[01-Aug-2023 11:28:57 America/New_York] PHP Fatal error: Uncaught TypeError: flock(): Argument #1 ($stream) must be of type resource, bool given in /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/JsonKeyValueStoreField.php:132
Stack trace:
#0 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/JsonKeyValueStoreField.php(132): flock(false, 2)
#1 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/BaseField.php(193): OPNsense\Base\FieldTypes\JsonKeyValueStoreField->actionPostLoadingEvent()
#2 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/BaseField.php(191): OPNsense\Base\FieldTypes\BaseField->eventPostLoading()
#3 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/BaseField.php(191): OPNsense\Base\FieldTypes\BaseField->eventPostLoading()
#4 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/BaseField.php(191): OPNsense\Base\FieldTypes\BaseField->eventPostLoading()
#5 /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/BaseField.php(191): OPNsense\Base\FieldTypes\BaseField->eventPostLoading()
#6 /usr/local/opnsense/mvc/app/models/OPNsense/Base/BaseModel.php(367): OPNsense\Base\FieldTypes\BaseField->eventPostLoading()
#7 [internal function]: OPNsense\Base\BaseModel->__construct()
#8 /usr/local/opnsense/mvc/script/run_migrations.php(52): ReflectionClass->newInstance()
#9 {main}
thrown in /usr/local/opnsense/mvc/app/models/OPNsense/Base/FieldTypes/JsonKeyValueStoreField.php on line 132
-
I have noted that the outgoing_interface setting disappears during migration. This causes domain override /query forwarding to stop working in my setup.
-
outgoing_interface has the same problem. I think the fix is this but only if the old data is still in place.
https://github.com/opnsense/core/commit/f5efffcf94a
# opnsense-patch f5efffcf94a
Cheers,
Franco
-
I can upgrade one more of my routers from 23.1.11 to 23.7 to verify the patch. But what sequence of actions/commands do you recommend?
-
Upgrade, apply patch, run migration script. Reboot if migration went ok.
Cheers,
Franco
-
Worked like a charm!
Thank you
root@reserv2:~ # /usr/local/opnsense/mvc/script/run_migrations.php
Migrated OPNsense\Unbound\Unbound from 1.0.4 to 1.0.6
-
Ok great, we will have to update the upgrade path to 23.7.1 once it is out but that's not a big deal.
Thanks for the details and testing.
Cheers,
Franco
-
I had a similar issue where Unbound failed to start after upgrade to 23.7.
I logged in using the IP address and found that Enable Unbound was unchecked. I checked that but it still failed to start. I found this thread and checked the active_interface in /conf/config.xml and noticed that there was an entry for ovpns1 but I had already deleted that interface as I switched from OpenVPN to Wireguard.
I was still logged into the UI via the IP address so I went to Unbound --> General and manually selected all the interfaces except Loopback under Network Interfaces. I no longer see an "All (recommended) option in that list for some reason. Saved the settings and then restarted Unbound and now I have DNS resolution again.
However, when I run_migrations via SSH, I still get the failure for Unbound Migration*** OPNsense\Unbound\Unbound Migration failed, check log for detailsNot sure how I can upgrade again, because running upgrade indicates that there are no updates available. I also ran an update from console which also indicated that all packages are up to date.
I will reboot once more and see if Unbound restarts automatically on boot without any issues.
EDIT : Subsequent reboots have kept everything working, however the run_migrations.php script still indicates that Unbound Migration failed, but updates don't bring in any new packages.
-
@Inxsible
https://github.com/opnsense/core/commit/f5efffcf94
# opnsense-patch f5efffcf94
# /usr/local/opnsense/mvc/script/run_migrations.php
-
Just posting in this thread because I had an issue when I upgraded from 22.11 to 23.7
Unbound would start and be running, but it wasn't able to talk to my upstream dns server and get/poll results. Not until:
-> Services: Unbound DNS: General
Network Interface: LAN <-- removed
-> Network Interface: All (Recommended)
Prior to this, the service would run and had no issues until I upgraded to 23.7