OPNsense Forum
Archive => 20.7 Legacy Series => Topic started by: deejacker on December 18, 2020, 09:22:56 am
-
Pretty much as the subject states. All working fine before update, updated fine, but then noticed that Unbound service had just stopped. Managed to log into the GUI to restart, but this is now happening routinely. Anybody else experiencing this?
-
Anything in the logs? Do you use DNSBL?
-
No don’t use DNSBLs, just a Pi-hole.
Looking at the Unbound logs, I can’t see anything obvious which would suggest a service failure, but that may be my limited knowledge.
2020-12-18T08:09:47 unbound[36701] [36701:0] info: start of service (unbound 1.13.0).
2020-12-18T08:09:47 unbound[9063] daemonize unbound dhcpd watcher.
2020-12-18T08:09:46 unbound[36701] [36701:0] notice: init module 0: iterator
2020-12-18T07:53:33 unbound[81533] [81533:2] notice: sendto failed: Permission denied
2020-12-18T07:29:56 unbound[81533] [81533:0] info: start of service (unbound 1.13.0).
2020-12-18T07:29:55 unbound[48254] daemonize unbound dhcpd watcher.
2020-12-18T07:29:55 unbound[81533] [81533:0] notice: init module 0: iterator
2020-12-18T01:04:07 unbound[9402] [9402:3] notice: sendto failed: Permission denied
-
Please check the interfaces it is listening to. Maybe there is something wrong. Change this setting and hit save. Then change it back and save again.
-
Same/comparable issue on my end. Although my configuration is using DNSBL.
2020-12-18T09:20:51 kernel -> pid: 63934 ppid: 1 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-12-18T09:20:51 kernel [HBSD SEGVGUARD] [unbound (63934)] Suspension expired.
2020-12-18T09:20:51 kernel pid 63934 (unbound), jid 0, uid 59: exited on signal 11
What additional info can I/we append to investigate this issue further? Should we revert to the previous version?
-
Same here, I've upgraded to 20.7.7 to get IPv6 prefix delegation working again, however, unbound crashed twice since I've upgraded:
root@fw:~ # dmesg | grep unbound
pid 85049 (unbound), jid 0, uid 59: exited on signal 11
Here is my unbound config:
<unbound>
<enable>1</enable>
<custom_options/>
<regdhcp>1</regdhcp>
<cache_max_ttl/>
<cache_min_ttl/>
<incoming_num_tcp>10</incoming_num_tcp>
<infra_cache_numhosts>10000</infra_cache_numhosts>
<infra_host_ttl>900</infra_host_ttl>
<jostle_timeout>200</jostle_timeout>
<log_verbosity>1</log_verbosity>
<msgcachesize>4</msgcachesize>
<num_queries_per_thread>4096</num_queries_per_thread>
<outgoing_num_tcp>10</outgoing_num_tcp>
<unwanted_reply_threshold/>
<hosts>
<host>******</host>
<domain>*********</domain>
<rr>A</rr>
<ip>***********</ip>
<mxprio/>
<mx/>
<descr/>
<aliases>
<item/>
</aliases>
</hosts>
<hosts>
<host>******</host>
<domain>*********</domain>
<rr>A</rr>
<ip>**********</ip>
<mxprio/>
<mx/>
<descr/>
<aliases>
<item/>
</aliases>
</hosts>
<hosts>
<host>******</host>
<domain>**********</domain>
<rr>A</rr>
<ip>**********</ip>
<mxprio/>
<mx/>
<descr/>
<aliases>
<item/>
</aliases>
</hosts>
<hosts>
<host>******</host>
<domain>*********</domain>
<rr>A</rr>
<ip>**********</ip>
<mxprio/>
<mx/>
<descr/>
<aliases>
<item/>
</aliases>
</hosts>
<regdhcpstatic>1</regdhcpstatic>
</unbound>
-
Interfaces setting was set to 'All' but that setting does not seem to be available anymore and I've enabled all interfaces manually.
-
Stability is now better?
-
I followed your suggestion of changing the interface and back again. Will monitor to see if this makes any difference.
-
I just had another crash. I'm considering downgrading at the end of our workday. As a workaround, I've disabled unbound and enabled Dnsmasq to do the DNS resolving.
-
can you try to set Log level verbosity to 5, disable DHCP registration (just in case), restart unbound and wait for crash?
share fresh logs please
looks like a unbound bug
-
related?
https://github.com/NLnetLabs/unbound/issues/376
-
can test with Log level 0?
-
Setting the interfaces manually appears to have stabilized the issue. Will report back if the situation changes.
-
Easy workaround for the affected:
# opnsense-revert -r 20.7.6 unbound
Looks like Unbound 1.13.0 has a number of issues but was necessary to fix CVE....
Cheers,
Franco
-
I also noticed that unbound wasn't restarted after the upgrade 20.7.6 -> 20.7.7, I had to do that manually. Is this intentional?
-
Setting the interfaces manually appears to have stabilized the issue. Will report back if the situation changes.
Spoke too soon :( It just crashed again.
Reverted to unbound 20.7.6 as suggested by Franco
-
I can confirm I have the same behaviour. This however only seems to happen when I reboot the firewall, not throughout the day.
Starting Unbound manually solves it for now.
Hope that a fix is available soon.
-
Changing and then reverting the interfaces didn’t resolve anything for me, experienced another Unbound service stop shortly afterwards.
-
It would be better to pull this update as it completely breaks the system and makes it unusable since DNS crashes over and over.
-
It would be better to pull this update as it completely breaks the system and makes it unusable since DNS crashes over and over.
opnsense-revert -r 20.7.6 unbound
-
I think @Animosity022 meant stop offering the broken unbound 1.13.0 package in System > Firmware > Updates, or releasing 20.7.7_2 with unbound pinned at 1.12.0, or withdrawing 20.7.7_1 altogether (I get there are security issues resolved in this release so that might not be the best option).
Is it possible to at least add a note in the release notes warning people about this issue if they are using unbound in their configuration? So they can then make a more informed decision about whether to upgrade or not.
My backup firewall hasn't been upgraded yet, and 20.7.7_1 with unbound 1.13.0 is still being offered when I check for upgrades. So even if I read the release notes carefully, if I hit upgrade on that firewall, it would be guaranteed to break until unbound is manually reverted (after figuring out unbound had crashed and then finding this thread).
-
related?
https://github.com/NLnetLabs/unbound/issues/376
Meanwhile, I reverted to Unbound 1.12.0 using the command that was posted here.
However, it looks like a patch is now included in FreeBSD-ports, based on the latest reply on GitHub.
@franco any chance to include this in a hotfix release, before you guys start to enjoy your much deserved Christmas break? :-) (otherwise indeed a good idea to update the release notes post. Could save some frustration).
-
It would be better to pull this update as it completely breaks the system and makes it unusable since DNS crashes over and over.
opnsense-revert -r 20.7.6 unbound
I was saying to stop pushing a patch that bricks your router.
-
Same here.
I updated my opnsense instance to v 20.7.7_1 yesterday.
Today my unbound 1.13.0 crashed, and I can't start it back.
Rollback to unbound 1.12.0 with opnsense-revert -r 20.7.6 unbound command didn't help.
I had no choice but completely disable unbound.
ְAny suggestions for alternative stable local DNS?
-
I was saying to stop pushing a patch that bricks your router.
That's not a reasonable way to describe a choice to update or revert. If Unbound wants 1.13.0 out being the only way to deal with a CVE it should make sure it works. Users need to accept the possibility that this is mostly the case, but not always.
Cheers,
Franco
-
I was saying to stop pushing a patch that bricks your router.
That's not a reasonable way to describe a choice to update or revert. If Unbound wants 1.13.0 out being the only way to deal with a CVE it should make sure it works. Users need to accept the possibility that this is mostly the case, but not always.
Cheers,
Franco
If an update breaks a device and makes it not functional and the vendor understands the problem being caused (3rd party or not), they usually remove the broken update so it stops creating more havoc for people since it was not intended to break the device.
-
If an update breaks a device and makes it not functional and the vendor understands the problem being caused (3rd party or not), they usually remove the broken update so it stops creating more havoc for people since it was not intended to break the device.
There is a distinction I think is being missed here: "affects all people" vs. "affects some people".
Also, "break a device" is used opportunistically here. The device isn't bricked. The admin can still do something (if actually necessary, see first point).
Also maybe this is a bit unexpected: there is no clean rollback of published repositories with FreeBSD package manager. It can break your dependency chain worst case, deinstalling the core package leaving the device really really dead in the water. It's not a risk to take vs. first point.
There are more points, but I fear they are not relevant to the desires of the perfect consumer of the perfect project.
Cheers,
Franco
-
Affected our firewalls too, from 12 of 10. So its pretty much affect almost everyone, not just a few people.
Disabled unbound and using dnsmasq solve the issue.
-
ְAny suggestions for alternative stable local DNS?
I would say DNSCrypt-proxy. That is what I use since I have (other) problems with Unbound (mainly DNSBL and network port going up/down with complete restart of Unbound and as a result DNS outage). However it is not as integrated with OPNsense...
-
[/quote]
Also, "break a device" is used opportunistically here. The device isn't bricked. The admin can still do something (if actually necessary, see first point).
Cheers,
Franco
[/quote]
My device was "not functional". I had to reinstall as it was not possible to login nor do anything as no screen would paint not anything else. I reinstalled and restored from a previous backup, choose not to use Unbound and things are back to normal.
I used my word choices on purpose as I want to make sure the feelings are conveyed as the update will no doubt cause problems for many other people until it is removed or a new update is provided. It's just bad form.
-
You should be able to ssh into the router still
you can then select the shell option
and issue opnsense-revert -r 20.7.6 unbound
then reboot the router manually or with sudo reboot
-
So its pretty much affect almost everyone, not just a few people.
I respectfully disagree with generalisation due to the aforementioned points.
Disabled unbound and using dnsmasq solve the issue.
Yes, that actually works, too.
My device was "not functional". I had to reinstall as it was not possible to login nor do anything as no screen would paint not anything else.
I thought we were talking about Unbound here. If this was Unbound it would to have caused a kernel panic and disintegrated the root file system due to a forced reboot. I have no reports that suggest that this is the case. That also would be scenario where a hotfix would be necessary if one existed for either Unbound or the kernel to stop crashing the OS itself.
Cheers,
Franco
-
My device was "not functional". I had to reinstall as it was not possible to login nor do anything as no screen would paint not anything else.
I thought we were talking about Unbound here. If this was Unbound it would to have caused a kernel panic and disintegrated the root file system due to a forced reboot. I have no reports that suggest that this is the case. That also would be scenario where a hotfix would be necessary if one existed for either Unbound or the kernel to stop crashing the OS itself.
Cheers,
Franco
[/quote]
No reports? So I'm making up my situation for what point? Others are making it up posting as well?
It didn't cause a kernel panic. It caused my system to be not functional as I said. Login screen didn't work. Bandwidth went to almost nothing as pages wouldn't display. I'd assume it was due to the repeated crashes and backlog on the system.
All in all, good luck as you tend to be combative with users that are reporting bugs and I'm moving on to a different, more stable solution when many users as seen in just this thread report something, you belittle our situation and refuse to listen to feedback.
-
Guys/Gals , please be courteous when reporting and be specific with the problem and make sure it's related to the open thread/topic.
Without giving more information how it crash/logs/etc , i dont think one can help much, except from guessing or by experience guessing.
For myself I dont have any issue at all with unbound since upgraded from 20.7 -> 20.7.6 -> 20.7.7_1. I only run max 23 hours per day and the system will be power-off. Since I saw the report of unbound terminate abnormally, i keep tab on the unbound processes & logs, it's been fine for me - no issue starting from cold boot, no issue of process terminating abnormally, no unbound SIGSEGV/segfault .
I do have custom setting for unbound; e.g.
Bind to certain interface only
DNS over TLS <few IPaddress@853>
DNSSEC enabled
Message Cache size 10MB
Access List <few internal IPaddress>
Of course I do want similar patched applied similar in freebsd ports (https://github.com/NLnetLabs/unbound/issues/376 OR https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251821) if this fixes for certain scenario. If there is a need of example on how people log an issue , please refer to the links. They provide at least some log info, description, and possible with scenario, etc.
For the login screen that didnt work, I'm guessing seperate issue that was posted in this forum and/or, something related to HTTP redirect that is fixed on 20.7.7_1 OR related to https://forum.opnsense.org/index.php?topic=20514.0
In all best cases all update/upgrade works, some times we have to work with temporary solution and ofcourse have a permanent solution at a later time. OPNSense provide us an option to roll back , so i dont see any issue with it. I am in no position to say in what scenario warrant a pull of package / release, I'm confident OPNSense team will be able to make the right decision.
If there's something critical to your production and not able to single handed and deal with the situation , i guess it's best to subscribe to the business support.
p/s: my first post , yes i register just for purpose of posting in this specific thread. I dont usually want to post and i just wanted to read topic that interest me. what makes me post this , i guess i cannot escape from my conscious of xxxxxx - does not matters :)
-
I also have perfectly working systems upgraded from 20.7.6 to 20.7.7_1. Unbound working without issues. So, I can confirm that this issue is not on "every system".
I have learned this the hard way: always have an 1:1 test system if you are running a critical production system. And 1:1 includes: identical hardware, identical software and identical config. Otherwise, be prepared for issues whatever OS you are running.
Systems have so many applications interacting with each other that it is difficult to see in advance the possible problems. It might be a small config difference which causes the systems to behave differently after upgrades.
First I upgrade the test system and leave it running for a while. If it is running ok I continue to the production systems, gradually. This makes it possible to identify the issues and figure the workarounds if the software patches are not yet ready.
I know that in many systems upgrades are done on fly to production systems (hurry, money, whatever the reason is). To be honest, I would not like to use such systems, even for free. There are issues with every OS and software. But the worst problem is that the admin is not sure what he or she is doing.
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
No reports? So I'm making up my situation for what point? Others are making it up posting as well?
I think you are overreacting. If you reply to a thread about Unbound that your upgrade was bad that has nothing to do with Unbound. In fact, any update can be bad however small if the file system or disk disintegrates or file system full. Since I haven't seen a health audit I can't possibly say how bad it was.
you belittle our situation and refuse to listen to feedback
Maybe that is true. But maybe listening goes both ways?
From day to day experience I just want to say that I have broken my production systems a number of times with preproduction testing. It's just the way it is and I am grateful for every bug that doesn't happen in production releases as some of them have forced a full reinstall these systems.
Cheers,
Franco
-
I updated 3 systems to 20.7.7_1 about 24 h ago, no problems with unbound here...
-
Came here with this problem... unbound had been crashing roughly 6 times a day since the update.
I tried the unbound regression, but it may have not solved the problem as unbound fell over again about an hour later.
Will keep monitoring this thread and I'll be watching to see if the same problem keeps repeating.
-
I too had Unbound stop after running for a period of time, quickly after a reboot, then after restarting the service it ran for longer before stopping. All other functions seem to work normally, but anything that relied on DNS failed (obviously). For me reverting to the previous version has stopped the service from frequently stopping.
Is this related to:
https://github.com/NLnetLabs/unbound/issues/376
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=251821
(Found these threads before finding this thread here.)
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
Cheers,
Franco
Updated unbound on my system to the provided revision. Will report back with feedback.
-
I only came here to say that I also have not experienced any issues with unbound crashing since upgrading to 20.7.7 on release day. I'm sorry to those that have had issues and based on the flurry of activity surrounding this, there clearly is an issue that is affecting *some* users. But not all users are having this problem.
-
I complained earlier about the "6 unbound failures a day".
Just wanted to say that regressing to the previous version of unbound appears to have solved the problem.
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
Cheers,
Franco
Updated unbound on my system to the provided revision. Will report back with feedback.
Unbound has been running stable for 24 hours now on the new revision. Issue appears resolved on my end.
-
Just to confirm I am observing this problem on 20.7.7 as well. I have reverted unbound as per the instructions above.
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
Cheers,
Franco
Updated unbound on my system to the provided revision. Will report back with feedback.
Unbound has been running stable for 24 hours now on the new revision. Issue appears resolved on my end.
I can confirm the same! Any chance this could make it into a hotfix, or will this have to wait until 21.1?
-
Im also having the same issue and cannot easily apply the patch right now :(
What can I do? Is there an eta till the next update? Im having to start the service every few hours :(
-
20.7.8 is needed to bundle the Unbound fix. I am afraid we have to give it more time for multiple reasons. The non-obvious reason is that there is still one bugfix missing and that may be one of the reasons netlabs hasn't released 1.13.1 yet.
Cheers,
Franco
PS: The patch apply is trivial and perfectly safe.
-
Thank you for your reply. I appreciate it. It is kind of you to reply directly to me.
I am not able to Reboot as I already had my reboot window this week which I used to upgrade the firewall. As the issue is not a security problem, but a problem that requires me to login and start the service it is seen as a problem that I have to live with until Saturday when I can reboot again.
Is there any way to apply this fix without rebooting?
Kind regards
Peter
-
I don’t think there is a reboot needed.
-
Ok thank you that is awesome. I will run the patch then :)
Im just going to turn on ssh and do this then:
opnsense-revert -r 20.7.6 unbound
-
Thank you all. You were correct. No reboot. Just typed that command and turned off ssh again. I did restart unbound just in case also after but whole process was only 1 minute long :)
-
Same for me
opnsense-revert -r 20.7.6 unbound
fixed the issue
-
Thank you all - I add the same issue. I applied 1.13.0_1 patch and I will see how it goes.
-
Same here, just FTR. For me it crashes about every second day. I will apply the patch.
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
I tried this and can confirm that I am still getting crashes. Here is the latest General log lines from the last crash, running the above patch.
2020-12-28T23:16:15 kernel -> pid: 15953 ppid: 1 p_pax: 0xa50<SEGVGUARD,ASLR,NOSHLIBRANDOM,NODISALLOWMAP32BIT>
2020-12-28T23:16:15 kernel [HBSD SEGVGUARD] [unbound (15953)] Suspension expired.
2020-12-28T23:16:15 kernel pid 15953 (unbound), jid 0, uid 59: exited on signal 11
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
-
I'm seeing this too! :(
-
Thanks for the advice everyone, the opnsense-revert command worked for me. No reboot needed.
opnsense-revert -r 20.7.6 unbound
I was going crazying trying to trace why my network kept dying till I discovered the DNS service shutting down over and over again. I originally thought it was due to my provider since they had a recent bad outage.
-
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
I installed the patch yesterday morning and since then, no problems anymore. Unbound is working again!
Thanks,
Timo
-
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
Hi,
I'm also newbie, and also having the same issue with the unbound service stopping. Where you able to create a monit service to restart unbound when stop? can you share the configuration?
Thank you very much
-
Just install the patch previously posted here
-
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
Hi,
I'm also newbie, and also having the same issue with the unbound service stopping. Where you able to create a monit service to restart unbound when stop? can you share the configuration?
Thank you very much
I'm newbie myself but found this topic in documentatin: https://docs.opnsense.org/manual/monit.html#example-1
-
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
Hi,
I'm also newbie, and also having the same issue with the unbound service stopping. Where you able to create a monit service to restart unbound when stop? can you share the configuration?
Thank you very much
I'm newbie myself but found this topic in documentatin: https://docs.opnsense.org/manual/monit.html#example-1
As I said before i'm newbie (learning with an opnsense at home), and I don't know which ''condition'' should I put to test that unbound is working and for the ''service settings'' which statements to put on ''PID File'', ''Start'' and ''Stop''.
Regarding the patch, i though it was good idea to learn how to use monit to restat a service that stop
-
thanks ; opnsense-revert -r 20.7.6 unbound did the trick for me as well
-
Please upgrade rather than downgrade :-)
New Unbound version (1.13.0) was released to deal with CVE issues, patch (1.13.0_1) is minor and keeps those improvements:
Here's the latest Unbound revision 1 from FreeBSD ports to try:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
Edits: Include version and package information
-
for the record,
I upgraded to 20.7.7_1 last week and immediately applied the patch:
# pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
No issues identified, everything works.
br
-
Same issue here, unbound keeps crashing. I tyed the opnsense-revert thing, we'll see if it holds
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
Meanwhile I tried monit for the first time as well.
I *think* also I got it, but it took me a lot of tries, so for those who might be even more lost than I am, here is what I did in the monit>settings>service settings : add a new service as you can see in the attached picture.
Attached as well, in the monit>status you'll see it has found the proper process id.
I even stopped unbound and it got restarted within the 120 seconds of polling interval.
If you want to make it check faster, it happens on the first settings page, I think that is what would be the polling interval.
Couldn't find a way to get an email notification when the service gets restarted though... At least it restarts ;)
-
Just to confirm I am observing this problem on 20.7.7 as well ("notice: sendto failed: Permission denied" 4 times in 4 days). I have reverted unbound to 1.12.0, waiting for a OPNsense fix (via 20.7.7_X or 20.7.8 ).
And YES, I am aware of the CVE-2020-28935, but this vulnerability is *only* CVSS-3 scoring 5.5 as this is *only* a local vulnerability that could create a DoS of the system Unbound/NSD is running on. A very limited security risk in my personal situation.
Thank you and stay safe.
-
Unbound crashes for me too. Have not tried downgrading at this point.
One surprising thing was to see it does not restart itself - I would imagine for key system services there would be some auto restart process. Is there not such a thing in opnsense or is it disabled for unbound ?
-
Unbound crashes for me too. Have not tried downgrading at this point.
One surprising thing was to see it does not restart itself - I would imagine for key system services there would be some auto restart process. Is there not such a thing in opnsense or is it disabled for unbound ?
On my system it does restart but after 5 crashes it triggers the HBSD SEGVGUARD which suspends processes for 600s after 5 crashes.
-
Unbound crashes for me too. Have not tried downgrading at this point.
One surprising thing was to see it does not restart itself - I would imagine for key system services there would be some auto restart process. Is there not such a thing in opnsense or is it disabled for unbound ?
No, but you can add this via monit
-
affected by this as well.
15min ago someone release a fixed version of the unbound package: https://github.com/mat813/freebsd-ports/commit/95a05e89eda2ed7629addb4a28117e463b69eeb0
could we just get that upgrade via official update repos for OpnSense as fast as possible?
-
Looks like you missed the post above - see post #37
-
Unbound crashes for me too. Have not tried downgrading at this point.
One surprising thing was to see it does not restart itself - I would imagine for key system services there would be some auto restart process. Is there not such a thing in opnsense or is it disabled for unbound ?
No service can restart itself by itself when it dead. Only an OS or an another service can do it.
-
Same issue here...
-
Hi toxic, thanks for sharing the set-up, but for what I know (really little as I'm learning), I think is missing the test to see if the unbound service is working. How the machine know that unbound has stop? Thats not done by the test?
Regards
Same issue here, unbound keeps crashing. I tyed the opnsense-revert thing, we'll see if it holds
For now, I *think* I've setup a monit test to restart unbound if it crashes. Never used monit before, so we'll see if I did it right...
Meanwhile I tried monit for the first time as well.
I *think* also I got it, but it took me a lot of tries, so for those who might be even more lost than I am, here is what I did in the monit>settings>service settings : add a new service as you can see in the attached picture.
Attached as well, in the monit>status you'll see it has found the proper process id.
I even stopped unbound and it got restarted within the 120 seconds of polling interval.
If you want to make it check faster, it happens on the first settings page, I think that is what would be the polling interval.
Couldn't find a way to get an email notification when the service gets restarted though... At least it restarts ;)
-
Same issue here with 20.7.7_1
Looks like this does not exist anymore:
pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
-
I have Unbound stable (in forwarding mode with DNS-over-TLS), not seeing and problems with latest version. Maybe this is worth a try before downgrading...
-
Looks like this does not exist anymore
looks like 21.1 on the way
-
Looks like this does not exist anymore:
pkg add -f https://pkg.opnsense.org/FreeBSD:12:amd64/20.7/misc/unbound-1.13.0_1.txz
Because it's part of the official 20.7.7 now. ;)
-
The problem is the same for me. it stops at all times.
I assign the interface manually and the problem is still there.
Versions OPNsense 20.7.7_1-amd64
FreeBSD 12.1-RELEASE-p11-HBSD
OpenSSL 1.1.1i 8 Dec 2020
here are the logs:
2021-01-04T12:41:23 unbound[473] [473:0] info: start of service (unrelated 1.13.0).
2021-01-04T12:41:23 unbound[473] [473:0] review: init module 0: iterator
2021-01-04T07:56:12 unbound[8518] [8518:0] review: sendto failed: Permission denied
2021-01-04T03:00:19 unbound[8518] [8518:0] info: start of service (unrelated 1.13.0).
2021-01-04T03:00:19 unbound[8518] [8518:0] review: init module 0: iterator
2021-01-04T03:00:19 unbound[8518] [8518:0] review: Reboot of 1.13.0 unrelated.
2021-01-04T03:00:19 unbound[8518] [8518:0] info: server statistics for thread 1: requestlist max 0 avg 0 exceeded 0 hustled 0
2021-01-04T03:00:19 unbound[8518] [8518:0] info: server statistics for thread 1: 0 queries, 0 cache responses, 0 recurrences, 0 preference, 0 rejected by ip ratelimiting
2021-01-04T03:00:19 unbound[8518] [8518:0] info: server statistics for thread 0: requestlist max 0 avg 0 exceeded 0 hustled 0
2021-01-04T03:00:19 unbound[8518] [8518:0] info: server statistics for thread 0: 0 queries, 0 cache responses, 0 recurrences, 0 prefeasing, 0 rejected by ip ratelimiting
2021-01-04T03:00:19 unbound[8518] [8518:0] info: service stopped (unrelated 1.13.0).
2021-01-04T03:00:18 unbound[8518] [8518:0] info: start of service (unrelated 1.13.0).
2021-01-04T03:00:18 unbound[8518] [8518:0] review: init module 0: iterator
2021-01-04T03:00:18 unbound[8518] [8518:0] review: Reboot of 1.13.0 unrelated.
2021-01-04T03:00:18 unbound[8518] [8518:0] info: server statistics for thread 1: requestlist max 0 avg 0 exceeded 0 hustled 0
2021-01-04T03:00:18 unbound[8518] [8518:0] info: server statistics for thread 1: 0 queries, 0 cache responses, 0 recurrences, 0 prefeasing, 0 rejected by ip ratelimiting
2021-01-04T03:00:18 unbound[8518] [8518:0] info: server statistics for thread 0: requestlist max 0 avg 0 exceeded 0 hustled 0
2021-01-04T03:00:18 unbound[8518] [8518:0] info: server statistics for thread 0: 0 queries, 0 cache responses, 0 recurrences, 0 prefeasing, 0 rejected by ip ratelimiting
2021-01-04T03:00:18 unbound[8518] [8518:0] info: service stopped (unrelated 1.13.0).
2021-01-04T03:00:18 unbound[8518] [8518:0] info: start of service (unrelated 1.13.0).
-
Again: fetch latest updates, restart Unbound from GUI.
Cheers,
Franco
-
Because it's part of the official 20.7.7 now. ;)
Nice!
-
Because it's part of the official 20.7.7 now. ;)
Fetch latest updates, restart Unbound from GUI.
unbound 1.13.0_1 deployed for 1 day without any trouble. Seems OK.
Thanks franco !
-
This was driving me nuts with all kinds of weird errors and slow responding servers until I noticed that unbound was stopped in the Dashboard. Applied patch and seems ok now. Thanks again @franco
-
Hi toxic, thanks for sharing the set-up, but for what I know (really little as I'm learning), I think is missing the test to see if the unbound service is working. How the machine know that unbound has stop? Thats not done by the test?
Regards
I was baffeled by this as well but my current understanding is that it's meant to keep a process alive anyway, and you give it the pid file if you look at my config. So it's able to read the file and check that there is indeed a process with this pid, if not, then it uses start command and probably stop before...
It did detect me killing the service and did restart it, so pidfile is enought from what I understand, but I agree that's either "too intuitive" or maybe not explicit enough in the config gui...
Nevertheless, I got the _1 update as well, dunno if the monit service or the update did the trick but I'm fine now ;)
Not sure why there is not a monit entry by default for all core services, even disabled, so we just have to enable it when needed...
-
Would be nice if someone could post a Monit How-To for core services like unbound. and restart the service.
-
This just happened to me again today. I am up with 20.7.7_1. This has happened quite a few times with me. I am a newbie with OPNsense. Please let me know what I can do to supply more info.
-
Check for updates as you may not have the patch - 20.7.7_1 was released before the patch, and then the patch later added
-
unbound 1.13.0_1 installed yesterday. Still crashed today.
unbound 1.13.0_1 deployed for 1 day without any trouble. Seems OK.
Thanks franco !
-
Would be nice if someone could post a Monit How-To for core services like unbound. and restart the service.
(See attachment) Here you go.
-
i am using 21.1 with unbound 1.13.0_1
still crash. always crash. sometime several times a day, sometime once every several days.
there is nothing in the log whatsoever that showing something that i can understand why it keep crashes and crashes.
until the unbound developer (or opnsense decide to change the dns resolver to other) fix this, i disable the service and manually assign the dns to each client (or assign public dns such as opendns, google etc etc in the dhcp settings).
-
@cybersans
dnsbl records sanitizing added by @AdSchellevis
https://github.com/opnsense/core/issues/4898
https://github.com/opnsense/core/commit/31a0c40e3f503528f3adb05fb1bdd8b139495c38
unbound should no longer crash due to invalid entries imho
-
nice work guys. after applying 31a0c40 patch, unbound works flawlessly!
bravo! ;D
-
@cybersans
it seems to me better to get 21.1.6 right away (it already contains all the changes)
or at least add 565688c and f6c0fa8 ;)
https://github.com/opnsense/core/commits/master/src/opnsense/scripts/unbound/download_blacklists.py
-
I have been experiencing Unbound freezing every 3-7 days roughly since I applied the 20.7.7 update earlier this year. I am currently on 21.7.1. I have not done any pkg revert or additions for unbound.
My unbound config includes several domain overrides and host overrides, but nothing else really.
Today it died again and looking at my system log, I see several hundred lines of getswapspace(\d+): failed like so:
2021-08-30T18:34:21 kernel swap_pager_getswapspace(31): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(9): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(16): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:20 kernel pid 35220 (php-cgi), jid 0, uid 0, was killed: out of swap space
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(22): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(25): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed
I'm running OpnSense in a HyperV vm with dynamic ram set, though, I never see it changing from the 1024 that is set on initial boot in VM manager:
(https://i.imgur.com/KuFgpxM.png)
I ended up doing the monit restart solution rather than revert pkg so at least my internet will get back online quick after DNS dies. Hope this info helps someone else in the future.
-
Memory ballooning (?) probably won't work in FreeBSD 12.1 yet. As far as crashes go these are almost always related to DoH and probably a particular DoH provider? Make sure to post your setup in your "me too's" as this gives hints as to what to do: maybe don't use that provider or DoH in general or find a DoH alternative like dnscrypt-proxy.
Cheers,
Franco
-
Oh interesting. I didnt think that would be relevant since i dont have that plugin installed even. Just dnssec support enabled and only dns server is 1.1.1.3 (cloudflare family node).
Here are my plugins:
(https://i.imgur.com/gDOuZUB.png)
(Note: several of these installed plugins say misconfigured because they were installed before I did the Configuration import from my previously OpnSense router. They all seem to work fine though)
It would make sense if FreeBSD doesn't know how to handle dynamic ram for it to use up all space instead of accepting the host requests to increase its RAM dynamically.
Seems like these started happening after I installed sensei when trying to see if that could help me figure out why Unbound kept crashing. Maybe I'll uninstall it and see if those out of swap errors go away.
-
Would be nice if someone could post a Monit How-To for core services like unbound. and restart the service.
(See attachment) Here you go.
Thanks, but what is the correct "Service Test Setting" to apply?
br
-
Oh in that case you are right and Unbound may just go out of memory during normal operation... no crashes or problems, just too few resources globally.
I would set it to 2 GB without Sensei and 4 GB with Sensei at least.
Cheers,
Franco
-
Today I have updated to the latest update OPNsense 21.7.6-amd64
IDS / Outboound DNS keeps crashing. Nothing in the log.
Any suggestions why is happening ?
-
Check dmesg output...
-
Check dmesg output...
Hi Marco,
i noticed when IDS is enable both services crashes, when i disable IDS i notice the DNS keeps working.
any reason why?
-
Sure, possible guess without any further info: too little RAM -> configuration error trying to use Hyperscan and other RAM eaters like full-blown rulesets.
It's sort of why I asked for dmesg specifically because if Unbound is killed for out of memory that's that.
Cheers,
Franco