Unbound service routinely stopping/crashing following 20.7.7 update

[LouieLouie]

Would be nice if someone could post a Monit How-To for core services like unbound. and restart the service.

(See attachment)  Here you go. 

[cybersans]

i am using 21.1 with unbound 1.13.0_1

still crash. always crash. sometime several times a day, sometime once every several days.
there is nothing in the log whatsoever that showing something that i can understand why it keep crashes and crashes.

until the unbound developer (or opnsense decide to change the dns resolver to other) fix this, i disable the service and manually assign the dns to each client (or assign public dns such as opendns, google etc etc in the dhcp settings).

[Fright]

@cybersans
dnsbl records sanitizing added by @AdSchellevis
https://github.com/opnsense/core/issues/4898
https://github.com/opnsense/core/commit/31a0c40e3f503528f3adb05fb1bdd8b139495c38

unbound should no longer crash due to invalid entries imho

[cybersans]

nice work guys. after applying 31a0c40 patch, unbound works flawlessly!

bravo!  ;D

[Fright]

@cybersans
it seems to me better to get 21.1.6 right away (it already contains all the changes)
or at least add 565688c and f6c0fa8  ;)
https://github.com/opnsense/core/commits/master/src/opnsense/scripts/unbound/download_blacklists.py

[Szeraax]

I have been experiencing Unbound freezing every 3-7 days roughly since I applied the 20.7.7 update earlier this year. I am currently on 21.7.1. I have not done any pkg revert or additions for unbound.

My unbound config includes several domain overrides and host overrides, but nothing else really.

Today it died again and looking at my system log, I see several hundred lines of getswapspace(\d+): failed like so:

Code Select Expand

2021-08-30T18:34:21 kernel swap_pager_getswapspace(31): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(9): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(24): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(16): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:21 kernel swap_pager_getswapspace(32): failed
2021-08-30T18:34:20 kernel pid 35220 (php-cgi), jid 0, uid 0, was killed: out of swap space
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(18): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(20): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(22): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(25): failed
2021-08-30T18:34:15 kernel swap_pager_getswapspace(4): failed

I'm running OpnSense in a HyperV vm with dynamic ram set, though, I never see it changing from the 1024 that is set on initial boot in VM manager:


I ended up doing the monit restart solution rather than revert pkg so at least my internet will get back online quick after DNS dies. Hope this info helps someone else in the future.

[franco]

Memory ballooning (?) probably won't work in FreeBSD 12.1 yet. As far as crashes go these are almost always related to DoH and probably a particular DoH provider? Make sure to post your setup in your "me too's" as this gives hints as to what to do: maybe don't use that provider or DoH in general or find a DoH alternative like dnscrypt-proxy.


Cheers,
Franco

[Szeraax]

Oh interesting. I didnt think that would be relevant since i dont have that plugin installed even. Just dnssec support enabled and only dns server is 1.1.1.3 (cloudflare family node).

Here are my plugins:


(Note: several of these installed plugins say misconfigured because they were installed before I did the Configuration import from my previously OpnSense router. They all seem to work fine though)

It would make sense if FreeBSD doesn't know how to handle dynamic ram for it to use up all space instead of accepting the host requests to increase its RAM dynamically.

Seems like these started happening after I installed sensei when trying to see if that could help me figure out why Unbound kept crashing. Maybe I'll uninstall it and see if those out of swap errors go away.

[Mks]

Would be nice if someone could post a Monit How-To for core services like unbound. and restart the service.

(See attachment)  Here you go.

Thanks, but what is the correct "Service Test Setting" to apply?

br

[franco]

Oh in that case you are right and Unbound may just go out of memory during normal operation... no crashes or problems, just too few resources globally.

I would set it to 2 GB without Sensei and 4 GB with Sensei at least.


Cheers,
Franco

[Julien]

Today I have updated to the latest update OPNsense 21.7.6-amd64
IDS / Outboound DNS keeps crashing. Nothing in the log.
Any suggestions why is happening ?

[franco]

Check dmesg output...

[Julien]

Check dmesg output...

Hi Marco,

i noticed when IDS is enable both services crashes, when i disable IDS i notice the DNS keeps working.
any reason why?

[franco]

Sure, possible guess without any further info: too little RAM -> configuration error trying to use Hyperscan and other RAM eaters like full-blown rulesets.

It's sort of why I asked for dmesg specifically because if Unbound is killed for out of memory that's that.


Cheers,
Franco