Messages

If you think about it, what you want is some kind of set subtraction, which is not an easy task if the first set can itself be constructed by a list of expressions. Imagine IPv6 ranges to get a clearer picture.

There are two ways of doing this:

a. If you can control the list yourself, exclude the ranges you do not want in it yourself (Maxminds GeoIP list with selectable country ranges is a good example). This is not set subtraction, but set addition, which is much easier.

b. What you probably really want to do is something to the extent of "whitelisting" something before you add generalized block rules. In this case, you can use a separate rule with the whitelisted set which triggers before the block list.

Yes of course - Large thanks, why did I not add a rule before to take care of this? stubbern I guess by reading the documentation and missing such an easy solution! And yes I understand how you mean with resolving part of range and that - it is nearly impossible to get that right. I should have used my mind a bit more.....

Sorry my mistake, and the confusion that follows...

Yes I do use the built in GeoBlock for Country Blocking. However I thought I do a test for exclusion, and since the documentation, under nesting gives an example on ip range exclusion:
https://docs.opnsense.org/manual/aliases.html#nesting

Here one example says that one can exclude IP range or just IP address (host) by using !.

So there is about where I started, and then I created rather private test file on github to filter a bunch of ranges I don't need (I thought) so here is that one:

Code Select Expand

https://raw.githubusercontent.com/larhedse/hostnamelistan/master/Blocklistan.txt

Now row 7 in the above file includes the "81.224.0.0/12" range - it is a pretty large one right...

So that breaks one thing for me, I need to access IP address "81.228.3.233"

So I thought in this scenario that I just exclude that according to the nested (yes I know, it shows nested in there - but do I really need to nest Alias to achieve this?), so the result was:

Code Select Expand

https://raw.githubusercontent.com/larhedse/hostnamelistan/master/Blocklistan.txt
!81.228.3.233
This does not work.

So I changed to IP range:

Code Select Expand

https://raw.githubusercontent.com/larhedse/hostnamelistan/master/Blocklistan.txt
!81.228.3.0/24

Still does not work. Oh and by the way, the only way for me to test this is to duplicate the Alias - just changing it will NOT work, that updates the definition in the content field, but the Alias is not reloaded so to speak. So the last one above results in zero rows.

So I guess what I am trying to figure out now, is if the documentation that mentions exclude of IP(/-address) - how does that work, since the documentation under "Nested" suggests it should work? somehow? or what is it that I am missing?

Howdy,

So I am using an Alias for "Country Block" (GeoIP). Now for some reason I seem to open up an IP range that I seem to now days need. Now anyone using country blocking IP ranges knows it might fail at some time. In this cas this country is blocked since ages in my firewall, I never had to open up things like this. So I am a newbie on this, and well how do I do this?

In Alias documentation it says to add an ! to exclude from a Alias list. So:

Code Select Expand

!xxx.yyy.zzz.www/24

Let's say I have an "URL Table (IPs)" so I enter the name of the table in the "Content" box, say something like this:

Code Select Expand

https://raw.git.com/larhedse/list.txt

And it will read in the lot, say 1000 rows.

So now I need to do an exclusion, if I add a 2nd row to the above:

Code Select Expand

https://raw.git.com/larhedse/list.txt
!xxx.yyy.zzz.www/24

And now I will have zero rows in that table - exlusion yes, but on ALL rows. So what do I do wrong?

well, if you have to, you could of course block the IPs of all DoH servers....

https://raw.githubusercontent.com/jpgpi250/piholemanual/master/DOHipv4.txt

If anyone cares: Had another 100% CPU from Unbound. This time around there was NO interface / link going up or down - my LAN PC had been turned off for hours before this incident.

So here I give up, the Monit script works by running kill -9 and then restart. It would be nice if this gets solved some day, but I have very low fate in that to happen. There does not seem to be anything to work on, no interface dependes, no blocklist, no DoT, no DNSSEC or anything - it just freaks out. Someone has made a change, and my prediction is that this will get worse.

Unbound not runnning...

This time it seems to have been triggered by med switching on my LAN PC, at least it is about the same time, but there is NO lines in the Unbound log that says anything usefull for me, except fatal error (which of course is something bad?):

Code Select Expand

2024-02-22T13:50:58 Critical unbound [43239:3] fatal error: Could not initialize thread
2024-02-22T13:50:58 Critical unbound [43239:2] fatal error: Could not initialize thread

It would be lovely if I knew why - just telling me the above gives no clue. The general log has some more details maybe, but I can not interpret what is going on:

Code Select Expand

2024-02-22T13:50:58 Notice kernel <6>pid 43239 (unbound), jid 0, uid 59: exited on signal 11
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns ()
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dhcp ()
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt2)
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt2'
2024-02-22T13:50:43 Notice opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt2(igb2)
2024-02-22T13:50:43 Notice kernel <6>igb2: link state changed to UP

hi

Do NOT get confused about the roots.hint error - it is not the problem, I have showed that earlier - it is a symptom maybe?

sorry, what make you think so? from what i see - its (one of ?) the reason

I did a firmware update

so 'use internal hints' patch is gone?

Well I can of course not say for 100% sure that the roots.hint is not the challenge - but I had this 100% CPU Unbound challenge even with that patch installed. That patch got overwritten I guess when I upgraded to latest, since Unbound was part of that firmware upgrade (ports: unbound 1.19.1) - the option that was installed with that patch got removed anyway so it is not available anymore (except if I re-install the patch of course).

Yeasterday I did a firmware update, so now I am running:

Code Select Expand

OPNsense 24.1.2-amd64
FreeBSD 13.2-RELEASE-p10
OpenSSL 3.0.13

Within less than 24h I got another 100% CPU - and my Monit setup kill Unbound and everything got back to normal. I have combined three logs: General (under System), Monit and Unbound - it is way easier to follow what happens (why is that not a built in option, to look at several logs at the same time - or is it and I just do not know how to combine logs?):

Code Select Expand

LOG Date Severity Process Line
Monit 2024-02-22T00:09:22 Informational monit 'UnboundHighCPU' status succeeded (1) -- exit: Expression Syntax.
Unbound 2024-02-22T00:07:29 Informational unbound [17657:1] info: generate keytag query _ta-4f66. NULL IN
Unbound 2024-02-22T00:07:29 Informational unbound [17657:0] info: start of service (unbound 1.19.1).
Unbound 2024-02-22T00:07:29 Notice unbound [17657:0] notice: init module 2: iterator
Unbound 2024-02-22T00:07:29 Notice unbound [17657:0] notice: init module 1: validator
Unbound 2024-02-22T00:07:22 Informational unbound [17657:0] info: dnsbl_module: blocklist loaded. length is 3687688
General 2024-02-22T00:07:18 Error opnsense /usr/local/sbin/pluginctl: The command '/bin/kill -'TERM' '59284''(pid:/var/run/unbound.pid) returned exit code '1', the output was 'kill: 59284: No such process'
General 2024-02-22T00:07:18 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure unbound_start (execute task : unbound_configure_do(1))
General 2024-02-22T00:07:18 Notice opnsense /usr/local/sbin/pluginctl: plugins_configure unbound_start (1)
Monit 2024-02-22T00:07:18 Informational monit 'UnboundHighCPU' start: '/usr/local/sbin/pluginctl -c unbound_start'
Unbound 2024-02-22T00:07:18 Informational unbound [17657:0] info: dnsbl_module: updating blocklist.
Unbound 2024-02-22T00:07:18 Notice unbound [17657:0] notice: init module 0: python
Monit 2024-02-22T00:07:13 Informational monit 'UnboundHighCPU' stop: '/usr/local/opnsense/scripts/OPNsense/Monit/Unbound_Kill.sh stop'
Monit 2024-02-22T00:07:13 Informational monit 'UnboundHighCPU' trying to restart
Monit 2024-02-22T00:07:08 Error monit 'UnboundHighCPU' status failed (100) -- no output
Unbound 2024-02-22T00:03:14 Informational unbound [59284:4] info: generate keytag query _ta-4f66. NULL IN
Unbound 2024-02-22T00:03:14 Critical unbound [59284:3] fatal error: Could not initialize thread
Unbound 2024-02-22T00:03:14 Informational unbound [59284:0] info: start of service (unbound 1.19.1).
Unbound 2024-02-22T00:03:14 Informational unbound [59284:3] info: server stats for thread 3: requestlist max 0 avg 0 exceeded 0 jostled 0
Unbound 2024-02-22T00:03:14 Informational unbound [59284:3] info: server stats for thread 3: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting
Unbound 2024-02-22T00:03:14 Error unbound [59284:3] error: Could not set root or stub hints
Unbound 2024-02-22T00:03:14 Error unbound [59284:3] error: reading root hints /root.hints 2:6: Syntax error, could not parse the RR's type
Unbound 2024-02-22T00:03:14 Notice unbound [59284:0] notice: init module 2: iterator
Unbound 2024-02-22T00:03:14 Notice unbound [59284:0] notice: init module 1: validator
Unbound 2024-02-22T00:03:07 Informational unbound [59284:0] info: dnsbl_module: blocklist loaded. length is 3687688
General 2024-02-22T00:03:06 Notice opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet detached event for opt2(igb2)
General 2024-02-22T00:03:06 Notice kernel <6>igb2: link state changed to DOWN
Unbound 2024-02-22T00:03:03 Informational unbound [59284:0] info: dnsbl_module: updating blocklist.
Unbound 2024-02-22T00:03:03 Notice unbound [59284:0] notice: init module 0: python
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns (execute task : unbound_configure_do())
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns (execute task : dnsmasq_configure_do())
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dns ()
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dhcp (execute task : dhcpd_dhcp_configure())
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure dhcp ()
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure ipsec (execute task : ipsec_configure_do(,opt2))
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: plugins_configure ipsec (,opt2)
General 2024-02-22T00:03:00 Notice opnsense /usr/local/etc/rc.linkup: ROUTING: entering configure using 'opt2'
General 2024-02-22T00:03:00 Notice kernel <6>igb2: link state changed to UP
Unbound 2024-02-22T00:03:00 Informational unbound [50504:0] info: service stopped (unbound 1.19.1).
General 2024-02-22T00:02:59 Notice opnsense /usr/local/etc/rc.linkup: DEVD: Ethernet attached event for opt2(igb2)
General 2024-02-22T00:01:38 Notice dhclient dhclient-script: Creating resolv.conf
General 2024-02-22T00:01:38 Notice dhclient dhclient-script: Reason RENEW on igb1 executing


Do NOT get confused about the roots.hint error - it is not the problem, I have showed that earlier - it is a symptom maybe? but it is not what is the challenge, it is just "a thing" that seems to happen....

Anyway, for the record: DNSSEC, BlockLists, DoT enabled and NO DHCP registration --> this kills my idea of something DHCP related. Instead I have to admit that this might very well be interface related. I have earlier written I do not think so, and part of me are still in that odd corner, however when 23.7 was release I had problems with my direct connected PC (LAN if you will) as one might recall: https://forum.opnsense.org/index.php?topic=36807.msg180191#msg180191

So it does indeed look like there is a connection with link up/down. However as I wrote earlier, this worked in 23.1 - and from 23.7 it is a bit of a challenge...

I will go and search for my DLink switch, so I can remove the link up/down from this and see if this works better. Do note however that I have for example my HomeAssistant rpi4 server direct attached to another port, and a Linux Server to yet another port. However they run 24x7 so no link up/down on them...

I'll be back  8)

I just had a strange thing happening: Unbound just stopped - I have no clue why, nothing in the log for Unbound, or in System/Log/General. My Monit setup started Unbound again so no biggi in that way, but why just stop?

Hope it was just a glitch...

I'm still waiting for the next 100% CPU. Currently I am running DoT, DNSSEC, BlockList and NO DHCP registration.

If this works for say 14 days (I will not update for any fixes until crash or something... bad idea partly I know, but then again...), I will attach a switch (I have a rather simple D-Link somewher, it will do for this test) to see if anything changes (with DHCP re-enabled for registration). I do agree with CJ about a switch that sometimes can move the problem away...

I'm not out of ideas to test, it just takes for ever...

EDIT: And I do run ONLY IP4 - no IP6 at all.

Regarding DoT: No not yet.

And there is a reason for that: This config I am running, as I mentioned before, was running just fine until I upgraded to 23.7.x - after that upgrade this Unbound challenge has been around. Different patches has been tested, but none so far have solved this challenge. So currently, as mentioned, I have removed DHCP registration within Unbound - which, fingers crossed and all that, seems to make this just so much more stable.

So why might this be an issue for say myself and not for others - of course this is a very good question, which we still are looking for answers for. My thinking is: for some reason I think this will happen sooner or later for everybody (that are using Unbound that is), but for some reason my setup is faster (!?) to trigger this. So say that you upgrade your OPNsense as soon as something is released, or just a reboot once a week or something, this will so to speak reset the eggtimer.... And I seem to drag my feet abit around before I upgrade...

I don't use overrides either.

I "only" use BlockList and DoT.

Could you try

Code Select Expand

cd /tmp
ktrace -p <pid of misbehaving unbound>
# wait a couple of seconds
ktrace -C
kdump

This will catch all system calls the process performs in that time and state. It will not catch if it's calculating "something" internally. But frequently this gives hints about problems. E.g. server processes trying to open a logfile in a nonexistent directory so they cannot log why the fail to start etc. For file accesses you want to look for NAMI calls, for example.

At my latest 100% CPU Unbound happening, I have added thoose two commands to my kill script so that it will create something at each restart - I might not be home and all that, so it was an easy way. Well the files are empty from the two ktrace commands - maybe I do something wrong?

Code Select Expand

pgrep "unbound" | grep -v "$$" | xargs ktrace -p > /home/lars/ktracep_`date +'%y%m%d_%T'`
sleep 5
ktrace -C > /home/lars/ktraceC_`date +'%y%m%d_%T'`


Development!!!! Yihaa - or maybe not, hang on ::)

So I just had another 100% CPU bound happening. This time NO roots.hint error. So that in a way is great, or is it?

Well, there is still that 100% Unbound CPU usage, I just do not get the roots.hint error anymore - and this might be related to the patch mentioned earlier:

Code Select Expand

opnsense-patch -a kulikov-a 2e2294c

So this removes an "error" about the roots.hint file that no one could relate to or anything - it simply put did not make anyone happier.

So I am still with the challenge that something gives Unbound a challenge at some point of time. My current test approach to see if I can get it more stable (do remember that it was a lot more stable in the end of my 23.7-testing with two patches (not applied) and me removing a few plugins (most likely not involved in anything related to Unbound), and then the final thing I did was to NOT update Unbound on DHCP. So now I will uncheck:
1) Register DHCP Leases
2) Register DHCP Static Mappings

This will of course sabotage my name resolution on my intranet - however out of pure luck I have never used name resolution at all on my intranet - everything goes over IP addresses... So the impact for me is slim to none, so this is an easy way to test/validate if the update of any IP address from DHCP might be involved in this Unbound challenge...... Stay tuned.... 8) Starting that egg timer, yet again ::)

Pretty much.

Since I do not run bridge ports and direct connected PC, and I have ALSO had this challenge with Unbound when my direct connected LAN PC has NOT been used (as in I am not home, nothing started that particular PC so no interface up/down). So No, your assumption is not correct in this Unbound case. There has been other things related to interface up/down, but that is a totally different story. Do also note that there is at least ONE installation which is a virtual installation, that have been experiencing this Unbound challenge.