Was checking up on my NTP server and noticed a connection from Censys in the logs. I already have an alias with every IP range Censys uses, setup for blocking in the Opnsense firewall, and that particular IP address is indeed in one of the ranges in the alias (66.132.186.0/24). However, as I stated, it did make it past Opnsense to the NTP server in my network.
I noticed a similiar behavior previously with the GeoIP alias as well. I would add in an entire country to ban from accessing one of my services via a GeoIP alias, but I would see in the logs of the daemon that IPs from that country were still coming through. I chaulked that up to maybe GeoIP wasn't up-to-date, but it kept happening very frequently, to the point where I was creating my own alias of the IP ranges (/24) to ban in a seperate alias. That usually took care of the issue, but occasionally a connection attempt from one of those blocked IP ranges would still get through.
Has anyone else experienced this?
Show-the-rules (i.e.: all of them). Order matters. Interfaces matter. Direction matters. New rules or old matter - especially, when old rules have not been deleted.
From what it sounds, you have a preceeding rule to allow that kind of traffic before your blocking rule can hit.
You can also get /tmp/rules.debug and throw it at an AI of your choice together with the packet you think it should block for an explanation. For those types of work, an AI is actually quite good.
Quote from: Mpegger on Today at 05:46:25 AMI noticed a similiar behavior previously with the GeoIP alias as well. I would add in an entire country to ban from accessing one of my services via a GeoIP alias, but I would see in the logs of the daemon that IPs from that country were still coming through. I chaulked that up to maybe GeoIP wasn't up-to-date, but it kept happening very frequently, to the point where I was creating my own alias of the IP ranges (/24) to ban in a seperate alias. That usually took care of the issue, but occasionally a connection attempt from one of those blocked IP ranges would still get through.
GeoIP can be a mess sometimes and not your own fault at all so I would not worry about that too much! ;)
I can remember many posts from people on different forums such as :
- Why is my IP Address shown as from Country A while I live in Country B ?!
- Why does Netflix say I am from Country X while I live in Country Z ?!
And so on...
In the end these were all issues that only the ISP that owns a certain subnet can solve by updating it's network information when you do a
whois query, so there is not much you can do about that.
Quote from: nero355 on Today at 02:28:46 PMGeoIP can be a mess sometimes and not your own fault at all so I would not worry about that too much! ;)
I can remember many posts from people on different forums such as :
- Why is my IP Address shown as from Country A while I live in Country B ?!
- Why does Netflix say I am from Country X while I live in Country Z ?!
And so on...
In the end these were all issues that only the ISP that owns a certain subnet can solve by updating it's network information when you do a whois query, so there is not much you can do about that.
That's why I didn't think much of IPs getting through when it was GeoIP related. I figured it was either A) my GeoIP alias hadn't updated to include a "new" IP range for that country, or B) GeoIP just has the wrong or missing IP ranges for that country. So I was constantly performing lookup via IPInfo to manually add ranges into my own alias blocklist.
Quote from: meyergru on Today at 09:09:54 AMShow-the-rules (i.e.: all of them). Order matters. Interfaces matter. Direction matters. New rules or old matter - especially, when old rules have not been deleted.
From what it sounds, you have a preceeding rule to allow that kind of traffic before your blocking rule can hit.
You can also get /tmp/rules.debug and throw it at an AI of your choice together with the packet you think it should block for an explanation. For those types of work, an AI is actually quite good.
I'm very particular about the rules and rule order. I'm always triple checking to make sure all is working as intended. That being said, you can take a look yourself:
censys1.png
This is the Censys alias which has all the ranges the company themself publish.
This is my firewall rules, starting below the auto generated rules I place most of my blocking rules as I do understand order is important:
censys2.png
You can see I have both a block for Censys and Modat scanners at the bottom of this first block section. Way further down in the firewall list after all my allows (and a couple of specific blocks for DNS redirection duty), is a manual IPv6 allow, and a automaticly generated IPv4 allow for the NTP service. Yes, there is a Destination NAT port forward rule for the NTP service, and I thought it may have been an issue of the NAT rule taking priority over the firewall Block rule, but if that were the case, I would have dozens of entries for Censys and Modat in the NTP server logs like it was before I blocked them. There is currently only a single entry in the NTP server log which shows that the scanners
are being blocked, except for that 1 single entry.
Hostname NTP Drop Int IntL Last Cmd Drop Int Last
===============================================================================
197.186.132.66.censys-sc> 1 0 - - 17h 0 0 - -
(This one is purely theoretical, because 66.132.186.197 has no open NTP port)
NTP is a UDP protocol, which is stateless by default. If that one IP is in one of the NTP time server specifications, which often are specified as pools, which in turn can have thousands of entries, you querying them will also allow them contacting you for a short time.
What I think the real reason is:
You did an NTP redirection which creates an implicit allow rule if you do not specify the "manual", but the "pass" rule. I think that goes before any interface-specific rules. This is not as easy to see with the new rules than with the old ones. It was the main reason for me to have such block rules in the "Floating" section of the old rules - even if they applied to just one (i.e. WAN) interface: Otherwise, any other "Floating" rule could not be blocked.
With the new rules, this is not possible any more: Once you only have one interface, the rules are not floating any more.
Quote from: meyergru on Today at 06:34:25 PM(This one is purely theoretical, because 66.132.186.197 has no open NTP port)
NTP is a UDP protocol, which is stateless by default. If that one IP is in one of the NTP time server specifications, which often are specified as pools, which in turn can have thousands of entries, you querying them will also allow them contacting you for a short time.
My NTP server is not in any pool, I have it open to the WAN just for myself. And it definetely isn't a time server from Censys contacting my NTP server as Censys is a port scanner. Either way, server/pool entries in the chrony.conf don't initiate contact with the client, the client (in this case that would be
my NTP server) initiates contact with the remote server to get the time response, and such server/pool entries being used by my NTP server wouldn't show up in the client log list, only the NTP clients contacting my server for time.
Quote from: meyergru on Today at 06:34:25 PMWhat I think the real reason is:
You did an NTP redirection which creates an implicit allow rule if you do not specify the "manual", but the "pass" rule. I think that goes before any interface-specific rules.
This is also what I was thinking, except as I stated, if that was the case there would be dozens of entries for Censys and Modat scanners in the client log list of the NTP server as there was previously when I didn't have the block rules in place. It's possible that these scanners only scan once every X number of days, or weeks, and
maybe that why I currently see only 1 entry after I started using the blocks. But without the blocks, I was getting a couple of hits a day from Censys and Modat.
Quote from: meyergru on Today at 06:34:25 PMThis is not as easy to see with the new rules than with the old ones. It was the main reason for me to have such block rules in the "Floating" section of the old rules - even if they applied to just one (i.e. WAN) interface: Otherwise, any other "Floating" rule could not be blocked.
With the new rules, this is not possible any more: Once you only have one interface, the rules are not floating any more.
I always understood floating rules as rules that apply to any interface, in any direction. So I never used or setup any floating rules as I was always making rules specific to a interface and direction. And I didnt know where in the order of firewall rules floating rules were, so to keep things better ordered, I just never used floating rules.
Quote from: meyergru on Today at 06:34:25 PM[...]NTP is a UDP protocol, which is stateless by default. If that one IP is in one of the NTP time server specifications, which often are specified as pools, which in turn can have thousands of entries, you querying them will also allow them contacting you for a short time.[...]
Specifically the outbound session will override the inbound policy. Do you have the block rule in both directions? I localize 'em...
Edit: Heh - I tend to forget that my setup is unusual. I have traffic in both directions. Bridges.
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
#* PPS 0 2 377 3 -184ns[ -200ns] +/- 575ns
#? GPS 0 4 377 17 -8894us[-8894us] +/- 3087us
^- 17.253.2.35 1 10 377 837 -151us[ -152us] +/- 1435us
^- 17.253.2.45 1 10 377 4 -154us[ -154us] +/- 1409us
^- 17.253.20.45 1 10 377 335 -114us[ -117us] +/- 4692us
^- 2620:149:a10:4000::31 1 10 377 447 -450us[ -450us] +/- 5030us
^- 23.186.168.127 2 10 266 47m +1428us[+1425us] +/- 54ms
^- 2602:fd37:508:73df::123 2 10 377 344 +2643us[+2643us] +/- 34ms
^- 167.248.62.201 3 10 17 438 -256us[ -256us] +/- 74ms
^- 2001:470:fe6f::123 1 10 377 322 +241us[ +242us] +/- 106ms
That is the current list of NTP servers my NTP server is using. And I get that *if* that Censys IP was in the server pool there is a possibility my server *might* have contacted it, but then it would only have shown up in the sources list, not in the client log list. The client log list will show any and every IP address (inclduing it's own local) that contacted the NTP server first, not my NTP server contacting the Censys server, which means that Censys IP made it through Opnsense to my NTP server.
As I said: That is a potential, theoretical cause, which is highly unlikely, because that IP does not have an open NTP port, so it will most surely not be in any NTP pool list.
However, if it was, then your firewall would be open once your NTP instance tried to contact it.
I highly doubt that only this single IP can get through at this time, as only this one contacting you means next to nothing - you do not know how their selection process for outbound connections is. Maybe that one is the only one trying. Unless you have proof the other ones are being successfully blocked, that is.
You can easily check which pf rules really apply or result from your GUI settings, as I said: Look at /tmp/rules.debug.
You can also check what IPs are in the alias that is built from your list, they are in the firewall status.
It is simply illogical to assume a single IP from a range to be "forgotten" - a misconfiguration is more likely, that is all I can say.
How about logs, then? Looks like you'd need to enable a few...
Quote from: pfry on Today at 08:00:15 PMHow about logs, then? Looks like you'd need to enable a few...
I've enabled them, but no idea when one will pop up in the logs.
Quote from: meyergru on Today at 07:53:23 PMI highly doubt that only this single IP can get through at this time, as only this one contacting you means next to nothing - you do not know how their selection process for outbound connections is. Maybe that one is the only one trying. Unless you have proof the other ones are being successfully blocked, that is.
The only proof I currently have is that which is made through obersavtion. When no blocklist was active, couple of contacts from Censys and Modat daily from random IPs in thier IP ranges. They just port scan for open ports. After blocklist is active, no contact from either except from this one single time.
Quote from: meyergru on Today at 07:53:23 PMYou can easily check which pf rules really apply or result from your GUI settings, as I said: Look at /tmp/rules.debug.
You can also check what IPs are in the alias that is built from your list, they are in the firewall status.
Specific instructions required.
Check /tmp/rules.debug from GUI? Wouldn't that be via CLI? And what would I be looking for?
I do not find any firewall 'status' anywhere in the Firewall section of the GUI. I assume that what I placed in the alias (as pictured earlier above) should be in the firewall itself. Yes, I did hit Apply, both in the Alias and Firewall sections, and this was also a couple of weeks ago that I applied these blocks.
Quote from: meyergru on Today at 07:53:23 PMIt is simply illogical to assume a single IP from a range to be "forgotten" - a misconfiguration is more likely, that is all I can say.
But that's why it has me confused (concerned? worried?). 66.132.186.197 is in the 66.132.186.0/24 range in that alias, which as you can see from my firewall rules
should be blocked, but it wasn't, and that should be concerning if it's also happening on other systems. Hence why I brought it up and want to try to figure out. If everything on my end is correctly setup in Opnsense,
why did it happen?The only thing I can think of is either A) Opnsense for some reason did not consider 66.132.186.197 to be in the 66.132.186.0/24 range, and allowed it through, or B) the Destination NAT rule is taking precedent over the blocking firewall rule, and the port scan of port 123 from that IP was allowed.
If it was B, I should see more Censys and Modat IPs hitting the client list on the NTP server, but I only see 1 single IP. Which leads me to think it's somehow A instead. Yes, it's illogical. Yes, it shouldn't be possible. Yes, it shouldn't happen. But it did.
Why?I am not a Linux guru. I do not know how to go into the CLI and start digging for information, but I can follow instructions if they are clear and consise. So if anyone wants to help to figure out the "Why?", I'm all ears.
In the middle of typing all the previous replies, I just got a hit from a Modat scanner:
o045.scanner.modat.io 1 0 - - 43m 0 0 - -
That host is 85.217.149.45, which is in the 85.217.149.40/29 range of the alias for Modat.
I'm starting to think the NAT rule is taking prescedent and skipping the firewall rule, which IMO, should not be the way a primary firewall operates. But if that is the case, it would mean setting up a firewall on all individual servers open to the WAN, instead of OPNsense doing all the firewalling.
It is not the NAT rule, but the "pass" setting you specified with it. When you select "manual", you can tailor and prioritize a specific rule to your liking, which is what I wrote in #4 already.
Quote from: meyergru on Today at 09:47:06 PMIt is not the NAT rule, but the "pass" setting you specified with it. When you select "manual", you can tailor and prioritize a specific rule to your liking, which is what I wrote in #4 already.
That's confusing to me then, because the auto generated Pass rule is
after the manual Block rule I created. And if I understand the docs and rule order, because the Block rule is earlier (higher priority), the moment an IP in that block rule comes across, OPNsense should block it and stop going through any further rules.
I'll disable the NAT auto firewall rule and recreate it manually and see if that makes any difference (if I see any further hits in the NTP client logs).
I think that may be because of the transition from the old to the new rules, which is not yet finished. With the old rules, this was more apparent, as I tried to explain. However, I do not know for sure and at this point, I do not bother to pursue this, because there is still work underway: For example, the firewall rules are "new", but the NAT rules are still "old" - and frankly, I do not know the exact implications, which is why I suggested to look at what is actually being made of it in /tmp/rules.debug.
P.S.: Some people like to exclude blocking aliases by using their negation in the allowed source of the NAT rules. My previous preferred way was blocking in the old "floating" rules, but maybe you have to do it differently now.
Quote from: meyergru on Today at 10:01:48 PMI think that may be because of the transition from the old to the new rules, which is not yet finished. With the old rules, this was more apparent, as I tried to explain. However, I do not know for sure and at this point, I do not bother to pursue this, because there is still work underway: For example, the firewall rules are "new", but the NAT rules are still "old" - and frankly, I do not know the exact implications, which is why I suggested to look at what is actually being made of it in /tmp/rules.debug.
Yes, I see that now. I disabled the Auto Firewall Rule option in the Destination Nat rule and set it to Manual instead. The Destination NAT rule was setup using both the WAN+LAN interface in my network, so that I could use the external FQDN to contact the NTP server, even from within my local LAN network. When I tried to recreate the Firewall rule the same way the NAT Auto rule was, it was created as a Floating Rule (because it was using the WAN+LAN interfaces in a single rule), which moves it the top of the all the manual Firewall rules I have in place, and there is no way to move it further down in the list because it is a Floating rule. This to me means that Floating rules would come before all manual single interface rules (in my case the blocking rules I have setup), which would explain how those IPs that should be blocked make it through. They hit a floating rule first, before hitting the non-floating block rule.
I'm going to instead recreate the rule as 2 seperate rules, one for the WAN and one for the LAN, as this would allow me to set the order of the rule and should still allow internal and external access to the NTP server external FQDN.
My guess is that even though the NAT Auto generated firewall rule is at the bottom of the list, because it is setup for both interfaces, it's considered a Floating Rule and will be parsed before the manual rules, even though it shows at the bottom of the list of firewall rules. I'll need to go through my other rules as I have similar rules setup for other services.
And your insight on creating blocking rules as floating rules makes sense if floating rules always come first before non-floating rules.
Quote from: Mpegger on Today at 10:15:36 PMAnd your insight on creating blocking rules as floating rules makes sense if floating rules always come first before non-floating rules.
When the new rules scheme was first discussed, this was my main gripe about it, because with the old rules, you could create floating rules purely for precedence reasons. Now, with the new rules, the role of a rule switches with the number of interfaces it applies to (and it can never be floating with just one interface like "WAN only").
We'll see how this plays out when everything is migrated.