Messages

[think]

I think that may be because of the transition from the old to the new rules, which is not yet finished. With the old rules, this was more apparent, as I tried to explain. However, I do not know for sure and at this point, I do not bother to pursue this, because there is still work underway: For example, the firewall rules are "new", but the NAT rules are still "old" - and frankly, I do not know the exact implications, which is why I suggested to look at what is actually being made of it in /tmp/rules.debug.

Yes, I see that now. I disabled the Auto Firewall Rule option in the Destination Nat rule and set it to Manual instead. The Destination NAT rule was setup using both the WAN+LAN interface in my network, so that I could use the external FQDN to contact the NTP server, even from within my local LAN network. When I tried to recreate the Firewall rule the same way the NAT Auto rule was, it was created as a Floating Rule (because it was using the WAN+LAN interfaces in a single rule), which moves it the top of the all the manual Firewall rules I have in place, and there is no way to move it further down in the list because it is a Floating rule. This to me means that Floating rules would come before all manual single interface rules (in my case the blocking rules I have setup), which would explain how those IPs that should be blocked make it through. They hit a floating rule first, before hitting the non-floating block rule.

I'm going to instead recreate the rule as 2 seperate rules, one for the WAN and one for the LAN, as this would allow me to set the order of the rule and should still allow internal and external access to the NTP server external FQDN.

My guess is that even though the NAT Auto generated firewall rule is at the bottom of the list, because it is setup for both interfaces, it's considered a Floating Rule and will be parsed before the manual rules, even though it shows at the bottom of the list of firewall rules. I'll need to go through my other rules as I have similar rules setup for other services.

And your insight on creating blocking rules as floating rules makes sense if floating rules always come first before non-floating rules.

It is not the NAT rule, but the "pass" setting you specified with it. When you select "manual", you can tailor and prioritize a specific rule to your liking, which is what I wrote in #4 already.

That's confusing to me then, because the auto generated Pass rule is after the manual Block rule I created. And if I understand the docs and rule order, because the Block rule is earlier (higher priority), the moment an IP in that block rule comes across, OPNsense should block it and stop going through any further rules.

I'll disable the NAT auto firewall rule and recreate it manually and see if that makes any difference (if I see any further hits in the NTP client logs).

In the middle of typing all the previous replies, I just got a hit from a Modat scanner:

Code Select Expand

o045.scanner.modat.io           1      0   -   -   43m       0      0   -     -
That host is 85.217.149.45, which is in the 85.217.149.40/29 range of the alias for Modat.

I'm starting to think the NAT rule is taking prescedent and skipping the firewall rule, which IMO, should not be the way a primary firewall operates. But if that is the case, it would mean setting up a firewall on all individual servers open to the WAN, instead of OPNsense doing all the firewalling.

[should]

How about logs, then? Looks like you'd need to enable a few...

I've enabled them, but no idea when one will pop up in the logs.

I highly doubt that only this single IP can get through at this time, as only this one contacting you means next to nothing - you do not know how their selection process for outbound connections is. Maybe that one is the only one trying. Unless you have proof the other ones are being successfully blocked, that is.

The only proof I currently have is that which is made through obersavtion. When no blocklist was active, couple of contacts from Censys and Modat daily from random IPs in thier IP ranges. They just port scan for open ports. After blocklist is active, no contact from either except from this one single time.

You can easily check which pf rules really apply or result from your GUI settings, as I said: Look at /tmp/rules.debug.

You can also check what IPs are in the alias that is built from your list, they are in the firewall status.

Specific instructions required.
Check /tmp/rules.debug from GUI? Wouldn't that be via CLI? And what would I be looking for?

I do not find any firewall 'status' anywhere in the Firewall section of the GUI. I assume that what I placed in the alias (as pictured earlier above) should be in the firewall itself. Yes, I did hit Apply, both in the Alias and Firewall sections, and this was also a couple of weeks ago that I applied these blocks.

It is simply illogical to assume a single IP from a range to be "forgotten" - a misconfiguration is more likely, that is all I can say.

But that's why it has me confused (concerned? worried?). 66.132.186.197 is in the 66.132.186.0/24 range in that alias, which as you can see from my firewall rules should be blocked, but it wasn't, and that should be concerning if it's also happening on other systems. Hence why I brought it up and want to try to figure out. If everything on my end is correctly setup in Opnsense, why did it happen?

The only thing I can think of is either A) Opnsense for some reason did not consider 66.132.186.197 to be in the 66.132.186.0/24 range, and allowed it through, or B) the Destination NAT rule is taking precedent over the blocking firewall rule, and the port scan of port 123 from that IP was allowed.

If it was B, I should see more Censys and Modat IPs hitting the client list on the NTP server, but I only see 1 single IP. Which leads me to think it's somehow A instead. Yes, it's illogical. Yes, it shouldn't be possible. Yes, it shouldn't happen. But it did. Why?

I am not a Linux guru. I do not know how to go into the CLI and start digging for information, but I can follow instructions if they are clear and consise. So if anyone wants to help to figure out the "Why?", I'm all ears.

[*if*]

Code Select Expand

MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#* PPS                           0   2   377     3   -184ns[ -200ns] +/-  575ns
#? GPS                           0   4   377    17  -8894us[-8894us] +/- 3087us
^- 17.253.2.35                   1  10   377   837   -151us[ -152us] +/- 1435us
^- 17.253.2.45                   1  10   377     4   -154us[ -154us] +/- 1409us
^- 17.253.20.45                  1  10   377   335   -114us[ -117us] +/- 4692us
^- 2620:149:a10:4000::31         1  10   377   447   -450us[ -450us] +/- 5030us
^- 23.186.168.127                2  10   266   47m  +1428us[+1425us] +/-   54ms
^- 2602:fd37:508:73df::123       2  10   377   344  +2643us[+2643us] +/-   34ms
^- 167.248.62.201                3  10    17   438   -256us[ -256us] +/-   74ms
^- 2001:470:fe6f::123            1  10   377   322   +241us[ +242us] +/-  106ms
That is the current list of NTP servers my NTP server is using. And I get that *if* that Censys IP was in the server pool there is a possibility my server *might* have contacted it, but then it would only have shown up in the sources list, not in the client log list. The client log list will show any and every IP address (inclduing it's own local) that contacted the NTP server first, not my NTP server contacting the Censys server, which means that Censys IP made it through Opnsense to my NTP server.

(This one is purely theoretical, because 66.132.186.197 has no open NTP port)

NTP is a UDP protocol, which is stateless by default. If that one IP is in one of the NTP time server specifications, which often are specified as pools, which in turn can have thousands of entries, you querying them will also allow them contacting you for a short time.

My NTP server is not in any pool, I have it open to the WAN just for myself. And it definetely isn't a time server from Censys contacting my NTP server as Censys is a port scanner. Either way, server/pool entries in the chrony.conf don't initiate contact with the client, the client (in this case that would be my NTP server) initiates contact with the remote server to get the time response, and such server/pool entries being used by my NTP server wouldn't show up in the client log list, only the NTP clients contacting my server for time.

What I think the real reason is:

You did an NTP redirection which creates an implicit allow rule if you do not specify the "manual", but the "pass" rule. I think that goes before any interface-specific rules.

This is also what I was thinking, except as I stated, if that was the case there would be dozens of entries for Censys and Modat scanners in the client log list of the NTP server as there was previously when I didn't have the block rules in place. It's possible that these scanners only scan once every X number of days, or weeks, and maybe that why I currently see only 1 entry after I started using the blocks. But without the blocks, I was getting a couple of hits a day from Censys and Modat.

This is not as easy to see with the new rules than with the old ones. It was the main reason for me to have such block rules in the "Floating" section of the old rules - even if they applied to just one (i.e. WAN) interface: Otherwise, any other "Floating" rule could not be blocked.

With the new rules, this is not possible any more: Once you only have one interface, the rules are not floating any more.

I always understood floating rules as rules that apply to any interface, in any direction. So I never used or setup any floating rules as I was always making rules specific to a interface and direction. And I didnt know where in the order of firewall rules floating rules were, so to keep things better ordered, I just never used floating rules.

GeoIP can be a mess sometimes and not your own fault at all so I would not worry about that too much! ;)

I can remember many posts from people on different forums such as :
- Why is my IP Address shown as from Country A while I live in Country B ?!
- Why does Netflix say I am from Country X while I live in Country Z ?!

And so on...

In the end these were all issues that only the ISP that owns a certain subnet can solve by updating it's network information when you do a whois query, so there is not much you can do about that.

That's why I didn't think much of IPs getting through when it was GeoIP related. I figured it was either A) my GeoIP alias hadn't updated to include a "new" IP range for that country, or B) GeoIP just has the wrong or missing IP ranges for that country. So I was constantly performing lookup via IPInfo to manually add ranges into my own alias blocklist.

Show-the-rules (i.e.: all of them). Order matters. Interfaces matter. Direction matters. New rules or old matter - especially, when old rules have not been deleted.

From what it sounds, you have a preceeding rule to allow that kind of traffic before your blocking rule can hit.

You can also get /tmp/rules.debug and throw it at an AI of your choice together with the packet you think it should block for an explanation. For those types of work, an AI is actually quite good.

I'm very particular about the rules and rule order. I'm always triple checking to make sure all is working as intended. That being said, you can take a look yourself:
You cannot view this attachment.
This is the Censys alias which has all the ranges the company themself publish.

This is my firewall rules, starting below the auto generated rules I place most of my blocking rules as I do understand order is important:
You cannot view this attachment.
You can see I have both a block for Censys and Modat scanners at the bottom of this first block section. Way further down in the firewall list after all my allows (and a couple of specific blocks for DNS redirection duty), is a manual IPv6 allow, and a automaticly generated IPv4 allow for the NTP service. Yes, there is a Destination NAT port forward rule for the NTP service, and I thought it may have been an issue of the NAT rule taking priority over the firewall Block rule, but if that were the case, I would have dozens of entries for Censys and Modat in the NTP server logs like it was before I blocked them. There is currently only a single entry in the NTP server log which shows that the scanners are being blocked, except for that 1 single entry.

Code Select Expand

Hostname                      NTP   Drop Int IntL Last     Cmd   Drop Int  Last
===============================================================================
197.186.132.66.censys-sc>       1      0   -   -   17h       0      0   -     -

Was checking up on my NTP server and noticed a connection from Censys in the logs. I already have an alias with every IP range Censys uses, setup for blocking in the Opnsense firewall, and that particular IP address is indeed in one of the ranges in the alias (66.132.186.0/24). However, as I stated, it did make it past Opnsense to the NTP server in my network.

I noticed a similiar behavior previously with the GeoIP alias as well. I would add in an entire country to ban from accessing one of my services via a GeoIP alias, but I would see in the logs of the daemon that IPs from that country were still coming through. I chaulked that up to maybe GeoIP wasn't up-to-date, but it kept happening very frequently, to the point where I was creating my own alias of the IP ranges (/24) to ban in a seperate alias. That usually took care of the issue, but occasionally a connection attempt from one of those blocked IP ranges would still get through.

Has anyone else experienced this?

If IPv6 is the issue, it may be because of the way your ISP delegates IPv6 to you. I am on Verizon Fios and receive a /56 prefix from them and use a similar setup as used in this post: Verizon Fios IPv6. If you are absolutely sure your ISP supports IPv6 and provides thier users with a IPv6 address, it's best to just do a simple duckduckgo search for "<ISPname> IPv6 opnsense". There may already be some posts or guides pointing out how to setup IPv6 with your ISP properly. Once you have IPv6 properly working on the WAN, from there you can start working on IPv6 for your LAN, though truthfully it doesn't matter unless you are planning on hosting services via IPv6.

I am probably misunderstanding how Crowdsec works, but from what I have read, it seems that Crowdsec doesn't monitor the packets going across the interfaces like other IDS/IPS software does, but instead just watches firewall logs for any known abusive patterns. Does this mean that if I have any IP blocking lists in the Opnsense firewall, that I need to enable logging on each entry in order for Crowdsec to "see" any potential patterns? Or is the enable logging option only for the users eyes, and internally Opnsense still keeps logs?

I ask because after adding in some block lists, my Crowdsec Console reports that it's been very quiet from my setup, which could mean either I screwed up the settings and its no longer reporting (not likely because it still sees the firewall and other systems reporting on my network), or as thier own popup help states, that it could just be there is nothing to report on (the blocklists are blocking any suspicious activity, but Crowdsec doesn't see it).

* OPNsense connection to Internet - WAN port(em0 Interface) is connected to a Nokia Fastmile 5G Gateway set to Bridge mode.

Just a observation, but I haven't seen anyone mention it. This Nokia Fastmile 5G sounds like a cellphone network adapter, which means it's possible their connection is CGNAT and (or?) IPv6 only.

[not]

Memtester is a user space application. It can never test RAM that is in use (allocated).

As far as I understand it (I am not a programmer), in simplest terms to be capable of doing such a thing as testing RAM that is in use, would require it to be baked into the OS kernal itself, as the OS would need to move data in RAM that is in use to other sections to allow such a RAM testing application to test those sectors. This would require a very high level of choreography between the OS, the RAM testing application, and all the other applications that are loaded in, running in, and using RAM as those programs need to be aware that the sectors of RAM allocated to them could change at any given time. The programs would need to be "Live RAM Testing" aware, or such an action would fail (the program would error out because the data it expects in a specific sector of the RAM is no longer there). It's the same reason some backup programs for VMs require the VM being backed up to be shutdown, cause it can't handle data on a drive being live (actively in use) or "locked" (only the OS can access said data).

There probably is a way to have memtester available in Opnsense, but again, running it while Opnsense is live will only result in the unused free RAM being tested. You would need to shutdown and boot into a RAM tester like Memtester86+ in order to test all the system RAM.

And again we are again discussing miniscule details of the IPv6 addressing schemes and nobody seems to want to answer the question if you should put the ULA you are using into the home networks ... 🙄

Isn't that what I just said in my previous post? Or do I need to specifically point it out and not assume the OP can't infer to use that result in thier Intrusion Detection Home Networks?

I get my post before that one was vague, but again, one should be able to infer the answer from that.

It's my understanding that fc00:: is not officially part of the LAN only routable range as it is still possible it could be assigned in the future, however miniscule (improbable? not in any of our lifetimes?) that possibility may be. Hence fc00::0/7 should not be used. fd00::0/8 is the official range to be used for LAN routeable only ULAs and is why I responded with using a proper ULA calculator. The OP can then use the calculated fdxx:xxxx:xxxx::/48 address for thier own network, or fd00::0/8 if they have plans in the future of merging thier LAN with another LAN using ULAs.

Also, use this calculator or this calculator (automatically enters your MAC). The one I linked previously appears to give out borked addresses.

We are using private IPv6 addresses along with IPv4 addresses on our local network.  I noticed that the IPv6 range "fc00::0/7" is not included in the default Home Networks list.  Why?  Does "Home Networks" not apply to IPv6?  Are there any special considerations when adding "fc00::0/7" to the Home Networks list?

In advance, thanks for guidance on this matter.

Why? Because fc00: shouldn't be use for ULA. Use a proper ULA calculator like this proper ULA generator (see my following post for generators that work properly) which explains in detail how it generates the ULA.