Messages

What I did is include the netflow storage path in the var/log ram disk by modifying the end of /usr/local/etc/rc.subr.d/var :

Code Select Expand

        echo -n "Setting up /var/log memory disk..."
        mount -t tmpfs -o size=$((MAX_MEM_SYS / 100 * MAX_MFS_VAR)) tmpfs /var/log
        echo "done."

ln -s /var/log/netflow /var/netflow
mkdir -p /var/log/netflow
chown root:wheel /var/log/netflow
chmod 750 /var/log/netflow

fi

# prep boog log
: > /var/log/boot.log

I also did something similar for the unbound.duckdb.

You'll need to re-apply these changes after each opnsense update as they'll be overwritten.

That's a nice idea, great many thanks!

I am wondering why this is not bound to /var/log anyways - so that it lands in RAM if you decide to configure a RAM disk for this.

I am seeing these log entries, but I'm not clear if they're a result of the high CPU:

Code Select Expand

2025-11-20T12:07:38-05:00
Error
lighttpd
(/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.82/src/gw_backend.c.533) connect() /var/lib/php/tmp/php-fastcgi.socket-1: Connection refused

That may well be the reason for the high CPU usage!

This is a socket:

root@OPNsense:~ # ls /var/lib/php/tmp/php-fastcgi.socket-1
srwxr-xr-x  1 root wheel 0 Nov 17 22:12 /var/lib/php/tmp/php-fastcgi.socket-1=


Does it exist?

What are the permissions?

Yes, that does not look sane!

I made a test, had "top" running and then logged in to the dashboard.

I saw just a small spike and then it went down to normal:

last pid: 62529;  load averages:  0.24,  0.15,  0.09                                                                                                         up 12+02:08:11  21:27:58
64 processes:  1 running, 63 sleeping
CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
Mem: 117M Active, 629M Inact, 2067M Wired, 104K Buf, 59G Free
ARC: 1198M Total, 815M MFU, 244M MRU, 1046K Anon, 17M Header, 118M Other
     873M Compressed, 2131M Uncompressed, 2.44:1 Ratio
Swap: 8192M Total, 8192M Free

root@OPNsense:~/smart #   diff collect_mb_2025-11-19 collect_mb_2025-11-20
1c1
< 4026285 MB
---
> 4026453 MB


root@OPNsense:~/smart # python3
Python 3.11.14 (main, Oct 21 2025, 21:38:48) [Clang 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd7080 on freebsd14
Type "help", "copyright", "credits" or "license" for more information.
>>> 4026453 - 4026285
168

So, we are down to 168 MB in about 20 hours.


Good enough for me :-)

Thanks!

Netflow. It's by far the generator of the largest amount of any data you can have on OPNsense. Export Netflow data to an external collector like ElastiFlow

But then again let's assume you write 40 GB per day. With a typical TBW of 200 for a smaller quality SSD that means you get 5.000 guaranteed days of operation or over 13 years.

Many thanks, I will disable NetFlow for now and check if this betters the situation.

While you may be right about the usage, I just don't like the idea that more data get's written to the SSD than I am transferring over the net in the same time.
That is just not right.

I noticed that the TBW (Tera-Byte Written) of my SSD used for OPNsense shows about 4 TB that got written.

I got the current values yesterday and today again:

root@OPNsense:~ # ls collect_mb_2025-11-18 collect_mb_2025-11-19
-rw-r--r--  1 root wheel 11 Nov 18 23:41 collect_mb_2025-11-18
-rw-r--r--  1 root wheel 11 Nov 19 17:01 collect_mb_2025-11-19

root@OPNsense:~ # diff collect_mb_2025-11-18 collect_mb_2025-11-19
1c1
< 3992034 MB
---
> 4024636 MB


This is about 32 GB in not even 21 hours!

I tried to catch the process that is responsible with iostat, procstat, top (top -b -m io -o write) and zpool iostat.
But I see no process that is doing this.

From the process list, one python3.11 for NetFlow and vnstatd are generating the most writes, but by far not enough for this 32 GB!

I have no idea what to do or check next.

Any idea?

I am using the regular zpool / zfs configuration and have "copies" set to 3, which should tripple write-IOs, but that is not the problem here.

The only suspect I have, is ZFS itself, with it's ARC cache that tries to predict future IO.
dtrace is of course installed, but does not seem to work:

root@OPNsense:~ # dtrace -n 'proc:::exec-success { trace(execname); }'
dtrace: invalid probe specifier proc:::exec-success { trace(execname); }: "/usr/lib/dtrace/psinfo.d", line 37: syntax error near "uid_t"

/var/log and /tmp are configured as RAM disks with 20% of memory each.

Otherwise, I am at version 25.7.7_4-amd64

Better update the subject line too!

Others may be discouraged to update reading the subject alone.

I use the following lists to block malware IPs between all my firewall networks:

https://feodotracker.abuse.ch/downloads/ipblocklist.txt
https://rules.emergingthreats.net/blockrules/compromised-ips.txt

And this to block malware IPs as Source or Destination on my WAN interface only (as it contains private networks):
https://iplists.firehol.org/files/firehol_level1.netset

Again and again, those lists contain regular IPs that should not be blocked.
Therefor, I added whitelisting rules ... but those should of course not allow anything from or to the whitelisted IPs!

Instead, I removed Quick from the both the (Floating) whitelisting and blocking rules, set the tag "blocked" on the blocking rules and "whitelisted" on the whitelisting rules.

The whitelisting rules come after the blocking rules, to that the tag get's changed from "blocked" to "whitelisted" - and then I have a final Floating rule that just blocks all traffic that has the "blocked" tag.

Is this the right way to handle this?

I first thought that inverting the tag "whitelisted" with "!whitelisted" would be the easiest way, but that syntax does not seem to be supported and I had no idea how to correctly implement whitelisting in other ways.

I have another example for the same underlying problem with wrong logging:

Here, a package gets apparently blocked by the rule "Block FRITZ to all non-routable".
This is clearly wrong, as it is a package from an IP to the broadcast address in the same network, which is allowed in an earlier rule!

Instead this package gets blocked by a Floating rule "NL Block Ports", where ports 135 to 139 are blocked!
But this rule is non-Quick!

This is the same wrong behavior we saw above!

A non-Quick rule does not get logged anymore, instead some other rule will be logged.

I implemented a new test case:

1) The Pass rule, non-logging
2) The logging Block rule, with Source "! FRITZ net"
3) Then again the same rule, with Source "any"
4) And just to check if that plays a role, a copy of this last rule - to check how it will be logged

I disabled logging for the default rule now.

Here is the screenshot:

I tested with this and finally saw those packages again, blocked by my "Block FRITZ all other".
To complete the test, I then disabled "Quick" for "Block FRITZ all other" and now I see the packaged logged with "Block FRITZ all other COPY" instead, which is a copy and just comes later.

So it is as I suspected, the non-Quick default deny / state violation rule will not be hit and logged, when there is a Quick-Rule somewhen later.
For me, this is unsuspected in that the Quick rule seems to apply state as the reason for blocking, which should instead be blocked by the default rule.

So, only when I delete both block rules, I can see the default rule login again.
I would rather prefer to have the default rule as Quick, so that it can directly apply!

But I now understand logging much better, thank you!

P.S. Somehow I see no way to attach the screenshots, will try later.

I implemented a new test case:

1) The Pass rule, non-logging
2) The logging Block rule, with Source "! FRITZ net"
3) Then again the same rule, with Source "any"
4) And just to check if that plays a role, a copy of this last rule - to check how it will be logged

I disabled logging for the default rule now.

Here is the screenshot:

Why do you need a block rule before the default block in the first place?

I thought that this is obvious:

I want to log any such package, if it ever occurs.

[not]

Let's try again:

I have a Pass rule that allows Source "FRITZ net" to any destination for the FRITZ interface.
This Pass rules did not apply because of state, as it seems - from the information seen with the "i" button. I see only "PA" or "A" TCP flags set.

The "Default deny / State violation" rule comes first, but it has "Quick" not set, so that other rules are applied as well.

My Block rule was meant to catch IPs that are not from FRITZ net, as it has Source "any", not Source "FRITZ net" like the Pass rules - to any destination.
So it should normally only apply when a spoofed IP tries to reach anything!

Here is where you go wrong: Source "any" catches everything, which includes FRITZ net. And why shouldn't it? It is a catch-all rule that is always being applied before the default block rule, just because it matches anything. You could have noticed this long ago by the fact that the log entry tells you it is because of this very rule matching (and blocking): Look at the log entries, the label says "Block Fritz all other", which is exactly your second rule.

If you wanted it to match only traffic that is not from FRITZ net, you would need to change the source to "!FRITZ net", not "any" (including FRITZ net). If you do that, then the default block rule would be applied.

Source "any" is used in the Block rule, not in the Pass rule.

But I know understood, that you are telling me that the Block rule catches packages with state problems too and will do so instead of the default rule?

That is a valid argument!
I will change the Block rule to Source "! FRITZ net" and enable logging for both block rules!

Many thank!

Whatever, regular default-deny /state-violation packages should be logged for this rule and not with some other, user-added rule!
That should be clear.

No, that's not how it works. If some rule you define has already matched a packet and acted upon it, that packet will not reach the default block, whether logging is enabled or not. I vaguely recall there being match type rules which allowed you to act on a packet but it still traverse the rest of the interface rules. But don't see that in current opnsense.

I never said something like you seem to assume!
You are mixing up my tests, where I enabled logging for the default block rule - in this case, the right rule will be logged!

[explicitely enabled logging]

So then, the only misunderstanding is that you seem to know why your allow rule does not fire (namely because of state violations - if you search this forum for that you will find many examples of people having created split routing, as I said). And yes, you have shown the log entries, but not the rule details, AFAICS.

Apart from that, the logs you showed last are clearly from the second, blocking rule, for which you have explicitely  enabled logging as shown in your first post, so what is the problem?

I can see still just one: Your packets are not being allowed for whatever cause (could be split routes, as indicated by invalid states), thus, this rule does not match and your second rule fires and gets logged, which is to be expected.

You seem to imply a problem with OpnSense or its logging and I do not see this, please explain how you come to your conclusion?

It is obvious that I was not able to bring over my point or my reasoning.

Let's try again:

I have a Pass rule that allows Source "FRITZ net" to any destination for the FRITZ interface.
This Pass rules did not apply because of state, as it seems - from the information seen with the "i" button. I see only "PA" or "A" TCP flags set.

The "Default deny / State violation" rule comes first, but it has "Quick" not set, so that other rules are applied as well.

My Block rule was meant to catch IPs that are not from FRITZ net, as it has Source "any", not Source "FRITZ net" like the Pass rules - to any destination.
So it should normally only apply when a spoofed IP tries to reach anything!

But we clearly see that is traffic from FRITZ net to some Internet IP - something that normally should have been allowed by the Pass rules already!
My Block rule should NOT apply here, as the source IP is clearly from FRITZ net!

What is finally blocking the package is the "Default deny / State violation" rules, from all what I can see.

Yet, the package get's blocked and will be logged with my Block rule "Block FRITZ all other".

And this is wrong!

It should be logged with the "Default deny / State violation" rules because this is the reason it got blocked!
But it seems that logging will not be done with the actual rule when this rule has "Quick" missing.

This is at least my interpretation.

To formulate this in a different way: For me it seems that OPNsense logs with the last rule that blocks, when the "real" blocking rule has "Quick" not set and logging disabled.
It kind of forget's the real reason for the block.

And yes, that feels very much like a bug!
The wrong rule will be logged.

Was that more clear?