Messages

I am seeing high CPU use and temperatures after upgrading to 23.7.9. Netflow seems to be the culprit. Resetting Netflow data and rebuilding the database have not helped, nor has rebooting the box.

Code Select Expand

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
15527 root          1 100    0    30M    20M CPU3     3  18:40  93.46% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)
88671 root         31  20    0   865M   203M uwait    2   1:13   3.42% /usr/local/AdGuardHome/AdGuardHome -s run
26850 root          1  52    0    58M    28M accept   2   0:02   1.76% /usr/local/bin/php-cgi
56115 root          1  20    0    13M  2548K bpf      3   0:09   1.27% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
32301 root          5  20    0    45M    12M kqread   0   0:19   0.88% /usr/local/sbin/syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid
18061 root          1  52    0    59M    28M accept   0   0:06   0.59% /usr/local/bin/php-cgi
23558 root          1  20    0    59M    28M accept   2   0:04   0.49% /usr/local/bin/php-cgi

Has anyone else seen this?

Thanks dinguz. I reset the Netflow data – no difference. I repaired the database – no difference. I rebooted OPNSense. After this, at first it seemed there was no difference, but then I noticed that ddclient seemed to feature high in the process list (sorted on WCPU).

I remembered that I had changed the ddclient back-end from ddclient to native after reading the release notes for 23.7.9, so I changed it back to ddclient. Now the overall CPU activity is ever so slightly lower, but there is still a lot more going on than before the upgrade.

Netflow remains with high WCPU in the list.

Code Select Expand

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
15527 root 96 0 30M 20M CPU1 1 14:19 99.55% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)

Thanks. This looks like a potential culprit:

Code Select Expand

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
80468 root 86 0 54M 33M CPU1 1 443:26 100.00% /usr/local/bin/python3 /usr/local/opnsense/scripts/netflow/flowd_aggregate.py (python3.9)

Has anything changed regarding python or netflow between 23.1 and 23.7?

Having just upgraded to 23.7.9, I am seeing much higher CPU usage than before, and the unit runs much warmer. While it isn't getting critical, I would like to understand what has changed and if there are parameters I should change or something I should fix.

Before the upgrade, the CPU usage would stay below 10% all the time except when there was heavy traffic such as a download or HD video streaming. Now, even when there is no traffic, it oscillates between 0% and 100%:


The CPU used to run at 61 degrees, now it is at 65.

Where should I start looking for the cause?

Thanks Bart. On my second question, I was really wondering why typing an interface name or partial IP address into the search field on the plain firewall log sends the GUI into an endless "Loading..." loop.

I will look at LibreNMS. Does that log all traffic for later inspection?

I have also done some further research and am considering using telegraf on OPNSense to send data to an influxDB on the QNAP and then use Grafana on a laptop to interrogate the influxDB. That way, I am thinking I can leverage the CPU and RAM of the laptop just when I want to get the analytics but keep collecting the underlying data on the QNAP without straining either the QNAP or the OPNSense box.

I am a relative newcomer to OPNSense. I use it in a home network setting and have modest hardware (APU2E4 with 4-core AMD GX-412TC SOC, 4GB RAM). I have a couple of questions relating to keeping an eye on traffic:

1. Are there any add-on solutions to improve the reporting/visibility of traffic, for example to see common web sites for outgoing traffic from a specific LAN IP? Any solution needs to either work on my modest hardware or on e.g. a RPi on the LAN, or on my QNAP x86 with Celeron 4-core J3455 and 8GB RAM. I have investigated ELK etc. but it seems these are too HW-demanding.
2. The other day my QNAP reported a suspicious connection attempt, even though I don't believe there should be a way for traffic from the WAN to get through the OPNSense FW. To check, I went to the OPNSense FW log file, plain view, and searched for the external IP of the suspicious attempt. That just left the interface saying "Loading..." forever. Initially, CPU use was quite high, but even after it had dropped back to ~5% the log file search screen still said "Loading...". The same happens if I search on an interface, such as wg1. Why is this?

Reviving this thread instead of starting a new one. I am trying to install pfELK on a machine on my LAN (following the how-to for docker-compose) and have a couple of questions:

1. My machine has modest hardware so I'd like to maximise performance. I thought that it would be a good idea to run pfELK on a single-node setup, so I wanted to modify the docker-compose.yml file accordingly. The only instruction I found on Github was to modify /etc/elasticsearch/elasticsearch.yml, but a) that file does not exist before you start the install and b) I would have thought that docker-compose.yml also needs to be modified. Do I need to change the create certs and environment sections, and if so how? Alternatively, if running three nodes does not consume more resources than a single node, please let me know.

2. Also, I'd like to set up MaxMind, and I'd like to do it on Docker since my machine is running Alpine Linux and I don't think there is a repository for MaxMind available. I have found a Docker container for the purpose, but I am not sure exactly how pfElk speaks to MaxMind so I need some more info to make sure the two can communicate. The pfELK how-to for MaxMind does not mention the required interface with MaxMind so I don't know what the prerequisite is when not installing MaxMind in the standard way.

I have recently implemented an OPNSense firewall and router for my home network. Looking at the built-in reporting available, I would like to see how I can get more comprehensive data and analysis of the traffic in/out, firewall actions, etc. Since my HW is limited, I suspect I'll set up the reporting on a separate machine, perhaps running the ELK stack or something similar. But, before I embark on this...

1. Are there more advanced reporting possibilities in OPNSense itself, perhaps with some added packages, even on modest hardware such as mine? I have seen mention on Routerperformance of Grafana, InfluxDB and other packages, but I am not sure if they would fit the bill.
2. If I go down the route of sending OPNSense data to an external reporting box, what would be a good way to start? I assume it should be possible to get some good data with what is already being generated on my OPNSense and without installing Zenarmor, right? Then should I go for pfELK, the integration from Elastic or something else?

When upgrading from 22.7.4 to 22.7.7, I got the following messages:

Code Select Expand

[71/74] Extracting os-wireguard-1.13_1: .......... done
Stopping configd...done
Starting configd.
Migrated OPNsense\Wireguard\Server from 0.0.3 to 0.0.4
Migrated OPNsense\Wireguard\Client from 0.0.6 to 0.0.7
Reloading plugin configuration

Fatal error: Uncaught Error: Class "phpseclib3\Crypt\Common\AsymmetricKey" not found in /usr/local/share/phpseclib/Crypt/RSA.php:69
Stack trace:
#0 /usr/local/etc/inc/certs.inc(34): require_once()
#1 /usr/local/etc/inc/config.inc(41): require_once('/usr/local/etc/...')
#2 /usr/local/etc/rc.configure_plugins(35): require_once('/usr/local/etc/...')
#3 {main}
  thrown in /usr/local/share/phpseclib/Crypt/RSA.php on line 69
Reloading template OPNsense/Wireguard: configd socket missing (@/var/run/configd.socket)
pkg-static: POST-INSTALL script failed

and

Code Select Expand

[72/74] Extracting os-ddclient-1.9_1: .......... done
configd not running? (check /var/run/configd.pid).
Starting configd.
Unable to lock on the pidfile.
/usr/local/etc/rc.d/configd: WARNING: failed to start configd
Migrated OPNsense\DynDNS\DynDNS from 1.4.0 to 1.5.0
Reloading plugin configuration

Fatal error: Uncaught Error: Class "phpseclib3\Crypt\Common\AsymmetricKey" not found in /usr/local/share/phpseclib/Crypt/RSA.php:69
Stack trace:
#0 /usr/local/etc/inc/certs.inc(34): require_once()
#1 /usr/local/etc/inc/config.inc(41): require_once('/usr/local/etc/...')
#2 /usr/local/etc/rc.configure_plugins(35): require_once('/usr/local/etc/...')
#3 {main}
  thrown in /usr/local/share/phpseclib/Crypt/RSA.php on line 69

Should I worry about these? Do I need to make any manual adjustments to my config?

I'd like to share a trick to solve one of the issue I had using sensei on my APU2.

Hi and thanks for the helpful post. I am a new OPNSense user and also using an APU2 (APU2E4 with 4 GB RAM). Before installing Zenarmor, I'd be really interested in what kind of reporting you have with the MongoDB database. Can you share any examples – either screenshots or just descriptions of the type of reporting that is possible? All the examples I have found on the internet are using Elastic, so I'm curious as to how the interface and data is different with MongoDB.

Thanks!

I am confused by the manual/howto, specifically the text under "Choosing an interface" that mentions NAT and the WAN interface.

Because my ISP's router can't be put in bridge mode, I have set it to regard my OPNSense router as DMZ, so all traffic gets passed through to the OPNSense router and the OPNSense WAN interface gets an IP address on the ISP router. In this situation, can I get proper benefit from activating Suricata on the WAN interface to catch and stop intrusion attempts, or is this not possible?

Updates work fine here from the webinterface

Hmmm...I get an error message "Auto-update failed."

What is the procedure for manual update? I am running OpnSense on an amd64 architecture, so I assume I need to download the latest AdGuardHome_freebsd_amd64.tar.gz from https://github.com/AdguardTeam/AdGuardHome/releases and replace /usr/local/AdGuardHome with it. But I also assume I first need to stop the AdGuard service; how do I do that? I tried ./AdGuardHome -s stop but that gave an error message. (I had already stopped AdGuardHome in the web interface, but on my previous setup that was not enough, the service also had to be stopped over SSH.)