Messages

I am in the process of printing up some rack mounting brackets and will be upgrading this WAN gateway to have more hardware resources and be highly-available via 3-node virtualized fail-over. Once something has my attention I prefer to rework it to not have to deal with it for another 5-10 years. lol

In the meantime I uninstalled ZA since we weren't getting a lot out of it anwyay. Post-upgrade I will be able to load it up again and with a lot more filtering rules.

I will report back if I continue to see high CPU usage after the upgrade. Assume no news is good news.

ty

4 CPU system. Big pipes but not a lot of traffic. Very edge-environment. I have been thinking about replacing it but that's another conversation. I've been running OPN + ZenArmor on it for years and never seen lighttpd red line the CPU for days.

I'm just digging around now to see if FBSD has an equiv to strace/ltrace/eBPF. Not seeing anything in the logs.

EDIT: 2 CPUs; 4 cores, I guess.

I just updated and rebooted and lighttpd is still idling north of 90% even without any connections to the management interface.

I'm not at liberty to share a lot of detail but I have 3 10Gb interfaces, relatively few custom fw rules, but 2 outbound VPN tunnels (Wireguard). Also a mostly stock ZenArmor config.

franco,

When you get a chance can you let me know what sizes might qualify as "large"? I'll then confirm fix after upgrade if I'm in the "large" category.

thx

Hey, folks.

I did my periodic update/upgrade this week and since doing so I've been seeing very high CPU usage by lighttpd. I noticed because ZenArmor reporting and other reporting interfaces are struggling to render in the admin interface. I've had some odd behavior related to ZA database in the past so I reset the database to fresh and rebooted but I'm still seeing high CPU and laggy interface.

The system load is ~ 1.5 or so so not terrible.

[1] CPU: Intel(R) Core(TM) i3-4360 CPU @ 3.70GHz (3691.56-MHz K8-class CPU)
hw.ncpu: 4
hw.physmem: 8432918528

Thoughts?

I was never able to shake loose enough free time to troubleshoot this properly. I just uninstalled Zenarmor, re-installed, and set the data retention periods to 24 hours. Hopefully that resolves and I can actually rely on the web filtering stuff again.

I had the Zenarmor section of the OPNsense admin console open and I was served a Zenarmor ad (For, like, OSX features? I don't run OSX and I don't care.). Is this normal? I find it rather distasteful.

So I'm investigating if that is actually the correct way to set the gateway IP.

Going off of what is in the official docs, I did not use the DNS server's IP address as the gateway. I instead did the -1 trick off of the Tunnel Address used in the WG Instance setup. However, the docs say to enter the -1 address in the Instance config itself and in newer OPN versions that setting is actually in the Gateway setting under IP Address.

I find this part of the setup to be really confusing. Particularly in the docs where it says that IP address is essentially arbitrary. I cannot figure out why I cannot get Gateway Monitoring to work using the Endpoint Address.

Also, even if I Reset State, I often have to reboot the firewall in order to get the firewall to route traffic out any of the WG tunnels.

For reference, I seem to be able to LB across 3 different Mullvad tunnels now. That may sound excessive however I'm playing region + ASN routing games with various sites and this is the most simple solution I have been able to formulate.

As soon as i remove the second gateway, disable the second wg interface and disable the second wg instance, everything works perfectly fine again.

I took some screenshots of my config: Screenshots

Do you have any idea for a solution to my problem?

Thank you very much for your help :)

That is hands-down the best and most up-to-date tutorial I have found on the topic. However, like you, mine breaks when I try to add a 2nd tunnel -- eventual goal is LB over multiples.

The biggest difference I see between the linked tutorial and some of the other guides I have found is that Andrew is using the IP address of what is listed as the DNS server in the WG config. Multiple Gateways with the same gateway IP address is not going to work. OPN won't even allow that as a config.

So I'm investigating if that is actually the correct way to set the gateway IP.

Unfortunately I'm back to nightly shutdowns.

I'm looking at what I should install on OPNSense and in my network for log and performance monitoring. Stand by...

Hi,

This is a temporary partition used to store trace logs for device updates and other activities. To identify the largest folder, navigate to the `/usr/local/zenarmor/run/tracefs/{interface_name}` directory and use the command `du -sh *`. Once identified, you can remove the folder if necessary. Zenarmor retains this data for troubleshooting purposes.

Alternatively, you can increase the partition size by going to Settings > Reporting & Data > Database and adjusting the Tracefs Partition Size. It is recommended to set it to 150 MB to adequately store trace data.

Confirming... I bumped up default values for Memory Disk Size and Tracefs Partition Size significantly and ZenArmor was still running as of this morning. So that's a good sign.

I didn't find much in the docs about doing this for small installations. Some somewhat related material for enterprise deployments.

Thanks for your help, @sy.

Sorry, for some reason I'm not getting notification emails from the forum about watched threads. Just happened to login for another reason and see this...

Re RAM disk, sorry I'm not a BSD person and so I'm not totally sure how to check about this and I'm not finding docs on it either. I don't **think** I have RAM disks except perhaps for this:

    tmpfs                        102400      252   102148     0%    /usr/local/zenarmor/run/tracefs

In the web console under Settings/System/Misc none of the RAM disk options are enabled. I did just enable swap there however.

This is as been ongoing issue for several months now and I'm not sure what to make of it. My Zenarmor Home shuts down every night due to what appears to be a false-positive warning about disk usage on the OPNsense box. In my monitoring I see no indication that any of the volumes on the firewall are going above 50%.

Other than this being a bit of a hassle -- restart every morning -- it also means that my family web filtering is not effective unless I stay on top of things daily.

Anyone else seeing this?

Previous update was in early February but I just updated as I submitted this post. If I don't say otherwise, assume the problem persists even with today's updates applied.

# OPNsense details:

Type opnsense
Version 25.1.3
Architecture amd64
Commit 6aa1d97b1
Mirror https://pkg.opnsense.org/FreeBSD:14:amd64/25.1
Repositories OPNsense (Priority: 11), SunnyValley (Priority: 7), mimugmail (Priority: 5)
Updated on Fri Mar 21 08:39:02 MDT 2025
Checked on N/A

# Zenarmor details

os-sensei (installed) 1.18.6 207MiB 3 SunnyValley Enterprise Security Extensions for OPNsense (ZENARMOR)
os-sensei-updater (installed) 1.17 4.03KiB 3 SunnyValley OPNsense ZENARMOR Plugin Updater
os-sunnyvalley (installed) 1.4_3 2.44KiB 3 OPNsense Vendor Repository for Zenarmor (a.k.a Sensei, Next Generation Firewall Extensions)
os-sensei-agent 1.18.6 115MiB 3 SunnyValley ZENARMOR (Sensei) Connectivity Agent for Cloud Central Management


thx

I see a few threads here which look pretty useful for helping validate some (too?) clever things I'm trying to do with Wireguard site-to-site VPNs and I will dive into those shortly. However, what I'm not finding is a lot of collateral on how to do low-level troubleshooting at the CLI. That is going to be the highest fidelity approach in some situations because the management console presents only a subset of the available info for debugging.

Does anyone have any suggestions on a resource? Primarily just the usual FreeBSD docs?

thx

+1 from me.

Until I get a more infra-wide dashboard in place I have same report up on a spare monitor and would love to ignore spikes from re-scaling the Y axis.