Messages

Hi @zzup,

This looks like a pretty decent system.

I can't see how much in there in ARC memory; but I'm guessing High RAM usage might be due to ZFS ARC cache. When there's available memory, ZFS will want to use it for caching. If it sees that some neighbor processors might need extra memory, it should happily return back some of the ARC memory.

More on this: https://wiki.freebsd.org/ZFSTuningGuide

@doug, sorry that it did not work out for you this time.

Trying to understand what went wrong WRT elasticsearch database, my best guess is our ES installation might be clashing with another Elasticsearch deployment. To our experience, this usually happens if you're also using Michael's community repository. If you also have that repo enabled as well, pkg might be installing the one which is also served by the community repository, instead of Zenarmor-managed ES database.

Is that the case for you?

Hi @fabricioguzzy,

You'll also need to install os-sensei, which is the actual zenarmor package. os-sunnyvalley just enables the repository which serves actual zenarmor packages.

Please see here for detailed instructions:
https://www.zenarmor.com/docs/opnsense/installing/install-zenarmor-ngfw-plugin-on-opnsense-web-ui#zenarmor-next-generation-firewall-plugin-installation

Hi @Taunt9930,

If you don't have any reliability issues, native netmap mode should produce better performance figures. But this will be only noticeable once you hit your CPU limits, under normal load, there should not be much difference (according to our tests and what we've received as feedback from our users).

We've introduced emulated improvements as an effort to provide netmap support for drivers which do not have their native support; as an effort to bring a driver-agnostic reliable netmap capability for all ethernet drivers. So, if you don't have issues with your native mode, no harm using that.

WRT RSS, you should be able to safely use RSS with either native or emulated mode. The only caveat is you won't see the performance improvements RSS is offering till we ship multi-core enabled zenarmor. Zenarmor will continue to run on a single CPU core (all queues for a single interface will be served through a single CPU core).

Hope this information is helpful.

Hi @almodovaris,

Thanks for the video. It was very helpful in understanding what's going on.

It looks like the crash is indeed caused by the 'sysctl' binary while trying to update net.inet.ip.intr_direct_queue_maxlen sysctl value.

Although the crash is not directly caused by zenarmor per se; something along the package install path seems to be triggering an operating system bug leading to an OS panic.

We're trying to reproduce this on our environment.

Hi @opnsenseuser,

You're correct. There were a few bug-fixes on the engine side of things which might have improved things on your end.

One bug was causing some DNS resolutions to fail in an ad-doc manner after the engine has stayed up for about 4-5 days. This was affecting small deployments and was a result of insufficient memory pre-allocation in small deployment tiers.

We advise updating to 1.13.2 if you're running on 1.13.x release series.

For the full list of updates, please see:

https://www.zenarmor.com/docs/support/release-notes#1132---june-14-2023

Hi @jf2001j, @TheForumTroll,

Thank you for sharing your feedback related to the Heartbeat.

We appreciate your feedback. As shared previously, we believe we are lucky to be serving a technically sophisticated and privacy-conscious user community.

Indeed, this is helping us very much in our quest to provide a great security product without sacrificing the privacy of our users. Compliance with applicable regulations, including CCPA and the General Data Protection Regulation (GDPR), is of utmost importance to us.

@jf2001j, as you correctly pointed out, there are several other technical points where heartbeat messages are helping us to improve the quality of the software:

The information about the number of unique installations provide us insights on several key performance metrics:

• Pro-active detection of software problems: After every release, an unexpected drop  in the number of deployments generally signals us that there might be a software problem affecting some of our users, which allows us to act proactively and start a remediation process without any delay which might impact an even greater number of users.
• Ability to understand how new capabilities shipped with a new release is perceived by our users: We are able to assess this from an increase in the rate of new deployments after a software release.
• Understanding how the software is performing on a newly introduced vs existing platforms: The platform information in the heartbeat provides us this information.
• Understanding which releases are being used and on which platforms & versions: so that we can decide safe EOL dates for supporting individual platforms and releases
• Understanding which platforms Zenarmor is being deployed on, helps us focus our porting and integration efforts.

Also, the number of unique installations information allows us to provide this KPI to our investors so that they have an understanding of how the software is perceived in the field. This is one of the reasons with which we can justify a free tier in the eyes of our investors.

We take it as homework to add this information to the help section of the Heartbeat selection.

Regarding whether IP addresses and similar information which do not directly identify the user:

While we were working on the latest version of the PP and ToS, although there are several interpretations whether it's personal information or not, we decided that it would be safer if we handled them as sensitive information.

To that end, we've placed the necessary security mechanisms in place. The relevant sections in the Privacy Policy ensures that this information is legally safeguarded.

We respect an individual's right to have a differing opinion with our view of this data as information  necessary to maintain our quality of service to our installed user base. We take very seriously our responsibility to protect all data from any purpose other than what's been stated. We appreciate that individuals or organizations may not wish to share any data when using our service. In this case we understand their decision to uninstall and discontinue the use of our software.

Your feedback is well noted and appreciated. We'll continue to carefully consider your input in our decision making processes as we continue providing the best possible service to our customers. Thank you.

@rfc805, @almodovaris, @TritonB7

Can you reach out to Zenarmor support? Use the "Bug Report" screen on the upper right hand corner of the screen - in case you're not familiar.

Make sure you send all OPN/Zenarmor config and logs.

This looks pretty much like another netmap thing. We want to investigate further.

Thanks.

Quick question: were you upgrading from 23.1.6 or from an earlier OPNsense version?

This seems netmap-related and I wonder if you were upgrading from an earlier version of OPNsense where the netmap emulated fixes were not there...

Hi @FullyBorked,

Yes, we're shipping this capability with 1.13 (OPNsense UI only). You'll be able to import your own CA key/cert.

Would that be helpful?

Hi @doug_phoenix,

Sorry to hear that. It looks like you're hitting a nasty kernel bug which might be triggered by netmap.

Any chances you can type trace in the ddb prompt and send us the kernel stack trace? That will be a lot more helpful.

Hi @DoBoY,

In our experience, intel is still a good choice in terms of driver compatibility. It's just that igc is fairly new.

The path we'll be following as @beki mentioned, will be a driver-agnostic approach. Roughly almost 95% of the time, the issue is with a new driver introduction (like the case of igc) or a driver update which breaks netmap compatibility.

In that regard, working with OPNsense and Klara teams, we've decided to improve netmap's emulated mode (which works driver agnostic) so that it performs reasonably equally well for all drivers. This work is still under development and hopefully if we can reach our goals, driver compatibility should be a discussion of history.

Correct.

You need all offloading settings and VLAN HW filtering disabled if you want to run Suricata in IPS mode/Zenarmor.

This is a netmap limitation. Netmap is the packet I/O interface that Suricata uses to grab packets off the wire. 

Not sure, why it was working before or if/how your settings got changed.

Hi @b.unkel, do you have interface offloadings disabled? It looks that you have checksum offloading enabled for igb2.

@SpinningRust, thanks for the feedback.

Which ethernet were you using for the ZA protected interface? Were there any VLANs involved?