Messages

What kind of traffic and host count are we talking about where you manage to practically kill the device?

I run a 1gb download/200 mbps upload, but it wasn't the traffic that was causing the crashes/freezes, it was SQLite litterally strangling the nvme through successive writing requests, since the nvme doesn't have a cache and relies on memory, soon it runs out of "borrowed" cache memory and starts queuing up the write requests, then using all of the swap, then making ZFS ARC memory balloon, Zenarmor didn't like to be waiting for SQLite so everything was super super slow.

I didn't try TMPFS because any reboot would have resulted in me losing my data plus I wanted to kick out anything database related from the firewall, that's how I always functionned, from my experience (even professional) it is never a good idea to place the database and apps on the same server.

Reinstalled yesterday. Ran the update. Everything worked fine. I ran the update on a backup server. Everything worked fine. The only detail that might be of any use is the last thing that appeared before the crash was the refresh.sh script. That's where the system appeared to hang and never came back. I even waited about ten minutes thinking that something complicated might be going on.

Things appear to be back to normal.

git backup is your best friend.
Setting up a git repository will take care not only of uploading your configuration but also show you the differences between versions.

Unrelated, but you're one of the lucky ones, my ISP never liked VLANs, while it works for months at some point the WAN interface going through the VLAN would inexplicably lose the public IP while the other WAN interface (direct connection) remains perfectly fine.

Maybe I'll have to see if my new hardware has an extra nvme slot and I can put in a database drive and just let it get beat on, but also be nice and fast.

That was my plan as well, but my machine doesn't have an extra nvme slot and I didn't want to go through the hassle of reinstalling and moving configurations and the license, so I thought why not use that laying around nvme on Proxmox as a dedicated storage for fast things? that's how it all started, in retro-perspective it was the better decision, so if you have an hypervisor, spinning a vm with a pci pass-through can alleviate shift the load to the vm instead of OPNsense. 

That is something to consider, I hadn't thought about the storage being any part of the slowness. There are other reason to buy the product, the policies flexibility is one of them. And when you find a product that you like, it's nice when you can support it and keep it moving forward.

For me what gave me the hint is these errors  :

Code Select Expand

[52805] 196.658358 [1089] netmap_obj_free          ouch, double free on buffer 18426

[52805] 196.658370 [1089] netmap_obj_free          ouch, double free on buffer 18428

[52805] 196.658381 [1089] netmap_obj_free          ouch, double free on buffer 18432

[52807] igb1: link state changed to UP

[52854] swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101141, size: 65536

[52866] swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2108016, size: 16384

[52874] swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2101141, size: 65536

[52886] swap_pager: indefinite wait buffer: bufobj: 0, blkno: 2108016, size: 16384
Also, I noticed that sometimes the zenarmor live  session dashboard would throw a : Error (200), and that anything related to zenarmor was way too slow (policies, reports), and this had a snowball effect, SQLite has a single writer, it cannot write concurrently write data, having a dramless nvme meant that SQLite would "run out" of IOPS meaning that while locking the database it was also making zenarmor wait and freeze, it  caused a cascading failure where SQLite would get stuck, dragging zenarmor and then the netmap driver making the nics flap very often.

I think the most reliable solution would be to externalize elasticsearch, even if you install it locally you would reserve a big amount of resources to elasticsearch and might hit a performance bottleneck.

I know the Zenarmor team have their reasons for removing remote database support from the free tier plan, but it is almost a must to fully experience the benefits of the product, and this is not mentioned anywhere, and some people, me included, simply underestimate how demanding to run SQLite as a back-end, which means that it would be difficult to really appreciate the free tier without remote database support.

Funny enough, last week I sent them an email to get current pricing for my work system... No replies yet. I have another support contract for a different product that we aren't renewing, and I wanted to see if ZA could fit in its spot.

I do wish they would make a reduced price lab license with full multithreaded support. Not sure how to keep people honest, but it would be nice.

On this part I am cutting them some slack as I think it is a smaller team, so normal to need some time, so far the only ticket I opened was dealt with acceptably swiftly.

As for the multi-core support unless you have some 2.5gb/s or more bandwidth I genuinely don't think it is needed, I have never enabled the option and when I thought about upgrading my CPU I was simply on the wrong troubleshooting path, I was wrongly confusing chocking IOPS with high cpu load, the latter was a direct result of queues forming up as a result of a slow nvme (lacked a dram cache), having a database like SQLite or elasticsearch on a slower nvme/ssd might give the false impression that the CPU is slow to process the traffic.

Is there a recording available for those who couldn't attend?
Thanks

I've put my sqlite wholly into RAM.
From my understanding the dash reads from db and stores those reads as temp files, so dashboards performance gain seem neiglible. Attempted to put that into ram too but it wasn't too happy with it.

Would also really want multicore support but many people in the home user space are demanding everything and anything at nearly no cost and complaining for no good reason. Seriously, ZA is one of the FEW (only?) good web & application filters available anymore.

Been using on and off for years at this point. Came back when Untangle killed their home license.

I'd be happy with ONLY the application + web filter (+ dash/reports ofc) and multicore.

Home pricing is very good for what you're getting when you consider other options. Fuck off with the crocodile tears, companies make pennies with these licenses compared to what enterprise pays.

I've learned that it is better to let the firewall do its thing, and let the vms/containers do their thing.

Having a process constantly writing data, especially a database, will have at some point a CPU load high enough to slow down other functionalities on the firewall, I am not even mentioning IOPS, sqlite was basically chocking my firewall and had to dig in the logs to find the culprit, I wasn't aware of putting sqlite in the ram other than tmpfs which I always found a bit too risky for data I cared about.

But once you externalize the database things become more predictable even under high load, this being said, the free version might not necessarily give the best user experience unless you have some specific hardware (fast nvme with dram cache), but in my experience sqlite+dramless nvme was a pure recipe for headaches, because the free version the last time I tested doesn't offer the option to use an external elasticsearch instance which would definitely have consequences if a user doesn't have the "right" hardware.

Out of curiosity: can't you use an off device Elastic installation? E.g I run Elastic as an "app" on TrueNAS for my FreeBSD jail based Nextcloud to use.

I did use an elasticsearch container running docker/portainer but I just feel it is an overkill for a home lab or even an SMB, sqlite being transactional nothing would stop mysql from replacing it and having far more benefits, as I said before I don't know if there are architectural limitations or roadblocks for the devs, but from my perspective I think it is far better to have a mysql/mariadb back end for performance and redundancy, homelabbers like me would choose mysql/mariadb for its "simplicity" and bigger players could use elasticsearch for their bigger databases.

I used to run a master/slave mariadb cluster years ago, and it was one of the ways to minimize the load on my production server, I can't see that happening with SQLite that lives in the same nvme as opnsense (I know I can always use a mount), or using elasticsearch for that purpose, it just gives me less possibilities overall.

Quick update, I modified my setup's architecture to leave the database outside of opnsense :

- PCI pass-through of a WD nvme with dram to a Proxmox debian vm.
- Run elasticsearch as a container with data volume on PCI pass-through nvme.
- Migrate from sqlite to the newly deployed remote elasticsearch on the same lan as opnsense, no vlans involved.
- Set retention to 7 days.

Results so far :

- No more SQLite database locked (error 200)
- ZFS memory usage back to an acceptable 2gb, it would skyrocket to 6gb or even 8gb when uncapped and when using sqlite.
- CPU runs way cooler now.
- Zenarmor is so far much faster and responsive.
- opnsense stopped hanging and freezing due to the limitation of the dramless nvme drive on which it is installed while running sqlite.

The lesson learned so far is that Zenarmor has some specific requirements to function optimally, outside of those requirements, the results can simply be too unreliable and cause more problems.

[native netmap mode]

If you plan to run Zenarmor in native netmap mode you have to widen the queues to avoid starvation and interfaces crashes down the road, look at my post where I implemented a solution that fixed my native netmap mode that caused incessant interfaces restarts, this fixed my igb interfaces issues and might work for igc providing the nic is compatible.

While the decision is yours, I respect that.
But it is expected to have a basic understanding of how Zenarmor works to really make an informed decision and ditch it.
Some settings are not suitable for some hardware and will break stuff, the provided documentation is far from ideal but solutions do exist and it takes some trial and errors to reach the "sweet spot", you can read my post and/or other members posts to find solutions to issues that people encountered before.
At the end of the day time is expensive and some of us have better uses for their time but if you decide to resume the troubleshooting, help is available here.

Have you tried a dmesg when the video freezes up? the freeze can happen for various reasons, one of them being the zenarmor engine mode used, it can also be an under powered cpu, or not enough ram, there are tons of possibilities and without a capture of those metrics at the time of the issue the best you would end up with are guesses.

Hi all,

I just want to share with you some tunables that fixed repetitive crashes when using zenarmor in native mode, there is a warning that say you should be using the emulated mode, but that wasn't enough of an explanation for me, queuese would run out after a few hours and force an interface restart, after a lot of AI fu I was finally given the correct value, these won't work from the web interface as those are applied once the firewall has started which is already late, this works with i350-t4 cards :

Code Select Expand

cat /boot/loader.conf.local
dev.igb.0.iflib.override_ntxds="4096"
dev.igb.1.iflib.override_ntxds="4096"
dev.igb.2.iflib.override_ntxds="4096"
dev.igb.3.iflib.override_ntxds="4096"
dev.igb.0.iflib.override_nrxds="4096"
dev.igb.1.iflib.override_nrxds="4096"
dev.igb.2.iflib.override_nrxds="4096"
dev.igb.3.iflib.override_nrxds="4096"
dmesg confirms the application of the new values :

Code Select Expand

[31] tun0: changing name to 'zen0'
[33] igb1: link state changed to UP
[34] igb0: link state changed to UP
[34] igb2: link state changed to UP
[35] igb3: link state changed to UP
[42] 137.783848 [ 805] iflib_netmap_config       txr 6 rxr 6 txd 4096 rxd 4096 rbufsz 2048
[42] 137.783906 [ 805] iflib_netmap_config       txr 6 rxr 6 txd 4096 rxd 4096 rbufsz 2048
[42] igb1: link state changed to DOWN
[42] igb0: link state changed to DOWN
[43] 138.429571 [ 805] iflib_netmap_config       txr 6 rxr 6 txd 4096 rxd 4096 rbufsz 2048
[43] 138.540666 [ 805] iflib_netmap_config       txr 6 rxr 6 txd 4096 rxd 4096 rbufsz 2048
[47] igb0: link state changed to UP
[47] igb1: link state changed to UP
as soon as I applied this, my issues were gone.

Out of curiosity: can't you use an off device Elastic installation? E.g I run Elastic as an "app" on TrueNAS for my FreeBSD jail based Nextcloud to use.

I did use an elasticsearch container running docker/portainer but I just feel it is an overkill for a home lab or even an SMB, sqlite being transactional nothing would stop mysql from replacing it and having far more benefits, as I said before I don't know if there are architectural limitations or roadblocks for the devs, but from my perspective I think it is far better to have a mysql/mariadb back end for performance and redundancy, homelabbers like me would choose mysql/mariadb for its "simplicity" and bigger players could use elasticsearch for their bigger databases.

I used to run a master/slave mariadb cluster years ago, and it was one of the ways to minimize the load on my production server, I can't see that happening with SQLite that lives in the same nvme as opnsense (I know I can always use a mount), or using elasticsearch for that purpose, it just gives me less possibilities overall.

I agree with most of what was said in this thread, the other thing I feel very limiting is the number of policies, having just 5 policies can become a serious limitation for the home licence subscribers.

I wish other database back ends were considered, obviously the developers have their own reasons, but I feel between elastic and sqlite the choices are restrictive as both could wear out the disk while having an external mariadb handling the database aspect would make things more flexible.

Anyway, this is unrelated to the first post but thought giving some feedback can help improve the product.