21.7.1 Maltrail OOM / Possible fix

[Leviathan]

Still no official solution to this problem?

[mimugmail]

No

[Leviathan]

After 3 months still no solution... :-\

[mimugmail]

You can try to build a port for pcapy-ng, then we can work on.

[Northguy]

It wont get fixed anytime soon as the pcapy lib is the root cause for it.

If it is pcapy that is offending, you are right. it seems unmaintained (last release 02 Jul 2019). Although Stamparm himself has forked pcapy and is maintaining pcapy-ng. I would assume that this was done to implement improvements in maltrail?

[opn_nwo]

That's too bad. I had to disable the service on both my servers as it was getting too cumbersome to manually kill the runaway processes every day. Hopefully it will get fixed at some point.

[mimugmail]

It wont get fixed anytime soon as the pcapy lib is the root cause for it.

If it is pcapy that is offending, you are right. it seems unmaintained (last release 02 Jul 2019). Although Stamparm himself has forked pcapy and is maintaining pcapy-ng. I would assume that this was done to implement improvements in maltrail?

Yep, but this needs a FreeBSD port to integrate it. Sadly I dont have the time

[javis]

It wont get fixed anytime soon as the pcapy lib is the root cause for it.

If it is pcapy that is offending, you are right. it seems unmaintained (last release 02 Jul 2019). Although Stamparm himself has forked pcapy and is maintaining pcapy-ng. I would assume that this was done to implement improvements in maltrail?

Yep, but this needs a FreeBSD port to integrate it. Sadly I dont have the time

Maltrail was almost completely unusable because of the memory leak. Unfortunately I have some company devices that generate crazy amount of telemetry requests, which greatly worsened the situation: 16GB of RAM and SWAP were completely filled up every half a day before my server box froze! Creating a cron job to restart the sensors actually accelerated the memory leak, because the old processes were simply orphaned rather than killed. This plugin in its current state is bound to crash your network sooner or later.

Thanks to the info in this thread, the fix is actually very simple (one doesn't have to install Python 2 to get it done) if people don't mind installing pip. Just follow Maltrail author's advice of using pcapy-ng, which simply added the PY_SSIZE_T_CLEAN macro definition to the abandoned pcapy codebase.

Code Select Expand


python3 -m ensurepip
pip3 install pcapy-ng


It's been a few days now, and my server box's memory usage has been very stable (around 30% ~ 60% without any swapping).

@mimugmail thank you for porting Maltrail to FreeBSD and creating the OPNsense plugin! If you don't have time bundling pcapy-ng with Maltrail (I'm not familiar with OPNsense development but it looks to me the plugin simply offers a web interface, and the actual Maltrail and its dependencies are pre-bundled into the system? How do folks go about submitting updates to them?), do you think you can revise the plugin description to include a short pcapy-ng installation guide? It shouldn't take one a couple days of crashing and researching to find out the culprit and the way to fix it :D

[mimugmail]

I already added a PR for pcapy-ng in December but noone with Commit bit merges it

[javis]

I already added a PR for pcapy-ng in December but noone with Commit bit merges it

I was trying to find your PR so did some digging (couldn't find it though), is this where OPNsense builds the Maltrail plugin: https://github.com/opnsense/ports/blob/91da3754f16f546456479e3a8790bccff33cf429/security/maltrail/Makefile#L11? Can we simply update this to depend on pcapy-ng or is there something else we need to do (like updating FreeBSD ports upstream etc)?

[mimugmail]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260732

[javis]

Ah right, the GitHub repo is just a mirror, would need to submit a ticket on FreeBSD instead. Thx for bumping up that bug!

[mimugmail]

Can you remind me in April again here? If its still not merged over there we can add it to opn mirror as an OPNsense port

[javis]

Sure, I'll set a reminder for myself on 4th.

[javis]

Can you remind me in April again here? If its still not merged over there we can add it to opn mirror as an OPNsense port

Here's your friendly human alarm clock reporting that the FreeBSD ticket regarding pcapy-ng has been closed / fixed :) However, I started seeing the memory leak again recently so am not entirely sure if it's simply because pcapy-ng still can't work with Python 3 in its current state (although I see some fixes have been pushed to pcapy-ng since last conversation). I have disabled Maltrail for now and will check if the latest port, once available, fixes the memory issues.