TUI for viewing and analysing OPNsense filter/firewall logs

[allddd]

Hi everyone

Since I couldn't find a simple way to quickly analyse filter logs, I've been writing a TUI in Go for this over the past few months. It's come a lot further than I first expected, so I decided to share it in case anyone else needs something like this.

It's called opnsense-filterlog and it's basically a TUI similar to a pager like less in terms of navigation, but with color output and search/filtering features that are better suited for firewall logs. The filter syntax is similar to tcpdump and pretty simple, but still lets you filter out all log entries you (don't) want to see, e.g.:

(src 192.168.1.1 or src 192.168.1.2) and action block and not proto udp

It's a simple binary with no dependencies that runs on OPNsense itself. I tried to make it as memory and resource efficient as possible, so it should be able to handle huge log files, even on low-spec devices.

In case anyone is interested, there is more documentation in the repo: https://gitlab.com/allddd/opnsense-filterlog

Figured I'd share this here, maybe it'll save someone a bit of time digging through logs.

[patient0]

Thanks for posting the viewer, I gave it a go and do like it.  I like the navigation in the TUI.

If I could have a wish :) or two:

• my screen is quite small (1280x800) and not all columns fit on the screen. It would be helpful if I could scroll horizontally with e.g. either the left/right arrow keys and/or 'h'/'l' (like in vim).
• right now filtering for 'proto ip6' doesn't show any results. But filtering for 'proto ip' shows only the ip6 traffic. I would prefer if 'proto ip' would show the ipv4 entries and 'proto ip6' the ipv6. Maybe even a shortcut like in 'pftop' 'ip' and 'ip6' showing the ipv4 and ipv6 entries.

[allddd]

I'm glad you liked it :)

my screen is quite small (1280x800) and not all columns fit on the screen. It would be helpful if I could scroll horizontally with e.g. either the left/right arrow keys and/or 'h'/'l' (like in vim).

This is already on my todo list because, even with larger screens, it's an issue if the terminal is not running in fullscreen mode, which is often the case. I even have a bit of code for this in a local branch, but I haven't really decided what would be the best way to do this.

One approach would be to dynamically truncate the columns based on window size, but that would cause an issue on smaller screens where you could not see part of the date, IP, etc., which isn't ideal.

Another approach, as you mentioned, would be to implement horizontal scrolling. This would be more tricky to implement and might not look as good, but at least it would not cut off parts of IPs or other fields.

right now filtering for 'proto ip6' doesn't show any results. But filtering for 'proto ip' shows only the ip6 traffic. I would prefer if 'proto ip' would show the ipv4 entries and 'proto ip6' the ipv6. Maybe even a shortcut like in 'pftop' 'ip' and 'ip6' showing the ipv4 and ipv6 entries.

Currently, it is not possible to filter based on IP version, but adding this as an option would be easy. Documentation on the filter.log format:

Code Select Expand

IPv4
====

[Packetfilter], ipversion, tos, ecn, ttl, id, offset, flags, protonum, protoname, length, src, dst

The protonum/protoname order is reversed compared to IPv6.

IPv6
====

[Packetfilter], ipversion, class, flow, hoplimit, protoname, protonum, length, src, dst

The protonum/protoname order is reversed compared to IPv4.

The proto filter is used to filter by protoname. The reason you get any results with a filter query such as proto ip, is because some protocol names contain ip* (e.g. ipv6-icmp) and the value does not have to be an exact match. To implement this, I would either have to abuse the proto keyword or add a new one used specifically for matching the ipversion field. The latter option would probably be less confusing.

If you have a Gitlab account, feel free to open an issue if you notice any bugs or have suggestions.

[patient0]

This is already on my todo list because, even with larger screens ... dynamically truncate the columns ...  implement horizontal scrolling

I think there is no way of truncating without loosing information. And with scrolling that would be prevented, like pftop does it. It scrolls by columns which is working excellent for me.

filter based on IP version, but adding this as an option would be easy

Since I'm a slow learner I like it if I can use my knowledge of another app, pfTop in that case. Something like 'ipver' or even 'ip' and 'ip6' without a field name. But that's only because I often want to look at either IPv4 or IPv6 traffic.

If you have a Gitlab account, feel free to open an issue if you notice any bugs or have suggestions.

I do have an GL account and will do that.

[allddd]

I've added both horizontal scrolling and IP version filtering in v0.3.0.

For now, the IP version filter works in the same way as any other field filter ("field value" syntax). Something like ip4/ip6 is also possible, but I'll look into it later because I have to make some small-ish changes to the filter module first. For docs on IP version filtering see https://gitlab.com/allddd/opnsense-filterlog#filter.

[patient0]

Thank you, I compiled and copied it over to a OPNsense instance. It's much easier now the scrolling and 'ip (4|6)' filtering. And the filter expression can be modified.

The filtering is remarkable fast. I run one of the OPNsense instances on a VM with 2 CPUs/4GB RAM on Proxmox. The filter file from yesterday has around 102'000 entries.

It takes around 7 seconds to start up with the filter file and filtering for 'ip 4' which results in 87'000 items is instant, execellent.

Edit: it takes around 7 seconds if started from the VM console. Running it from a SSH session, it only takes about 1.5 to 3 seconds to start.

[franco]

Hi allddd,

Nice work on this!  If you want we can work on including this in a future release as an optional binary package and see how it goes from there?


Cheers,
Franco

[allddd]

Hi allddd,

Nice work on this!  If you want we can work on including this in a future release as an optional binary package and see how it goes from there?


Cheers,
Franco

Hi Franco,

Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.

[allddd]

And the filter expression can be modified.

I finally replaced my input field implementation with the one provided by the TUI library I use, it now supports all standard key bindings. Tbh, I should've done that from the beginning...

The filtering is remarkable fast.

Should be even faster now with v0.4.0. I've optimized both the filter and streamer code and removed some redundant stuff left over from when I first started writing this.

The filter file from yesterday has around 102'000 entries.

Shouldn't be an issue. I'm mainly testing on a few busy OPNsense instances I run, where even on quiet days, the filter log has a few million entries.

It takes around 7 seconds to start up with the filter file and filtering for 'ip 4' which results in 87'000 items is instant, execellent.

Edit: it takes around 7 seconds if started from the VM console. Running it from a SSH session, it only takes about 1.5 to 3 seconds to start.

Yeah it's not bad, but I'm still trying to figure out how to make it better. Right now, the loading screen remains visible until the file is indexed because some functions (e.g. filtering) don't work without the index.

One way to improve it could be to skip the loading screen and go straight to the TUI, with indexing happening in the background. The advantage of this would be that the TUI opens instantly, and while some features might not be available right away, most users probably wouldn't even notice this since indexing is quick.

Another option, at least in theory, would be to skip building the index altogether and just process everything on the fly. We could cache the lines we've already interacted with to avoid doing the same thing over and over again. However, I'm not sure how well this would perform on low spec devices.

[Seimus]

@allddd

Let me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.

Well done!

Regards,
S.

[patient0]

Should be even faster now with v0.4.0.

It certainly feels faster. And in general I very much admire the fact that you update the application and the documentation!

One way to improve it could be to skip the loading screen and go straight to the TUI, with indexing happening in the background. The advantage of this would be that the TUI opens instantly, and while some features might not be available right away, most users probably wouldn't even notice this since indexing is quick.

It would be helpful if it could read multiple file or a directory and then it may become more of an issue.

You got time until someone presses <Enter> in a filter field :)

Another option, at least in theory, would be to skip building the index altogether and just process everything on the fly. We could cache the lines we've already interacted with to avoid doing the same thing over and over again. However, I'm not sure how well this would perform on low spec devices.

That would need some testing, I assume that it will be very slow with lots of data. It would not only depend on the CPU/RAM specs but also on the read speed of the disk.

[allddd]

It would be helpful if it could read multiple file or a directory

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. Since we need to show entries from multiple files in one view, I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files. For now I'll stick with the single file limit until I find better solution. I'm open to any suggestions though.

Let me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.

Well done!

You're welcome :) Looking through the filter logs was bad enough, but searching through them was even worse. CSV is not a very human friendly format, and it isn't good for scripting either when the order of values changes with each entry...

I just released v0.5.0, which adds support for JSON output. E.g.:

Code Select Expand

opnsense-filterlog -j
opnsense-filterlog -j /path/to/filter.log

You can also filter in the same way as in the TUI by using the -f flag, e.g.:

Code Select Expand

opnsense-filterlog -j -f '(dport 80 or dport 443) and proto tcp and not action pass'


[Seimus]

Nice!

I hope this gets into a official package (But honestly having this inbuilt in OPN would be even more awesome). If it does it will reach larger audience and I believe it will help a lot of users.

Regards,
S.

[patient0]

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. ... I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files.

Wouldn't it be enough to read the first and the last line(s) of a file, determine the time stamps of these entries and you'd know the order of the files?