Hi everyone
Since I couldn't find a simple way to quickly analyse filter logs, I've been writing a TUI in Go for this over the past few months. It's come a lot further than I first expected, so I decided to share it in case anyone else needs something like this.
It's called opnsense-filterlog and it's basically a TUI similar to a pager like less in terms of navigation, but with color output and search/filtering features that are better suited for firewall logs. The filter syntax is similar to tcpdump and pretty simple, but still lets you filter out all log entries you (don't) want to see, e.g.:
(src 192.168.1.1 or src 192.168.1.2) and action block and not proto udp
It's a simple binary with no dependencies that runs on OPNsense itself. I tried to make it as memory and resource efficient as possible, so it should be able to handle huge log files, even on low-spec devices.
In case anyone is interested, there is more documentation in the repo: https://gitlab.com/allddd/opnsense-filterlog (https://gitlab.com/allddd/opnsense-filterlog)
Figured I'd share this here, maybe it'll save someone a bit of time digging through logs.
Thanks for posting the viewer, I gave it a go and do like it. I like the navigation in the TUI.
If I could have a wish :) or two:
- my screen is quite small (1280x800) and not all columns fit on the screen. It would be helpful if I could scroll horizontally with e.g. either the left/right arrow keys and/or 'h'/'l' (like in vim).
- right now filtering for 'proto ip6' doesn't show any results. But filtering for 'proto ip' shows only the ip6 traffic. I would prefer if 'proto ip' would show the ipv4 entries and 'proto ip6' the ipv6. Maybe even a shortcut like in 'pftop' 'ip' and 'ip6' showing the ipv4 and ipv6 entries.
I'm glad you liked it :)
Quote from: patient0 on November 27, 2025, 10:24:48 AMmy screen is quite small (1280x800) and not all columns fit on the screen. It would be helpful if I could scroll horizontally with e.g. either the left/right arrow keys and/or 'h'/'l' (like in vim).
This is already on my todo list because, even with larger screens, it's an issue if the terminal is not running in fullscreen mode, which is often the case. I even have a bit of code for this in a local branch, but I haven't really decided what would be the best way to do this.
One approach would be to dynamically truncate the columns based on window size, but that would cause an issue on smaller screens where you could not see part of the date, IP, etc., which isn't ideal.
Another approach, as you mentioned, would be to implement horizontal scrolling. This would be more tricky to implement and might not look as good, but at least it would not cut off parts of IPs or other fields.
Quote from: patient0 on November 27, 2025, 10:24:48 AMright now filtering for 'proto ip6' doesn't show any results. But filtering for 'proto ip' shows only the ip6 traffic. I would prefer if 'proto ip' would show the ipv4 entries and 'proto ip6' the ipv6. Maybe even a shortcut like in 'pftop' 'ip' and 'ip6' showing the ipv4 and ipv6 entries.
Currently, it is not possible to filter based on IP version, but adding this as an option would be easy. Documentation on the filter.log format:
IPv4
====
[Packetfilter], ipversion, tos, ecn, ttl, id, offset, flags, protonum, protoname, length, src, dst
The protonum/protoname order is reversed compared to IPv6.
IPv6
====
[Packetfilter], ipversion, class, flow, hoplimit, protoname, protonum, length, src, dst
The protonum/protoname order is reversed compared to IPv4.
The
proto filter is used to filter by
protoname. The reason you get any results with a filter query such as
proto ip, is because some protocol names contain
ip* (e.g. ipv6-icmp) and the value does not have to be an exact match. To implement this, I would either have to abuse the
proto keyword or add a new one used specifically for matching the
ipversion field. The latter option would probably be less confusing.
If you have a Gitlab account, feel free to open an issue if you notice any bugs or have suggestions.
QuoteThis is already on my todo list because, even with larger screens ... dynamically truncate the columns ... implement horizontal scrolling
I think there is no way of truncating without loosing information. And with scrolling that would be prevented, like pftop does it. It scrolls by columns which is working excellent for me.
Quotefilter based on IP version, but adding this as an option would be easy
Since I'm a slow learner I like it if I can use my knowledge of another app, pfTop in that case. Something like 'ipver' or even 'ip' and 'ip6' without a field name. But that's only because I often want to look at either IPv4 or IPv6 traffic.
QuoteIf you have a Gitlab account, feel free to open an issue if you notice any bugs or have suggestions.
I do have an GL account and will do that.
I've added both horizontal scrolling and IP version filtering in v0.3.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases).
For now, the IP version filter works in the same way as any other field filter ("field value" syntax). Something like ip4/ip6 is also possible, but I'll look into it later because I have to make some small-ish changes to the filter module first. For docs on IP version filtering see https://gitlab.com/allddd/opnsense-filterlog#filter (https://gitlab.com/allddd/opnsense-filterlog#filter).
Thank you, I compiled and copied it over to a OPNsense instance. It's much easier now the scrolling and 'ip (4|6)' filtering. And the filter expression can be modified.
The filtering is remarkable fast. I run one of the OPNsense instances on a VM with 2 CPUs/4GB RAM on Proxmox. The filter file from yesterday has around 102'000 entries.
It takes around 7 seconds to start up with the filter file and filtering for 'ip 4' which results in 87'000 items is instant, execellent.
Edit: it takes around 7 seconds if started from the VM console. Running it from a SSH session, it only takes about 1.5 to 3 seconds to start.
Hi allddd,
Nice work on this! If you want we can work on including this in a future release as an optional binary package and see how it goes from there?
Cheers,
Franco
Quote from: franco on December 02, 2025, 11:20:09 AMHi allddd,
Nice work on this! If you want we can work on including this in a future release as an optional binary package and see how it goes from there?
Cheers,
Franco
Hi Franco,
Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an
install target? A man page would also be nice.
Quote from: patient0 on November 30, 2025, 11:42:07 AMAnd the filter expression can be modified.
I finally replaced my input field implementation with the one provided by the TUI library I use, it now supports all standard key bindings. Tbh, I should've done that from the beginning...
Quote from: patient0 on November 30, 2025, 11:42:07 AMThe filtering is remarkable fast.
Should be even faster now with v0.4.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases/v0.4.0). I've optimized both the filter and streamer code and removed some redundant stuff left over from when I first started writing this.
Quote from: patient0 on November 30, 2025, 11:42:07 AMThe filter file from yesterday has around 102'000 entries.
Shouldn't be an issue. I'm mainly testing on a few busy OPNsense instances I run, where even on quiet days, the filter log has a few million entries.
Quote from: patient0 on November 30, 2025, 11:42:07 AMIt takes around 7 seconds to start up with the filter file and filtering for 'ip 4' which results in 87'000 items is instant, execellent.
Edit: it takes around 7 seconds if started from the VM console. Running it from a SSH session, it only takes about 1.5 to 3 seconds to start.
Yeah it's not bad, but I'm still trying to figure out how to make it better. Right now, the loading screen remains visible until the file is indexed because some functions (e.g. filtering) don't work without the index.
One way to improve it could be to skip the loading screen and go straight to the TUI, with indexing happening in the background. The advantage of this would be that the TUI opens instantly, and while some features might not be available right away, most users probably wouldn't even notice this since indexing is quick.
Another option, at least in theory, would be to skip building the index altogether and just process everything on the fly. We could cache the lines we've already interacted with to avoid doing the same thing over and over again. However, I'm not sure how well this would perform on low spec devices.
@allddd
Let me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.
Well done!
Regards,
S.
Quote from: allddd on December 05, 2025, 10:24:05 PMShould be even faster now with v0.4.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases/v0.4.0).
It certainly feels faster. And in general I very much admire the fact that you update the application and the documentation!
QuoteOne way to improve it could be to skip the loading screen and go straight to the TUI, with indexing happening in the background. The advantage of this would be that the TUI opens instantly, and while some features might not be available right away, most users probably wouldn't even notice this since indexing is quick.
It would be helpful if it could read multiple file or a directory and then it may become more of an issue.
You got time until someone presses <Enter> in a filter field :)
QuoteAnother option, at least in theory, would be to skip building the index altogether and just process everything on the fly. We could cache the lines we've already interacted with to avoid doing the same thing over and over again. However, I'm not sure how well this would perform on low spec devices.
That would need some testing, I assume that it will be very slow with lots of data. It would not only depend on the CPU/RAM specs but also on the read speed of the disk.
Quote from: patient0 on December 07, 2025, 08:34:13 PMIt would be helpful if it could read multiple file or a directory
I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. Since we need to show entries from multiple files in one view, I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files. For now I'll stick with the single file limit until I find better solution. I'm open to any suggestions though.
Quote from: Seimus on December 07, 2025, 01:03:40 PMLet me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.
Well done!
You're welcome :) Looking through the filter logs was bad enough, but searching through them was even worse. CSV is not a very human friendly format, and it isn't good for scripting either when the order of values changes with each entry...
I just released v0.5.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases/v0.5.0), which adds support for JSON output. E.g.:
opnsense-filterlog -j
opnsense-filterlog -j /path/to/filter.log
You can also filter in the same way as in the TUI by using the -f flag, e.g.:
opnsense-filterlog -j -f '(dport 80 or dport 443) and proto tcp and not action pass'
Nice!
I hope this gets into a official package (But honestly having this inbuilt in OPN would be even more awesome). If it does it will reach larger audience and I believe it will help a lot of users.
Regards,
S.
Quote from: allddd on December 07, 2025, 11:43:17 PMI've thought about this, and the main issue is sorting the entries since I can't load everything into memory. ... I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files.
Wouldn't it be enough to read the first and the last line(s) of a file, determine the time stamps of these entries and you'd know the order of the files?
Quote from: patient0 on December 08, 2025, 06:06:19 AMQuote from: allddd on December 07, 2025, 11:43:17 PMI've thought about this, and the main issue is sorting the entries since I can't load everything into memory. ... I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files.
Wouldn't it be enough to read the first and the last line(s) of a file, determine the time stamps of these entries and you'd know the order of the files?
Good idea, the best approach is probably a mix of your suggestion and checking the file creation time.
I'm still not sure how well this will work though since the filter log directory can contain >30GB of files. If indexing takes 10-20 (or more) seconds, it's not something I would consider usable.
Quote from: allddd on December 09, 2025, 08:32:48 PMI'm still not sure how well this will work though since the filter log directory can contain >30GB of files.
What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)
Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.
Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.
Quote from: patient0 on December 10, 2025, 10:40:00 PMQuote from: allddd on December 09, 2025, 08:32:48 PMI'm still not sure how well this will work though since the filter log directory can contain >30GB of files.
What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)
Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.
Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.
I recently got around to doing some benchmarking/profiling of the stream package, and it turns out that the code I added early on to save memory (back when I thought it was a good idea to load everything into memory :D) was actually making everything way slower.
I swapped that code out and replaced some of Go's built ins with faster/simpler functions, and now the performance is much better. A log file with a few million lines now gets indexed almost instantly, and filtering is much much faster than before.
Here are some benchmarks of the function that parses the logs:
before:
BenchmarkParse-2 3287289 1102 ns/op 216 B/op 11 allocs/op
after:
BenchmarkParse-2 8188898 437.7 ns/op 160 B/op 1 allocs/op
Once I finish refactoring the TUI, I'll look into adding an option to open multiple files at once. With the recent changes, the code should now easily handle multiple files or even large dirs. Also, the TUI now looks much nicer (in my opinion) and is more compact. I've also added a feature where the user can select an entry to view more details that I can't show in the default view, since the filter log contains so much info. The details view is curremtly very minimal, but I'm already working on adding all the fields from the filter log CSV to it.
If you or anyone else is interested in doing some testing, you'll need to compile the development branch for now, as I still need to make a few changes before tagging the release.
It's really a lot faster, well done! I had to record the srceen to see the pop-up with the version :)
And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?
Below is the output (btw, on NetBSD arm64) of a mixed IPv4 and IPv6 view. As you can see it fit's perfectly, with a little room for improvement regarding the 'Source > Destination' column :). And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?
Time Action Protocol Interface Source > Destination
Nov29-00:00:02 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:02 pass ipv6-icmp <vtnet0 fdaa:b2b4:d8b2:1000::46 > fdaa:b2b4:d8b2:1000::1
Nov29-00:00:02 pass ipv6-icmp >vtnet0 fdaa:b2b4:d8b2:1000::1 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:03 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:12 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:12 pass ipv6-icmp >vtnet0 fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:12 pass ipv6-icmp <vtnet0 fe80::be24:11ff:fe64:7ecf > fe80::d475:84ff:feec:bb35
Nov29-00:00:15 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:16 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:17 pass ipv6-icmp <vtnet0 fdaa:b2b4:d8b2:1000::46 > fe80::d475:84ff:feec:bb35
Nov29-00:00:17 pass ipv6-icmp >vtnet0 fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
...
Nov29-00:00:29 block carp >vtnet0 10.101.102.9 > 224.0.0.18
Nov29-00:00:31 block carp >vtnet0 10.101.102.9 > 224.0.0.18
position: 1/102735
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter
The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.
Time Action Protocol Interface Source > Destination
Nov29-00:04:11 pass udp <vtnet0 10.101.102.46 57429 > 128.140.109.119 123
Nov29-00:04:17 pass udp <vtnet0 fdaa:b2b4:d8b2:1000::46 37449 > 2a01:239:25e:bd00::1 123
Nov29-00:05:31 pass udp <vtnet0 10.101.102.46 21946 > 185.207.105.38 123
Nov29-00:06:51 pass udp <vtnet0 10.101.102.46 34219 > 84.16.73.33 123
Nov29-00:07:49 pass udp <vtnet0 fdaa:b2b4:d8b2:1000::46 44293 > 2a01:4f8:120:14ed:1::100 123
Nov29-00:08:01 pass udp <vtnet0 fdaa:b2b4:d8b2:1000::46 10256 > 2603:c020:8017:3eee:123:123:123:123 123
Nov29-00:08:13 pass udp <vtnet0 fdaa:b2b4:d8b2:1000::46 11354 > 2a02:168:420b:4::7b:12 123
Nov29-00:08:21 pass udp <vtnet0 10.101.102.46 52554 > 85.195.210.125 123
Nov29-00:10:59 pass udp >vtnet2 fe80::be24:11ff:fe19:804e 546 > ff02::1:2 547
Nov29-00:10:59 pass udp <vtnet2 fe80::be24:11ff:fe6b:7a06 547 > fe80::be24:11ff:fe19:804e 546
...
Nov29-00:34:54 pass udp <vtnet0 10.101.102.46 48905 > 85.195.210.125 123
Nov29-00:39:35 pass udp <vtnet0 fdaa:b2b4:d8b2:1000::46 1345 > 2a01:239:25e:bd00::1 123
Nov29-00:39:35 pass udp <vtnet0 10.101.102.46 46626 > 128.140.109.119 123
position: 1/1492 | filter: "udp"
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter | esc: clear
But as said before: I really it and you are doing an tremendous job!
Thanks for checking it out :)
Quote from: patient0 on December 28, 2025, 02:15:25 PMAnd the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?
I don't think a separate column is necessary, since the direction always appears before the interface name and it's therefore just as easy to spot as it would be in a separate column. I'll try adding a space or using another char to see if it improves readability.
Quote from: patient0 on December 28, 2025, 02:15:25 PMAnd I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?
Jumping to the beginning/end would require checking the view length on every render, and with such a compact view it's just not worth it. Preventing scrolling past the right edge has the same issue. I looked at how
less handles it to get an idea, and even there you can scroll infinitely, so in my opinion it's fine to do the same here, especially now that horizontal scrolling is rarely needed.
Quote from: patient0 on December 28, 2025, 02:15:25 PMThe scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.
Yes, as plain text that view isn't very easy to read. Does your terminal support formatting/colors? I haven't updated the screenshot in the repo yet since the view may still change a bit, but I've added formatting that makes it easy to see the difference between IPs and ports:
Screenshot_2025-12-29_13-34-40.png
If I make the ">" line up at the same position for all lines, we basically end up with the old view again.
Quote from: allddd on December 29, 2025, 01:53:12 PMJumping to the beginning/end would require checking the view length on every render ... I looked at how less handles it to get an idea, and even there you can scroll infinitely...
Fair point and you can plant an easter egg if someone scroll for 10000 characters the OPNsense maskot jumps up :)
QuoteDoes your terminal support formatting/colors? I haven't updated the screenshot in the repo yet since the view may still change a bit, but I've added formatting that makes it easy to see the difference between IPs and ports:
Yep, the formatting work, 'block' is red and the IPs are in bold, that work well. Did you experiment with the ports being in color and/or the direction being bold or in color?
Quote from: patient0 on December 29, 2025, 06:14:21 PMDid you experiment with the ports being in color and/or the direction being bold or in color?
The styling library I use supports adaptive styles, so you can define separate color schemes for light and dark backgrounds (even using different colors for 256-color and truecolor terminals). I wasted so much time trying to find the perfect color scheme, only to find out that half the terminals report the wrong background color... After that, I tried using just bold and faint styles, but even faint doesn't work on light backgrounds...
I've now added colors to ports and IPs in v0.7.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases/v0.7.0), which should hopefully make it more readable. No fancy adaptive styles though, I'm just using 2 colors that should work on both light and dark backgrounds (and don't look that bad on dark).
I also changed the direction indicators to I/O (with space), and it's indeed much more readable. Thanks for the suggestion!
demo.png
Quote from: allddd on January 01, 2026, 04:06:08 PMI've now added colors to ports and IPs in v0.7.0 (https://gitlab.com/allddd/opnsense-filterlog/-/releases/v0.7.0), which should hopefully make it more readable. No fancy adaptive styles though, I'm just using 2 colors that should work on both light and dark backgrounds (and don't look that bad on dark).
It looks great in my terminal with the colors and I|O, thank you!
Quote from: allddd on December 02, 2025, 06:11:25 PMQuote from: franco on December 02, 2025, 11:20:09 AMHi allddd,
Nice work on this! If you want we can work on including this in a future release as an optional binary package and see how it goes from there?
Cheers,
Franco
Hi Franco,
Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.
Is this a likely event for, say, 26.1?
Quote from: passeri on January 03, 2026, 12:59:22 AMQuote from: allddd on December 02, 2025, 06:11:25 PMQuote from: franco on December 02, 2025, 11:20:09 AMHi allddd,
Nice work on this! If you want we can work on including this in a future release as an optional binary package and see how it goes from there?
Cheers,
Franco
Hi Franco,
Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.
Is this a likely event for, say, 26.1?
The project itself is "ready", I've added both the manpage and Makefile targets a few versions ago. I don't know when it will happen though, @franco might be able to say more.
@allddd
I think you could create a PR here that adds it to the opnsense sub directory:
https://github.com/opnsense/ports/tree/master/opnsense
If you need inspiration check out recently added ports there (eg ndp-proxy-go or hostwatch)
Quote from: Monviech (Cedrik) on January 04, 2026, 05:45:37 PM@allddd
I think you could create a PR here that adds it to the opnsense sub directory:
https://github.com/opnsense/ports/tree/master/opnsense
If you need inspiration check out recently added ports there (eg ndp-proxy-go or hostwatch)
https://github.com/opnsense/ports/pull/252
I hope I've gotten everything right, it builds and installs without issue in a clean FreeBSD VM.