TUI for viewing and analysing OPNsense filter/firewall logs

[patient0]

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

[allddd]

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

I recently got around to doing some benchmarking/profiling of the stream package, and it turns out that the code I added early on to save memory (back when I thought it was a good idea to load everything into memory :D) was actually making everything way slower.

I swapped that code out and replaced some of Go's built ins with faster/simpler functions, and now the performance is much better. A log file with a few million lines now gets indexed almost instantly, and filtering is much much faster than before.

Here are some benchmarks of the function that parses the logs:

before:

Code Select Expand

BenchmarkParse-2    3287289       1102 ns/op      216 B/op       11 allocs/op

after:

Code Select Expand

BenchmarkParse-2    8188898        437.7 ns/op      160 B/op        1 allocs/op

Once I finish refactoring the TUI, I'll look into adding an option to open multiple files at once. With the recent changes, the code should now easily handle multiple files or even large dirs. Also, the TUI now looks much nicer (in my opinion) and is more compact. I've also added a feature where the user can select an entry to view more details that I can't show in the default view, since the filter log contains so much info. The details view is curremtly very minimal, but I'm already working on adding all the fields from the filter log CSV to it.

If you or anyone else is interested in doing some testing, you'll need to compile the development branch for now, as I still need to make a few changes before tagging the release.

[patient0]

It's really a lot faster, well done! I had to record the srceen to see the pop-up with the version :)

And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?

Below is the output (btw, on NetBSD arm64) of a mixed IPv4 and IPv6 view. As you can see it fit's perfectly, with a little room for improvement regarding the 'Source > Destination' column :). And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?

Code Select Expand

Time            Action  Protocol  Interface  Source > Destination
Nov29-00:00:02  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:02  pass    ipv6-icmp <vtnet0    fdaa:b2b4:d8b2:1000::46 > fdaa:b2b4:d8b2:1000::1
Nov29-00:00:02  pass    ipv6-icmp >vtnet0    fdaa:b2b4:d8b2:1000::1 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:03  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:12  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:12  pass    ipv6-icmp >vtnet0    fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:12  pass    ipv6-icmp <vtnet0    fe80::be24:11ff:fe64:7ecf > fe80::d475:84ff:feec:bb35
Nov29-00:00:15  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:16  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:17  pass    ipv6-icmp <vtnet0    fdaa:b2b4:d8b2:1000::46 > fe80::d475:84ff:feec:bb35
Nov29-00:00:17  pass    ipv6-icmp >vtnet0    fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
...
Nov29-00:00:29  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:31  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
position: 1/102735
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter


The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.

Code Select Expand

Time            Action  Protocol  Interface  Source > Destination
Nov29-00:04:11  pass    udp       <vtnet0    10.101.102.46 57429 > 128.140.109.119 123
Nov29-00:04:17  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 37449 > 2a01:239:25e:bd00::1 123
Nov29-00:05:31  pass    udp       <vtnet0    10.101.102.46 21946 > 185.207.105.38 123
Nov29-00:06:51  pass    udp       <vtnet0    10.101.102.46 34219 > 84.16.73.33 123
Nov29-00:07:49  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 44293 > 2a01:4f8:120:14ed:1::100 123
Nov29-00:08:01  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 10256 > 2603:c020:8017:3eee:123:123:123:123 123
Nov29-00:08:13  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 11354 > 2a02:168:420b:4::7b:12 123
Nov29-00:08:21  pass    udp       <vtnet0    10.101.102.46 52554 > 85.195.210.125 123
Nov29-00:10:59  pass    udp       >vtnet2    fe80::be24:11ff:fe19:804e 546 > ff02::1:2 547
Nov29-00:10:59  pass    udp       <vtnet2    fe80::be24:11ff:fe6b:7a06 547 > fe80::be24:11ff:fe19:804e 546
...
Nov29-00:34:54  pass    udp       <vtnet0    10.101.102.46 48905 > 85.195.210.125 123
Nov29-00:39:35  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 1345 > 2a01:239:25e:bd00::1 123
Nov29-00:39:35  pass    udp       <vtnet0    10.101.102.46 46626 > 128.140.109.119 123
position: 1/1492 | filter: "udp"
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter | esc: clear

But as said before: I really it and you are doing an tremendous job!

[allddd]

Thanks for checking it out :)

And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?

I don't think a separate column is necessary, since the direction always appears before the interface name and it's therefore just as easy to spot as it would be in a separate column. I'll try adding a space or using another char to see if it improves readability.

And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?

Jumping to the beginning/end would require checking the view length on every render, and with such a compact view it's just not worth it. Preventing scrolling past the right edge has the same issue. I looked at how less handles it to get an idea, and even there you can scroll infinitely, so in my opinion it's fine to do the same here, especially now that horizontal scrolling is rarely needed.

The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.

Yes, as plain text that view isn't very easy to read. Does your terminal support formatting/colors? I haven't updated the screenshot in the repo yet since the view may still change a bit, but I've added formatting that makes it easy to see the difference between IPs and ports:

You cannot view this attachment.

If I make the ">" line up at the same position for all lines, we basically end up with the old view again.