TUI for viewing and analysing OPNsense filter/firewall logs

[patient0]

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

[allddd]

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

I recently got around to doing some benchmarking/profiling of the stream package, and it turns out that the code I added early on to save memory (back when I thought it was a good idea to load everything into memory :D) was actually making everything way slower.

I swapped that code out and replaced some of Go's built ins with faster/simpler functions, and now the performance is much better. A log file with a few million lines now gets indexed almost instantly, and filtering is much much faster than before.

Here are some benchmarks of the function that parses the logs:

before:

Code Select Expand

BenchmarkParse-2    3287289       1102 ns/op      216 B/op       11 allocs/op

after:

Code Select Expand

BenchmarkParse-2    8188898        437.7 ns/op      160 B/op        1 allocs/op

Once I finish refactoring the TUI, I'll look into adding an option to open multiple files at once. With the recent changes, the code should now easily handle multiple files or even large dirs. Also, the TUI now looks much nicer (in my opinion) and is more compact. I've also added a feature where the user can select an entry to view more details that I can't show in the default view, since the filter log contains so much info. The details view is curremtly very minimal, but I'm already working on adding all the fields from the filter log CSV to it.

If you or anyone else is interested in doing some testing, you'll need to compile the development branch for now, as I still need to make a few changes before tagging the release.

[patient0]

It's really a lot faster, well done! I had to record the srceen to see the pop-up with the version :)

And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?

Below is the output (btw, on NetBSD arm64) of a mixed IPv4 and IPv6 view. As you can see it fit's perfectly, with a little room for improvement regarding the 'Source > Destination' column :). And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?

Code Select Expand

Time            Action  Protocol  Interface  Source > Destination
Nov29-00:00:02  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:02  pass    ipv6-icmp <vtnet0    fdaa:b2b4:d8b2:1000::46 > fdaa:b2b4:d8b2:1000::1
Nov29-00:00:02  pass    ipv6-icmp >vtnet0    fdaa:b2b4:d8b2:1000::1 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:03  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:12  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:12  pass    ipv6-icmp >vtnet0    fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
Nov29-00:00:12  pass    ipv6-icmp <vtnet0    fe80::be24:11ff:fe64:7ecf > fe80::d475:84ff:feec:bb35
Nov29-00:00:15  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:16  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:17  pass    ipv6-icmp <vtnet0    fdaa:b2b4:d8b2:1000::46 > fe80::d475:84ff:feec:bb35
Nov29-00:00:17  pass    ipv6-icmp >vtnet0    fe80::d475:84ff:feec:bb35 > fdaa:b2b4:d8b2:1000::46
...
Nov29-00:00:29  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
Nov29-00:00:31  block   carp      >vtnet0    10.101.102.9 > 224.0.0.18
position: 1/102735
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter


The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.

Code Select Expand

Time            Action  Protocol  Interface  Source > Destination
Nov29-00:04:11  pass    udp       <vtnet0    10.101.102.46 57429 > 128.140.109.119 123
Nov29-00:04:17  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 37449 > 2a01:239:25e:bd00::1 123
Nov29-00:05:31  pass    udp       <vtnet0    10.101.102.46 21946 > 185.207.105.38 123
Nov29-00:06:51  pass    udp       <vtnet0    10.101.102.46 34219 > 84.16.73.33 123
Nov29-00:07:49  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 44293 > 2a01:4f8:120:14ed:1::100 123
Nov29-00:08:01  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 10256 > 2603:c020:8017:3eee:123:123:123:123 123
Nov29-00:08:13  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 11354 > 2a02:168:420b:4::7b:12 123
Nov29-00:08:21  pass    udp       <vtnet0    10.101.102.46 52554 > 85.195.210.125 123
Nov29-00:10:59  pass    udp       >vtnet2    fe80::be24:11ff:fe19:804e 546 > ff02::1:2 547
Nov29-00:10:59  pass    udp       <vtnet2    fe80::be24:11ff:fe6b:7a06 547 > fe80::be24:11ff:fe19:804e 546
...
Nov29-00:34:54  pass    udp       <vtnet0    10.101.102.46 48905 > 85.195.210.125 123
Nov29-00:39:35  pass    udp       <vtnet0    fdaa:b2b4:d8b2:1000::46 1345 > 2a01:239:25e:bd00::1 123
Nov29-00:39:35  pass    udp       <vtnet0    10.101.102.46 46626 > 128.140.109.119 123
position: 1/1492 | filter: "udp"
q: quit | hjkl: move | ud: page | gG: jump | enter: details | /: filter | esc: clear

But as said before: I really it and you are doing an tremendous job!