Messages

@allddd

I think you could create a PR here that adds it to the opnsense sub directory:

https://github.com/opnsense/ports/tree/master/opnsense

If you need inspiration check out recently added ports there (eg ndp-proxy-go or hostwatch)

https://github.com/opnsense/ports/pull/252

I hope I've gotten everything right, it builds and installs without issue in a clean FreeBSD VM.

Hi allddd,

Nice work on this!  If you want we can work on including this in a future release as an optional binary package and see how it goes from there?

Cheers,
Franco

Hi Franco,

Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.

Is this a likely event for, say, 26.1?

The project itself is "ready", I've added both the manpage and Makefile targets a few versions ago. I don't know when it will happen though, @franco might be able to say more.

Did you experiment with the ports being in color and/or the direction being bold or in color?

The styling library I use supports adaptive styles, so you can define separate color schemes for light and dark backgrounds (even using different colors for 256-color and truecolor terminals). I wasted so much time trying to find the perfect color scheme, only to find out that half the terminals report the wrong background color... After that, I tried using just bold and faint styles, but even faint doesn't work on light backgrounds...

I've now added colors to ports and IPs in v0.7.0, which should hopefully make it more readable. No fancy adaptive styles though, I'm just using 2 colors that should work on both light and dark backgrounds (and don't look that bad on dark).

I also changed the direction indicators to I/O (with space), and it's indeed much more readable. Thanks for the suggestion!

You cannot view this attachment.

Thanks for checking it out :)

And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?

I don't think a separate column is necessary, since the direction always appears before the interface name and it's therefore just as easy to spot as it would be in a separate column. I'll try adding a space or using another char to see if it improves readability.

And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?

Jumping to the beginning/end would require checking the view length on every render, and with such a compact view it's just not worth it. Preventing scrolling past the right edge has the same issue. I looked at how less handles it to get an idea, and even there you can scroll infinitely, so in my opinion it's fine to do the same here, especially now that horizontal scrolling is rarely needed.

The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.

Yes, as plain text that view isn't very easy to read. Does your terminal support formatting/colors? I haven't updated the screenshot in the repo yet since the view may still change a bit, but I've added formatting that makes it easy to see the difference between IPs and ports:

You cannot view this attachment.

If I make the ">" line up at the same position for all lines, we basically end up with the old view again.

Does it have to be an LAN host, or would it be OK for an external service to notify you?

You could use a service like https://healthchecks.io in combination with Monit. This would be even more reliable, since you would receive a notification regardless of whether you are currently using the system or not.

You can configure Monit to send an HTTP request to healthchecks.io every time a check is successful. If it fails for any reason, or if OPNsense cannot reach healthchecks.io at all, you will be notified. They offer a generous free tier, you can even receive calls and SMS.

and triggers an action if either of the upstream gateways is down.

Not sure about the API, but have you considered using Monit? It's designed to do exactly that.
It can notify you, execute a script, or basically do anything else you want it to.

https://docs.opnsense.org/manual/monit.html

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

I recently got around to doing some benchmarking/profiling of the stream package, and it turns out that the code I added early on to save memory (back when I thought it was a good idea to load everything into memory :D) was actually making everything way slower.

I swapped that code out and replaced some of Go's built ins with faster/simpler functions, and now the performance is much better. A log file with a few million lines now gets indexed almost instantly, and filtering is much much faster than before.

Here are some benchmarks of the function that parses the logs:

before:

Code Select Expand

BenchmarkParse-2    3287289       1102 ns/op      216 B/op       11 allocs/op

after:

Code Select Expand

BenchmarkParse-2    8188898        437.7 ns/op      160 B/op        1 allocs/op

Once I finish refactoring the TUI, I'll look into adding an option to open multiple files at once. With the recent changes, the code should now easily handle multiple files or even large dirs. Also, the TUI now looks much nicer (in my opinion) and is more compact. I've also added a feature where the user can select an entry to view more details that I can't show in the default view, since the filter log contains so much info. The details view is curremtly very minimal, but I'm already working on adding all the fields from the filter log CSV to it.

If you or anyone else is interested in doing some testing, you'll need to compile the development branch for now, as I still need to make a few changes before tagging the release.

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. ... I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files.

Wouldn't it be enough to read the first and the last line(s) of a file, determine the time stamps of these entries and you'd know the order of the files?

Good idea, the best approach is probably a mix of your suggestion and checking the file creation time.

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files. If indexing takes 10-20 (or more) seconds, it's not something I would consider usable.

It would be helpful if it could read multiple file or a directory

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. Since we need to show entries from multiple files in one view, I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files. For now I'll stick with the single file limit until I find better solution. I'm open to any suggestions though.

Let me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.

Well done!

You're welcome :) Looking through the filter logs was bad enough, but searching through them was even worse. CSV is not a very human friendly format, and it isn't good for scripting either when the order of values changes with each entry...

I just released v0.5.0, which adds support for JSON output. E.g.:

Code Select Expand

opnsense-filterlog -j
opnsense-filterlog -j /path/to/filter.log

You can also filter in the same way as in the TUI by using the -f flag, e.g.:

Code Select Expand

opnsense-filterlog -j -f '(dport 80 or dport 443) and proto tcp and not action pass'


And the filter expression can be modified.

I finally replaced my input field implementation with the one provided by the TUI library I use, it now supports all standard key bindings. Tbh, I should've done that from the beginning...

The filtering is remarkable fast.

Should be even faster now with v0.4.0. I've optimized both the filter and streamer code and removed some redundant stuff left over from when I first started writing this.

The filter file from yesterday has around 102'000 entries.

Shouldn't be an issue. I'm mainly testing on a few busy OPNsense instances I run, where even on quiet days, the filter log has a few million entries.

It takes around 7 seconds to start up with the filter file and filtering for 'ip 4' which results in 87'000 items is instant, execellent.

Edit: it takes around 7 seconds if started from the VM console. Running it from a SSH session, it only takes about 1.5 to 3 seconds to start.

Yeah it's not bad, but I'm still trying to figure out how to make it better. Right now, the loading screen remains visible until the file is indexed because some functions (e.g. filtering) don't work without the index.

One way to improve it could be to skip the loading screen and go straight to the TUI, with indexing happening in the background. The advantage of this would be that the TUI opens instantly, and while some features might not be available right away, most users probably wouldn't even notice this since indexing is quick.

Another option, at least in theory, would be to skip building the index altogether and just process everything on the fly. We could cache the lines we've already interacted with to avoid doing the same thing over and over again. However, I'm not sure how well this would perform on low spec devices.

Hi allddd,

Nice work on this!  If you want we can work on including this in a future release as an optional binary package and see how it goes from there?


Cheers,
Franco

Hi Franco,

Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.

I've added both horizontal scrolling and IP version filtering in v0.3.0.

For now, the IP version filter works in the same way as any other field filter ("field value" syntax). Something like ip4/ip6 is also possible, but I'll look into it later because I have to make some small-ish changes to the filter module first. For docs on IP version filtering see https://gitlab.com/allddd/opnsense-filterlog#filter.

I'm glad you liked it :)

my screen is quite small (1280x800) and not all columns fit on the screen. It would be helpful if I could scroll horizontally with e.g. either the left/right arrow keys and/or 'h'/'l' (like in vim).

This is already on my todo list because, even with larger screens, it's an issue if the terminal is not running in fullscreen mode, which is often the case. I even have a bit of code for this in a local branch, but I haven't really decided what would be the best way to do this.

One approach would be to dynamically truncate the columns based on window size, but that would cause an issue on smaller screens where you could not see part of the date, IP, etc., which isn't ideal.

Another approach, as you mentioned, would be to implement horizontal scrolling. This would be more tricky to implement and might not look as good, but at least it would not cut off parts of IPs or other fields.

right now filtering for 'proto ip6' doesn't show any results. But filtering for 'proto ip' shows only the ip6 traffic. I would prefer if 'proto ip' would show the ipv4 entries and 'proto ip6' the ipv6. Maybe even a shortcut like in 'pftop' 'ip' and 'ip6' showing the ipv4 and ipv6 entries.

Currently, it is not possible to filter based on IP version, but adding this as an option would be easy. Documentation on the filter.log format:

Code Select Expand

IPv4
====

[Packetfilter], ipversion, tos, ecn, ttl, id, offset, flags, protonum, protoname, length, src, dst

The protonum/protoname order is reversed compared to IPv6.

IPv6
====

[Packetfilter], ipversion, class, flow, hoplimit, protoname, protonum, length, src, dst

The protonum/protoname order is reversed compared to IPv4.

The proto filter is used to filter by protoname. The reason you get any results with a filter query such as proto ip, is because some protocol names contain ip* (e.g. ipv6-icmp) and the value does not have to be an exact match. To implement this, I would either have to abuse the proto keyword or add a new one used specifically for matching the ipversion field. The latter option would probably be less confusing.

If you have a Gitlab account, feel free to open an issue if you notice any bugs or have suggestions.

Hi everyone

Since I couldn't find a simple way to quickly analyse filter logs, I've been writing a TUI in Go for this over the past few months. It's come a lot further than I first expected, so I decided to share it in case anyone else needs something like this.

It's called opnsense-filterlog and it's basically a TUI similar to a pager like less in terms of navigation, but with color output and search/filtering features that are better suited for firewall logs. The filter syntax is similar to tcpdump and pretty simple, but still lets you filter out all log entries you (don't) want to see, e.g.:

(src 192.168.1.1 or src 192.168.1.2) and action block and not proto udp

It's a simple binary with no dependencies that runs on OPNsense itself. I tried to make it as memory and resource efficient as possible, so it should be able to handle huge log files, even on low-spec devices.

In case anyone is interested, there is more documentation in the repo: https://gitlab.com/allddd/opnsense-filterlog

Figured I'd share this here, maybe it'll save someone a bit of time digging through logs.