Messages

For anyone interested, opnsense-filterlog is now also available as a binary package (thanks @franco!) and can be installed via pkg:

Code Select Expand

pkg install opnsense-filterlog
The package comes with a man page that's got pretty much the same info as the README:

Code Select Expand

man opnsense-filterlog
Currently v0.8.0 is in the repo, v0.9.0 will come with the 26.1.6 release and can now parse all filter log fields/protocols, filter by time, etc.

Kommt wahrscheinlich drauf an, wie komplex das Setup ist. Mir ist bisher nur aufgefallen, dass Captive Portal Sessions nicht gesynct werden. Beim Failover fliegt die Verbindung raus und man muss sich neu einloggen, damit der Traffic wieder durchkommt.

Welchen DNS Server verwendet der Client? Was steht in der Client Config unter AllowedIPs? Poste einfach mal die Config als Screenshot.

Mullvad support also told me that they wouldn't support psk-exchange anymore.

Makes sense. I haven't tried it myself, but it seems like it could be (ab)used to get around the five-device limit, since you get both a PSK and a primary key that isn't visible in the account settings.

Laut src/if_wg.c ist die Zahl nur eine interne ID. Irgendwie komisch, dass sie fürs Logging verwendet wird, sich aber nicht mit einer IP/einem Pubkey verknüpfen lässt. Wahrscheinlich bringt sie den Entwicklern was, für End User ist sie anscheind nicht gedacht.

In case you aren't aware: psk-exchange can be used to obtain a PSK that you can simply put in the WG config, like you would with a "normal" WG server, no mullvad-upgrade-tunnel needed. You need to compile it yourself though, since this isn't really officially supported, but it works.

@allddd

I think you could create a PR here that adds it to the opnsense sub directory:

https://github.com/opnsense/ports/tree/master/opnsense

If you need inspiration check out recently added ports there (eg ndp-proxy-go or hostwatch)

https://github.com/opnsense/ports/pull/252

I hope I've gotten everything right, it builds and installs without issue in a clean FreeBSD VM.

Hi allddd,

Nice work on this!  If you want we can work on including this in a future release as an optional binary package and see how it goes from there?

Cheers,
Franco

Hi Franco,

Thanks! I'd be honored, just let me know how I can help :) Would you need any changes to the Makefile/build process, maybe an install target? A man page would also be nice.

Is this a likely event for, say, 26.1?

The project itself is "ready", I've added both the manpage and Makefile targets a few versions ago. I don't know when it will happen though, @franco might be able to say more.

Did you experiment with the ports being in color and/or the direction being bold or in color?

The styling library I use supports adaptive styles, so you can define separate color schemes for light and dark backgrounds (even using different colors for 256-color and truecolor terminals). I wasted so much time trying to find the perfect color scheme, only to find out that half the terminals report the wrong background color... After that, I tried using just bold and faint styles, but even faint doesn't work on light backgrounds...

I've now added colors to ports and IPs in v0.7.0, which should hopefully make it more readable. No fancy adaptive styles though, I'm just using 2 colors that should work on both light and dark backgrounds (and don't look that bad on dark).

I also changed the direction indicators to I/O (with space), and it's indeed much more readable. Thanks for the suggestion!

You cannot view this attachment.

Thanks for checking it out :)

And the view is a lot more compact, I do like >/< as indicator for incoming or outgoing traffic. It does make it a bit harder to read, I'd prefer to have the direction in it's own column or maybe a space between it and the interface (or O/I, not sure)?

I don't think a separate column is necessary, since the direction always appears before the interface name and it's therefore just as easy to spot as it would be in a separate column. I'll try adding a space or using another char to see if it improves readability.

And I can scroll to the right for infinity, maybe it would make sense to not go over the right end? Jumping to the beginning and end of a line is removed, I assume because it's hardly necessary since the view is already very compact?

Jumping to the beginning/end would require checking the view length on every render, and with such a compact view it's just not worth it. Preventing scrolling past the right edge has the same issue. I looked at how less handles it to get an idea, and even there you can scroll infinitely, so in my opinion it's fine to do the same here, especially now that horizontal scrolling is rarely needed.

The scenario when filtering for protocols with ports I think the view is not easy to read in regards to spotting the source port, with IPv4 and IPv6 address.
If filtering for one or the other it's a lot better.
Though I still think it could help to place the '>' in 'Source > Destination' at the same position for all columns.

Yes, as plain text that view isn't very easy to read. Does your terminal support formatting/colors? I haven't updated the screenshot in the repo yet since the view may still change a bit, but I've added formatting that makes it easy to see the difference between IPs and ports:

You cannot view this attachment.

If I make the ">" line up at the same position for all lines, we basically end up with the old view again.

Does it have to be an LAN host, or would it be OK for an external service to notify you?

You could use a service like https://healthchecks.io in combination with Monit. This would be even more reliable, since you would receive a notification regardless of whether you are currently using the system or not.

You can configure Monit to send an HTTP request to healthchecks.io every time a check is successful. If it fails for any reason, or if OPNsense cannot reach healthchecks.io at all, you will be notified. They offer a generous free tier, you can even receive calls and SMS.

and triggers an action if either of the upstream gateways is down.

Not sure about the API, but have you considered using Monit? It's designed to do exactly that.
It can notify you, execute a script, or basically do anything else you want it to.

https://docs.opnsense.org/manual/monit.html

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files.

What behaviour would you expect from an application in such an situation? Many apps just freeze or crash :)

Personally I would try to evaluate if the app can handle it and if not show a pop-up. Either abort with that pop-up so that the user can select less files. Or fallback to load only what you can load and let the user know what was loaded and what not.

Although the fallback is probably not a good idea since the user had an idea what she wanted to look through and then wouldn't be sure what is included and what not. It is probably better to let the user know and abort so she knows that it will require two steps to look through all files.

I recently got around to doing some benchmarking/profiling of the stream package, and it turns out that the code I added early on to save memory (back when I thought it was a good idea to load everything into memory :D) was actually making everything way slower.

I swapped that code out and replaced some of Go's built ins with faster/simpler functions, and now the performance is much better. A log file with a few million lines now gets indexed almost instantly, and filtering is much much faster than before.

Here are some benchmarks of the function that parses the logs:

before:

Code Select Expand

BenchmarkParse-2    3287289       1102 ns/op      216 B/op       11 allocs/op

after:

Code Select Expand

BenchmarkParse-2    8188898        437.7 ns/op      160 B/op        1 allocs/op

Once I finish refactoring the TUI, I'll look into adding an option to open multiple files at once. With the recent changes, the code should now easily handle multiple files or even large dirs. Also, the TUI now looks much nicer (in my opinion) and is more compact. I've also added a feature where the user can select an entry to view more details that I can't show in the default view, since the filter log contains so much info. The details view is curremtly very minimal, but I'm already working on adding all the fields from the filter log CSV to it.

If you or anyone else is interested in doing some testing, you'll need to compile the development branch for now, as I still need to make a few changes before tagging the release.

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. ... I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files.

Wouldn't it be enough to read the first and the last line(s) of a file, determine the time stamps of these entries and you'd know the order of the files?

Good idea, the best approach is probably a mix of your suggestion and checking the file creation time.

I'm still not sure how well this will work though since the filter log directory can contain >30GB of files. If indexing takes 10-20 (or more) seconds, it's not something I would consider usable.

It would be helpful if it could read multiple file or a directory

I've thought about this, and the main issue is sorting the entries since I can't load everything into memory. Since we need to show entries from multiple files in one view, I need a reliable way to order them, and using file names/creation times isn't reliable because users can rename/move files. For now I'll stick with the single file limit until I find better solution. I'm open to any suggestions though.

Let me say thanks. This is a very nice idea, and it eases the pain to look thru the filter logs which is hard on the eyes.

Well done!

You're welcome :) Looking through the filter logs was bad enough, but searching through them was even worse. CSV is not a very human friendly format, and it isn't good for scripting either when the order of values changes with each entry...

I just released v0.5.0, which adds support for JSON output. E.g.:

Code Select Expand

opnsense-filterlog -j
opnsense-filterlog -j /path/to/filter.log

You can also filter in the same way as in the TUI by using the -f flag, e.g.:

Code Select Expand

opnsense-filterlog -j -f '(dport 80 or dport 443) and proto tcp and not action pass'