Messages

1

Tutorials and FAQs / Simple Filter Log Export via Telegraf

« on: April 24, 2024, 02:23:26 am »

I do very minimal filter logging with my OPNsense implementation for home use but I always wanted slightly better filter log views. I have been using Telegraf, Influxdb, Packetbeat & Grafana for general system and DNS monitoring. I finally decided to spend a little time to work on adding in the filter log and it turned out be relatively straight forward for the limited fields I needed (source, destination, destination port, rule number and label) for IPV4, tcp/udp, and blocks only. Here is the snippet that I added for Telegraf to /usr/local/etc/telegraf.d/custom.conf:

[[inputs.tail]]
  files = ["/var/log/filter/latest.log"]
  from_beginning = false
  pipe = false
  watch_method = "poll"
  name_override = "filterlog"
  data_format = "grok"
  grok_patterns=["<%{NONNEGINT}>%{NONNEGINT} %{TIMESTAMP_ISO8601:timestamp:ts} %{HOSTNAME} %{WORD} %{NONNEGINT} \\- \\[%{GREEDYDATA}\\] %{INT:rulenum},%{DATA},%{DATA},%{DATA:label:tag},%{DATA:interface:tag},%{DATA},%{DATA},%{DATA},%{INT},%{BASE16NUM},%{DATA},%{INT},%{INT},%{INT},%{DATA},%{INT},%{DATA:proto:tag},%{INT},%{IPV4:src:tag},%{IPV4:dst:tag},%{INT},%{INT:dst_port:tag}"]

Attached is the view I configured in Grafana. I just did the rule label translation in Grafana since I only have a few rules which are logging.

The bulk of the effort was fine tuning the grok pattern. I had found several examples but most were from syslog export tools and didn't include everything needed to read directly from the filter files. So far it has been working fine for me.

2

General Discussion / Re: Adblocker - Need Help

« on: August 25, 2022, 02:54:21 pm »

the solution pointed to so far is to use AdguardHome and there is a plugin for it, so yes.
If you mean, is there a "something" like a plugin that puts the configuration of the two or three parts in opn that make the whole as well work well i.e. changing the port for unbound, etc. the no.
But maybe I misunderstand what you mean.

I was just providing an alternative option that is completely service based that can be used by OPNsense/Unbound as well as any device when away from the home network.

3

General Discussion / Re: Adblocker - Need Help

« on: August 24, 2022, 06:44:22 am »

Take a look at the NextDNS service (https://nextdns.io). I connect to it from Unbound using DNS over TLS. Works great for me and doesn't require a separate plugin (though an agent is available as well). If you decide to try it out with Unbound, make sure to disable DNS rebinding checks in OPNsense if you set it to return 0.0.0.0 for blocked sites.

4

22.7 Legacy Series / Re: telegraf output bytes send end recv are cummulativ

« on: August 24, 2022, 06:21:20 am »

That is expected and how Telegraf captures the data from FreeBSD in the case of OPNsense (Telegraf net plugin documentation: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/net/README.md). It is up to you with whatever timeseries database and graphing tool you are using to interpret the metrics using rate of change functions or difference functions to get transfer rates or bytes send/received over a period of time. In my case I have Grafana with Influx 1.8 and use the following queries to get WAN port transfer rates:

Received: SELECT non_negative_derivative(mean("bytes_recv"), 1s) * 8 FROM "net" WHERE ("interface" = 'igb0') AND $timeFilter GROUP BY time($__interval) fill(none)

Sent: SELECT non_negative_derivative(mean("bytes_sent"), 1s) * 8 FROM "net" WHERE ("interface" = 'igb0') AND $timeFilter GROUP BY time($__interval) fill(none)

igb0 is my WAN port.

5

General Discussion / Re: Anyone setup NextDNS?

« on: July 26, 2022, 04:41:40 pm »

The ability to do that custom DNS over TLS configuration via the GUI was added last year with the hostname field. I have been using it since it was added and have had no issues with NextDNS. Just set the hostname to <client identifier>-<next_dns_profile_id>.dnsX.nextdns.io.

6

22.1 Legacy Series / Re: Unbound Constantly Restarting when WAN Gateway Down

« on: June 09, 2022, 07:19:55 pm »

At least for me right now it seems to be restarting no matter what interfaces I select. I have tried LAN only and All and in both cases Unbound is restarting. Here is a screen of my Unbound log from this morning during a cable modem reset with the Interface setting with LAN only.

7

22.1 Legacy Series / Re: Unbound Constantly Restarting when WAN Gateway Down

« on: June 08, 2022, 01:20:38 pm »

I didn't change the monitoring IP (it is blank). My main concern is that the firewall never recovers when gateway monitoring is on and the WAN comes back online. It seems to be linked to the unbound restarts and python CPU usage.

8

22.1 Legacy Series / Unbound Constantly Restarting when WAN Gateway Down

« on: June 07, 2022, 04:33:13 pm »

I am running 22.1.8_1 and anytime my WAN gateway (home cable modem) goes down Unbound constantly restarts until the Gateway comes back online. If I have Gateway monitoring enabled, the firewall doesn't recover on its own. I have to either reboot or manually disable/enable the gateway. With Gateway monitoring disabled, the firewall recovers fine. I don't remember this being an issue in previous releases. I changed Unbound network interfaces to All and that didn't help.

Is this behavior expected or are there other changes I can make to stop Unbound from constantly restarting when my cable modem resets. I am dealing with the cable modem resets separately with Optimum but would like to clear this up on the OPNsense side as well.

9

22.1 Legacy Series / Re: Http TLS/SSL Navigation logs - suricata or web proxy

« on: April 06, 2022, 09:18:38 pm »

You can add permanent Suricata custom configuration settings that are not available in the GUI plugin to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml. There should be a sample file in that directory to use as a starting template.

Sensei (Zenarmor) can also log all HTTP and HTTPS connections as well as DNS. It can be configured to store its data in an external Elasticsearch database (or local if the hardware is good enough) so you can run whatever analytics you want on the data with Kibana in addition to the reporting it provides. It can also block content based on categories and security threats.

There are also other tools like Zeek (formerly Bro) and Packetbeat (from Elastic) that can capture that level of information. Zeek is available in the community repository for OPNSense (https://www.routerperformance.net/opnsense-repo/).

10

General Discussion / Re: Historical view for Top Talkers

« on: March 28, 2022, 03:34:02 pm »

There is the built-in Insight reporting that maintains historical data based on NetFlow. If you want more detail, there is NTOPNG combined with InfluxDB for its timeseries data.

11

General Discussion / Re: Track historical usage per user

« on: March 12, 2022, 05:06:01 pm »

I don't use nProbe. It isn't required for the client statistics. Also I think it requires a license.

You should be able to import my dashboard json into grafana and then customize it as needed.

There is a telegraf plugin for opnsense which makes it simple to send the firewall metrics to influx.

12

General Discussion / Re: Track historical usage per user

« on: March 12, 2022, 02:47:55 pm »

ntopng does support both ipv4 and ipv6. influxdb is very efficient at storing timeseries data since that is all it does. Also, ntopng provides several options for what data to retain and how often to write it out (every minute or every 5 minutes). Other than the timeseries data, ntopng maintains everything else in memory. I just reset my environment but I was using almost no disk space to store 2 weeks worth of data and I enable most of the timeseries details. I doubt you would have any issue running ntopng, influxdb & grafana (if you want your own custom dashboards which I recommend) on your firewall with those specs.

13

General Discussion / Re: Track historical usage per user

« on: March 07, 2022, 11:50:53 pm »

Attached is my Grafana dashboard for the OPNsense Telegraf metrics and ntopng timeseries. It has two InfluxDB datasource variables to point to for OPNsense telegraf and ntopng databases. I stripped out all panels that require the custom Telegraf things I do for DHCP and DNS/Packetbeats.

For ntopng, I have the following timeseries settings:

• Resolution set to 1 minute
• Interface Layer-7 applications set to Both
• Host Timeseries set to Full
• Host Layer-7 applications set to Both

For the OPNsense CPU temperature, I manually added the following to the telegraf script:
[[inputs.exec]]
  commands = ["/sbin/sysctl dev.cpu | grep temperature | sed 's/[a-z\.]*/systemp cpu/;s/\.[a-z]*\: /=/;s/.$//'"]
  timeout = "5s"
  data_format = "influx"

14

General Discussion / Re: Track historical usage per user

« on: March 07, 2022, 10:00:55 pm »

I run ntopng and store my timeseries data in InfluxDB for easy reporting/graphing in Grafana for client bandwidth usage statistics as well as application/category statistics. I only store the data for 2 weeks but could easily keep it for 30+ days if needed. Attached is a screenshot from my Grafana dashboard for the last 7 days for my ntopng data.

I also send OPNsense metrics to InfluxDB via Telegraf and I use Packetbeats to monitor DNS traffic. It makes for a pretty comprehensive dashboard for my home network. I run this all on a Raspberry Pi 4 with a mirror port since I didn't want to interfere with the firewall except for the Telegraf agent. I am assuming it could all be run the firewall itself as well using mimugmail's repository for InfluxDB & Grafana.

15

General Discussion / Re: NextDNS

« on: February 07, 2022, 04:58:03 am »

The format for the Hostname in the Unbound DNS over TLS tab is: XXXXX.dns1.nextdns.io or XXXXX.dns2.nextdns.io where XXXXX is your endpoint ID. If you want the endpoint to have a name, you would do: name-XXXXX.dns1.nextdns.io.

Your dot.conf forward addr lines should look like these:

forward-addr: 45.90.28.0#XXXXX.dns1.nextdns.io
forward-addr: 45.90.30.0#XXXXX.dns2.nextdns.io

Fixing the hostname format should clear up the certificate verification error you are getting.