Messages

25.1.5 fixed the problem of pass rules logging blocks. But it still looks like pf is having issues with port forwarding parallel DNS or NTP UDP requests for the same client to different destinations. When my Ring wired floodlight cam sends 3 UDP DNS requests in parallel most of the time only 2 of the 3 are answered (one is always my local DNS server and the other is one of the two port forwards). I can confirm the issue with packet captures, Adguard Home logging and NTOPNG DNS Request vs Reply Ratio.

No clue how long this issue has been going on. I only started noticing the problem because of the pass rule block logging that started with 25.1.1. My DNS port forward is to Adguard Home running on the appliance and my NTP port forward is to Chrony running also on the appliance. Luckily this doesn't stop anything from working since the client does get several DNS or NTP responses back just not all of them.

Are there any limitations with pf on parallel UDP port forwards from the same client to different destinations?

If there is any additional information I can provide, please let me know.

Thanks for pointing out that thread. This makes a lot more sense now with the timing of releases. I don't think I had upgraded my old appliance to 25.1.1 since I was planning on replacing it. That would explain why I never saw this issue before on the old appliance. I built the new appliance with the current release of 25.1 in mid March and noticed the issue immediately.

I recently upgraded my OPNsense firewall hardware that I had been running for 5+ years and I am running into a weird issue with DNS port forwarding that I didn't see on my previous appliance. I am running Adguard Home via the repository and I configured port forwarding for TCP/UDP 53 to 127.0.0.1. It works fine but I have a few IoT devices that are sending parallel DNS requests to the firewall LAN address (DHCP DNS address), 8.8.8.8 and 1.1.1.1. Most of the time 2 of 3 requests are responded to and the third on is dropped. In the firewall log I see a block for the port forward PASS rule (if I change to an associated rule I see a drop for that). I checked the Adguard Home logs and I don't see the third request. I never had this issue with my previous appliance so I am trying to figure out what could be different (other than the system being faster). I have done packet captures for those devices and can see the 3 DNS requests hitting the firewall and one will be flagged with DNS response missing. Is there anything that can be tweaked to handle simultaneous UDP DNS port forwards from the same client to 2 different destinations. This is more of cosmetic issue since the client always gets at least 2 responses and nothing breaks. But getting a block for a PASS rule in the log is annoying.

I do very minimal filter logging with my OPNsense implementation for home use but I always wanted slightly better filter log views. I have been using Telegraf, Influxdb, Packetbeat & Grafana for general system and DNS monitoring. I finally decided to spend a little time to work on adding in the filter log and it turned out be relatively straight forward for the limited fields I needed (source, destination, destination port, rule number and label) for IPV4, tcp/udp, and blocks only. Here is the snippet that I added for Telegraf to /usr/local/etc/telegraf.d/custom.conf:

[[inputs.tail]]
  files = ["/var/log/filter/latest.log"]
  from_beginning = false
  pipe = false
  watch_method = "poll"
  name_override = "filterlog"
  data_format = "grok"
  grok_patterns=["<%{NONNEGINT}>%{NONNEGINT} %{TIMESTAMP_ISO8601:timestamp:ts} %{HOSTNAME} %{WORD} %{NONNEGINT} \\- \\[%{GREEDYDATA}\\] %{INT:rulenum},%{DATA},%{DATA},%{DATA:label:tag},%{DATA:interface:tag},%{DATA},%{DATA},%{DATA},%{INT},%{BASE16NUM},%{DATA},%{INT},%{INT},%{INT},%{DATA},%{INT},%{DATA:proto:tag},%{INT},%{IPV4:src:tag},%{IPV4:dst:tag},%{INT},%{INT:dst_port:tag}"]

Attached is the view I configured in Grafana. I just did the rule label translation in Grafana since I only have a few rules which are logging.

The bulk of the effort was fine tuning the grok pattern. I had found several examples but most were from syslog export tools and didn't include everything needed to read directly from the filter files. So far it has been working fine for me.

the solution pointed to so far is to use AdguardHome and there is a plugin for it, so yes.
If you mean, is there a "something" like a plugin that puts the configuration of the two or three parts in opn that make the whole as well work well i.e. changing the port for unbound, etc. the no.
But maybe I misunderstand what you mean.

I was just providing an alternative option that is completely service based that can be used by OPNsense/Unbound as well as any device when away from the home network.

Take a look at the NextDNS service (https://nextdns.io). I connect to it from Unbound using DNS over TLS. Works great for me and doesn't require a separate plugin (though an agent is available as well). If you decide to try it out with Unbound, make sure to disable DNS rebinding checks in OPNsense if you set it to return 0.0.0.0 for blocked sites.

That is expected and how Telegraf captures the data from FreeBSD in the case of OPNsense (Telegraf net plugin documentation: https://github.com/influxdata/telegraf/blob/master/plugins/inputs/net/README.md). It is up to you with whatever timeseries database and graphing tool you are using to interpret the metrics using rate of change functions or difference functions to get transfer rates or bytes send/received over a period of time. In my case I have Grafana with Influx 1.8 and use the following queries to get WAN port transfer rates:

Received: SELECT non_negative_derivative(mean("bytes_recv"), 1s) * 8 FROM "net" WHERE ("interface" = 'igb0') AND $timeFilter GROUP BY time($__interval) fill(none)

Sent: SELECT non_negative_derivative(mean("bytes_sent"), 1s) * 8 FROM "net" WHERE ("interface" = 'igb0') AND $timeFilter GROUP BY time($__interval) fill(none)

igb0 is my WAN port.

The ability to do that custom DNS over TLS configuration via the GUI was added last year with the hostname field. I have been using it since it was added and have had no issues with NextDNS. Just set the hostname to <client identifier>-<next_dns_profile_id>.dnsX.nextdns.io.

At least for me right now it seems to be restarting no matter what interfaces I select. I have tried LAN only and All and in both cases Unbound is restarting. Here is a screen of my Unbound log from this morning during a cable modem reset with the Interface setting with LAN only.

I didn't change the monitoring IP (it is blank). My main concern is that the firewall never recovers when gateway monitoring is on and the WAN comes back online. It seems to be linked to the unbound restarts and python CPU usage.

I am running 22.1.8_1 and anytime my WAN gateway (home cable modem) goes down Unbound constantly restarts until the Gateway comes back online. If I have Gateway monitoring enabled, the firewall doesn't recover on its own. I have to either reboot or manually disable/enable the gateway. With Gateway monitoring disabled, the firewall recovers fine. I don't remember this being an issue in previous releases. I changed Unbound network interfaces to All and that didn't help.

Is this behavior expected or are there other changes I can make to stop Unbound from constantly restarting when my cable modem resets. I am dealing with the cable modem resets separately with Optimum but would like to clear this up on the OPNsense side as well.

You can add permanent Suricata custom configuration settings that are not available in the GUI plugin to /usr/local/opnsense/service/templates/OPNsense/IDS/custom.yaml. There should be a sample file in that directory to use as a starting template.

Sensei (Zenarmor) can also log all HTTP and HTTPS connections as well as DNS. It can be configured to store its data in an external Elasticsearch database (or local if the hardware is good enough) so you can run whatever analytics you want on the data with Kibana in addition to the reporting it provides. It can also block content based on categories and security threats.

There are also other tools like Zeek (formerly Bro) and Packetbeat (from Elastic) that can capture that level of information. Zeek is available in the community repository for OPNSense (https://www.routerperformance.net/opnsense-repo/).

There is the built-in Insight reporting that maintains historical data based on NetFlow. If you want more detail, there is NTOPNG combined with InfluxDB for its timeseries data.

I don't use nProbe. It isn't required for the client statistics. Also I think it requires a license.

You should be able to import my dashboard json into grafana and then customize it as needed.

There is a telegraf plugin for opnsense which makes it simple to send the firewall metrics to influx.

ntopng does support both ipv4 and ipv6. influxdb is very efficient at storing timeseries data since that is all it does. Also, ntopng provides several options for what data to retain and how often to write it out (every minute or every 5 minutes). Other than the timeseries data, ntopng maintains everything else in memory. I just reset my environment but I was using almost no disk space to store 2 weeks worth of data and I enable most of the timeseries details. I doubt you would have any issue running ntopng, influxdb & grafana (if you want your own custom dashboards which I recommend) on your firewall with those specs.