Messages

[6) Reboot System]

Solved.
Thank you SanjivSharma for your suggestion regarding IPv6.

In System: Settings: General -- Networking (did not find anything in Ubound DNS settings),
the "Prefer to use IPv4 even if IPv6 is available" was unchecked.  After a Console Reboot (i.e., 6) Reboot System), the execution of the software worked as expected.  This package's dedicated domain has two IPv4 addresses in the DNS.

1)  I needed to do a OPNsense reboot.   The software package (accessed by a DNS domain) failed to execute until I rebooted.
2)  I needed to change which browser I used to invoke the software package.  Vivaldi failed while Firefox worked.  Exploring whether network related settings and behaviors are different in these two specific browsers may yield interesting insights when OPNsense is the firewall.
3)  The software package does not have IPv6 addresses, but {up.railway.app} does use them.  {up.railway.app} with some browsers does something differently when a domain has no IPv6 addresses registered with OPNsense as the firewall.  A reason to to have several current browsers on hand in case a complicated interaction of API's, browsers, IPv6 vs. Ipv4, and OPNsense!
4)  There was no reason to have the "Prefer to use IPv4 even if IPv6 is available" checked.  It was originally checked when OPNsense became my firewall in order to minimize learning complications for a firewall newbie to just about anything network/Internet.  With IPv6 now in much wider use, there may no reason to ever have this checkbox checked for today's Internet (newbie or not).
5)  A second software package from the same vendor (no IPv6 addresses and with only 1 IPv4 addess) still failed; even though the first software package now works without a workaround.   Requires a workaround executing under Vivaldi with the built-in Proton VPN temporarily enabled.  After the initialization phase, the Protoon VPN is disabled.  Everything works as expected.  This workaround using Vivaldi also works the the original software package I had problems with.  I will launch one software package in Firefox, then use Vivaldi with Proton VPN temporarily enabled to startup the second software package (these two software packages do not like each others company when executing in the same browser it turns out).
6)   Private (incognito) browser sessions may affect this problem, but I have been unable to determine anything specifically.
7)   There may be other browser settings at play, such as caching, javascript (WebAssembly, JIT), browser vs. Unbound DNS blocklists, extensions, local storage, etc. may be at play .

Delighted to have both software packages now working as expected, albeit with a lot of unanswered questions as to exactly why and why they behave differently with different browsers.

The problem appears not to be in OPNsense.
The hosting website that provides the access to the target webservers is not prepared to keep the DNS state, so key information is lost when OPNsense intercepts DNS lookups (which causes the hosting website to discard DNS and other authorizing information).

Until the target website owner (e.g., Railway) keeps the DNS state between DNS server accesses (i.e., those DNS server accesses done by OPNsense and hidden from the target website), I will need to rely on a workaround which bypasses OPNsense and the NAT interventions which confuse the target website that causes the target website to discard information it needs to provide me with access.

There may be tactics to use with OPNsense to provide simulated DNS "A" records, etc., but I am not hopeful.

ISP is the AT&T Fiber service.  The AT&T Fiber BGW320-500 modem offers WiFi and has four RJ45 ports which are assigned addresses in the 192.168.0.0/24 AT&T Fiber modem/firewall's local network.  This Fiber box effectively is a configureable firewall appliance with a built-in fiber modem doing the translations between the external fiber port and the internal RJ45 WAN port.   The AT&T WiFi option is unused and is secured with a password.  The OPNsense Protectli firewall appliance has a Grandstream WiFi device on a Virtual LAN connected to the OPNsense device via a TP-Link switch.  This OPNsense WiFi configuration works as expected.

Knowing that it would cause a double-NAT, I have chosen to not use the IP passthrough option through the AT&T Fiber modem/firewall for my OPNsense appliance, a Protectli device with adequate hardware resources.  The AT&T Fiber firewall has effective silent blocking of attempts of external actors to solicit responses to unsolicted protocols, ports, and packets.  As the OPNsense firewall appliance is a both an functional operational and a blood-sweat-and-tears learning tool, it helps a lot to have the OPNsense log not filled primarily with thousands of entries of unsolicted protocols and packets (roughly from 64 to 512 per hour).

My problem is that a website used by a Windows laptop (W10, updates current as of 2025.10 update) and a Windows desktop (W11, updates current as of 2026.07 update) references a domain name – dms.xxxxxx.com - generates a SERVFAIL because: 1) "all servers for this domain failed, at zone up.railway.app at [a valid IPv6 address] no server to query nameserver addresses not useable"; and per Services : Unbound DNS : Log File - 2) there is no public DNS records for the domain name – dms.xxxxxx.com – available per DNSchecker's "DNS Lookup" service (all worldwide authoritive servers including ICANN's agree on this).

Domains involved have been placed on the Services : Unbound DNS : Blocklists : Allowlist Domains to no effect.

When the Windows desktop bypasses OPNsense by directly by a dedicated use one of the AT&T Fiber's modem/firewall ports, the SERVFAIL does not occur!

OPNsense is at 26.1.11_10-amd64 and is current per the CLI Update from console and the OPNsense Web Gui's System : Firmware : Updates.  No IP's involved are on any IP DNSchecker blacklist.

CrowdSec and Unbound DNS OPNsense services do not affect the SERVFail in any combination of being Enabled or Disabled. The complete current list of plugin services added to the base OPNsense configuration is (with "*" is installed/configured/running, "**" is installed/not-configured; "***" is installed/configured/stopped):

System : Firmware : Plugins -  installed list (plugins not yet configured are labeled "misconfigured")
*     os-chrony – Chrony time synchronization (chrony daemon)
*** os-crowdsec – Lightweight and collaborative security engine
*** os-debug – Debugging Tools
*     os-demidecode – Display  hardware information on the dashboard
**   os-hw-probe (misconfigured) – Collect hardware diagnostics
**   os-iperf (misconfigured) – Connection speed tester
*     os-isc-dhcp – ISC DHCPv4/v6 server
**   os-maltrail (misconfigured) – Malicious traffic detection system
**   os-netdata (misconfigured) – Real-time performance monitoring
*     os-smart – SMART tools
*     os-telegraf (misconfigured) – Agent for collecting metrics and data
*     os-vnstat (misconfigured) – Network traffic monitor


Lobby : Dashboard : Services
*     chrony daemon
*     System Configuration Daemon
*     Cron
*** Crowdsec
*     ISC DHCPv4
*     ISC DHCPv6
*     Gateway monitor watcher
*     Gateway monitor (WAN_DHCP6)
*     Gateway monitor (WAN_DHCP4)
*     Insight Aggregator
*     Host discovery service
*** iperf Performance Test
*     Users and Groups
*   Packet Filter
*   Router Advertisements
*   System routing
*   NetFlow Distributor
*   Intrusion Detection
*   System tunables
*   Syslog-ng Daemon
*   Unbound DNS
*   vnStat Daemon
*   Web GUI 


Any thoughts on why OPNsense may be causing a SERVFAIL?

Is there an alternative to using UNBOUND DNS on OPNsense?

Could this be a ISC DHCPv4 and ISC DHCPv6 related issue?

Thoughts are appreciated, thank you.

A workaround for unwanted DNSBL blocks is to use a VPN (I use the free version of ProtonVPN).

Remember to disable the VPN ASAP after you access an Unbound DNS DNSBL blocked web site as the VPN completely bypasses the DNS protections afforded by OPNsense.

1)  Whitelist problem.  Using "cloreautomotive.com" as an example.  This domain appears in several DNSBL.  Using the Whitelist entries of "cloreautomotive.com" and "www.cloreautomotive.com" (and "*.cloreautomotive.com) does not enable any Whitelist action to override the DNSBL blocking domain entries.  Does Unbound DNS examine the Whitelist first or is the Whitelist ignored as the blocklist overrides are looked at after the DNSBL domains have been checked (which effectively ignores the Whitelist entries)?

2)  Suggestion:  Quad9 has a web page where you may check for blocked domains (use  https://quad9.com/result/?url=cloreautomotive.com  with  Services: Unbound DNS: General  "Enable Unbound" unchecked).  A similar query for Unbound DNS which lists the DNSBLs involved if Unbound DNS is blocking a domain would be very helpful for investigating unwanted blocks and creating a Whitelist entry.

Skipping any update in any released sequence of updates has proven to be problematic for devices running software as "firmware" such as OPNsense.   A question for the OPNsense maintainers is whether OPNsense should prevent skipping of even minor update releases to insure that all updates require and may assume that all the prior updates have been executed in chronological sequence.

This could be done by OPNsense's update logic when an updater attempts to skip over an update (often to jump directly to a major release such as 24.7). 

Testing with both of my Linux/FF ESR and Windows 10/Pale Moon laptops indicates that the OPNsense javascript does not recover when the Screen Saver becomes active either automatically (elapsed time) or manually (lock).  This indicates that the OPNsense's javascript keyboard handler or the javascript interface with the OPNsense appliance server for these situations is problematic (i.e., non-functional).  OPNsense may think that the laptop's running the OPNsense GUI interface have timed out when they have not.

1)  A Linux laptop (Debian 12.5, FF ESR browser) fails (often within seconds) after startup of the dashboard widgets (dashboard stops working).  A Windows 10 laptop pressed into duty (old Windows 1709 version, but running the latest Pale Moon and Iridium browsers) runs the dashboard & widgets fine.  The various monitoring OPNsense displays (Live logfile, etc.) sometimes work and sometimes do not even start on the Linux laptop, but always do run on the Windows laptop.
2)  The Linux laptop loses execution focus when the OPNsense provided javascript is interrupted by engaging the screen saver (either by elapsed time or by a manual lock) and then unlocking the screen saver.  The OPNsense javascript never recovers from the screen saver lock on the Linux laptop, but always recovers on the Windows laptop.
3)  Eventually, OPNsense runs out of memory (even when simply running pftop on the OPNsense console of the firewall appliance) after complaining that the swapspace requests have failed - the 10 GB swap space is allocated - and a panic reboot is done.
4) Processing of the Hagezi block lists completed in under an hour in background (pre 24.7).  Does not complete at all starting with 24.7.2 (appears to be related to memory not being freed by OPNsense).  OPNsense is generating every 2-3 minutes a backup configuration file while processing block lists (not just Hagezi).  Disabling Hagezi block list processing has eliminated the generation of hundreds of spurious backup configuration files (which used up 30% of a 500 GB SSD).    24.7.2 is also generating multiple requests to 127.0.0.1:53 (which never occurred pre-24.7.x) when processing block lists - perhaps a surriata <--> OPNsense conflict regarding memory usage and releases.
5)  Memory non-release problems may be related to the Python3 updates . . . . .
In addition, per top running in the OPNsense console shell, Suricata has 297M (increasing about 3M every hour) of resident RAM and 3330M of total  memory when no block lists are being actively processed. The suricata swap footprint goes up when processing blocklists such that eventually even pftop is denied swap space (both 8GB and 10GB max swap sizes have been tried). 

Grep memory /var/run/dnesg.boot   displays:
agp0: aperture size is 256M, detected 8188k stolen memory
pid 69794 (pftop), jid 0, uid 0, was killed: failed to reclaim memory

These messages started appearing with an upgrade to 24.7 and first noticed with an upgrade to 24.7.2 (even with the cessation of Hagezi blocklist processing).  Note that Hagezi blocklist processing (as a background task) pre-24.7 took less than an hour to always complete.

With 24.7.2, there apparently is a memory leak.  Eventually, OPNsense reboots itself automatically.  The VGA console display presents a Login prompt (and a corresponding login needs to be done for the GUI OPNsense management interface).

I noticed that in System: Settings: Miscellaneous, the Swap File setting to "add a 2GB swap file to the system" was unchecked.  It is unclear whether a 24.7.x update disabled my swap file settings, I remember having an 8GB swap file for my OPNsense appliance.

When the swap file fills up (or is not established by a setting), it would be nice to have a more elegant way to add more swap space dynamically as well as a FYI message as part of the login messages displayed why an automatic restart was done by OPNsense (such as the swap file filing up).

It is possible that when the blocklists are being processed that invariably the swap file space will fill up (my configuration is 4GB Ram, 8GB swap space).   It seems that blocklist processing under 24.7.2 (Hagezi specifically) has never successfully completed since I applied the 24.7.2 changes.  OPNsense is also creating multiple history backups (every 2-3 minutes!) while processing the blocklists - so I have lost my original configuration backups (my generous backup of count of 2048 means that OPNsense has trashed all of my pre 24.7 history backups and has used up 30% of my disk space for useless history backups).   While processing blocklists, there is no need for interim history backups!!!

It would be very helpful to add a 14th option to the console selection menu:
14)  Display status of settings for RAM, Swap Space, Disk Space

I am disabling my usage of Hagezi block lists until the 24.7.x problems are sorted out.

The upgrade from 24.7.1 to 24.7.2:
* A Linux Debian 12.5 laptop (dedicated to being a firewall management device) with Firefox ESR is unable to run:
  - /ui/diagnostics/firewall/log#Lobby
  - /?url=ui/core/dashboard
  An error message is eventually displayed:  The Connection has timed out
  Linux is no longer usable as a firewall management device for OPNsense.
* My OPNsense firewall appliance has a USB port for keyboard and a VGA port for a monitor.
* The swap file in OPNsense (console displayed error message onto the VGA display running pfTop) shows that the FreeBSD swap file for OPNsense kept expanding in one execution until the swap space filled up.
* A Windows 10 laptop executing the Pale Moon browser is able to run as expected the  Lobby  and  Firewall:Log Files:Live View  displays.  Missing is a display of the swap file size.  The graphics now work smoothly in 24.7.2. 

The pre-24.7 lobby displayed memory status (disk,ram,swap) in text - please return this display capability as an option.

The Windows 10/Iridium's OPNsense dashboard is still working fine after two hours have passed -- even after a forced screen saver (i.e., a Lock).   This is on an old Windows 10 laptop (16 GB RAM) that is missing more than a few OS updates; the Iridium browser is the latest version.  Other activity in addition to the Iridium browser are not affecting the dashboard's widget displays (when focus returns to Iridium's dashboard page, the widgets restart correctly and as expected).

The Linux/Firefox ESR laptop's OPNsense dashboard stops working sometimes only after a few seconds even though there is no other keyboard/mouse activity after an OS restart and only the Firefox ESR is started up.   Whether or not the public Internet access is enabled or not does not appear to affect OPNsense dashboard widgets behavior.

My LANNER PFW600 (at OPNsense 24.7_9) starts the Lobby: Dashboard OK, but eventually stops working on my Linux laptop dedicated to being a OPNsense management laptop connected to a local network access only port on the LANNER (Browser is the current Linux Firefox ESR variant running on a System 76 EduBook Starling 2; EduBook's OS is Debian 12.5 with updates).  A 2nd browser just activated for 24.7 dashboard testing is an Iridium (Chrome fork) running on a Windows 10 system on a Toshiba Portege laptop.

The Linux machine, even when the only program that is active is the browser running the OPNsense dashboard and a Firewall: Log Files: Live View eventually ends up with the OPNsense dashboard no longer working when the dashboard no longer has focus and always if the screen saver kicks in.  The Firefox ESR complains about the drag on browser performance due to the dashboard javascript.  The CPU widget appears to stop working first, then every other widget quickly comes to a halt.

There may be a problem with the widgets re-initializing after being suspended by the OS due to not having focus (swapping and laptop/browser memory constraints are not a factor).     The dashboard responsiveness slowly degrades even if focus is kept on the dashboard web page -- there may be a problem with flags not being cleared, memory leakage, memory not being freed.

Testing of the Windows 10/dashboard is ongoing, will update this post with a reply if insights occur . . . . . (the Windows 10/Iridium machine's dashboard display is still working after 30 minutes . . . . .).

Having an option to run the old dashboard on my Linux laptop would be a workaround.