SERVFAIL when domain has no Authoritative Server or public DNS records

[NonGough]

ISP is the AT&T Fiber service.  The AT&T Fiber BGW320-500 modem offers WiFi and has four RJ45 ports which are assigned addresses in the 192.168.0.0/24 AT&T Fiber modem/firewall's local network.  This Fiber box effectively is a configureable firewall appliance with a built-in fiber modem doing the translations between the external fiber port and the internal RJ45 WAN port.   The AT&T WiFi option is unused and is secured with a password.  The OPNsense Protectli firewall appliance has a Grandstream WiFi device on a Virtual LAN connected to the OPNsense device via a TP-Link switch.  This OPNsense WiFi configuration works as expected.

Knowing that it would cause a double-NAT, I have chosen to not use the IP passthrough option through the AT&T Fiber modem/firewall for my OPNsense appliance, a Protectli device with adequate hardware resources.  The AT&T Fiber firewall has effective silent blocking of attempts of external actors to solicit responses to unsolicted protocols, ports, and packets.  As the OPNsense firewall appliance is a both an functional operational and a blood-sweat-and-tears learning tool, it helps a lot to have the OPNsense log not filled primarily with thousands of entries of unsolicted protocols and packets (roughly from 64 to 512 per hour).

My problem is that a website used by a Windows laptop (W10, updates current as of 2025.10 update) and a Windows desktop (W11, updates current as of 2026.07 update) references a domain name – dms.xxxxxx.com - generates a SERVFAIL because: 1) "all servers for this domain failed, at zone up.railway.app at [a valid IPv6 address] no server to query nameserver addresses not useable"; and per Services : Unbound DNS : Log File - 2) there is no public DNS records for the domain name – dms.xxxxxx.com – available per DNSchecker's "DNS Lookup" service (all worldwide authoritive servers including ICANN's agree on this).

Domains involved have been placed on the Services : Unbound DNS : Blocklists : Allowlist Domains to no effect.

When the Windows desktop bypasses OPNsense by directly by a dedicated use one of the AT&T Fiber's modem/firewall ports, the SERVFAIL does not occur!

OPNsense is at 26.1.11_10-amd64 and is current per the CLI Update from console and the OPNsense Web Gui's System : Firmware : Updates.  No IP's involved are on any IP DNSchecker blacklist.

CrowdSec and Unbound DNS OPNsense services do not affect the SERVFail in any combination of being Enabled or Disabled. The complete current list of plugin services added to the base OPNsense configuration is (with "*" is installed/configured/running, "**" is installed/not-configured; "***" is installed/configured/stopped):

System : Firmware : Plugins -  installed list (plugins not yet configured are labeled "misconfigured")
*     os-chrony – Chrony time synchronization (chrony daemon)
*** os-crowdsec – Lightweight and collaborative security engine
*** os-debug – Debugging Tools
*     os-demidecode – Display  hardware information on the dashboard
**   os-hw-probe (misconfigured) – Collect hardware diagnostics
**   os-iperf (misconfigured) – Connection speed tester
*     os-isc-dhcp – ISC DHCPv4/v6 server
**   os-maltrail (misconfigured) – Malicious traffic detection system
**   os-netdata (misconfigured) – Real-time performance monitoring
*     os-smart – SMART tools
*     os-telegraf (misconfigured) – Agent for collecting metrics and data
*     os-vnstat (misconfigured) – Network traffic monitor


Lobby : Dashboard : Services
*     chrony daemon
*     System Configuration Daemon
*     Cron
*** Crowdsec
*     ISC DHCPv4
*     ISC DHCPv6
*     Gateway monitor watcher
*     Gateway monitor (WAN_DHCP6)
*     Gateway monitor (WAN_DHCP4)
*     Insight Aggregator
*     Host discovery service
*** iperf Performance Test
*     Users and Groups
*   Packet Filter
*   Router Advertisements
*   System routing
*   NetFlow Distributor
*   Intrusion Detection
*   System tunables
*   Syslog-ng Daemon
*   Unbound DNS
*   vnStat Daemon
*   Web GUI 


Any thoughts on why OPNsense may be causing a SERVFAIL?

Is there an alternative to using UNBOUND DNS on OPNsense?

Could this be a ISC DHCPv4 and ISC DHCPv6 related issue?

Thoughts are appreciated, thank you.

[nero355]

My problem is that a website used by a Windows laptop and a Windows desktop references a domain name – dms.xxxxxx.com - generates a SERVFAIL because:

1) "all servers for this domain failed, at zone up.railway.app at [a valid IPv6 address] no server to query nameserver addresses not useable"; and per Services : Unbound DNS : Log File -

2) there is no public DNS records for the domain name – dms.xxxxxx.com – available per DNSchecker's "DNS Lookup" service (all worldwide authoritive servers including ICANN's agree on this).

Domains involved have been placed on the Services : Unbound DNS : Blocklists : Allowlist Domains to no effect.

When the Windows desktop bypasses OPNsense by directly by a dedicated use one of the AT&T Fiber's modem/firewall ports, the SERVFAIL does not occur!

What happens when you grab one of your working and not working Clients and use :
- nslookup
- tracert
In combination with the domain ?!

The same for when connected to the ISP supplied Router and then OPNsense and eventually compare all the different outputs ?

[NonGough]

The problem appears not to be in OPNsense.
The hosting website that provides the access to the target webservers is not prepared to keep the DNS state, so key information is lost when OPNsense intercepts DNS lookups (which causes the hosting website to discard DNS and other authorizing information).

Until the target website owner (e.g., Railway) keeps the DNS state between DNS server accesses (i.e., those DNS server accesses done by OPNsense and hidden from the target website), I will need to rely on a workaround which bypasses OPNsense and the NAT interventions which confuse the target website that causes the target website to discard information it needs to provide me with access.

There may be tactics to use with OPNsense to provide simulated DNS "A" records, etc., but I am not hopeful.

[SanjivSharma]

That IPv6-specific error is the big clue here — Unbound trying to reach an IPv6 nameserver for up.railway.app and failing while the direct-to-AT&T-modem path works fine strongly points to broken IPv6 connectivity or DNS resolution specifically inside OPNsense's network, not a blocklist or plugin issue. Worth checking if Unbound has "Prefer IPv4 for outgoing queries" available under Unbound's advanced settings, since forcing it to skip that broken IPv6 path entirely would confirm the theory fast. CrowdSec and the DHCP services are almost certainly red herrings here given the error is happening at the DNS resolution layer itself, pointing straight at Unbound's IPv6 handling.

[NonGough]

Solved.
Thank you SanjivSharma for your suggestion regarding IPv6.

In System: Settings: General -- Networking (did not find anything in Ubound DNS settings),
the "Prefer to use IPv4 even if IPv6 is available" was unchecked.  After a Console Reboot (i.e., 6) Reboot System), the execution of the software worked as expected.  This package's dedicated domain has two IPv4 addresses in the DNS.

1)  I needed to do a OPNsense reboot.   The software package (accessed by a DNS domain) failed to execute until I rebooted.
2)  I needed to change which browser I used to invoke the software package.  Vivaldi failed while Firefox worked.  Exploring whether network related settings and behaviors are different in these two specific browsers may yield interesting insights when OPNsense is the firewall.
3)  The software package does not have IPv6 addresses, but {up.railway.app} does use them.  {up.railway.app} with some browsers does something differently when a domain has no IPv6 addresses registered with OPNsense as the firewall.  A reason to to have several current browsers on hand in case a complicated interaction of API's, browsers, IPv6 vs. Ipv4, and OPNsense!
4)  There was no reason to have the "Prefer to use IPv4 even if IPv6 is available" checked.  It was originally checked when OPNsense became my firewall in order to minimize learning complications for a firewall newbie to just about anything network/Internet.  With IPv6 now in much wider use, there may no reason to ever have this checkbox checked for today's Internet (newbie or not).
5)  A second software package from the same vendor (no IPv6 addresses and with only 1 IPv4 addess) still failed; even though the first software package now works without a workaround.   Requires a workaround executing under Vivaldi with the built-in Proton VPN temporarily enabled.  After the initialization phase, the Protoon VPN is disabled.  Everything works as expected.  This workaround using Vivaldi also works the the original software package I had problems with.  I will launch one software package in Firefox, then use Vivaldi with Proton VPN temporarily enabled to startup the second software package (these two software packages do not like each others company when executing in the same browser it turns out).
6)   Private (incognito) browser sessions may affect this problem, but I have been unable to determine anything specifically.
7)   There may be other browser settings at play, such as caching, javascript (WebAssembly, JIT), browser vs. Unbound DNS blocklists, extensions, local storage, etc. may be at play .

Delighted to have both software packages now working as expected, albeit with a lot of unanswered questions as to exactly why and why they behave differently with different browsers.