[SOLVED] Cannot ping ISPs DNS servers since upgrade from 17.1.11 to 17.7

[tarly123]

Hi,

since the upgrade from 17.1.11 to 17.7 I cannot resolve external hostnames anymore.

My network:
Internet <--> ISP box (in routing mode, DHCP server enabled) <--> OPNsense box (retrieves DNS server IPs per DHCP from the ISP box) <--> Clients

After some troubleshooting I found out that the OPNsense box is unable to ping my ISPs DNS servers.

There are two weird entries in the routing table of my OPNsense box:

Code Select Expand

Destination        Gateway            Flags     Netif Expire
195.34.133.21      00:0d:b9:XX:XX:XX  UHS        igb1
212.186.211.21     00:0d:b9:XX:XX:XX  UHS        igb1
The IPs are the two DNS servers from my ISP, and the gateway is the MAC address of the igb1 interface of my OPNsense box.
Shouldn't the gateway point to the IP/MAC of the ISP Box?

There are also two weird ARP entries:

Code Select Expand

? (195.34.133.21) at (incomplete) on igb1 expired [ethernet]
? (212.186.211.21) at (incomplete) on igb1 expired [ethernet]
The ISPs DNS server is obviously not in my local network, why does the OPNsense box have an ARP entry for it?

When I delete the two routes for the ISPs DNS servers manually, I can ping the DNS servers from the OPNsense box again and everything works. But this route gets recreated on each boot.
Has anybody a tip what's the issue here?


Thanks,
Andreas

[franco]

Hi Andreas,

Are you using IDS in IPS mode? We have multiple reports of this since Suricata 3.2.3. It may also have to do with Hyperscan. We're closing in on the issue...


Cheers,
Franco

[tarly123]

Hi Franco,

no, I don't have Intrusion Detection enabled.
Is there any other service which could create this route? imho the problem has to do something with this route, because without the route everything works (i.e. the ISPs DNS server get routed by the default gateway, which is the ISP box).


best regards,
Andreas

[franco]

It would indicate something in the form of gateway rules redirects your requests. Do you have firewall rules with gateways set? Have you checked the firewall log for blocked packets on port 53?


Cheers,
Franco

[tarly123]

What are gateway rules? Firewall rules where the gateway field is something other than default? I don't have any of these.

I already checked if the firewall is the issue, but there are no blocked requests, requests to destination port 53 go through by the "@63 pass out log all flags S/SA keep state allow-opts label "let out anything from firewall host itself" rule.

Some update: During night the route reappeared and I had to delete it again.


best regards,
Andreas

[tarly123]

Alright, I've found the line where this route gets created: https://github.com/opnsense/core/blob/86996d7bf74d7eadcd0879d8edb5aa3d7f807b32/src/sbin/dhclient-script.ext#L274
This line adds this route with the wrong gateway (the MAC address of the specified interface of the OPNsense box). Probably the gateway should be specified explicitly?

The only thing that wonders me is that this file wasn't touched for about 2 years. And it was working with version 17.1.11 ???


best regards,
Andreas

[deviantintegral]

I've got exactly the same problem since upgrading. Deleting the two routes for each DNS server fixes the problem, and renewing the WAN IP brings them back.

[franco]

Unset [ x ] Allow DNS server list to be overridden by DHCP/PPP on WAN under System: Settings: General.

Do you guys have nameservers set on that general settings page?

[franco]

PS: Please also try this patch... https://github.com/opnsense/core/commit/051e44ca7

# opnsense-patch 051e44ca7

Either with override on or off (let us know which works).

[deviantintegral]

The patch didn't do anything, regardless of the override setting. I did have that enabled.

However, this pointed out a better workaround for now - if I statically set DNS, the bad routes aren't added.

[qwerty]

Hello everyone,

just did the upgrade to 17.7 yesterday evening and no internet, respectively can't ping nor do a DNS lookup on my ISP DNS, too. Workaround with a static DNS (for example Google) works.

I have no floating rules and don't use surricata (anymore).

regards

[martap]

I have this exact same issue..

Manually deleted the routes for my ISP's DNS servers and all works fine. Obviously that's not really a fix,
so for now I just unchecked the "Override DNS by ISP DNS servers" and manually entered google's DNS servers.

[luhe]

Just wanted to note that I too was affected by this issues after upgrading to 17.7.
If there is anything else to help debuggiung/fixing it, feel free to ask me.
Workaround with overriding DNS servers helped.

[franco]

Sure, try this patch workaround:

# opnsense-patch 0b38eff5f
# /usr/local/etc/rc.filter_configure

We're considering moving back in 17.7.1 the rules generation we changed for 17.7 slightly, but in our analyzed cases it points to a setup quirk in multi-wan. The search is ongoing. Any feedback helps.


Thanks,
Franco

[luhe]

Thanks for your quick response!
I re-enabled "Override DNS by ISP DNS servers" and can confirm that 0b38eff5f fixed the problems - DNS queries are now responding again.

Thank you for your great help,
Lukas