Hi,
A few weeks ago I migrated from pfSense to OPNsense 23.1.7. It was running without issue. Yesterday I updated to 23.1.8, and a lot of clients on my network started to be assigned an IPv6 DNS server (in addition to the IPv4 one). This seemed to cause the network to go a bit haywire, not sure why? Currently using Unbound on OPNsense for the DNS server.
I completely disabled IPv6, the clients stopped getting the IPv6 DNS server, and everything seems smooth again.
Cheers,
Zolt
I have the same issue.
When visiting https://ipv6-test.com/ previously I constantly got 18/20 (missing two points are due to ICMP6), but with 23.1.8 it dropped down to 14 or even lower because the DNS part became very flaky.
On Windows clients when doing ipconfig /all I realized that with 23.1.8 a DNS IPv6 is handed out, which was not the case with 23.1.7_3.
On GNU/Linux and Android clients however, also with 23.1.7_3 the DNS IPv6 is handed out.
Most likely this is due to the different ways of handing out IPv6 to the OS (DHCPv6 vs. SLAAC, etc.).
I have not debugged any further but also reverted to 23.1.7_3 because that makes my network much more stable.
BTW: I'm using Dnsmasq on OPNsense as DNS server which forwards to a Pi-hole. Both, OPNsense and Pi-hole can be DNS-queried using their IPv6 ULA successfully when trying with dig/host/nslookup.
Quote from: Zoltrix on May 28, 2023, 05:42:47 AM
I completely disabled IPv6, the clients stopped getting the IPv6 DNS server, and everything seems smooth again.
Sounds like IPv6 started working after update. Disable if not required and that's it. The thing with defaults is that you have them set so they could be working or not... DHCPv6/Track6 are the default for WAN/LAN.
@sbellon how is that "the same issue" when you report the reverse of what OP said?
Cheers,
Franco
Now I'm confused ...
QuoteA few weeks ago I migrated from pfSense to OPNsense 23.1.7.
So did I.
QuoteIt was running without issue.
So was it for me.
QuoteYesterday I updated to 23.1.8, and a lot of clients on my network started to be assigned an IPv6 DNS server (in addition to the IPv4 one).
Exactly the same in my setup (for the Windows clients).
QuoteThis seemed to cause the network to go a bit haywire, not sure why?
Same here.
QuoteCurrently using Unbound on OPNsense for the DNS server.
Ok, I'm using Dnsmasq instead.
Sorry, but why do you think I reported the opposite?
Quote from: sbellon on May 28, 2023, 12:34:29 PM
QuoteA few weeks ago I migrated from pfSense to OPNsense 23.1.7.
So did I.
Ok sorry I cannot keep track of your on and off relationship with OPNsense.
Cheers,
Franco
I think we are confusing each other now.
I do NOT have an on and off relationship with OPNsense. I am consistently using OPNsense since 2021 when I migrated from Ubiquiti Security Gateway.
My "So did I" was misleading ... I was just pointing out what I did w.r.t. 23.1.7 and what has now changed with 23.1.8 ... I wanted to express "a few weeks ago I upgraded to OPNsense 23.1.7 and everything was fine with 23.1.7 like for the OP", not that I migrated to OPNsense 23.1.7 from pfsense (as the OP).
One of the reasons I migrated, was because the forums over at pfSense were borderline abusive, and not very supportive. Hoping to have a different experience over here.
Thank you for the comments franco, but why a default installation would cause network issues when IPv6 starts working, seems a little strange to me?
@sbellon
Apologies for misreading then. I'd always wish for more concise reports qmbiguity is the enemy of community support.
@Zoltrix
It's not unusual that the IPv6 defaults can be a factor of connectivity issues with certain ISPs. We have such reports regularly, mostly pertaining to problems with firmware updates. I don't think IPv6 connectivity issues reach the internal clients too often if such a fundamental issue exists. Also it has been known that some ISPs meddle with DNS resolution to the point where it looks like it's broken.
As such it's impossible to give a generalised assessment of what the issue is. Thus the rule of thumb is if you don't want or use IPv6 clear the IPv6 modes of your interfaces.
Cheers,
Franco
PS: as for IPv6 issues an overview of WAN and LAN IPv6 modes on your install helps as well as any relevant logs. If DHCPv6 (dhcp6c) is involved on the WAN it would make sense to enable the debug mode for IPv6 and reboot and gather the actual connection info from the client.
Quote from: franco on May 28, 2023, 04:50:37 PM
@sbellon
Apologies for misreading then. I'd always wish for more concise reports qmbiguity is the enemy of community support.
To be honest, I didn't even realize that the OP was coming from
pfsense, my brain just saw OPNsense, the same version number and "the same" (or similar) issue than mine.
Back to the actual issue: Would it help (or would you want that at all?) if I sent you my configuration via some private channels?
Today even my wife (using iOS) admitted that yesterday she turned off Wifi and used 5G in the house as "everything was unbelievable slow". Since downgrading to 23.1.7_3 everything is back to normal.
So, there IS something going on with IPv6 from 23.1.7_3 to 23.1.8 ... not saying it is the fault of OPNsense, perhaps its just the broken configuration that worked in the past and now needs proper adjustments.
With 23.1.8 a few things changed for IPv6 and how to handle it. So when you guys didn't have IPv6 with 23.1.7_3 at all and after 23.1.8 IPv6 came in I'd like to say it's showing that IPv6 is on the right way in OPNsense.
When your clients receive a DNS IPv6 server now and if this server is unreliable this can lead into issue. Of course IPv6 heavily relies on properly configured MTU and MSS. For example I have to setup MTU and MSS to 1492 as I'm using PPPoE.
Which IPv6 DNS server are your clients getting? Is it the OPNsense IPv6 or another one? How is your Router Advertisments service configured (Unmanaged, Assisted etc.)? Or do you have configured a traffic shaper without considering IPv6?
Edit: Apple devices, for example, first use DNS IPv6 if present and if it fails it will use DNS IPv4 as required by RFC. This can slow down connections sometimes. As france already mentioned, please provide us an overview of your IPv6 config.
I'll try to do a write-up what and how I configured my network regarding IPv6:
System / Settings / General
- DNS servers: IPv4 of Pi-hole
- NOT SET: Prefer to use IPv4 even if IPv6 is available
- NOT SET: Allow DNS server list to be overridden by DHCP/PPP on WAN
- NOT SET: Do not use the local DNS service as a nameserver for this system
Interfaces / WAN:
- IPv4 is PPPoE
- IPv6 is DHCPv6
- MTU is left empty but says "Calculated PPP MTU: 1492" (that's why I assume I can leave it empty)
- Request only an IPv6 prefix
- Prefix delegation size 56
- Send IPv6 prefix hint
- Use IPv4 connectivity
Interfaces / LAN:
- IPv4: Static IPv4
- IPv6: Track Interface
- MTU is left empty
- Track interface: WAN
- IPv6 Prefix ID: 0x0
- Allow manual adjustment of DHCPv6 and Router Advertisements
Interfaces / Settings:
- IPv6 DHCP: Prevent Release
Interfaces / Virtual IPs / Settings:
- IP Alias LAN for ULA of OPNsense
Services / DHCPv4:
- DNS servers: IPv4 of OPNsense
Services / DHCPv6:
- DNS server: IPv6 ULA of OPNsense (virtual IP alias)
Services / Dnsmasq DNS / Settings:
- Enabled
- Register DHCP leases
- Register DHCP static mappings
- Resolve DHCP mappings first
- Query DNS servers sequentially
- Require domain
- Do not forward private reverse lookups
Services / Router Advertisements / LAN:
- Stateless
- Use the DNS configuration of the DHCPv6 server
- NOT SET: Do not send any DNS configuration to clients
Again, this works perfectly fine with 23.1.7_3 and I do get a score of 18/20 on https://ipv6-test.com/, and if I even enable an ICMP6 rule in the firewall, I get a full 20/20.
After upgrading to 23.1.8, DNS resolution from various clients became slow (most likely running into various timeouts) up to completely unreliable. I noticed that on the Windows client, the IPv6 ULA of the OPNsense is handed out as DNS server to the clients, which is not the case with 23.1.7_3. On GNU/Linux I get the IPv4 and the IPv6 ULA of the OPNsense in /etc/resolv.conf with 23.1.7_3 and with 23.1.8 (same on Android). Have not checked with iOS, but as already mentioned, my wife's experience was reduced to an extend that she switched to 5G instead of Wifi...
I hope I did not forget anything important, otherwise please ask.
Thanks for the overview.
Is your Pi-hole able to resolve AAAA records and can your Pi-hole also provide an IPv6 address? Under settings general you can then enter the IPv6 address as DNS and assign it to DHCP6 gateway. The IPv4 DNS server can be assigned to PPPoE.
Try setting a MSS value of 1492 at WAN interface. Furthermore I would enable parallel DNS queries by disabling ,,Query DNS servers sequentially". If it doesn't work you could check ,,Do not send any DNS configuration to clients".
Hope it will help.
Quote from: Cyberturtle on May 28, 2023, 08:45:17 PM
Is your Pi-hole able to resolve AAAA records and can your Pi-hole also provide an IPv6 address?
Yes, to both.
QuoteUnder settings general you can then enter the IPv6 address as DNS and assign it to DHCP6 gateway. The IPv4 DNS server can be assigned to PPPoE.
This is not possible, as it results in an error message in the OPNsense GUI if I try to configure it:
The following input errors were detected:
You can not assign a gateway to DNS server "192.168.1.9" which is on a directly connected network.
You can not assign a gateway to DNS server "fe80::f7:b7ff:fe24:55ca" which is on a directly connected network.
And I think this makes sense, if I assign a DNS server to a gateway, then the meaning is "use this gateway to reach the DNS server", isn't it? But in my case the Pi-hole is on the local network, not outside.
I added the ULA IPv6 of the Pi-hole to System / Settings / General as a second entry in addition to the IPv4. I can verify that it gets written into /etc/resolv.conf of the OPNsense and I can query via its IPv6 and also AAAA type from within the CLI of the OPNsense.
Quote
Try setting a MSS value of 1492 at WAN interface.
So, you are saying that I should explicitly configure 1492 for both, MTU and MSS?
QuoteFurthermore I would enable parallel DNS queries by disabling ,,Query DNS servers sequentially".
I had this disabled because it shouldn't have made any difference if I only have one DNS server configured, should it? Now with the IPv4 and the IPv6 it would perhaps make sense to use the faster of the two (although as it is the same device, this would be for IPv4/IPv6 redundancy only and not performance).
QuoteIf it doesn't work you could check ,,Do not send any DNS configuration to clients".
But that's not what I want. I do want clients to configure their IPv4 and IPv6 DNS server via DHCPv6 and DHCPv6 (which works with 23.1.7_3).
Configuring at least MSS clamping with a value of 1492 should be done as IPv6 does not fragment packets like IPv4 does. Only the source is allowed to fragment but has to know which value has to be considered. I have also set MTU even it should be detected automatically.
I understand you that way that the IPv6 DNS server showed up initially with 23.1.8. So that's why I suggested disabling sending IPv6 DNS server in the router advertisement. So your clients are only receiving the IPv4 one. Just for testing and trial and error.
I don't have any knowledge about Pi-hole config so someone else can help hopefully.
Did dnsmasq receive an update with 23.1.8?
Edit: Can you please try test-ipv6.com? This site was able to detect MTU problems in the past.
Thanks for all your help and insights.
With test-ipv6.com I always get a full 10/10 (was always the case and still is with 23.1.8).
Will try to update to 23.1.8 once more and then see whether MTU/MSS makes a difference (but would this be expected on an update like 23.1.7_3 -> 23.1.8?).
I doubt the Pi-hole is the issue here. I have it running for years, and as mentioned, it resolves A and AAAA, listening on both IPv4 and IPv6, just fine.
DNS resolution has also become very unstable for me after upgrading to 23.1.8. Using Unbound and Cloudflare as upstream DNS.
I've been running dual stack IPv4/IPv6 on my current ISP with no issues for more than half a year, and nothing seams to have changed on their side.
Looking at Smokeping, resolving test.test on unbound from my local network, I see a huge difference after upgrading to 23.1.8. Spikes going over 800ms and even some timeouts. Internal latency is <0.7 ms.
DNS resolution from my wired laptop are now fairly consistent > 40ms (even for cached results) and before the upgrade they were < 1ms for cached results.
I used to have 20/20 on ipv6-test.com, but now various tests time-out (inconsistent between refreshes) so I end up somewhere between 10/20 and 18/20.
I'll try to downgrade to 23.1.7_3 to see if it helps.
Quote from: squarky on May 30, 2023, 10:50:25 AM
DNS resolution has also become very unstable for me after upgrading to 23.1.8. Using Unbound and Cloudflare as upstream DNS.
I've been running dual stack IPv4/IPv6 on my current ISP with no issues for more than half a year, and nothing seams to have changed on their side.
Looking at Smokeping, resolving test.test on unbound from my local network, I see a huge difference after upgrading to 23.1.8. Spikes going over 800ms and even some timeouts. Internal latency is <0.7 ms.
DNS resolution from my wired laptop are now fairly consistent > 40ms (even for cached results) and before the upgrade they were < 1ms for cached results.
I used to have 20/20 on ipv6-test.com, but now various tests time-out (inconsistent between refreshes) so I end up somewhere between 10/20 and 18/20.
I'll try to downgrade to 23.1.7_3 to see if it helps.
i have the same problem with 23.1.7_3.
i advice to go back to 23.1.6 most of people has tested this
Quote from: Julien on May 30, 2023, 11:15:59 AM
i have the same problem with 23.1.7_3.
i advice to go back to 23.1.6 most of people has tested this
Thanks for the tip. I actually just disabled IPv6 (as it's not critical for me for the moment - and have to get some work done) and everything is now working as a charm. DNS resolution back down to ~1ms for locally cached results (and 4ms for results fetched from Cloudflares cache).
I applied the patch mentioned in https://forum.opnsense.org/index.php?topic=34241.msg165713#msg165713 (https://forum.opnsense.org/index.php?topic=34241.msg165713#msg165713) and it fixed some issues, but no the DNS lookup issue.
I'm still confused what DNS changes have come with 23.1.8.
Yesterday I had slow DNS lookups too, but as far as I know Google public DNS servers had a few problems in Germany from 10 pm until 11 pm (local time zone: UTC+2).
Quote from: squarky on May 30, 2023, 11:58:05 AM
Quote from: Julien on May 30, 2023, 11:15:59 AM
i have the same problem with 23.1.7_3.
i advice to go back to 23.1.6 most of people has tested this
Thanks for the tip. I actually just disabled IPv6 (as it's not critical for me for the moment - and have to get some work done) and everything is now working as a charm. DNS resolution back down to ~1ms for locally cached results (and 4ms for results fetched from Cloudflares cache).
I applied the patch mentioned in https://forum.opnsense.org/index.php?topic=34241.msg165713#msg165713 (https://forum.opnsense.org/index.php?topic=34241.msg165713#msg165713) and it fixed some issues, but no the DNS lookup issue.
when you say disabled IPV6 do you mean on Firewall: Settings: Advanced and uncheck the IPV6 ?
on the page you provided i don't see a patch, which one do you mean?
Today 23.1.9 has been released with further IPv6 improvements. Maybe it solves your problem?
Quote from: Cyberturtle on May 31, 2023, 09:22:07 PM
Today 23.1.9 has been released with further IPv6 improvements. Maybe it solves your problem?
While it now seems to work again for GNU/Linux and Windows clients, my wife's iPhone is still unusable with 23.1.9 on Wifi.
I now took a look and the iPhone is correctly connected to Wifi, but says "No internet connection".
I checked the settings and they all look perfectly fine.
I think I'll revert back to 23.1.7_3 once more in order to see whether the iPhone reports some different setting in there.
I noticed that the iPhone reports "router" as the IPv4 LAN address of the OPNsense *and* the link-local IPv6 address as well. My Android phone only reports the IPv4 LAN address of the OPNsense as gateway. The Windows client also has IPv4 LAN and link-local IPv6 of OPNsense as gateway - and works.
But this indicates rather a gateway issue than a DNS issue now ...
Which Access Points do you use? My iPhone is working perfectly fine with 23.1.9. It also reports link local IPv6 address as router and the IPv6 address of the DNS at the DNS section.
TP Link APs do not handle IPv6 correct and are blocking some IPv6 traffic for example (especially on Apple devices). Had this in the past and switched to UniFi because of this.
Android does only support SLAAC where Apple supports DHCPv6 and SLAAC (can be important if DNS servers are pushed via DHCPv6).
"Sorry" to say, but UniFi UAP-HD and UAP-Pro here.
Hm, perhaps I should try NOT pushing the IPv6 DNS via DHCPv6 then ...
Do you have any multicast enhancement or IGMP snooping enabled? With recent iOS changes this can lead into issues with UniFi as well. I have turned off any enhancements. Only plain WiFi for private and guest.
Just an idea.
"Enable multicast enhancement (IGMPv3)" in UniFi is NOT turned on.
I'm using only SLAAC in unmanaged mode and my iPhone is setting the correct DNS IPv6 server of the router itself. So it's worth a try to disable sending DNS info via DHCPv6.
Ok, what I actually did to (hopefully) fix it:
I had previously entered the ULA IPv6 of the OPNsense that I have configured via Virtual IP in the DHCPv6 DNS servers to hand out.
Now I removed that setting and left "DNS servers" in the DHCPv6 configuration empty, thinking that then *no* IPv6 DNS server will be handed out, but instead the global IPv6 from WAN interface tracking is handed out via DHCPv6 to the clients.
But this works!
So, my assumption for now: Dnsmasq did not listen on the Virtual IP. In Dnsmasq settings I have only two of my network interfaces selected, but there is no way to additionally selecting the Virtual IP.
My explanation cannot be the reason because with
root@opnsense:~ # ps auwx | grep dnsmasq
I do see the Virtual IP on the LAN interface listed on the command line as --listen-address=fd01:... perhaps iOS just does not like that ...
Quote from: Cyberturtle on June 01, 2023, 04:38:57 PM
Do you have any multicast enhancement or IGMP snooping enabled? With recent iOS changes this can lead into issues with UniFi as well. I have turned off any enhancements. Only plain WiFi for private and guest.
Just an idea.
@Cyberturtle, thank you, I had also the problem that IPv6 was not working. The fallback to IPv4 worked in my case.
I am using also UniFi APs (U6 Pro). In my case the MAC address from the OPNsense NIC I am now using was missing in the setting of the
Multicast and Broadcast Control Exceptions list.
Quote from: sbellon on June 01, 2023, 07:42:04 PM
My explanation cannot be the reason because with
root@opnsense:~ # ps auwx | grep dnsmasq
I do see the Virtual IP on the LAN interface listed on the command line as --listen-address=fd01:... perhaps iOS just does not like that ...
Im using a ULA for my DNS server (Adguard Home) and it works with iOS now again. And it used to work till I reworked my OPNsense and changed the NICs.
KH
Quote from: sbellon on June 01, 2023, 06:17:38 PM
Ok, what I actually did to (hopefully) fix it:
I had previously entered the ULA IPv6 of the OPNsense that I have configured via Virtual IP in the DHCPv6 DNS servers to hand out.
Now I removed that setting and left "DNS servers" in the DHCPv6 configuration empty, thinking that then *no* IPv6 DNS server will be handed out, but instead the global IPv6 from WAN interface tracking is handed out via DHCPv6 to the clients.
But this works!
So, my assumption for now: Dnsmasq did not listen on the Virtual IP. In Dnsmasq settings I have only two of my network interfaces selected, but there is no way to additionally selecting the Virtual IP.
Entering no address uses always the interface address as far as I know. Apple devices uses link-local addresses for router and the prefix ones for DNS. Background was many years ago the problem of AdvRDNSS advertisements. Sometimes they are getting lost and than either the IPv4 address for DNS is used or the IPv6 router address for getting info for DNS resolution. Don't know if it is still like this. Nevertheless it's nice that it is working now on your side.
Quote from: KHE on June 01, 2023, 08:20:12 PM
@Cyberturtle, thank you, I had also the problem that IPv6 was not working. The fallback to IPv4 worked in my case.
I am using also UniFi APs (U6 Pro). In my case the MAC address from the OPNsense NIC I am now using was missing in the setting of the Multicast and Broadcast Control Exceptions list.
KH
Nice that it is working now. You're welcome :)