Android -- some apps load very slowly after upgrade to 23.1

[jwketchum]

Following upgrading to 23.1, some apps running on two Android devices started loading very slowly.
Android devices:  Samsung Tablet, Pixel 5a phone
Slow apps:
-- loading an Excel file from OneDrive -- this can take several >>minutes<< unless I am running NordVPN on the tablet, in which case loading is immediate, as it was before upgrade to 23.1.  File size is 240k.
-- Starting Google Drive and loading a file -- not quite as slow as Excel File, but it can take up to a minute for Google Drive to display the directory, then another extended wait to load a pdf or word file.  Not a problem with the VPN running.
-- Running Wunderground weather app on either the tablet or the phone.  The app starts immediately, but takes an extended time to load any information.  Again, not a problem when running the VPN.  Also, not a problem for the phone when connected the the cellular network, not the LAN with the opnsense router.
Other devices on the network include Linux desktop running  Ubuntu 22.04, Windows 10 laptops, iphone, ipad.  None of these other devices are having similar problems.
Opnsense configuration:
OPNsense 23.1.5_4-amd64
Intel(R) Atom(TM) CPU E3845 @ 1.91GHz (4 cores, 4 threads)
Unbound DNS
I did not operate any of the intermediate versions of 23.1, but went directly to 23.1.5_4 release.
Is there something funky about how Android does DNS that causes a problematic interaction with Opnsense?
Thanks in advance for any help.

[Galaxy]

This seems similar to the issue I'm having:
https://forum.opnsense.org/index.php?topic=33559.0

Although most of mine aren't just slow, but time out entirely. All work fine on cellular.

[jwketchum]

Galaxy -- thanks for your reply. 
Suspecting (with little to no evidence) that this is a problem with DNS, I tried changing DNS servers, and related settings, to no avail.  Then I fired up tcpdump with a Wireshark client and tried to find some obvious anomaly in DNS exchanges related to the apps that I am having problems with, also to no avail.  Lots of DNS traffic between DNS clients on the LAN and opnsense, much more sparse DNS traffic on the WAN.
Final attempt today was to remove the deprecated dyndns plugin and instead install os-ddclient but leave it deactivated, and parked the mostly unused domain name that was pointing to my OPNsense router.  This seemed to improve things for some, but not all, apps that are having problems on my phone and tablet.
Have you been using dynamic DNS?
I don't know why dynamic DNS would be causing problems, and it is frustrating that making that change led to apparent partial resolution, so that I don't really know if it has anything to do with the problem.
DNS settings on Android don't seem to make any difference either, but how DNS works in current Android releases is pretty opaque to me.
I would love it if someone could suggest other debugging steps or opine about what the root cause of this issue might be.

[Galaxy]

I have AdGuardHome as a plugin and all my DNS routes to that through Unbound. Disabling the AdGuard service doesn't seem to change anything so I ruled that out as a cause. At first I thought this was a DNS issue too but now I'm not so sure.

Does everything work ok for you in the few minutes following an OPNSense reboot? That's one of the distinguishing features of this issue for me, and I feel like it's a big hint as to the cause.

Also I've noticed its only a problem through the official apps. If I open the CoinGecko app it spins and spins then times out, but if I open a browser on that same device and go to coingecko.com it loads instantly. Is your experience the same?

[jwketchum]

I rebooted last night for the first time since completing the upgrade a few weeks ago, and it is not clear... one of the apps on my phone cleared up right after the reboot, but after that no improvement.  So if the reboot fixed anything, it only lasted for one or two minutes.
And, yes, it appears that the problem only pertains to apps, not through a browser such as Firefox, which is my default.  For example I use  the Weather Underground app.  If I browse to wunderground.com in FireFox on my phone, weather info loads immediately.  When I open the Weather Underground app on the same phone, the app opens immediately, but it takes literally a few minutes for any weather info to load.  Watching LAN DNS traffic while the app is loading info, it is clear that there is a lot of DNS activity related to ads in the app.  Not sure that is relevant.

[Galaxy]

Well after a dozen hours of troubleshooting I'm throwing in the towel. If you manage to figure anything out please post what you find here, I'll keep checking back.

If nobody figures out what's going on here I guess I'll just switch back to pfsense. I don't know what else to try and I'm getting almost 0 help from the community.

[jwketchum]

Yes, I am a bit frustrated too, but I don't plan to go back to pf.  I can work around this either by using a VPN or turning off wifi on my phone so it uses cellular.  Hardly ideal, but I would rather support this open source effort and give the issue a chance to play out.  Though I wish I at least knew whether the problem is a configuration issue or a bug of some sort in the code.
If there are only two of us having the issue (if it is, in fact, the same issue), it's going to end up pretty low priority.

[jwketchum]

Last night, I updated to 23.1.6, from 23.1.5_4.  My issues with >>extremely<< slow load observed in some apps on my two Android devices seem to have been resolved by the update.  My hunch that this problem was a DNS issue is supported by the fact that the update included a new version of os-ddclient.
The two Android devices are a Pixel 5a phone and a Samsung Galaxy Tab S7.  None of the non-Android devices on the network appeared to have similar problems -- iPhone, iPad, Windows 10 laptops, linux workstation and servers.
Still curious to know what the problem was and why it was constrained to Android devices -- what is different about how DNS works on Android (if in fact it was a DNS problem)?
I also have doubts about Galaxy's idea that the problem goes away briefly after a reboot.  When the opnsense router reboots, Android drops the wifi connection and reverts to cellular -- on the phone anyway, no cellular on the tab.  With the cellular connection, everything works normally.  Once the reboot is complete, it takes a few minutes for the phone to find wifi again and reconnect.
And that is one difference between Android and iPhone -- when the router WAN connection goes away for any reason -- either the router is offline or ISP WAN connection is down, the iPhone keeps its wifi connection and complains about lack of internet access.  Android drops the wifi connection when WAN internet access disappears and reverts to cellular.
My thanks anyway for the fix.  I try to stay away from the bleeding edge of major releases, to avoid issues like this.  I guess I jumped the gun by a few weeks this time.

[ritztro]

Had similar problems. Adguard was my primary DNS with unbound as my upstream. Certain apps on my Android phone did not work after I forced it to use my local DNS and disabled private DNS. I fixed this by disabling DNS64 support on Unbound.

I found that I had previously enabled DNS64 support on unbound because I was trying to set it up on my network, but gave up on DNS64 and disabled it across OPNSense. However, by leaving it on in Unbound, DNS was synthesizing fake IPv6 DNS responses for a DNS64 prefix which was not being actively forwarded. Other devices were apparently robust to this, but not Android apps.

After disabling DNS64 I had to flush Adguard and Unbound caches and restart apps + clear their caches.