Hello every one,
I have installed my opnsense box a few time ago and since the beginning i have an issue with the android devices on my network.
They are all connected to wifi by an access point connected by ethernet on my network.
If we use them to browse with http/https sites, all is fine . Youtube App is also working.
But a lot other apps are not working. It seems theses apps are having some time out : bank app, Deezer app, Microsoft authenticator, ...
As soon i deconnect the device from wifi they are working properly.
All the phones and tablet are concerned.
What confuses me it's the lack of logs to make a diagnostic: i don't see in the firewall logs any blocked queries for these devices when i made some test, neither in the dns logs.
Anybody could give me some advice to investigate my problem as i am searching from several months without any success.
I have already checked my firewal rules, dns configuration, ipV6 configuration.
I have made a lot of tries, all have failed. Thanks to the backup configuration functionality.
I thank in advance everyone who agrees to help me because I have exhausted all my ideas.
Mins
No one has an idea of what i can do to log more accurately what's happen between a specific device and my router while i make some tests to diagnose my problem ?
Mins
Ping
The problem is that there are too many variables to consider and nobody knows your setup. You need to narrow down the problem, post your relevant setup. if you don't know what is relevant, imagine how everyone else in the forum is unknowing of it :)
I'd start with DNS. What's your DNS setup? The whole of it: what provides DNS to your network (is it OPN dhcp, unbound, dnsmasq, pi-hole in another device), what is your network infra like, VLANs, switches, etc.
And yes, there are separate logs for separate services. So my advice is grab a pc/laptop into the wifi and start diagnosing from it. Ideally not Windows (why I hear you ask? because I for one don't know how to use diagnostics with commands from it).
Hi,
I'm aware my problem is not easy to investigate.
That's why my last question is what i can activate as logs in my opnsense box to see where the trouble begins because currently i see no request failed neither in dns or firewall logs.
My setup is this one :
- my opnsense box is a little vault protectli like this one : https://eu.protectli.com/vault-4-port/ My mistake was to think opnsense is an all in box system and the dns was provided.
- the Dns is UnboundDns running on the vault
- the vault is linked to an access point netgear WAX 214 like this one https://www.netgear.com/business/wifi/access-points/wax214/
- i don't us any vlan
What i noticed is the problem seems to affect all the android devices using wifi.
For example when i connect a smartphone to my pc to diagnose the problem using adb, i used to share the pc network connection to the device and the problem suddenly vanished.
All the devices acquire an ip v4 and ip v6 from the dhcp running on the vault but I haven't been able to determine if the issue is related to ipv6 or not but ping -4 and ping -6 to google are ok.
I know this is a difficult question so i thank you very much for any help you can provide.
Mins
Different services each has its own log. They are mostly in /var/log/. For instance Unbound is /var/log/resolver/latest.log. This one has a UI to look at too. Then there are settings for the service to increase the verbosity and include additional messages: Services > Unbound DNS > Advanced. There you can dial up the logging whilst diagnosing. Uses more storage so be sure to reduce it later.
So if you were to diagnose DNS, you could increase this logging for failures like NXDOMAIN. If you can see the name resolution when using the app, then you know the name resolution part is OK.
Then you can move to the firewall side. Similarly you can look in the UI for the incoming requests.
Additional logging which is default off is in System > Settings > General. Again, careful with storage. Go back to defaults afterwards.Tooltips will help.
Finally, the problem could be with IPv6 which I don't use, so can't advise on that.
Could the DNS silently drop the queries ?
I observe the clients waiting a long time before displaying an error as if they were waiting the name resolution without never getting it.
In case there is an active functionality dropping the queries (i think a kind of adblock), Would this cause this kind of issue ?
Mins
I have increased the loglevel and i am trying to forward them to another host with some sort of kibana to analyse them more easily
Work still in progress ...
Mins
> Could the DNS silently drop the queries ?
Not by default.
You should describe your full infrastructure setup. There might be other elements in play.
Everybody else can just plug a wireless AP into a switch that is connected to OPN by ethernet and has no problems:
Internet -> WAN - OPN - LAN-> switch -> AP -> wifi clients
|
---------------> wired clients
Indeed that's exactly my setup :
Internet -> WAN - OPN - LAN-> switch -> AP -> wifi clients
| |
| ---------------> wired clients
- OPT1-> Synology
Same here !
set-up:
LAN1 192.168.101.101/24 => Switch1
LAN2 192.168.102.101/24 => WiFi AP
LAN3 192.168.103.101/24 => Swicth2
Fpor some reason each LAN can't communicate with the others (but that's another thread)
Devices on WiFi AP are all Android or IoT, and all have full Internet access.
But I have 2 app that can't connect to Internet, but can't see/find any packet dropped, any blocked traffic (or I don't look in the right place)
FW rules are all defaults
IDS/IPS on/off doesn't change anything
FW disable leads to no Internet access, so I couldn't try without it :/
> Fpor some reason each LAN can't communicate with the others (but that's another thread)
Only first LAN interface has an allow all out rule. New interfaces and networks need it creating explicitly.
> Devices on WiFi AP are all Android or IoT, and all have full Internet access.
> But I have 2 app that can't connect to Internet, but can't see/find any packet dropped, any blocked traffic (or I don't look in the right place)
If all else works on the Android device except the app and no other services on the firewall enabled, then it suggests the problem not on OPN, no ?
Zenarmor enabled ?
Quote from: cookiemonster on October 10, 2024, 06:26:32 PM
> Fpor some reason each LAN can't communicate with the others (but that's another thread)
Only first LAN interface has an allow all out rule. New interfaces and networks need it creating explicitly.
Yes, they have it, the two extra rules "Allow all" on LAN1 cloned to each other LAN
Quote from: cookiemonster on October 10, 2024, 06:26:32 PM
> Devices on WiFi AP are all Android or IoT, and all have full Internet access.
> But I have 2 app that can't connect to Internet, but can't see/find any packet dropped, any blocked traffic (or I don't look in the right place)
If all else works on the Android device except the app and no other services on the firewall enabled, then it suggests the problem not on OPN, no ?
Zenarmor enabled ?
No Zenarmor, no extra application, it's pretty much all stock,
The only mod I did is to anable IDS/IPS, but as said in previous, disabling it didn't change anything :/
I'm pretty sure (as it is a secured app, like banking or gouvernement or crypto or such) it's a matter of port and layer used and being bloked, but I can't see which
Ah!. Unlikely unless you hare doing something wacky with certificates and breaking TLS.
Some of those use certificate pinning.
To me the next step in diagnostic is to do a packet capture and analysis.
You are using Unbound, right ?
And do they (the apps) give some error or some indication of the problem?
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
Ah!. Unlikely unless you hare doing something wacky with certificates and breaking TLS.
Some of those use certificate pinning.
If it is so, then it's completely out of my wish/controle, I just use the tablet the same way I did while connected to my previous router, which was much of a strainer, letting most going through, hence me on OPNsense now
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
To me the next step in diagnostic is to do a packet capture and analysis.
I would be happy to oblige, using the search I found "packet capture" and set it up to interface LAN2, IP 192.168.102.103 (Tablet)
And will post it below
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
You are using Unbound, right ?
And do they (the apps) give some error or some indication of the problem?
Unbound, yes
with or without blocklist (AdWare, ...) doesn't change anything
The app asks for passphrase, then spin for about a minute and then drop saying: -"Sorry, there was an issue processing your request, please try again later" kinda standard msg
Packets captured on LAN2 for IP 192.168.102.103
View capture
Interface Timestamp SRC DST output
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:13.458246 Tablet MAC LAN2 MAC IPv4, length 688: 192.168.102.103.48346 > Public IP : tcp 622
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:13.570098 LAN2 MAC Tablet MAC IPv4, length Public IP > 192.168.102.103.48346: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:13.580465 LAN2 MAC Tablet IP IPv4, length 480: Public IP > 192.168.102.103.48346: tcp 414
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:13.584274 Tablet MAC 192.168.102.103 LAN2 MAC IPv4, length 66: 192.168.102.103.48346 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:17.972276 Tablet MAC 192.168.102.103 LAN2 MAC IPv4, length 74: 192.168.102.103.33891 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:18.875502 Tablet MAC Unknown MAC or IPv6 ? IPv4, length 133: 192.168.102.103.5353 > Public IP.5353: UDP, length 91
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:19.089311 Tablet MAC LAN2 MAC IPv4, length 66: 192.168.102.103.38107 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:21.742308 Tablet MAC LAN2 MAC IPv4, length 66: 192.168.102.103.60959 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:21.863101 LAN2 MAC Tablet MAC IPv4, length 66: Public IP.443 > 192.168.102.103.60959: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:21.866320 Tablet MAC LAN2 MAC IPv4, length 66: 192.168.102.103.60959 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:24.643595 Tablet MAC LAN2 MAC IPv4, length 74: 192.168.102.103.46383 > Public IP.443: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:24.665528 LAN2 MAC Tablet MAC IPv4, length 74: Public IP.443 > 192.168.102.103.46383: tcp 0
IGC2_Cisco6co_ETH3_CAT7blue
igc2 2024-10-09
18:45:24.669333 Tablet MAC LAN2 MAC IPv4, length 66: 192.168.102.103.46383 > Public IP.443: tcp 0
Is it normal that the public IPs have 5 sections, the last one being .443 ? it looks like the port, but it should be :443 rather than .443, right ?
Quote from: MarieSophieSG on October 11, 2024, 12:10:04 AM
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
Ah!. Unlikely unless you hare doing something wacky with certificates and breaking TLS.
Some of those use certificate pinning.
If it is so, then it's completely out of my wish/controle, I just use the tablet the same way I did while connected to my previous router, which was much of a strainer, letting most going through, hence me on OPNsense now
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
To me the next step in diagnostic is to do a packet capture and analysis.
I would be happy to oblige, using the search I found "packet capture" and set it up to interface LAN2, IP 192.168.102.103 (Tablet)
And will post it below
Quote from: cookiemonster on October 10, 2024, 10:04:00 PM
You are using Unbound, right ?
And do they (the apps) give some error or some indication of the problem?
Unbound, yes
with or without blocklist (AdWare, ...) doesn't change anything
The app asks for passphrase, then spin for about a minute and then drop saying: -"Sorry, there was an issue processing your request, please try again later" kinda standard msg
Right, thanks for this.
5 sections seem right from the header: Interface Timestam SRC DST output.
Do me a favour. Can you provide the capture in the download format. In other words, set it and when finished, provide the file instead of this pasted output. That way I can put it in wireshark and see more easily and quickly.
Interfaces: Diagnostics: Packet Capture (what you did).
Select the interface where the traffic is coming IN from the firewall's perspective, which is the netowork your client is.
Select promiscuous. All other as any/ and defaults except the count. Set it to say 10000 just to ensure it won't runaway.
Then start the capture, trigger what you want to test with the client attached to that network/interface, wait for it to do its thing, could then move to open a tab on browser for instance and navigate to a known place, say google.com so we have a postitive one to compare against on the same capture.
Then stop the capture, download the file (will be a compressed one possibly). Attach or send to me if you're more comfortable and I'll give it a check. If you want.
Will do, as soon as I get my GUI access back .. :/
Ha !
I've re-started the VPN, and now I have access to App2 !
Still not with App1 (gov.cnct), but it's going in the right direction (at least as user, not so much as admin)
So my VPN going through, that means, I suppose, the IPs the apps was trying to reach was block/denied by OPNsense, but I've tried without IDS/IPS on, and without Blocklist, so IDK what could have blocked these ?
I imagine there are no blocks or they would still be failing. More likely to be some sort of routing problem.
Remember a VPN is another network, albeit one that is inside/alongside another depending on your point of view.
Why restarting it unclogged things ? Dunno. Some connection became stale, gremlins, will prob never know.
What you do want to know as an admin is what are your network routing setup with those i.e. their interactions.
I've mentioned it before, draw yourself a diagram and keep it updated. Easier for everyone when trying to convey a message.
My understanding is that the VPN is only 1 IP, so OPNsense only sees that one, and not those that are reached out from the VPN server, therefore invisible to OPNsense, which can't blcok them
A diagram ? I guess that means much more detailed than just:
LAN1 192.168.101.101/24 => switch1 DHCP 192.168.101.116-122
But rather this, plus all the FW rules, the DNS, the redirection, the grinlins and goblins, the packet filter, blocklists, IDS/IPS,
I guess that would be a good exercise indeed, and not only for this forum, but also for my own little self to undertand and have a clear picture of my gates.
Great. That diagram works.
So where is the VPN, an app installed on a device, which one?
I thought you meant the VPN was set as a VPN client on OPN to a provider like say Surfshark or even a rented vps. Can you elaborate?
p.s. you seem to have two ips on the same network for the same device (NAS). That can cause problems, unrelated to these apps though.
Quote from: cookiemonster on October 12, 2024, 12:16:38 AM
Great. That diagram works.
So where is the VPN, an app installed on a device, which one?
I thought you meant the VPN was set as a VPN client on OPN to a provider like say Surfshark or even a rented vps. Can you elaborate?
p.s. you seem to have two ips on the same network for the same device (NAS). That can cause problems, unrelated to these apps though.
VPN all all of them, except a bunch of IOT for which I will direct the Alias _IOT to Wireguard, but that's for later
My VPN account allow me for ten devices, the 7th will be the OPNsense box to cover all of them
The two IPs goes to the two network interfaces of the NASes, then managed internally, so to the router, it's two NASes, even if it's the same physical one
I did a full re-install, but neither apps are connecting, so it is not an error on my side in whatever configuration I did before :-p
In case you missed it:
Quote from: Patrick M. Hausen on October 11, 2024, 12:58:03 AM
Quote from: MarieSophieSG on October 10, 2024, 11:43:12 PM
The NASes have two network interfaces,
NAS1 has 2x 2,5 GbE and NAS2 has 2x 1GbE, with a failover (if one is down, or one is overloaded, traffic goes to the other)
Each independant from the other, so I can, if I want, connect 1 laptop to 192.168.101.111 as root, and 1 laptop to 192.168.101.112 as user
This is fundamentally impossible in networking. A system cannot have two interfaces in a single network. Period.
One possible cause of your problems.
As for the apps failing with their VPN app whilst in your network, back to packet capture for clues.
As you mentioned Zenarmor, I went to see github and their website, they claim they can filter by application ... isn't that what I would need, or is it way above my grade ?
For now, since re-install, I have both apps connecting through VPN, but still not (neither) without VPN
Remaining of the msg moved to its rightfull thread https://forum.opnsense.org/index.php?topic=43205.45
Quote from: MarieSophieSG on October 12, 2024, 06:56:15 PM
But *I am* connected to both interface ...
Then everything works as you intend it to do and we can close all the threads. Right?
The fact that you can plug two interfaces of one box into the same switch and have both get an IP address via DHCP does in no way confirm that this is in any way a supported topology in IP networking. It's not.
Read this article, please:
https://www.truenas.com/community/resources/multiple-network-interfaces-on-a-single-subnet.45/
If you think you know better than me, I am obviously in no position to help you.
Kind regards,
Patrick
Quote from: Patrick M. Hausen on October 12, 2024, 07:30:26 PM
Then everything works as you intend it to do and we can close all the threads. Right?
For this thread, about Android apps, yes.-ish, after I get the box back up to running as expected, I will have a look at Zenarmor to investigate further what these two have so different from the others that they can't connect through OPNsense without a VPN
Remaining of the msg moved to its rightfull thread https://forum.opnsense.org/index.php?topic=43205.45
I've installed Zenarmor
It doesn't say much about apps and such, at least not in the "free" version ...
So I guess I wil wait for the next blockage to re-run some deeper tests
I'm having similar symptoms.
- Apps on Android clients often (but not always) time-out when they're connected to wifi.
- Switching off wifi and back on again can usually restore app connectivity temporarily, even for apps
- HTTP/HTTPS browsing is always fine.
I've tried to trace any blocked requests from android clients, but haven't had any luck finding anything yet.
Because of #2, it leads me to believe it's a DNS issue of some sort. It's as if the local network is blocking DNS calls and once it's on mobile, DNS requests are cached temporarily. In my case, I started with unbound DNS forwarding to ad block, but by now, I've removed ad block functions and even removed DNS forwarding with no change.
@minskaya I know this is an old thread, but did you ever find the cause?
Subscribing for updates.
Finally fixed by disabling ipv6 on WAN and LAN interfaces. All android clients (seem to be) working now after several days of observation.
Hope this helps someone else down the road!