Trouble with Wifi access points

[WolfpactVI]

Good morning!  Hoping you all can help me with a head-scratcher.

I have a small office of a few computers served by Comcast Business internet.  I finally got Opnsense with Comcast's crap system and crap modem/router box.  However, something weird happened along the way.  For years I'd been using three TP-link wifi routers (two C7 v2's and one C9 v6) in access point mode (plugged into switches via a LAN port, WAN ports unplugged, DHCP disabled, each assigned a static IP outside the router's DHCP range, each with different SSID's, etc).  Once I got the Opnsense box working, suddenly no device (mostly phones) connected via wifi to the TP-link boxes could access the internet.  They are all receiving IPs in the Opnsense DHCP range, they all have the Opnsense IP as the gateway.  I can ping all TP-link boxes from Opnsense and vice versa.  I can ping all phones from Opnsense.  And suddenly one phone (and still only the one) can now access the internet through one of the TP-link boxes.

The two things that are completely baffling me are 1) what would be causing traffic from plain old access points to be blocked, especially when everything can ping everything else?  and 2) the inconsistency of one phone getting through but no other (even an identical phone configured identically).

Anyone have any ideas where to start?

[WolfpactVI]

Well, I've narrowed down the problem.  When I switch one of the PC to use DHCP, it immediately loses internet.  If I manually put in Google's DNS or Cloudflare's DNS in the ethernet adapter settings, it gets internet again.  Seems everything going through Opnsense's DHCP is not getting DNS.  Everything on the Opnsense box is still default.  Unbound DNS is on.  In testing things, it seems to make no difference whether I put in 1.1.1.1 under the LAN DHCP settings or not, or under System:Settings:General or not.  I'm not super familiar with Unbound.  Is there somewhere I should be putting in a DNS like 1.1.1.1 in Unbound's settings, or does Unbound use it's own DNS IP address?  Or maybe Unbound is trying to use a DNS from upstream on Comcast's box, which is now turned off?  What is a good method to test if Unbound is working OK? 

[cookiemonster]

Unbound by default if enabled will go to root servers.
> In testing things, it seems to make no difference whether I put in 1.1.1.1 under the LAN DHCP settings or not,
leave them blank so Unbound serves from root servers

> or under System:Settings:General or not.
Leave also blank as above if you wish to continue with Unbound being your recursor.

There are tooltips on each of those above. The fields are to override.

Check that Unbound is listening to all interfaces (default). Default Rules will prevent it from being reached from WAN.
After that, restart the dhcp service on the client and look in the live firewall view. It should show the hits with the allow all rule. Then you can check unbound logs.

[WolfpactVI]

@cookiemonster, thanks for the suggestions.  When I had read through this article, https://homenetworkguy.com/how-to/confused-about-dns-configuration-in-opnsense/, what I thought it was saying, along with some other posts in this forum, was that even if DNS IP's were put in those two locations, it didn't matter because Unbound would ignore them.  Seems I had it exactly backwards.

So a couple of things.  When I switched a PC over to "obtain everything automatically", I did see the "allow access to DHCP server" in the log live view.  Unbound's log did not show anything related to that PC, and no errors or warnings, just Informational and Notice items.  Closest I could find was an occasional "daemonize unbound dhcpd watcher".  However, under Reporting:Unbound DNS:Reporting tab, everything is red with a return code of SERVFAIL and a resolve time of 0ms.  Bunch of entries there from the PC trying to reach different websites, and a bunch of stuff from Opnsense itself trying to check for updates (which incidentally no longer works after I removed all of the DNS IP's).

What should I try next?  Also, do you need screenshots of anything?

[cookiemonster]

It does matter but some overrides are for different services or for the system itself.
SERVFAILs explains that is the problem for the clients. DNS is failing.
So I suggest to go for a minimal configuration to begin with.
- system >general : leave DNS fields blank, post what options you have set in "DNS server options".
- services > DHCPv4 (I don't unse v6 so can't advise on that) > LAN : DNS servers leave blank.
- Services > UNbound DNS > General : what port are you using there?; Network interfaces (all recommended); what other options you have there?
- What other Unbound settings have you set?

- Restart Unbound if needed.
- In Unbound advanced section you can select to log queries and replies (only temporarily, lots of logging to disk) and increase the verbosity too for diagnostic.
See what it says.
With these settings, clients will get OPN's LAN ip as their given DNS server, Unbound will get them (firewall default rules allow it), and send them out to root servers, then returning them to your clients.

[WolfpactVI]

Under System: Settings: General:  all DNS entries are blank, all "Use Gateway" dropdown menus are set to none.  Under "DNS server options", everything is unchecked.

Under Services: ISC DHCPv4: [LAN], "Enable DHCP server on the LAN interface" is checked, and the Range is set from .125 to .200.  Otherwise everything is blank or unchecked including DNS.

Under Services: Unbound DNS: General:  Enable Unbound is checked, port is left at default of 53, three boxes are checked (which were not initially, but I added a few days ago to try and solve the DHCP DNS issue):   Enable DNSSEC Support, Register ISC DHCP4 Leases, Register ISC DHCP Static Mappings.

After the current config listed above, I did reboot the box, which did not seem to change anything.

For lack of knowing a better way, I did try scanning my network with AngryIP the other day just to see if something else was using port 53, but nothing showed using it at all.

I will try restarting Unbound service and enabling the logging you suggested.

I did notice under Services: Unbound DNS: Advanced that "Aggressive NSEC" was checked, though I did not check that, so I don't know if that is a default setting.

[WolfpactVI]

Here are two screenshots.  You can see the PC (.175) trying to reach youtube and google, and Opnsense itself trying to reach the update servers.

[WolfpactVI]

Second screenshot.

[cookiemonster]

Settings sound fine for Unbound itself. Aggresive NSEC I would disable for now.
So let's have a look around for the network setup. This only happens to clients connected to Wifi and those APs are in bridge mode and plugged into the OPN machine's LAN, right?

Modem --> WAN OPN ---> LAN OPN port --> Switch --Switch port 1 --> AP1
                                                                            -- Switch port 2 --> AP2

Is this correct ? If not, how is it? Could it be AP2 is is plugged into another OPN port and therefore in another network not yet setup?
And you have definitively verified that the APs are not doing DHCP, right?

[WolfpactVI]

@cookiemoster, amazing guess on the network setup without my having explicitly laid it out for you.  <smacks face>  Yes, that is exactly correct.  I have verified multiple times that no AP is doing DHCP.

As a further test, on my own desktop PC, (plugged into the switch) I changed it entirely over to DHCP.  Soon as I do that, no internet access.  It correctly chooses the Opnsense box for both gateway and DNS.  If I manually change just the DNS on the desktop PC to 1.1.1.1 or 8.8.8.8, immediately it has internet access again.

[cookiemonster]

Ok next is to check the firewall rules on the LAN are default. Enable logging of default rules (I don't think they log by default) and then you should follow them in the firewall live log.
If this exercise doesn't yield a diagstic result, next step IMHO is to do packet capture. Do you have a linux client in your lan for diagnosing? Ask because I don't know how well to diagnose with other OSes.

[WolfpactVI]

OK, I will turn on logging of default rules.  Is there a specific one I should be looking at?

I have a laptop with Mint installed on it that I could use for packet capture.

Thanks for hanging in with me through this!  I really appreciate it.

[WolfpactVI]

Logging was already on.  When switching the test PC to DHCP, a green log "allow access to the DHCP server" showed up, along with the "anti lockout rule" allowing access to the WebUI.  That was it.  Bunch of 127.0.0.1 from and to different ports (including port 53 of unbound) when attempting to check for updates.  I thought maybe it would show 127.0.0.1 trying to reach the WAN or the update server address when checking for updates, but it's always 127.0.0.1:port to 127.0.0.1:some other port.  Don't know if that matters any.  Would it be worth picking some other port for Unbound just to try?

If you have a link for a packet capture tutorial, or feel like explaining it yourself, I'd be grateful.

[cookiemonster]

what's the content of resolv.conf on opnsense, just to check?
For the capture, we want to do it on OPN. Luckily for us it's built in: Interfaces > Diagnostics > Packet capture.
Interface: LAN and WAN
Tick promiscuous
Address family: IPV4 (I assume you have IPV6 disabled everywhere unless you are on it)
Protocol: any
Leave default until count: 5000
Start the capture. Start from a client's browser a browse for a known website, say google.com
Go and stop the capture (little square on the line showing the capture is running). Then download it (cloudy icon on the capture line) and save it to your machine with wireshark installed.
The download will be compressed with zip. Uncompress it. Inside there'll be three files, one for each interface selected. Proceed to open both with wireshark.
In top field in wireshark you can then filter the packets to and from your client ip and for DNS. Try this expression:
ip.addr == 192.168.5.186 and  (udp.port == 53 || tcp.port == 53)
replace that ip with the one for your client. Let me know what you get. First for the LAN file. That should show the conversation between the client and OPN. We expect the failures but at least confirmation the two are "talking dns".
The other file is the wan and therefore the conversation between OPN and "something" upstream from it. I can't be specific yet because I don't know yet what you have setup on OPN for DNS. For instance in my case all those conversations are over TLS but that is not default. However see if you can spot the dns servers by looking on the protocol column for dns.

[WolfpactVI]

Sorry for the delay.  Busy weekend.

resolv.conf has only the following in it:

domain localdomain
nameserver 127.0.0.1
search localdomain

Packet capture had the following pair for google.com from igb0 (LAN) when the PC is switched to DHCP (and identical responses for any other website attempted to visit).  .2 = Opnsense LAN, .175 = desktop PC:

492   7.344390   192.168.10.175   192.168.10.2   DNS   74   Standard query 0xc9a5 A www.google.com
494   7.345286   192.168.10.2   192.168.10.175   DNS   74   Standard query response 0xc9a5 Server failure A www.google.com

On the WAN side (em0) packet capture, there are no DNS entries from any local device, just a bunch of DNS entries back and forth between the WAN address to outside DNS like Cloudflare (1.1.1.1) or Comcast (75.75.5.75).   Which is to be expected I guess.  Any device on the network with a static LAN IP is set to use our server as DNS first (which is then set to use Cloudflare) or Cloudflare second if the server's down.  Weird aside, the server is supposed to be using 1.1.1.1 for DNS too, but the timestamps indicate it seems to be the only device querying 75.75.75.75.  Not sure why, I can't seem to find any evidence of 75.75.75.75 in any settings on the server.  But I did inherit this setup, so I'm sure there are oddities left over from the previous setter-upper. 

Also, there were no entries on the WAN side in the packet capture at the 7.34 second mark to match the entries on the LAN side above.

It's like once on DHCP, a device's DNS queries are getting as far as Unbound on the Opnsense box and then just getting blocked or disappearing.  Thanks again for sticking with me on this.  I hardly ever post in forums any more because 99.9% of my issues have already been posted about and solved.  But this one really has me stumped.