OPNsense Forum
Archive => 19.1 Legacy Series => Topic started by: princ3ssa on April 11, 2019, 06:18:39 pm
-
I started a thread on Reddit (https://www.reddit.com/r/OPNsenseFirewall/comments/bb76nw/how_to_troubleshoot_client_and_server_dns/) and thought perhaps I should bring it over to the forum for further discussion.
I don't really want to repeat my initial remarks from Reddit here, so I figure I'll summarize the most salient points (unless someone would like for me to pull everything into this).
As indicated, I'm following this guide for pfSense (https://www.techhelpguides.com/2017/06/12/ultimate-pfsense-openvpn-guide/) (but it's been nearly a one-to-one translation so kudos to you guys for keeping things so clearly similar while improving the UI!!) and have gotten stuck with DNS issues, specifically in STEP 11.
I'm on OPNsense 19.1.5_1-amd64 and using Unbound (Dnsmasq is disabled) and I have verbosity set to level 3 for Unbound. I'm able to watch the Unbound logs with "clog -f /var/log/resolver.log".
I have OPNsense set up in an isolated test network with 2 clients attached (1 Windows 10 with DHCP & 1 Linux with a static IP). OPNsense is running at 192.168.1.1. The Windows and Linux clients are both able to ping OPNsense AND 8.8.8.8 (so traffic is fine both locally and out to the internet). When I assign DNS manually to either client (8.8.8.8 ) they can resolve hostnames fine and browse the web, but when 192.16.1.1 (OPNsense) is used as the DNS server on either system, no name resolution occurs.
There are no updates in resolver.log when websites are visited or name queries should be made.
I found that when I do "telnet 8.8.8.8 53" I am able to get a response (well it hangs up on me since it needs a binary understanding client vs telnet), but when I do this for OPNsense ("telnet 192.168.1.1 53") it times out and there is nothing within the resolver.log.
I don't know all the fields in /var/log/filter.log, but saw that filter.log was being updated pretty quickly and so I filtered for "53" ("clog -f /var/log/filter.log | grep 53") and saw this:
(https://i.ibb.co/XZVW3Nq/telnet-result.png)
Each of the 3 longer lines there is a time I did "telnet 192.168.1.1 53".
I have to think there's some kind of firewall or some other issue blocking Unbound (port 53) from replying..... What could be going on?
-
DNS uses UDP normally and not TCP so you can't telnet to a UDP port.
As I noted in the other post, you need to allow Unbound on your OpenVPN Interface if you want it to work and make sure the network has access to it.
https://imgur.com/JAZzrSJ
-
I thought you were saying to enable it on Network Interfaces from your other reply (https://www.reddit.com/r/OPNsenseFirewall/comments/bb76nw/how_to_troubleshoot_client_and_server_dns/ekx2wsk/):
(https://i.ibb.co/Pcts3vr/interfaces-1.png)
But from your screenshot it seems you meant to go to Services > Unbound DNS > Access Lists. This is very confusing since this is not a VPN with a known IP address (not even a known subnet), but rather a privacy VPN (like Private Internet Access or PIA) functioning with various VPN nodes and various name servers around the world.
This is why the other pfSense guide is more pertinent and nearly spot on except for this one particular DNS issue so far it seems. Up to this step (Step 11, Method 2 of 12 steps) this has gone relatively smoothly and it's amazing at how direct the interfaces correlate except for a few relatively minor quirks. This one probably isn't a big deal either if I can just figure out what's going on here and understand how to implement this proper access or maybe interface listening.
If I try "nc -uv 192.168.1.1 53" I do get "Connection to 192.168.1.1 53 port [udp/domain] succeeded!" and firewall traffic appears:
(https://i.ibb.co/jRqMz4k/udp-traffic.png)
It also looks like from the Unbound log that something did happen there when I ran netcat:
(https://i.ibb.co/hWPGwCH/unbound-log.png)
So I guess I am not understanding something correctly like you're saying with the VPN access and then how to deal with Access Lists....
-
I still am not following what you are trying to do.
If you want to setup PIA for VPN for clients, you normally would push PIA's VPN to your clients so you don't 'leak' any DNS information and everything is contained.
If you want your PIA clients to resolve using your local DNS (DNSMasq or Unbound), you can point to that, but you need to make sure those clients have access.
That guide is fairly dated and seems to be missing some steps.
PIA has their own guide for pfSense which would be a better starting point.
https://www.privateinternetaccess.com/helpdesk/guides/routers/pfsense/pfsense-2-4-3-setup-guide
as that routes all traffic through.
Routing certain clients through takes a bit more effort as there a number of post relating to that on the pfSense forums.
If you hit search on the forums here, you found hits on how to do that on OPN.
https://forum.opnsense.org/index.php?topic=8998.0
-
No, I do not want to forward that information to clients, but instead want to use the method outlined in the pfSense document that many PIA users are also interested in. The people who do use pfSense right now are thrilled with the results and even one has expressed interest in moving to OPNsense.
Their (PIA's) pfSense guide had some problems and I have been in touch with support about this and hope they fix the guide.
The idea is to use load balancing in order to provide a few to several connections to their network that will then augment speed. One user, for example, who uses pfSense and that guide has seen their download speeds for everything from Windows updates to multi-threaded web downloads and even Netflix go from under 100 Mbps to over 500 Mbps. It seems that PIA has been having trouble with ISPs and otherwise and can't provide much more than 50-100 Mbps for many users now on a single VPN connection. Again, by using load balancing these limitations are able to be mitigated.
So in my particular case right now and where I'm having trouble (the guide has been fantastic and OPNsense has been very cooperative with a few minor tweaks) is on the STEP 11, METHOD 2 for "LEAK PREVENTION" (for this reason: "...or you still want to be able to resolve DNS addresses in the event that the VPN server you specified in Method 1 goes offline. The port forward method I showed in Method 1 cannot be load-balanced across multiple VPN gateways so if the one you decide to use for it were to be down you wouldn’t be able to resolve any domain names, essentially limiting your internet access.") Here are the instructions that are causing me trouble:
(https://i.ibb.co/VxYJxgL/pfsense-instructions.png)
-
I read through the guide and from what I'm reading, if you are using method 2, you are using the local OPN resolver at 192.168.1.1 and it's being let out via the VPN gateways and not the normal internet gateway.
You can test directly from a client.
nslookup
> server 192.168.1.1
Default server: 192.168.1.1
Address: 192.168.1.1#53
> google.com
Server: 192.168.1.1
Address: 192.168.1.1#53
Non-authoritative answer:
Name: google.com
Address: 172.217.6.238
If you aren't able to resolve, you are missing a rule letting the clients to your firewall.
Multiple VPNs or connections really don't do much for speed other than perhaps mitigate peering issues. In most cases, it can make things worse if the VPN providers are slow too. I only use the VPN to mask traffic.
-
Alright, I'm getting:
$ nslookup
> server 192.168.1.1
Default server: 192.168.1.1
Address 192.168.1.1#53
> google.com
;; connection timed out; no servers could be reached
Therefore this indicates that I'm missing a rule letting the clients to (or through?) the firewall....
So does this mean I need to add a rule to the Firewall? Or add an Access List? It seems you must mean that the client system cannot penetrate the firewall of OPNsense because it's blocking access.
These rules are also set up as explained in the guide. Here's what I have:
(https://i.ibb.co/6XxBT3g/existing-rules.png)
My rules are set up as per STEP 8 in the guide.
Here are the details of these 2 active rules I set up from that (In order as they appear above. Please scroll to the right as they are side by side):
(https://i.ibb.co/RBK1Ym2/firewall-rules.png)
(As a side note HTTP pipelining and various download speedup opportunities today, aside from peering, are why people are saying that the converse is happening - this load balancing will greatly speed up networking when used in conjunction with a VPN like PIA.)
-
The clients don't "penetrate" the opnsense for DNS, they contact Unbound on port 53 and this service does the DNS stuff and hands back the results to the client.
Allow on LAN (your first rule) access to port 53 of the sense (LANadress), so: add port 53 to your first rule in the LAN screenshot of your last post and report what the DNS lookup gives you...
-
Yeah, you need a rule from your VPN_Recipients to access your local firewall on UDP 53 via the GW_WAN so you can resolve DNS on from 192.168.1.1.
That rule should go above your VPN_Recipients / Domains to Bypass rule.
You should be able to test DNS resolution on a IP that isn't in your VPN_Recipients to validate DNS is working as that should be covered by the second the last rule in your screenshot that allows LAN traffic to everything although that seems to be using the default GW so that may need to be tweaked to use the GW_WAN to not go out the VPN GWs.
-
Oh @chemlud I think I overlooked this because it is the "Anti-Lockout Rule" and I never even thought about modifying it.
When I click on it, it redirects me to Firewall > Settings > Advanced and unfortunately there doesn't appear to be any settings here to control the port number. Is there something I am overlooking that would allow me to edit this? Or should I simply allow being locked out in order to get rid of this first automatic rule altogether?
If I do need to eliminate automatic rules such as this anti-lockout, should I create additional rules? And if so, can you give some explicit settings? I can give some examples of why this is a problem right now for me, but I figure I should wait to hear back first before I go any further.
-
Sorry as I don't think modifying the anti lockout rule is the right thing to do as that's there for a different reason. I thought I shared some specific steps to add a DNS rule, but good luck.
-
Add a new rule and place below the anit lock-out rule with port 53 and LANaddress as target (source: at least your VPN clients).
-
Oh! @Animosity022 you're saying to add a rule while @chemlud was saying to modify the rule. I don't see a way to modify or even eliminate the Anti-Lockout Rule so I guess your (@Animosity022's) method is the only workable (and preferred) method.
When I go to create a rule, I think you mean these settings right? But when I do this I get a "not allowed" on the "Destination port range" (and even on the Source port range if I expand it with the advanced button):
(https://i.ibb.co/DghhgVW/port.png)
-
Change the protocol from any to TCP/UDP and use port 53 to 53.
-
OK, so I have this for the specific settings:
(https://i.ibb.co/sqCTY3w/settings.png)
I applied moved the rule just below the Anti-Lockout Rule and applied the update:
(https://i.ibb.co/sgN89ZP/applied.png)
Verified that the IP I'm working from is in the VPN_Recipients Aliases list.
Did the lookup and got ";; connection timed out; no servers could be reached" again.
Looked at the firewall log and not really seeing anything indicating this rule is being hit (.104 is the Windows system that has the DHCP lease that I'm using to test nslookup with here):
(https://i.ibb.co/L8cBxwc/firewall-log.png)
-
Try to change the GW as I had noted about from the * to GW_WAN
-
Hrm, well good catch and I'm sorry that slipped by but I'm seeing the same result it seems:
(https://i.ibb.co/dkq3sBt/log-and-rules.png)
-
I'm a bit confused how your interfaces are setup.
It looks like your traffic is going through the PIAUSEast Interface as I'm not sure what that is.
I'm assuming 10.10.10.6 is your Windows machine?
I usually turn on logging on all my rules when I'm testing so I can trap the rule that's blocking it if something is.
What does the network topology look like?
-
So I have the internet coming from 192.168.137.1 into my isolated test OPNsense network. OPNsense is 192.168.1.1. The Windows machine (in the screenshot) is being assigned 192.168.1.104 and I have another linux peer that statically set to 192.168.1.20.
-
So where does the machine here come from in the log?
(https://i.imgur.com/MTeMzsY.png)
What's 10.10.10.6? That looks to be trying to get DNS to 192.168.1.1
-
I don't know. It doesn't exist as far as I can tell..... I've been wondering that myself and that subnet doesn't even exist anywhere in my topology anywhere else.... I can't help but think it must be somewhere outside of us-eastprivateinternetaccess.com, but that subnet is private and shouldn't show up like this from what I understand.... so yeah :-\
-
So let's take a step back, if you just turn on logging for all the rules and filter on the Windows machine, you should see it in the logs.
I did that as an example for my laptop which is 192.168.1.99.
You can see the rule hit when I did a DNS lookup.
(https://i.imgur.com/SBCrsGc.png)
You can even just ping 192.168.1.1 and see if that drops as it should and try that way too.
-
First, I just want to thank you for your patience and effort here. I was becoming frustrated and desperate after a long wait and your help and attention is just so nice.
So I went into each rule and turned on "Log packets that are handled by this rule" for ALL the rules, including the pre-existing ones.
I then simply pinged a couple times with the Windows client (192.168.1.104) in the filter:
(https://i.ibb.co/PNV0D8K/ping.png)
I also did some more looking and found that 10.10.10.6 is from the ovpnc1 interface: "inet 10.10.10.6 --> 10.10.10.5 netmask 0xffffffff"
-
Oh, I see the issue now.
Go back to your DNS rule and make the source port any as what normally happens is the client will any source port number, but the destination port is going to be 53.
I see on the DNS rule you have the source port set to 53, which would make it not work.
-
OK yes, I see what you're saying since random ports are used from the client to connect to the server's 53. But :( sadly I still get a timeout:
(https://i.ibb.co/s9CzBQJ/source-any-result.png)
-
Just to validate, do you have a host not in that VPN list that can test and make sure Unbound is working and give you a response back? The logs indicate the hit is going through, but seems like either Unbound isn't running or there is an ACL issue with it.
-
Yes, so as I mentioned earlier, I do have 192.168.1.20 which is a Linux system. I made sure that it was not in my "VPN_Recipients" list and I then tried a couple things that both worked. Here's what I did in Linux:
(https://i.ibb.co/6X4VbB0/linux.png)
Here's the firewall live log output:
(https://i.ibb.co/kQ6XXvC/linux-log-output.png)
-
So you have the 192.168.1.20 and the host you were having a problem with was the 192.168.1.104.
If you take 192.168.1.104 out of the VPN list and validate if it now works, that would ensure the host works normally before putting it in the VPN list.
You could also add 192.168.1.20 into the list and see if it still works for DNS as well.
-
So first, I've taken 192.168.1.20 and put it in the alias list to receive the VPN and got the same results:
(https://i.ibb.co/phYp0Qr/linux-on-vpn-list.png)
(https://i.ibb.co/TYT958V/log.png)
Windows then behaves just like the Linux computer when it is removed from the VPN_Recipients alias list:
(https://i.ibb.co/6cMBHVJ/windows.png)
-
Your DNS rule still isn't firing based on the logs you've shared so something seems still not right with it.
Can you share that?
-
Summary:
(https://i.ibb.co/c25Twmy/summary.png)
Details:
(https://i.ibb.co/zmYrgBz/all-settings.png)
-
And the full screen of all your rules?
-
Summaries:
(https://i.ibb.co/x88GH70/all-rules-summaries.png)
Did you also want the detailed views?
-
So your rules appear in the right order, but I'm still super confused as to why your setup generates both of these rule hits:
(https://i.imgur.com/qC63u6J.png)
There is no reason why your LAN hit would traverse the WAN interface. I have all my logging on as well and a similar DNS rule and I just see the WAN hit.
(https://i.imgur.com/cJZ2IZs.png)
For some reason in your setup, which doesn't seem obvious to me at the moment, the internal hit is going out the WAN interface. I'm guessing for that reason is why you are seeing the hit, but the return isn't coming back as something is off.
-
What does your System->Routes->Status look like?
-
(https://i.ibb.co/G01jPtk/routes-status.png)
-
Show us your unbound configuration ("General"), especially the "Networking Interfaces" and the "Outgoing Network Interfaces".
-
Yeah, your routes and IPs are really off somewhere as your Unbound config is fine.
I don't understand why your WAN interface has a GW of your LAN IP.
(https://i.imgur.com/iZRjIKm.png)
Your default gateway should be on your WAN interface.
Example WAN DHPC interface.
(https://i.imgur.com/IY24wyx.png)
Example LAN interface.
(https://i.imgur.com/gl55mWr.png)
Do you have something configured in your Routes->Configuration?
Is your WAN interface a DHCP interface or static? What's the config for that look like?
What's your LAN interface config?
-
@chemlud as you asked (I think @Animosity022 looked at previous the posting for this info and @Animosity022 I have your answers below):
(https://i.ibb.co/FXF4Y2s/unbound.png)
So @Animosity022 I have 192.168.137.1 as the WAN in this case since it's a test. My ISP only allows 1 IP lease and so I have a router pushing traffic to OPNsense until I can get it functional and replace it with OPNsense. So for now I have: ISP (Public IP) --> Router (192.168.137.1) --> OPNsense (192.168.1.1)
This Router is NOT offering DHCP, it's just static so I configured OPNsense statically (screenshot of WAN config (https://i.ibb.co/FWxmSyt/wan.png)).
Routes->Configuration: No, "No results found!"
LAN config:
(https://i.ibb.co/54Nk24K/LAN.png)
-
Based on this screenshot though, you have your WAN interface seeing LAN traffic so you have something on your network configured or setup incorrectly in your test environment.
(https://i.imgur.com/qC63u6J.png)
You can validate that by running a tcpdump -i on your WAN interface and capturing the traffic.
-
It is a little strange. I do see a lot of PIA East traffic on WAN, which is right since that connection is connecting to that.
I do also see some other odd traffic (?) like this NTP traffic:
(https://i.ibb.co/3sZkm5g/ntp-traffic.png)
I guess this confuses me since I still have 192.168.1.20 on the VPN_Recipients alias list. I tested and I still can't get any name resolution:
(https://i.ibb.co/wgJcXRy/failure.png)
I then noticed a bunch of other UDP traffic from firefox pour across from .20 (I did have it open, but wasn't resolving):
(https://i.ibb.co/rkhy6ZB/firefox.png)
I will say that while I was watching it I found that a lot of the traffic (nearly all of it after I closed down Firefox and disabled NTP on the Linux machine), was PIA VPN traffic on UDP. I checked this by reverse greping the PIA IP that was being used so that I was only seeing traffic (except for the time/informational header info) for non PIA bound or source traffic.
-
Yeah, you have something misconfigured somewhere along the way on your network setup as the WAN should not see LAN traffic.
How do you have the interfaces setup? What are you using to do that?
In a physical setup, the LAN cable is different from the WAN cable so they don't see each other :)
If you run a tcpdump on the LAN interface, I'm sure you can see the traffic coming in, but since you aren't getting a reply, it's definitely not coming back out.
-
I'm using Proxmox with an OVS Bridge. So I have The ISP traffic coming in to a Windows 2012 Server that shares this connection out from 192.168.137.1 to a virtual switch (we'll call vswitch1). This virtual switch is also set up as the WAN port for OPNsense (192.168.137.95). I then have a second virtual switch (we'll call vswitch2) providing LAN connections for the Windows and Linux machines I've been doing the testing on.
I do have another Linux server on vswitch1 (wan 192.168.137.250) that provides Internet access to the rest of my network.
Clearly the Windows 2012 gateway is just temporary. I was using it to test some various scenarios for PIA connectivity that just didn't work out well. My goal is to replace it with OPNsense, but I'd like to actually get OPNsense working properly before I swap it out since we have actual time sensitive work to do and can't afford the downtime right now... :'(
-
Sadly, I've never used proxmon or any of the virtual switching but that seems to be where I would look.
I've setup IP routing using aliases like you have on pfSense and OPNSense through TorGuard VPN using the exact same setup you have.
Your config looks solid and the only things that really strike me as being off is the fact the WAN can see LAN traffic so to me, that points to something in the switch config as the traffic is bleeding over.
The part that also has me is you are seeing it on the LAN interface so it's coming in both the WAN and LAN. I'm not sure if that causes a drop or something else going on.
If you can sort out the virtual switching, I think your config seems solid as there isn't much to it as the guide you posted/went through looks correct.
-
I'm curious about a test here. Is there a way to save the configuration of OPNsense so that I can load it up on a physical machine attached to my network to see if this is still happening? I've got some extra systems sitting around I could load up OPNsense onto and then see where that goes...
-
System->Configuration->Backups
You can download a copy of your config and upload to a new install.
-
Oh that's really good. Thank you. It seems it would make sense if someone were to make a config that works with PIA for example with as generic as possible options (DHCP everywhere), maybe that would help other people out instead of having to go through so many steps. I'll give this a shot here and see how it works.
-
.
-
So I've mixed this up a little bit. I thought I'd try to test a few things. I've gone ahead and put OPNsense right on the head of my network with no other possible traffic at this point. I'm using a vanilla setup with really no customization. I thought I'd see if I could at least get Unbound responding to local traffic FIRST before setting up anything else whatsoever.
I'm able to talk to the internet just fine, no additional firewall rules, nothin' at all. Unbound is supposedly on:
(https://i.ibb.co/f9XbZ8D/unbound.png)
But when I even test this I get a timeout:
nslookup google.com 192.168.1.1
;; connection timed out; no servers could be reached
Now that has me really scratching my head! Any thoughts on this?
-
Can you ping 192.168.1.1?
If that returns, you should be able to use DNS.
Did you turn on logging and do you see any hits from the rules?
-
Ah, lol, got it. Crazy me. I unthinkingly set the subnet to 192.168.0 instead of .1 this time around just powering through things. Thanks for the sanity catch there!!!! So nslookup DOES work now, that's really good!
-
I had some things come up and haven't had much time to work on this, but I've managed to find some time to come back to this issue today. I thought I'd do some testing at each step in the process and unfortunately while I think I have a better picture of what's going on, I'm even more confused now.
First, one thing that has confused me is that I've managed to get the VPN working, but while IPv4 shows the right remote IP address, IPv6 appears to be skipping over the VPN and just using the local ISP/WAN interface and I can never get it to go through the VPN. I figured maybe I need to block it and I can't even figure out how to do that so just IPv4 is only used in case that's an issue (but I don't think it is since, I THINK, PIA has used IPv6 before).
So now I have things arranged a little differently. I have OPNsense as the primary gateway now for my network.
I have one test system assigned to the VPN_Recipients alias which means I can keep tests isolated for it. Everything else is just on the LAN and my goal is to make them all just behave normally and pass through the WAN port only without touching the VPN.
For some details:
- 192.168.0.211 is the only IP within VPN_Recipients
- The VPN client is "PIA East" (screenshot (https://i.ibb.co/WWR9S4T/pia.png))
- Unbound is on and listening at 53 (screenshot (https://i.ibb.co/mRxB0vH/1.png))
- I have 3 Interfaces: LAN (https://i.ibb.co/4t8DLYv/lan.png), PIAEast (https://i.ibb.co/y8tcqvV/pia.png), and WAN (https://i.ibb.co/TBvn1yv/wan.png).
- Outbound NAT (Overview (https://i.ibb.co/47mCDB7/outbound-nat.png), Details (https://i.ibb.co/k5RJKHF/details.png))
- Firewall Rules (Overview (https://i.ibb.co/GCq5pwn/firewall-rules.png))
Notice I put the first two rules under the Anti-Lockout Rule in order to pass all traffic from the LAN that's NOT within the VPN_Recipients list on out to the default WAN. I wanted to keep my network working while doing the configuration work and seems sane to do something like this.
So I've found a couple of things particularly strange and I'm hoping you can elucidate the issues. I really was hoping that I could do an nslookup in all cases, but here's what I'm seeing:
When I have the VPN turned OFF, I can, from regular LAN systems, ping anything fine and "nslookup google.com 192.168.0.1" works great. Same is true of 192.168.0.211.
When I turn on the VPN, I cannot ping anything, but I CAN run "nslookup google.com 192.168.0.1" fine. From 192.168.0.211 I CANNOT run nslookup (times out), and I cannot ping a domain (like google.com), but I CAN ping 8.8.8.8.
Leaving VPN on, if I change the "NON-VPN - Default allow LAN to any rule" rule so that Gateway is NOT "default" but is instead set to "WAN_DHCP", I can then ping 8.8.8, but nslookup and name resolution still won't work.
I'm just not really seeing exactly what's going on here. Why would DNS resolution work and it seems that at other times when I was playing here with the settings I got DNS resolution for OPNsense to work while the VPN was on, but I lost track of the settings that did this. I'm obviously not understanding a crucial part of the puzzle.
-
I got to the point here where I thought it would be a good idea to test the pfSense (https://www.techhelpguides.com/2017/06/12/ultimate-pfsense-openvpn-guide/) tutorial on pfSense itself to see if that even worked for me with the same setup. (Unfortunately?) It did.
I followed the instructions and went right through the tutorial without any issues. pfSense is working correctly and I'm finding that VPN speeds are working as advertised with greatly enhanced performance that exceed the ISP limitations. It's very impressive actually, but I really wish I could have figured out what the differential is between the tutorial DNS section working fine with pfSense and not working with OPNsense.
Is there anything that can be done with this information? Can the configuration file be reviewed somehow to see what's being missed and migrated to OPNsense?