OPNsense Forum
English Forums => General Discussion => Topic started by: Margio on June 04, 2019, 09:38:07 pm
-
Hi everyone, after days of troubleshooting and trial and error I finally decided to ask your help,
bear in mind that I’m not a network engineer and I don’t have advanced knowledge of networking.
Goal
Setup a TAP OpenVPN connection between my router and my laptop, routing all the traffic (internet included) through it.
Yes, I need tap for mDNS and bonjour, and I want to route all the traffic so that one day i can add a VPN service on the server WAN side for secure internet browsing.
Problem
Boujour, mDNS and “local” networking works flawlessly and internet browsing outside the VPN also works, but if I check “Redirect Gateway” or put the command push “redirect-gateway def1” in the advanced configuration of the OpenVPN Server and ping 8.8.8.8 I get “Network is unreachable”, I tried to tcpdump on the server machine but I can’t find the icmp traffic on any of the available interfaces.
Configuration
Hardware
OPNsense 19.1.8 on a VM in ESXi with two ports:
vmx0 connected to modem (vSwitch1)
vmx1 connected to switch for LAN (vSwitch0)
Software
ESXi
both vSwitch1 and vSwitch0 have:
Promiscuous mode Accept
MAC address changes Accept
Forged transmits Accept
OPNsense
System > Gateways
WAN_PPPOE 192.168.1.100 (this is automatic from the pppoe connection)
System > Settings > General
I left blank all the DNS since i want to use Unbound as my resolver
System > Settings > Tunables
net.link.bridge.pfil_member = 0
net.link.bridge.pfil_bridge = 1
Interfaces
Interface vmx0 assigned to WAN with pppoe enabled (block private networks and block bogon networks unchecked)
Interface vmx1 assigned to LAN with static ip 10.0.1.1/24
interface ovpns1 (created by OpenVPN Server) assigned to TAP with no ip
Interface bridge0 (bridge created by combining LAN and TAP) assigned to BRIDGE with no ip
Firewall > Rules
BRIDGE: allow ipv4 any to any
LAN: default created by system (anti-lockout ecc.)
OpenVPN: allow ipv4 any to any
TAP: none
WAN: allow traffic to WAN address through port 1194 (for OpenVPN)
Firewall > NAT > Outbound
Interface Source NAT address
TAP 10.0.1.0/24 LAN address
Firewall > Settings > Advanced
Disable reply-to: checked
VPN > OpenVPN > Servers
Tunnel Settings
Bridge DHCP: checked
Bridge Interface: LAN
Client Settings
Dynamic IP: checked
Address Pool: unchecked
Advanced: push ”redirect-gateway def1”
push ”route-gateway 10.0.1.1"
push "dhcp-option DNS 10.0.1.1"
Please, I need your help, I’m going mad.
Thanks
-
can you connect to the internet if you set 10.0.1.1 as your default gateway on the laptop while connected to the VPN?
Bart...
-
can you connect to the internet if you set 10.0.1.1 as your default gateway on the laptop while connected to the VPN?
Bart...
Thank you Bart for the suggestion, i tried deleteting the existing default route and adding 10.0.1.1 as the default, now the ping say, "no route to host".
I previously seen "no route to host" as a ping response in one of the many configuration, with that said the server should already push the new gateway through the command push ”route-gateway 10.0.1.1", i think i can even add push "route default 10.0.1.1".
Anyway, that doesn't solve the problem, and i don't know what to do. I pushed the configuration as far as setting firewall rules on all the interfaces to pass any to any (properly isolating the system from the LAN) but with no success.
-
Do you see the ICMP packets come out of the tunnel towards the 10.0.1.1 gateway? Run a packet trace on each end to see where the ping is getting stuck.
Bart...
-
Do you see the ICMP packets come out of the tunnel towards the 10.0.1.1 gateway? Run a packet trace on each end to see where the ping is getting stuck.
Bart...
I run a bunch of tests again, here the results:
With the above configuration, using Wireshark on my laptop the icmp packets are nowhere to be seen as well as any other type of connection while pinging 8.8.8.8, meanwhile icmp packets from 10.0.1.1 ping are clearly visible on the vtap0 interface.
I decided to inspect more on the routes then and I found a mess caused by many many wrong configurations, so I decided to try a new configuration to clean that mess.
In the OpenVPN Server configuration:
- added the DNS server directly from the UI checkbox and removed push "dhcp-option DNS 10.0.1.1"
- added DNS Default Domain "lab.home"
- modified push "redirect-gateway def1" to push "redirect-gateway" (the old command created 0.0.0.0/1 and 128.0.0.0/1 rather than 0.0.0.0/0 in the routing table)
This is the routing table
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.1.1 UGScI 1 0 vtap0
10.0.1/24 link#19 UCS 0 0 vtap0 !
10.0.1.1/32 link#19 UCS 1 0 vtap0 !
10.0.1.1 x:x:xx:xx:xx:xx UHLWIir 3 3 vtap0 1198
10.0.1.35/32 link#19 UCS 1 0 vtap0 !
xx.xx.xxx.xxx/32 192.168.1.1 UGSc 1 0 en0
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 1 256 lo0
169.254 link#8 UCS 1 0 en0 !
169.254 link#19 UCSI 0 0 vtap0 !
192.168.1 link#8 UCS 0 0 en0 !
192.168.1.1/32 link#8 UCS 1 0 en0 !
192.168.1.1 xx:xx:x:xx:xx:xx UHLWIi 6 7 en0 1198
192.168.1.9/32 link#8 UCS 1 0 en0 !
224.0.0/4 link#8 UmCS 1 0 en0 !
224.0.0/4 link#19 UmCSI 0 0 vtap0 !
224.0.0.251 x:x:xx:x:x:xx UHmLWI 0 0 en0
255.255.255.255/32 link#8 UCS 0 0 en0 !
255.255.255.255/32 link#19 UCSI 0 0 vtap0 !
Note: 192.168.1.1 is the gateway of the wifi connection that I'm using on my laptop (some IP and MAC addresses are replaced by X for obvious reasons)
ping to 8.8.8.8 still gives No route to host
IMPORTANT NOTE ABOUT DNS
with the original configuration I couldn't resolve DNS, now I can but with this strange behaviour:
With the above configuration (in this post), I can resolve local addresses like test.lab.home (it obviously uses my Unbound DNS Server at 10.0.1.1) but if I try to resolve google.it it uses the wifi router DNS server (192.168.1.1).
If I add push "dhcp-option DNSMODE full" (specific of Viscosity, the client I'm using, it should force the client to use the specified DNS servers), I can now resolve both internal and external address with my Unbound DNS Server but the routing table look like this:
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 192.168.1.1 UGScI 6 0 en0
10.0.1/24 link#19 UCS 0 0 vtap0 !
10.0.1.1/32 link#19 UCS 1 0 vtap0 !
10.0.1.1 x:x:xx:xx:xx:xx UHLWIir 3 10 vtap0 1198
10.0.1.34/32 link#19 UCS 1 0 vtap0 !
xx.xx.xxx.xxx/32 192.168.1.1 UGSc 1 0 en0
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 1 250 lo0
169.254 link#19 UCS 1 0 vtap0 !
169.254 link#8 UCSI 0 0 en0 !
192.168.1 link#8 UCS 1 0 en0 !
192.168.1.1/32 link#8 UCS 1 0 en0 !
192.168.1.1 xx:xx:x:xx:xx:xx UHLWIir 8 6 en0 1197
192.168.1.9/32 link#8 UCS 0 0 en0 !
224.0.0/4 link#19 UmCS 0 0 vtap0 !
224.0.0/4 link#8 UmCSI 1 0 en0 !
224.0.0.251 x:x:xx:x:x:xx UHmLWI 0 0 en0
255.255.255.255/32 link#19 UCS 0 0 vtap0 !
255.255.255.255/32 link#8 UCSI 0 0 en0 !
Note that the table is identical from the previous one except the default gateway which is 192.168.1.1 (why?)
No ping to 8.8.8.8 even with this last one configuration.
Any suggestions?
-
I would advise you tackle one problem at a time. DNS can wait ;-)
The 8.8.8.8 ping doesn't enter your tunnel because your DG is on en0, not tap0. Concentrate on that.
Check the DHCP scope for the laptop to see what default route it set. Try a different OpenVPN client or even a different OS. From your network device names and Bonjour requirement I take it you're running macos? if so, gIve tunnelblick a go. https://tunnelblick.net/downloads.html
You can even boot the laptop into a Ubuntu live desktop and use the command line client. If you run it in the foreground (openvpn client.ovpn) it will give you verbose details, especially if you set verb 4 or 5 in the config. https://business.tutsplus.com/tutorials/how-to-create-a-bootable-ubuntu-usb-drive-for-mac-in-os-x--cms-21253
Bart...
-
Absolutely, DNS can wait :D
One question, why did you say that mu DG is on en0?
Maybe i'm missing something but from the routing table of my previous post it seems to me that is on vtap0
[...]
This is the routing table
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.1.1 UGScI 1 0 vtap0
10.0.1/24 link#19 UCS 0 0 vtap0 !
10.0.1.1/32 link#19 UCS 1 0 vtap0 !
10.0.1.1 x:x:xx:xx:xx:xx UHLWIir 3 3 vtap0 1198
10.0.1.35/32 link#19 UCS 1 0 vtap0 !
xx.xx.xxx.xxx/32 192.168.1.1 UGSc 1 0 en0
127 127.0.0.1 UCS 0 0 lo0
127.0.0.1 127.0.0.1 UH 1 256 lo0
169.254 link#8 UCS 1 0 en0 !
169.254 link#19 UCSI 0 0 vtap0 !
192.168.1 link#8 UCS 0 0 en0 !
192.168.1.1/32 link#8 UCS 1 0 en0 !
192.168.1.1 xx:xx:x:xx:xx:xx UHLWIi 6 7 en0 1198
192.168.1.9/32 link#8 UCS 1 0 en0 !
224.0.0/4 link#8 UmCS 1 0 en0 !
224.0.0/4 link#19 UmCSI 0 0 vtap0 !
224.0.0.251 x:x:xx:x:x:xx UHmLWI 0 0 en0
255.255.255.255/32 link#8 UCS 0 0 en0 !
255.255.255.255/32 link#19 UCSI 0 0 vtap0 !
[...]
(Maybe you get confused by the two different tables, sorry, i'll try to make shorter posts.)
This is the strange part to me, if the DG is set correctly, why the traffic doesn't even leave my laptop?
-
I would advise to packet trace on the laptop first. Ping the far gateway and you should see the arp request for 10.0.1.1 and the reply.
If you don't, then the tap tunnel isn't working properly. Try a different client to see if the issue is with the VPN or the Mac.
Bart...
-
Ok, everyone, SOLVED!
I browsed some very old threads and turns out that Mac OS (and I think Linux too) needs some time to negotiate DHCP addresses, so if the route is pushed before the machine gets the IP address, the routing table becomes a mess.
SOLUTION
Adding a delay to the route
route-delay n(In the client config file)
Or
push “route-delay n”In the server config file / advanced configuration in OPNsense (hoping that it won't be removed in a future release)
Where n is the value in seconds,
(I set 15 but you should do your own tests)
Windows has this value set to 30 by default while Mac OS (and I think Linux too) has 0 by default.
Maybe this solution was obvious but since I didn't found a TAP road warrior / all traffic through the tunnel guide anywhere, it took me some time.
Hope this is helpful to someone else too.
Bart, thank you for the support and troubleshooting tips!
-
good going, thanks for tying up the thread