OPNsense myteriously blocking OpenVPN traffic

[lenard220]

Hi,

i have some problems getting my openvpn server running behind opnsense.
(custom debian server, not the opnsense integrated one)

My Setup:

ISP Router --> OPNsense --> OpenVPN server
In my ISP Router I have forwarded a bunch of ports including 1194 to my OPNsense and then forwarded 1194 to my openvpn server from there.
Interfaces:

• hn0 = WAN interface of OPNsense, connected to my ISP router
• hn3 = LAN interface, opnvpn server is connected to this


My Problem is the following:
When trying to connect to my OpenVPN server outside of OPNsense some of the incomming packets are mysteriously dropped.
I tested the port forwarding with my phone by manually sending UDP Packets. All of the packets came through without any problem. See test_phone_dummy.png
All my other port forwaring works without any problems.

Now if I try to connect to the openvpn server through the same device something differenet happens.
As you can see in screenshot ovpn_connect_trace.png all of the packets are received at the WAN interface. The addresse is correct so I don't see anything wrong there.
If we look at the LAN interface we see that only the first incoming packet manages to get to the openvpn server. The packets later sent to the openvpn server are blocked for some reason and thats just what I dont't get why.
The only difference is the size of the packet but that shouldn't be a problem since you can see in my earlier screenshot test_phone_dummy.png that the packet sent by the phone during testing was even bigger.

So my next idea was to look at the live view of the firewall maybe something can be found there.
In my sules setting I have set all sules which block / reject packets to log so if something is blocking we should see it here.
I have also set the port forwarind to log packets as well and created an additional allow anything rule.
In the next screenshot firewall_log.png you can see that the first packet manages to get to the openvpn server.
All the other packets aren't even logged here so it seems to me that they are blocked before opnsense get's the chance to port forward them to the openvpn server.

A list of things I have tried to solve this issue (sorry if I forgot one)

• recreating the port forwarding rule
• creating a floating allow all rule
• checked all blocking firewall rules ( they are all set to log )
• manually sent udp packets to the openvpn server
• I have checked and can connect to the openvpn server just fine aslong as I stay within opnsense

Thanks in advance

Regards

Lenard

tldr: opnsense is somehow blocking every incoming openvpn packet after the first packet.

[bartjsmit]

Have a look at your routing - there are three subnets in play:

- what your ISP router thinks of as the LAN
- what OPNsense thinks of as the LAN
- what the OpenVPN server brings up as a tunnel

All three routers need to know where all three subnets live or they'll just drop the packets.

Bart...

[lenard220]

Hi,

thanks for your answer.

- what your ISP router thinks of as the LAN

I'm not really sure what you mean by this. The only device on the LAN of the ISP router is the opnsense firewall / router.

- what OPNsense thinks of as the LAN

OPNsense has several LAN's attached. "LAN_ext" which openvpn is connected to is one of them.
OPNsense has an outbound NAT configured so all packets gouing outside are addressed by the OPNsenses WAN IP.
OPNsenses WAN IP is in the subnet of the ISP routers lan.

- what the OpenVPN server brings up as a tunnel

Do you mean the subnet? The openvpn tunnel has a seperate subnet (tun). The openvpn server nats the subnet so all packets going out of the openvpn server are addressed by the openvpn' s servers address. I have tested that the nat works and all outgoing packets are addressed by the openvpns servers ip address.

All three routers need to know where all three subnets live or they'll just drop the packets.

Yes they can reach all subnets. On the ISP router I configured and tested the appropriate routes so I can reach all devices from the ISP routers LAN.

Regards

Lenard

[bartjsmit]

Hi Lenard,

Yes they can reach all subnets. On the ISP router I configured and tested the appropriate routes so I can reach all devices from the ISP routers LAN.

It is not the reaching that is the problem if the router won't try it. For example take these as your subnet ranges:

internet (ISP) 192.168.123.0/24 (OPNsense) 10.43.36.0/24 (OpenVPN) 172.24.180.0/24

Let's say that OPNsense has 192.168.123.2 for its WAN and 10.43.36.1 assigned to the LAN, while the OpenVPN server has 10.43.36.2 for its Ethernet interface and 172.24.180.1 for its TUN interface.

The ISP router must have two static routes:
10.43.36.0/24 via 192.168.123.2
172.24.180.0/24 via 192.168.123.2

OPNsense must have the ISP router as its default gateway and one static route:
172.24.180.0/24 via 10.43.36.2

The OpenVPN server must have OPNsense as its default gateway and push 172.24.180.1 as the default gateway to its clients.

Your Debian server returns 1 on this command, doesn't it?

sudo cat /proc/sys/net/ipv4/ip_forward

Bart...

[lenard220]

Hi,

my configuration is almost exactly like that.
By almost I mean that the only difference is that I NAT the openvpn tun subnet.
This means that as far as I know I do not need to have a static route to the openvpn subnet since nat repleaces the address with the openvpn servers address.
I have tested this by connecting to the openvpn server inside opnsense and then connecting to my webserver which reported the connected IP as the openvpns server ip. Later on I tested this with wireshark as well and it showed the same result so my NAT seems to work.

The problem is not that I can connect to the openvpn server and then can't reach anything. That would be the case if there would be a missing route to the openvpn subnet.
My problem is that while establishing the connection some packets are dropped, which causes the establishing of the connection to timeout.
The dropped packets have the exact same source and target address and the exact same port as the first packet which can go through the firewall. (Please see attached screenshots on the first post).

Regarding your last point:

Your Debian server returns 1 on this command, doesn't it?

sudo cat /proc/sys/net/ipv4/ip_forward

Yes it does.


Regards

Lenard

[lenard220]

Update.

So I got recommened to try it out with tcp instead of udp.
For some reason tcp works completely fine.

[lenard220]

Does anyone have an idea why udp is not working?

[miroco]

These are my settings.


miroco

[lenard220]

Hi,

thanks for your answer.
I did test your settings and the problem is still there.

Regards

Lenard

[bartjsmit]

Hi Lenard, I think your problem is with the double NAT.

OPNsense has an outbound NAT configured so all packets gouing outside are addressed by the OPNsenses WAN IP.

I have forwarded a bunch of ports including 1194 to my OPNsense and then forwarded 1194 to my openvpn server from there.

Unlike TCP, UDP is stateless and not easy to track across two NAT's. Can you try OPNsense without NAT (with the static routes), in bridge mode, or set your ISP router in modem mode?

Bart...

[lenard220]

Hi Bart,

thanks for your answer.
So my ISP Router sadly doesn't allow using it as a modem only.
Setting the static routes works just fine.

So I did what you said, disabled NAT and only used Port Forwarding on OPNsense.
Mind you the port is still forwaded to the OPNsense's WAN IP.
Aaaaaaand it's fixed.

So for some reason the double NAT bricks OpenVPN.
But only especially OpenVPN and only when establishing a connection.

Thanks.

Regards

Lenard

[bartjsmit]

Hi Lenard, glad you got it fixed - you should try VoIP over NAT