Routing problem with policy based IPsec tunnel

[frispete]

In a public cloud environment, an OPNsense 23.7.12_5 instance was installed as a simple IPsec gateway.

The OPT2 interface is connected to a 10.2.2.0/24 network with IP address 10.2.2.5/32, because autoconfigured ubuntu servers with similar /32 addresses reside there, e.g., 10.2.2.2/32:

server:

Code Select Expand

$ ip r | grep 10.2.2
10.2.2.0/24 via 10.2.2.1 dev enp7s0
10.2.2.0/24 via 10.2.2.1 dev enp7s0 proto dhcp src 10.2.2.2 metric 100
10.2.2.1 dev enp7s0 scope link
10.2.2.1 dev enp7s0 proto dhcp scope link src 10.2.2.2 metric 100


In order to support this setup, I figured to configure the OPNsense (hopefully) similarly:

Code Select Expand

$ netstat -nr4 | grep 10.2.2
10.2.2.0/24        10.2.2.1           UGS      vtnet3
10.2.2.1           link#4             UHS      vtnet3
10.2.2.5           link#4             UH          lo0


It took defining a custom IPv4 Upstream Gateway with 10.2.2.1 in the interface and a route from 10.2.2.0/24 to 10.2.2.1.

Using this setup, it is possible to ping 10.2.2.2 from 10.2.2.5 and vice versa directly.

In legacy tunnel settings, an IPsec phase 2 tunnel IPv4 was created with local network 10.2.2.0/24.

In IPsec firewall rules, IPv4 ICMP to 10.2.2.0/24 is allowed.

I can ping the OPNsense (10.2.2.5) via tunnel just fine, but none of the systems behind the tunnel.

Live view shows two green entries for attempts to ping 10.2.2.2 behind the tunnel, one from the IPsec inwards (with my custom label), and one on the interface outwards (let out anything from firewall host itself). A tcpdump doesn't show any ICMP packages arriving at 10.2.2.2.

The local OPNsense 23.7.12_5 IPsec endpoint has a phase 2 entry to remote subnet 10.2.2.0/24.

It looks like a routing problem between the IPsec and the physical interface.

Any ideas, what I'm missing here?

[Monviech (Cedrik)]

Could you provide a small layer 3 network plan that can show how a paket is routed from your test client to your test server?

[frispete]

Sure, with pleasure:

client <172.16.23.100> <-> local Os <172.16.23.1> <-> IPsec Tunnel/remote Os <10.2.2.5> <-> server <10.2.2.2>

Does this fit the purpose?

Thank you for your interest!

[Monviech (Cedrik)]

- Is the OPNsense the default gateway for the Ubuntu VMs in that remote network?
- Do the VMs have an active Firewall like ufw or iptables that could block the paket from source 172.16.23.0/24?

[frispete]

- Is the OPNsense the default gateway for the Ubuntu VMs in that remote network?

No, it's not.

- Do the VMs have an active Firewall like ufw or iptables that could block the paket from source 172.16.23.0/24?

No, iptables -L returns an empty rule set.

[Monviech (Cedrik)]

If the OPNsense is only connected with one interface (e.g. the WAN interface) and it has a gateway set on it, it forces all traffic back to its own default gateway. My guess is that its a reply-to problem. If you can tcpdump on your main router there, you might be able to see the pakets going there.

Next would be, that if the OPNsense is not the default gateway, all the Ubuntu VM clients will need static routes set on them to find the route to 172.16.23.0/24 when they initiate traffic. Otherwise the tunnel only functions one way (to the VM net).

What can solve most of these issues is creating a seperate network just for the opnsense. And setting a static route to 172.16.23.0/24 on that main router to the OPNsense. That way you have total symmetric routing with no shortcuts.

EDIT: One final trick is to Source NAT when you cant change the infrastructure. That would only enable the inbound connection (You can reach VMs), not the outbound (VMs cant initiate connections to you), though. On the OPNsense in the VM network, create a Source NAT that NATs 172.16.23.0/24 to the OPNsense IP address 10.2.2.5. That way, they are communicating with the OPNsense without knowing about the 172.16.23.0/24 network.