Policy based IPSec with routed networks and (S)NAT

[bamypamy]

Hey, so here is my setup and question.

TL;DR
Why do outbound or 1:1 NAT rules not work for routed networks (not directly connected)?

Setup:

• 2 Firewalls (pfsense and opsense)
• Servernetwork: 10.10.0.0/24
• Transfernetwork: 172.16.255.0/24
• NAT Mode: Hybrid
• IPSec Tunnel:
  
  • Local: 2.2.2.88/29
  • Remote: 1.1.0.0/14

The Server network is on the pfsense, and the IPsec Tunnel is on the opnsense.
pfsense and opnsense are connected via the Transfernetwork.

Transfernetwork IPs:

• pfsense: 172.16.255.2/24
• opnsense: 172.16.255.3/24

Routes:
pfsense:

• 1.1.0.0/14 GW 172.16.255.3

opnsense:

• 10.10.0.0/24 GW 172.16.255.2

FW rules (for testing):
pfsense:

• Servernetwork: any any allow
• Transfernetwork: any any allow

opnsense:

• ipsec: any any allow
• Transfernetwork: any any allow

2.2.2.88/29 is NOT a network on the opnsense but just something to NAT to. I got this from the remote end, they want NAT in the Tunnel.

So all traffic from 10.10.0.0/24 to 1.1.0.0/14 must be nattet to 2.2.2.88/29.
I created manual SPD entries for 10.10.0.0/24 and for 172.16.255.2/32 on the IPSec Tunnel (I am using connections NOT legacy).

Outbound NAT rule:

Interface: IPSec
Protcol: icmp (just for testing)
Source IP: 10.10.0.0/24
Src Port: any
Dest IP: 1.1.0.0/14
Dest Port: any
NAT IP: 2.2.2.88/32
NAT Port: empty
Log: enabled

Interface: IPSec
Protcol: icmp (just for testing)
Source IP: 172.16.255.2/32
Src Port: any
Dest IP: 1.1.0.0/14
Dest Port: any
NAT IP: 2.2.2.88/32
NAT Port: empty
Log: enabled

When I start a ping from any of the servers in the 10.10.0.0/24 network to 1.1.0.10 I can see these packets arrive on the opnsense on the transferinterface and I can also see them on the IPSec Interface but I do not see a hit on the NAT rule in the log.

When I start a ping from the pfsense to 1.1.0.10 coming from 172.16.255.2 (transfernetwork IP of the pfsense) I can also see these packets arrive on the opnsense on the transfernetwork interface and also see the packets on the IPSec Interface but here I see a hit on the according outbound NAT rule and also see ping replies from 1.1.0.10 going to 2.2.2.88 and the icmp reply is shown on the pfsense.
So exactly as expected.
But why does it not work for the routed server network?

I first thought opnsense does not NAT not directly connected networks but then I found this which tells me it should work the way I am doing it.
https://forum.opnsense.org/index.php?topic=34171.0

I also tested this with 1:1 NAT rules. It behaves the same with the only difference that I do not see a hit of the 1:1 NAT rule in the log for both IPs. So it works coming from 172.16.255.2 without showing a rule hit in the log but it does not work for traffic coming from 10.10.0.0/24.

I hope the setup is clear and you understand what my question is. 

[bamypamy]

UPDATE:
I just tested a ping from the pfsense with the servernetwork as a source and this works as well.
So this is even more strange.
It only does not work when the ping is started from one of the servers.

[bamypamy]

UPDATE 2:
I set up a less granular outbound NAT rule specifying the whole 10.10.0.0/24 network and started a ping from two other Servers.
Result: One of the servers works the other does not.

And I now see strange behavior in the Firewall log. It allows but then also blocks...? What is the IPsec internal host to host rule?

Code Select Expand

      Interface         Time                 Source         Destination Proto Label
BLOCK IPsec 2025-04-10T09:08:24 2.2.2.88 1.1.0.10 icmp IPsec internal host to host
PASS  IPSec 2025-04-10T09:08:24 2.2.2.88 1.1.0.10 icmp IPsec internal host to host
NAT   IPsec 2025-04-10T09:08:24 10.10.0.143 1.1.0.10 icmp nat rule

[bamypamy]

UPDATE 3:
I switched the IPSec Tunnel to legacy but it behaves the same.
Currently pinging from three servers, one goes through and two don't.
The one that currently works did not work before and vice versa.
From the pfsense the ping always works.

I also activated this option "Disable legacy auto-added VPN rules" but it is still the "IPsec internal host to host" rule that keeps blocking.


I am out of ideas... :-(

[bamypamy]

UPDATE 4: When I NAT the three servers to different IPs from the 2.2.2.88/29 all three of them work.
Is it not possible to hide all of the 10.10.0.0/24 behind one IP?

So basically what I do now is

Server1:
10.10.0.143 NAT to 2.2.2.88

Server2:
10.10.0.159 NAT to 2.2.2.89

Server3:
10.10.0.162 NAT to 2.2.2.90

But I do not have enough IPs in the /29 thus wanted to hide all behind one IP.

[bamypamy]

UPDATE 5:
I disabled the outbound NAT rules and created a 1:1 NAT rule (set to nat NOT binat) for the whole network 10.10.0.0/24 to nat to 2.2.2.88/29.
I then let a ping from four different IPs (from within 10.10.0.0/24) on the pfsense run towards 1.1.0.10.
This worked flawlessly.

Did the same from the three Servers and again only one worked.