IPSEC + BiNAT: manual spd entries added wrong

[ruffy91]

I have the following config:

Firewall A:
- Local Interface "LAN" 10.1.0.1/24

- 2 IPSEC Tunnels A+B

- Phase 1 to Firewall B: irrelevant
- Phase 2 to Firewall B
   - Tunnel A is 10.1.0.0/24 local to 10.2.0.0/24 remote

- Phase 1 to Firewall C: irrelevant
- Phase 2 to Firewall C
  - Tunnel B is 10.3.0.0/24 local to 10.4.0.0/24 remote
  - Tunnel B also has Manual SPD Entries: 10.1.0.0/24

Firewall C:
- Phase 1 to Firewall A: irrelevant
- Phase 2 to Firewall A:
  - Tunnel B is 10.4.0.0/24 local to 10.4.0.0/24 remote


We now have a BiNAT setup for:
Interface: IPSEC
Type: BiNAT
External Net: 10.3.0.0/24
Source: 10.1.0.0/24
Destination: 10.4.0.0/24

We no send a packet from 10.1.0.100 to 10.4.0.1.100
We see the packet arrive on "LAN" in a packet dump
We do NOT see the packet on the "IPSEC" interface

As a 2nd test we send a packet from 10.1.0.100 to 10.2.0.100, we can see it on "LAN" and also on the "IPSEC" interface.

I conclude that our Packet gets lost between LAN and IPSEC interfaces on our OPNSense.
Can anyone give me a hint on how to troubleshoot this issue?

Edit:
- clarified configuration

[ruffy91]

OK,

We are affected by:
https://github.com/opnsense/core/issues/2173
https://github.com/opnsense/core/issues/1773

We manually added the correct spd and it works.
But this does not survive a reboot.

How can we make the spdadd entry persistent?

Edit:
We added a sh script in /usr/local/etc/rc.syshook.d/99-spdadd.start

Code Select Expand

#!/bin/sh
/sbin/setkey -f /usr/local/scripts/ipsec/spd.conf
in spd.conf we added the required entries:

Code Select Expand

spdadd <src_net> <dst_net> any -P out ipsec esp/tunnel/<local_wan_ip>-<remote_wan_ip>/unique:<id>;

The spd entry now survives reboots.

[franco]

Not sure how we can put this into a feature yet.

Are these dynamic IPs? Do these belong to a particular phase 2? Why is this manual creation necessary?


Cheers,
Franco

[ruffy91]

I have 2 independent problems which both lead to wrong entries:

Multiple phase 2 entries -> spd entry belongs to the specific phase 2 where I added it in the UI

Remote is behind NAT -> peer identifier is a private ip (10.200.x.x) in this case which is added to the spd instead of the real IP (62.x.x.x) which should be used for the spd.


The desired behaviour would be to take the part after "out ipsec" from the spd entry of the existing phase 2 tunnel where we added the Additional SPD in the UI.