Cannot set up route-based (VTI) ipsec vpn between OPNsense <-> VyOS

[onnieoneone]

Hi, I am trying to set up a route-based (VTI) ipsec site-to-site tunnel between OPNsense (the A site) and VyOS (well, a Unifi USG-3P, the B site) using the "new > 23.1" setup using this guide: https://docs.opnsense.org/manual/how-tos/ipsec-s2s-conn-route.html

I have previously had a policy-based ipsec vpn working since about 19.1 but I think the problem referenced here hindered me (https://forum.opnsense.org/index.php?topic=30525.0) in that after 22.7 only one pair of subnets at each site could communicate at any one time and I had to manually take CHILD_SAs up and down.

I'm finally deciding to move to a route-based setup in the hope that with only one CHILD_SA for 0.0.0.0/0 -> 0.0.0.0/0 communication that it will work properly. I'd move to wireguard but there is no Unifi support for it on the USG-3P.

So, I have followed the guide for OPNsense for site A, and set up the other site B according to some VyOS/Unifi instructions.

I see what looks to me like a good SA setup (yes, both sides WAN interfaces are behind NAT on a 192* "DMZ" network, but the IKE and ESP seems to work):

Code Select Expand

# swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
ca24ede1-a222-4a6e-91dc-2c5720143fe6: #1, ESTABLISHED, IKEv2, 7d53711e7fea6ef4_i* 62a0c1809bc25095_r
  local  'A.A.A.A' @ 192.168.178.20[4500]
  remote '192.168.188.20' @ B.B.B.B[4500]
  AES_CBC-256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
  established 2172s ago, rekeying in 11323s
  afbe2c6c-fb46-4a1d-a9f4-facecc5ff2f9: #1, reqid 10, INSTALLED, TUNNEL-in-UDP, ESP:AES_CBC-256/HMAC_SHA1_96
    installed 2172s ago, rekeying in 1169s, expires in 1788s
    in  cc44dc3d,      0 bytes,     0 packets
    out c033ca54,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
Here is my OPNsense ipsec10 interface (do I have my 'tunnel inet' and 'inet' addresses mixed up?):

Code Select Expand

# ifconfig ipsec10
ipsec10: flags=1008011<UP,POINTOPOINT,MULTICAST,LOWER_UP> metric 0 mtu 1400
        options=0
        tunnel inet A.A.A.A --> 192.168.188.20
        inet 10.111.0.1 --> 10.111.0.2 netmask 0xfffffffc
        inet6 fe80::dead:beef:dead:beef%ipsec10 prefixlen 64 tentative scopeid 0x1a
        groups: ipsec
        reqid: 10
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
My A site's local networks are /24s contained in a 10.1.0.0/16 range and the B site is likewise in 10.2.0.0/16

Here's my A site routing table:

Code Select Expand

# netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.178.1      UGS            igb3
10.1.2.0/24        link#12            U      lagg0_vlan10
10.1.2.1           link#5             UHS             lo0
10.1.4.0/24        link#14            U      lagg0_vlan10
10.1.4.1           link#5             UHS             lo0
10.1.5.0/24        link#15            U      lagg0_vlan10
10.1.5.1           link#5             UHS             lo0
10.1.6.0/24        link#16            U      lagg0_vlan10
10.1.6.1           link#5             UHS             lo0
10.1.8.0/24        link#18            U      lagg0_vlan10
10.1.8.1           link#5             UHS             lo0
10.1.9.0/24        link#19            U      lagg0_vlan10
10.1.9.1           link#5             UHS             lo0
10.1.46.1          link#5             UHS             lo0
10.1.64.0/24       link#27            US            nat64
10.1.64.1          link#27            UH            nat64
10.2.0.0/16        10.111.0.2         UGS         ipsec10
10.111.0.1         link#5             UHS             lo0
10.111.0.2         link#26            UH          ipsec10
127.0.0.1          link#5             UH              lo0
192.168.1.0/24     link#1             U              igb0
192.168.1.10       link#5             UHS             lo0
192.168.178.0/24   link#4             U              igb3
192.168.178.20     link#5             UHS             lo0
So it looks as though I have set up the gateway and route to send 10.2.0.0/16 to the far side of the ipsec10 tunnel.

But it's not working.

I try a packet capture on ipsec10 and a ping to the far network (at the same time) and I get this:

Code Select Expand

$ ping -4 -S 10.1.8.1 10.2.4.1
PING 10.2.4.1 (10.2.4.1) from 10.1.8.1: 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.2.4.1 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

# tcpdump -i ipsec10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ipsec10, link-type NULL (BSD loopback), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

I'm not sure how this should behave, but I get this:

Code Select Expand

# ping 10.111.0.1
PING 10.111.0.1 (10.111.0.1): 56 data bytes
64 bytes from 10.111.0.1: icmp_seq=0 ttl=64 time=0.329 ms
64 bytes from 10.111.0.1: icmp_seq=1 ttl=64 time=0.158 ms
^C
--- 10.111.0.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.158/0.243/0.329/0.086 ms
# ping 10.111.0.2
PING 10.111.0.2 (10.111.0.2): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 10.111.0.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss
When using policy-based ipsec I used to be able to tcpdump on the enc0 interface and see the encapsulated ESP traffic, but that also shows nothing.

Perhaps the far end needs to be configured with that tunnel address, but it looks like VyOS doesn't do it like that and their vti interface is simply unnumbered (see this old unaswered post https://forum.opnsense.org/index.php?topic=38062.0)

What am I doing wrongly?

Does anyone have such an OPNsense <-> VyOS setup working?