IPSec VTI

[volga629]

Good Day Community !!!!
If I am writing or asking question then the issue pretty sure is close to bug state. I am building networks over 18 years, for me create VTI set is normally takes between 15 to 20 min max, include FRR eBGP.
I ran into issue where latest version OPNsense 26.7.2_2-amd64 FreeBSD 15.1-RELEASE-p2, VTI is not passing any traffic. All traffic 0 in or out in bytes.
Tunnel is in state ESTABLISHED and no retransmissions.
That state before firewall where are I don't see any traffic on tcpdump enc0 interface.
I see that in opnsense bind VTI interface to loopback, but nothing is routed through.


Any hint what else I can check. Really appreciate on hard work of all people who keeping lights on !!!

root@fw01:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
da254f91-ea8e-49ef-b1ba-c1727fa06ec9: #10, ESTABLISHED, IKEv2, e1c7516a937db119_i 30276a85361654fa_r*
  local  'local pub ip' @ local pub ip[500]
  remote 'remote pub ip' @ remote pub ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_384/MODP_2048
  established 2319s ago, rekeying in 11528s
  a9d737db-caa5-46a3-abd7-23b4a655c683: #8, reqid 10, INSTALLED, TUNNEL, ESP:AES_GCM_16-256
    installed 2319s ago, rekeying in 966s, expires in 1641s
    in  c46dd99b,      0 bytes,     0 packets
    out c56e5526,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
bf665be7-7931-4993-9c03-50d54c1f5c6a: #13, ESTABLISHED, IKEv2, b4c20eab1d170db2_i 7e733c1ad763743a_r*
  local  'local pub ip' @ local pub ip[500]
  remote 'remote pub ip' @ remote pub ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_256/ECP_521
  established 988s ago, rekeying in 13095s
  2c9e658d-075b-478d-832e-118dfc95d592: #11, reqid 20, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 988s ago, rekeying in 2397s, expires in 2972s
    in  c93e99c6,      0 bytes,     0 packets
    out c9a00496,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
root@fw01:~ # 

ipsec10: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1400
        description: IPSECVTI1 (ipsec10)
        options=0
        tunnel inet local pub ip --> remote pub ip
        inet 10.0.31.1 --> 10.0.31.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 10
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ipsec20: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1400
        options=0
        tunnel inet local pub ip --> remote pub ip
        inet 192.41.100.1 --> 192.41.100.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 20
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

As side not, please add some warning that will be good visual when "Policies" check mark is set.

[volga629]

Additional sysctl

Code Select Expand


root@fw01:~ # /sbin/sysctl -a | grep ipsec
<118>[4] .ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.42/mach/CORE
<6>[17] ipsec0: changing name to 'ipsec10'
<118>[23] Setting up routes for ipsec10...done.
<118> IPSECVTI1 (ipsec10) -> v4: 10.0.31.1/30
<6>[823] ipsec1: changing name to 'ipsec11'
<6>[71870] ipsec1: changing name to 'ipsec20'
<6>[80585] ipsec1: changing name to 'ipsec20'
<6>[84757] ipsec10: promiscuous mode enabled
<6>[84774] ipsec10: promiscuous mode disabled
<6>[85026] ipsec10: promiscuous mode enabled
kern.features.ipsec_natt: 1
kern.features.ipsec: 1
net.inet.ipsec.def_policy: 1
net.inet.ipsec.esp_trans_deflev: 1
net.inet.ipsec.esp_net_deflev: 1
net.inet.ipsec.ah_trans_deflev: 1
net.inet.ipsec.ah_net_deflev: 1
net.inet.ipsec.ah_cleartos: 1
net.inet.ipsec.dfbit: 0
net.inet.ipsec.ecn: 0
net.inet.ipsec.debug: 0
net.inet.ipsec.min_pmtu: 576
net.inet.ipsec.random_id: 0
net.inet.ipsec.offload.verbose: 0
net.inet.ipsec.filtertunnel: 1
net.inet.ipsec.natt_cksum_policy: 0
net.inet.ipsec.check_policy_history: 0
net.inet.ipsec.async_crypto: 0
net.inet.ipsec.crypto_support: 50331648
net.inet6.ipsec6.def_policy: 1
net.inet6.ipsec6.esp_trans_deflev: 1
net.inet6.ipsec6.esp_net_deflev: 1
net.inet6.ipsec6.ah_trans_deflev: 1
net.inet6.ipsec6.ah_net_deflev: 1
net.inet6.ipsec6.ecn: 0
net.inet6.ipsec6.debug: 0
net.inet6.ipsec6.filtertunnel: 1
net.enc.out.ipsec_bpf_mask: 1
net.enc.out.ipsec_filter_mask: 0
net.enc.in.ipsec_bpf_mask: 2
net.enc.in.ipsec_filter_mask: 0
root@fw01:~ #


[volga629]

As I mention that opnsense BUG which I can reproduce 100%. Incorrect route is installed.

Opnsense Team please let me know what the process to get fix it permanently.

Reproduce
1. Setup UI VTI tunnel per doc
2. No traffic is pass through sometimes bidirectional, sometimes not.
3. Inspect route tables ( as you can see that route incorrect that will cause route loop )

Code Select Expand

[b]root@fw01:~ # route -n get 10.0.31.2
   route to: 10.0.31.2
destination: 10.0.31.2
    gateway: 10.0.31.1
        fib: 0
  interface: lo0[/b]
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0


Code Select Expand

root@fw01:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
da254f91-ea8e-49ef-b1ba-c1727fa06ec9: #1, ESTABLISHED, IKEv2, 99f3bd62c6f5692a_i 3220d485f8f7d532_r*
  local  'local_pub_ip' @ local_pub_ip[500]
  remote 'remote_pub_ip' @ remote_pub_ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_384/MODP_2048
  established 137s ago, rekeying in 12896s
  a9d737db-caa5-46a3-abd7-23b4a655c683: #3, reqid 10, INSTALLED, TUNNEL, ESP:AES_GCM_16-256
    installed 137s ago, rekeying in 3213s, expires in 3823s
    in  c77db5ec,   7780 bytes,   264 packets
    out c692722b,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
root@fw01:~ # ping 10.0.31.2
PING 10.0.31.2 (10.0.31.2): 56 data bytes
92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks       Src       Dst
 4  5  00 0054 32d6   0 0000  01  01 0000 127.0.0.1 10.0.31.2

92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks       Src       Dst
 4  5  00 0054 f40f   0 0000  01  01 0000 127.0.0.1 10.0.31.2

^C
--- 10.0.31.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss


Solution

Code Select Expand


root@fw01:~ # route -n delete 10.0.31.2
delete host 10.0.31.2

root@fw01:~ # route add -host 10.0.31.2 -iface ipsec10
add host 10.0.31.2: gateway ipsec10
root@fw01:~ # ping 10.0.31.2
PING 10.0.31.2 (10.0.31.2): 56 data bytes
64 bytes from 10.0.31.2: icmp_seq=0 ttl=64 time=33.805 ms
64 bytes from 10.0.31.2: icmp_seq=1 ttl=64 time=34.316 ms
64 bytes from 10.0.31.2: icmp_seq=2 ttl=64 time=30.993 ms
64 bytes from 10.0.31.2: icmp_seq=3 ttl=64 time=38.312 ms
64 bytes from 10.0.31.2: icmp_seq=4 ttl=64 time=32.543 ms
^C
--- 10.0.31.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 30.993/33.994/38.312/2.445 ms
root@fw01:~ #