Messages

As I mention that opnsense BUG which I can reproduce 100%. Incorrect route is installed.

Opnsense Team please let me know what the process to get fix it permanently.

Reproduce
1. Setup UI VTI tunnel per doc
2. No traffic is pass through sometimes bidirectional, sometimes not.
3. Inspect route tables ( as you can see that route incorrect that will cause route loop )

Code Select Expand

[b]root@fw01:~ # route -n get 10.0.31.2
   route to: 10.0.31.2
destination: 10.0.31.2
    gateway: 10.0.31.1
        fib: 0
  interface: lo0[/b]
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0     16384         1         0


Code Select Expand

root@fw01:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
da254f91-ea8e-49ef-b1ba-c1727fa06ec9: #1, ESTABLISHED, IKEv2, 99f3bd62c6f5692a_i 3220d485f8f7d532_r*
  local  'local_pub_ip' @ local_pub_ip[500]
  remote 'remote_pub_ip' @ remote_pub_ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_384/MODP_2048
  established 137s ago, rekeying in 12896s
  a9d737db-caa5-46a3-abd7-23b4a655c683: #3, reqid 10, INSTALLED, TUNNEL, ESP:AES_GCM_16-256
    installed 137s ago, rekeying in 3213s, expires in 3823s
    in  c77db5ec,   7780 bytes,   264 packets
    out c692722b,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
root@fw01:~ # ping 10.0.31.2
PING 10.0.31.2 (10.0.31.2): 56 data bytes
92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks       Src       Dst
 4  5  00 0054 32d6   0 0000  01  01 0000 127.0.0.1 10.0.31.2

92 bytes from 127.0.0.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks       Src       Dst
 4  5  00 0054 f40f   0 0000  01  01 0000 127.0.0.1 10.0.31.2

^C
--- 10.0.31.2 ping statistics ---
2 packets transmitted, 0 packets received, 100.0% packet loss


Solution

Code Select Expand


root@fw01:~ # route -n delete 10.0.31.2
delete host 10.0.31.2

root@fw01:~ # route add -host 10.0.31.2 -iface ipsec10
add host 10.0.31.2: gateway ipsec10
root@fw01:~ # ping 10.0.31.2
PING 10.0.31.2 (10.0.31.2): 56 data bytes
64 bytes from 10.0.31.2: icmp_seq=0 ttl=64 time=33.805 ms
64 bytes from 10.0.31.2: icmp_seq=1 ttl=64 time=34.316 ms
64 bytes from 10.0.31.2: icmp_seq=2 ttl=64 time=30.993 ms
64 bytes from 10.0.31.2: icmp_seq=3 ttl=64 time=38.312 ms
64 bytes from 10.0.31.2: icmp_seq=4 ttl=64 time=32.543 ms
^C
--- 10.0.31.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 30.993/33.994/38.312/2.445 ms
root@fw01:~ #



Additional sysctl

Code Select Expand


root@fw01:~ # /sbin/sysctl -a | grep ipsec
<118>[4] .ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.42/mach/CORE
<6>[17] ipsec0: changing name to 'ipsec10'
<118>[23] Setting up routes for ipsec10...done.
<118> IPSECVTI1 (ipsec10) -> v4: 10.0.31.1/30
<6>[823] ipsec1: changing name to 'ipsec11'
<6>[71870] ipsec1: changing name to 'ipsec20'
<6>[80585] ipsec1: changing name to 'ipsec20'
<6>[84757] ipsec10: promiscuous mode enabled
<6>[84774] ipsec10: promiscuous mode disabled
<6>[85026] ipsec10: promiscuous mode enabled
kern.features.ipsec_natt: 1
kern.features.ipsec: 1
net.inet.ipsec.def_policy: 1
net.inet.ipsec.esp_trans_deflev: 1
net.inet.ipsec.esp_net_deflev: 1
net.inet.ipsec.ah_trans_deflev: 1
net.inet.ipsec.ah_net_deflev: 1
net.inet.ipsec.ah_cleartos: 1
net.inet.ipsec.dfbit: 0
net.inet.ipsec.ecn: 0
net.inet.ipsec.debug: 0
net.inet.ipsec.min_pmtu: 576
net.inet.ipsec.random_id: 0
net.inet.ipsec.offload.verbose: 0
net.inet.ipsec.filtertunnel: 1
net.inet.ipsec.natt_cksum_policy: 0
net.inet.ipsec.check_policy_history: 0
net.inet.ipsec.async_crypto: 0
net.inet.ipsec.crypto_support: 50331648
net.inet6.ipsec6.def_policy: 1
net.inet6.ipsec6.esp_trans_deflev: 1
net.inet6.ipsec6.esp_net_deflev: 1
net.inet6.ipsec6.ah_trans_deflev: 1
net.inet6.ipsec6.ah_net_deflev: 1
net.inet6.ipsec6.ecn: 0
net.inet6.ipsec6.debug: 0
net.inet6.ipsec6.filtertunnel: 1
net.enc.out.ipsec_bpf_mask: 1
net.enc.out.ipsec_filter_mask: 0
net.enc.in.ipsec_bpf_mask: 2
net.enc.in.ipsec_filter_mask: 0
root@fw01:~ #


Good Day Community !!!!
If I am writing or asking question then the issue pretty sure is close to bug state. I am building networks over 18 years, for me create VTI set is normally takes between 15 to 20 min max, include FRR eBGP.
I ran into issue where latest version OPNsense 26.7.2_2-amd64 FreeBSD 15.1-RELEASE-p2, VTI is not passing any traffic. All traffic 0 in or out in bytes.
Tunnel is in state ESTABLISHED and no retransmissions.
That state before firewall where are I don't see any traffic on tcpdump enc0 interface.
I see that in opnsense bind VTI interface to loopback, but nothing is routed through.


Any hint what else I can check. Really appreciate on hard work of all people who keeping lights on !!!

root@fw01:~ # swanctl --list-sas
no files found matching '/usr/local/etc/strongswan.opnsense.d/*.conf'
da254f91-ea8e-49ef-b1ba-c1727fa06ec9: #10, ESTABLISHED, IKEv2, e1c7516a937db119_i 30276a85361654fa_r*
  local  'local pub ip' @ local pub ip[500]
  remote 'remote pub ip' @ remote pub ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_384/MODP_2048
  established 2319s ago, rekeying in 11528s
  a9d737db-caa5-46a3-abd7-23b4a655c683: #8, reqid 10, INSTALLED, TUNNEL, ESP:AES_GCM_16-256
    installed 2319s ago, rekeying in 966s, expires in 1641s
    in  c46dd99b,      0 bytes,     0 packets
    out c56e5526,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
bf665be7-7931-4993-9c03-50d54c1f5c6a: #13, ESTABLISHED, IKEv2, b4c20eab1d170db2_i 7e733c1ad763743a_r*
  local  'local pub ip' @ local pub ip[500]
  remote 'remote pub ip' @ remote pub ip[500]
  AES_GCM_16-256/PRF_HMAC_SHA2_256/ECP_521
  established 988s ago, rekeying in 13095s
  2c9e658d-075b-478d-832e-118dfc95d592: #11, reqid 20, INSTALLED, TUNNEL, ESP:AES_CBC-256/HMAC_SHA2_256_128
    installed 988s ago, rekeying in 2397s, expires in 2972s
    in  c93e99c6,      0 bytes,     0 packets
    out c9a00496,      0 bytes,     0 packets
    local  0.0.0.0/0
    remote 0.0.0.0/0
root@fw01:~ # 

ipsec10: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1400
        description: IPSECVTI1 (ipsec10)
        options=0
        tunnel inet local pub ip --> remote pub ip
        inet 10.0.31.1 --> 10.0.31.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 10
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

ipsec20: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1400
        options=0
        tunnel inet local pub ip --> remote pub ip
        inet 192.41.100.1 --> 192.41.100.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 20
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

As side not, please add some warning that will be good visual when "Policies" check mark is set.

Hello Everyone,
I am using haproxy plugin and can't started, based on shell debug limit has wrong arguments  to start haproxy proxy

Code Select Expand


+ _cd=''
+ _doit='/usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid '
+ [ -n '' ]
+ [ -n '' ]
+ [ -n '' ]
+ _doit=' limits -C daemon  /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid '
+ _run_rc_doit ' limits -C daemon  /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid '
+ debug 'run_rc_command: doit:  limits -C daemon  /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid '
+ eval ' limits -C daemon  /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid '
+ limits -C daemon /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid
+ _return=1
+ [ 1 -ne 0 ]
+ [ -z '' ]
+ return 1
+ warn 'failed to start haproxy'
+ [ -x /usr/bin/logger ]
+ logger '/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy'
+ echo '/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy'
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy
+ return 1
root@fw01:~ # usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pidusr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haprox
usr/local/sbin/haproxy: Command not found.
root@fw01:~ # limit -c daemon /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid
limit: Too many arguments.
root@fw01:~ # limit -C daemon /usr/local/sbin/haproxy -q -f /usr/local/etc/haproxy.conf -p /var/run/haproxy.pid
limit: Too many arguments.



Hello Everyone,
After upgrade IPsec vpn route based is not coming up properly or going down all the time.

I tried set Opnsense side as responder, but didn't made any difference.

Version 22.7

Ipsec  Version

strongswan   5.9.9_1

In log

Code Select Expand


14[KNL] received invalid PF_ROUTE message


Yes, let me try.

I log in to cli and be able reproduce the issue

Code Select Expand




fw01.networklab.prod(config-router)# no ip prefix-list DMZ-1 seq 14 permit 10.41.100.0/24 le 32
fw01.networklab.prod(config)# end
fw01.networklab.prod# wr
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Can't open configuration file /usr/local/etc/frr/zebra.conf.1cTIyM.
Can't open configuration file /usr/local/etc/frr/bgpd.conf.YknQcF.
fw01.networklab.prod#


Permissions

Code Select Expand


ls -la /usr/local/etc/frr/
total 94
drwxr-xr-x   2 root  wheel    28 Oct  4 19:35 .
drwxr-xr-x  42 root  wheel   127 Oct  3 23:25 ..
-rw-r--r--   1 root  wheel   665 Sep  4 21:59 babeld.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 19:39 bfdd.conf
-rw-r--r--   1 root  wheel    39 Sep  4 21:59 bfdd.conf.sample
-rw-r--r--   1 root  wheel  1937 Oct  4 19:55 bgpd.conf
-rw-r--r--   1 root  wheel   600 Sep  4 21:59 bgpd.conf.sample
-rw-r--r--   1 root  wheel  2801 Sep  4 21:59 bgpd.conf.sample2
-rw-r--r--   1 root  wheel  1889 Sep  4 21:59 bgpd.conf.vnc.sample
-rw-r--r--   1 root  wheel   186 Sep  4 21:59 eigrpd.conf.sample
-rw-r--r--   1 root  wheel   501 Sep  4 21:59 fabricd.conf.sample
-rw-r--r--   1 root  wheel   789 Sep  4 21:59 isisd.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 ospf6d.conf
-rw-r--r--   1 root  wheel  1110 Sep  4 21:59 ospf6d.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 ospf6d_carp.conf
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 ospfd.conf
-rw-r--r--   1 root  wheel   182 Sep  4 21:59 ospfd.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 ospfd_carp.conf
-rw-r--r--   1 root  wheel   325 Sep  4 21:59 pbrd.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 ripd.conf
-rw-r--r--   1 root  wheel   343 Sep  4 21:59 ripd.conf.sample
-rw-r--r--   1 root  wheel   325 Sep  4 21:59 ripngd.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 sa_policies.conf
-rw-r--r--   1 root  wheel    88 Sep  4 21:59 staticd.conf.sample
-rw-r--r--   1 root  wheel     0 Oct  4 10:57 vtysh.conf
-rw-r--r--   1 root  wheel   128 Sep  4 21:59 vtysh.conf.sample
-rw-r--r--   1 root  wheel   174 Oct  4 10:57 zebra.conf
-rw-r--r--   1 root  wheel   305 Sep  4 21:59 zebra.conf.sample
root@fw01:/etc/rc.d # ps uax | grep frr
frr     78555   0.0  0.1 97080 10536  -  Is   19:39      0:00.01 /usr/local/sbin/zebra -d
frr     81041   0.0  0.1 66916 12140  -  Ss   19:39      0:00.03 /usr/local/sbin/bgpd -d
root    73183   0.0  0.0 12748  2348  0  S+   20:02      0:00.00 grep frr


Hello Everyone,
On latest update FRR package BGP section unable write any change to disk.
Getting errors

Code Select Expand


2022-10-04T13:56:18-04:00 Notice frr_carp FRR received carp configuration event.
2022-10-04T13:56:18-04:00 Error bgpd [EC 100663299] buffer_flush_available: write error on fd 2: Bad file descriptor


Any help thank you.
volga629