[Solved] Second IPsec Site2Site Tunnel down

[BusinessTux]

Hi at all,

I have a problem with a setup on three locations with with two ipsec S2S tunnels to the main office.

I've configured two routed IPSec Tunnels, like described here: https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html.

Homeoffice 1          Main Office                Homeoffice 2
100.64.21.2/30      100.64.21.1/30
                              100.64.22.1/30        100.64.22.2/30
                   
                   
The tunnel to Homeoffice 1 works like a charm. The tunnel to Homeoffice2 is active, but routing isn't functionally.

In short:
- WAN-Rules in Firewall (IPSec, ISAKMP, ESP) are active on all three locations
- Gateways for both home office are created and configured as "far gateway"
- Routes for the remote networks of both home offices are created in the main office
- Routes for the networks of the main office are created in both home offices
- Firewall-Rules on ipsec interface in the main office are created
- Firewall-Rules on ipsec interface in the home offices are created


Traceroute main office to Homeoffice 1: works
Traceroute main office to Homeoffice 2: hangs on main office gateway

The route to home office 2 is in the active routing table of then main office gateway.

But the mainofficerouter says network is down:

Code: [Select]

root@mainofficerouter:~ # ping -t 3 100.64.22.2
PING 100.64.22.2 (100.64.22.2): 56 data bytes
ping: sendto: Network is down
I doublechecked all configurations twice, but I can't figure it out.

The box versions are

mainofficerouter: OPNsense 21.10.2 (amd64/OpenSSL)
homeoffice 1:       OPNsense 21.7.7 (amd64/OpenSSL)
homeoffice 2:       OPNsense 21.10.2 (amd64/OpenSSL)

My ipsec.conf (completely generated)

Code: [Select]

root@mainofficerouter:~ # cat /usr/local/etc/ipsec.conf
# This file is automatically generated. Do not edit
config setup
  uniqueids = yes

conn con1
  aggressive = no
  fragmentation = yes
  keyexchange = ikev2
  mobike = yes
  reauth = yes
  rekey = yes
  forceencaps = no
  installpolicy = no
  type = tunnel





  left = 80.153.119.52
  right = custwar02.edvnet.biz
  rightallowany = yes
  leftid = userfqdn:site2siteHQBN@cust-bonn.de
  ikelifetime = 28800s
  lifetime = 3600s
  ike = aes256-sha512-modp2048!
  leftauth = pubkey
  rightauth = pubkey
  leftcert = /usr/local/etc/ipsec.d/certs/cert-1.crt
  leftsendcert = always
  rightca = "/C=DE/ST=NRW/L=Bonn/O=cust XXXXXX GmbH/OU=cust XCA/CN=custVpnCA/emailAddress=edv@cust-bonn.de/"
  rightid = userfqdn:site2sitehowa@cust-bonn.de
  reqid = 1
  rightsubnet = 0.0.0.0/0
  leftsubnet = 0.0.0.0/0
  esp = aes256-sha512-modp2048!
  auto = add

conn con2
  aggressive = no
  fragmentation = yes
  keyexchange = ikev2
  mobike = yes
  reauth = yes
  rekey = yes
  forceencaps = no
  installpolicy = no
  type = tunnel





  left = 80.153.119.52
  right = custror02.edvnet.biz
  rightallowany = yes
  leftid = userfqdn:site2siteHQBN@cust-bonn.de
  ikelifetime = 28800s
  lifetime = 3600s
  ike = aes256-sha512-modp2048!
  leftauth = pubkey
  rightauth = pubkey
  leftcert = /usr/local/etc/ipsec.d/certs/cert-2.crt
  leftsendcert = always
  rightca = "/C=DE/ST=NRW/L=Bonn/O=cust XXXXXX GmbH/OU=cust XCA/CN=custVpnCA/emailAddress=edv@cust-bonn.de/"
  rightid = userfqdn:site2sitehoro@cust-bonn.de
  reqid = 2
  rightsubnet = 0.0.0.0/0
  leftsubnet = 0.0.0.0/0
  esp = aes256-sha512-modp2048!
  auto = add

include ipsec.opnsense.d/*.conf
Where is my error?

[BusinessTux]

Here more screenshots

[BusinessTux]

Not good for debugging, but good for me.

This morning the ipsec connection to home office 1 was down. After I restartet ipsec vpn (in settings) the tunnel was online, but there was no routing. I saw the ping from home office 1 in the firewall log of the main office as passed to the lokal intranet of the main office. But there was no reply.

For my understanding the tunnel was online, but the main office gateway doesn't route.

After a restart of the OPNsense in main office both tunnels where working. I hope, this will be for a long time.

Where can I find additionial informations about routing problems in the OPNsense?
What can I restart to get routing back to work without restart the hardware?

Thanks
Ulf

[BusinessTux]

After about two weeks and very stable tunnels notice to myself: a reboot will not hurt

[BusinessTux]

Today there was no routing again.
After some restarts of all three devices the tunnels where up, but no routing.

I've doublecheck System/Routes/Status. The static routes I entered were not present. 

My workaround: Edit one static route and save it without to change someting.

After that both ipsec routings where online again.

[BusinessTux]

The error is back. 

Meantime I've read the GIT issues Static route to route-based IPsec gateway does not get configured after reboot and IPSec Route missing after WAN DHCP Renew (#3414 related?)

So reconfigured my phase 1 to use a static IP to the central site and certificates for authentication, like described here: https://nwildner.com/posts/2019-09-24-how-to-site2site-opnsense/ with the difference, that I use routed tunnels (VTI).

But homeoffice 2 can't route. If I stop the IPsec-Tunnel in homeoffce 2 und start it again the folling lines are in the ipsec logfile on the mainoffice router (redacted)

Code: [Select]

Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <8> received packet: from 79.XXX.XXX.190[500] to 80.XXX.XXX.52[500] (464 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> 79.XXX.XXX.190 is initiating an IKE_SA
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <8> selected proposal: IKE:AES_CBC_256/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/MODP_2048
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> sending cert request for "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerRootCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> sending cert request for "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerVpnCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> sending cert request for "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerMgmtCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <8> sending packet: from 80.XXX.XXX.52[500] to 79.XXX.XXX.190[500] (537 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <8> received packet: from 79.XXX.XXX.190[4500] to 80.XXX.XXX.52[4500] (1236 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> parsed IKE_AUTH request 1 [ EF(1/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> received fragment #1 of 3, waiting for complete IKE message
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <8> received packet: from 79.XXX.XXX.190[4500] to 80.XXX.XXX.52[4500] (1236 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> parsed IKE_AUTH request 1 [ EF(2/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> received fragment #2 of 3, waiting for complete IKE message
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <8> received packet: from 79.XXX.XXX.190[4500] to 80.XXX.XXX.52[4500] (548 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> parsed IKE_AUTH request 1 [ EF(3/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> received fragment #3 of 3, reassembled fragmented IKE message (2832 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <8> parsed IKE_AUTH request 1 [ IDi CERT CERTREQ IDr AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> received cert request for "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerVpnCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <8> received end entity cert "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, E=site2sitehoro@custdomain.de, CN=Customer_HORO"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <8> looking for peer configs matching 80.XXX.XXX.52[site2siteHQBN@custdomain.de]...79.XXX.XXX.190[site2sitehoro@custdomain.de]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> selected peer config 'con2'
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8>   using certificate "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, E=site2sitehoro@custdomain.de, CN=Customer_HORO"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8>   using trusted intermediate ca certificate "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerVpnCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> checking certificate status of "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, E=site2sitehoro@custdomain.de, CN=Customer_HORO"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> certificate status is not available
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8>   using trusted ca certificate "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerRootCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> checking certificate status of "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, OU=Customer XCA, CN=CustomerVpnCA, E=edv@custdomain.de"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> certificate status is not available
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8>   reached self-signed root ca with a path length of 1
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> authentication of 'site2sitehoro@custdomain.de' with RSA_EMSA_PKCS1_SHA2_256 successful
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> peer supports MOBIKE
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> authentication of 'site2siteHQBN@custdomain.de' (myself) with RSA_EMSA_PKCS1_SHA2_256 successful
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|7> schedule delete of duplicate IKE_SA for peer 'site2sitehoro@custdomain.de' due to uniqueness policy and suspected reauthentication
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> IKE_SA con2[8] established between 80.XXX.XXX.52[site2siteHQBN@custdomain.de]...79.XXX.XXX.190[site2sitehoro@custdomain.de]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> scheduling reauthentication in 28070s
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> maximum IKE_SA lifetime 28610s
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> sending end entity cert "C=DE, ST=NRW, L=Bonn, O=Customer GmbH, E=site2siteHQBN@custdomain.de, CN=Customer_HQBN"
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[CFG] <con2|8> selected proposal: ESP:AES_CBC_256/HMAC_SHA2_512_256/NO_EXT_SEQ
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|8> CHILD_SA con2{14} established with SPIs c4725931_i c15fe5c5_o and TS 0.0.0.0/0 === 0.0.0.0/0
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|8> generating IKE_AUTH response 1 [ IDr CERT AUTH N(ESP_TFC_PAD_N) SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_6_ADDR) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|8> splitting IKE message (2720 bytes) into 3 fragments
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|8> generating IKE_AUTH response 1 [ EF(1/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|8> generating IKE_AUTH response 1 [ EF(2/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|8> generating IKE_AUTH response 1 [ EF(3/3) ]
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <con2|8> sending packet: from 80.XXX.XXX.52[4500] to 79.XXX.XXX.190[4500] (1236 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <con2|8> sending packet: from 80.XXX.XXX.52[4500] to 79.XXX.XXX.190[4500] (1236 bytes)
Jan 15 14:11:30 mainofficerouter.custdomain.de charon[31072]: 15[NET] <con2|8> sending packet: from 80.XXX.XXX.52[4500] to 79.XXX.XXX.190[4500] (420 bytes)
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|7> deleting IKE_SA con2[7] between 80.XXX.XXX.52[site2siteHQBN@custdomain.de]...79.XXX.XXX.190[site2sitehoro@custdomain.de]
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|7> sending DELETE for IKE_SA con2[7]
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|7> generating INFORMATIONAL request 0 [ D ]
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[NET] <con2|7> sending packet: from 80.XXX.XXX.52[4500] to 79.XXX.XXX.190[4500] (96 bytes)
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[NET] <con2|7> received packet: from 79.XXX.XXX.190[4500] to 80.XXX.XXX.52[4500] (96 bytes)
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[ENC] <con2|7> parsed INFORMATIONAL response 0 [ ]
Jan 15 14:11:40 mainofficerouter.custdomain.de charon[31072]: 15[IKE] <con2|7> IKE_SA deleted

For me it seems to be a successfully connect.

But in contrast to the IPsec interface to homeoffice 1 there is no tunnel line in ifconfig:

Code: [Select]

root@pmainofficerouter:~ # ifconfig ipsec1
ipsec1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
        tunnel inet 80.XXX.XXX.52 --> 91.XXX.XXX.162
        inet6 fe80::42a6:b7ff:fe3c:f8cd%ipsec1 prefixlen 64 scopeid 0x14
        inet 100.64.21.1 --> 100.64.21.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 1
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
root@pmainofficerouter:~ # ifconfig ipsec2
ipsec2: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1400
        inet 100.64.22.1 --> 100.64.22.2 netmask 0xfffffffc
        inet6 fe80::42a6:b7ff:fe3c:f8cd%ipsec2 prefixlen 64 tentative scopeid 0x15
        groups: ipsec
        reqid: 2
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

mainofficerouter ipsec1 = homeoffice 1
mainofficerouter ipsec2 = homeoffice 2

The static routes via the IPsec tunnel a online on both routers

Code: [Select]

root@mainofficerouter:~ # date ; netstat -rn | grep ipsec
Sat Jan 15 14:28:45 CET 2022
10.0.1.0/24        100.64.21.2        UGS      ipsec1
10.0.2.0/24        100.64.21.2        UGS      ipsec1
10.0.5.0/24        100.64.21.2        UGS      ipsec1
10.1.12.0/24       100.64.22.2        UGS      ipsec2
10.1.21.0/24       100.64.21.2        UGS      ipsec1
10.1.21.253        100.64.21.2        UGHS     ipsec1
10.1.22.0/24       100.64.22.2        UGS      ipsec2
10.1.22.2          100.64.22.2        UGHS     ipsec2
10.1.62.0/24       100.64.22.2        UGS      ipsec2
100.64.21.2        ipsec1             UHS      ipsec1
100.64.22.2        ipsec2             UHS      ipsec2
fe80::%ipsec1/64                  link#20                       U        ipsec1
fe80::42a6:b7ff:fe3c:f8cd%ipsec1  link#20                       UHS         lo0
fe80::%ipsec2/64                  link#21                       U        ipsec2
fe80::42a6:b7ff:fe3c:f8cd%ipsec2  link#21                       UHS         lo0


Code: [Select]

root@homeoffice2router:~ # date ; netstat -rn | grep ipsec
Sat Jan 15 14:29:55 CET 2022
10.0.1.0/24        100.64.22.1        UGS      ipsec1
10.0.2.0/24        100.64.22.1        UGS      ipsec1
10.1.1.0/24        100.64.22.1        UGS      ipsec1
10.1.2.0/24        100.64.22.1        UGS      ipsec1
10.1.2.2           100.64.22.1        UGHS     ipsec1
10.1.4.0/24        100.64.22.1        UGS      ipsec1
100.64.22.1        ipsec1             UHS      ipsec1
fe80::%ipsec1/64                  link#18                       U        ipsec1
fe80::de58:bcff:fee0:38ca%ipsec1  link#18                       UHS         lo0

route on both opnsense shows the correct routings

Code: [Select]

root@homeoffice2router:~ # route -n show 10.1.1.18
   route to: 10.1.2.2
destination: 10.1.2.2
    gateway: 100.64.22.1
        fib: 0
  interface: ipsec1
      flags: <UP,GATEWAY,HOST,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1400         1         0



Code: [Select]

root@mainofficerouter:~ # route -n show 10.1.22.111
   route to: 10.1.22.111
destination: 10.1.22.0
       mask: 255.255.255.0
    gateway: 100.64.22.2
        fib: 0
  interface: ipsec2
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1400         1         0

Nevertheless, a traceroute from both sides ends on the respective opensense

Traceroute from homeoffice 2 pc to domain controller

Code: [Select]

C:\WINDOWS\system32>ipconfig

Windows-IP-Konfiguration


Ethernet-Adapter Ethernet:

   Verbindungsspezifisches DNS-Suffix: intra.customdomain.de
   IPv6-Adresse. . . . . . . . . . . : XXXX.XX:XXXX:XXXX::2000
   IPv6-Adresse. . . . . . . . . . . : XXXX.XX:XXXX:XXXX:8821:11ff:6c2a:716
   Verbindungslokale IPv6-Adresse  . : fe80::8821:11ff:6c2a:716%7
   IPv4-Adresse  . . . . . . . . . . : 10.1.22.111
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . : fe80::de58:bcff:fee0:38cb%7
                                       10.1.22.1

C:\WINDOWS\system32>tracert -d 10.1.2.2

Routenverfolgung zu 10.1.2.2 über maximal 30 Hops

  1     2 ms     1 ms     9 ms  10.1.22.1
  2     *        *        *     Zeitüberschreitung der Anforderung.
  3  ^C
C:\WINDOWS\system32>
Traceroute from domain controller to homeoffice 2 pc

Code: [Select]

root@10.1.2.2:~# traceroute -n 10.1.22.111
traceroute to 10.1.22.111 (10.1.22.111), 30 hops max, 60 byte packets
 1  10.1.2.1  0.177 ms  0.161 ms  0.139 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *^C

Which information do you need additionally?

Please give me a hint, where I can search for my error.

[BusinessTux]

I found one more detail.

The mainoffice router said, the tunnel network to homeoffice 2 is down. But why?

Code: [Select]

root@mainofficerouter:~ # ping -t 3 100.64.21.2
PING 100.64.21.2 (100.64.21.2): 56 data bytes
64 bytes from 100.64.21.2: icmp_seq=0 ttl=64 time=34.851 ms
64 bytes from 100.64.21.2: icmp_seq=1 ttl=64 time=35.068 ms
64 bytes from 100.64.21.2: icmp_seq=2 ttl=64 time=35.031 ms

--- 100.64.21.2 ping statistics ---
4 packets transmitted, 3 packets received, 25.0% packet loss
round-trip min/avg/max/stddev = 34.851/34.983/35.068/0.095 ms
root@mainofficerouter:~ # ping -t 3 100.64.22.2
PING 100.64.22.2 (100.64.22.2): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down

--- 100.64.22.2 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
The ipsec status is online for both tunnels


The box versions are

mainofficerouter: OPNsense 21.10.2 (amd64/OpenSSL)
homeoffice 1:       OPNsense 21.7.7 (amd64/OpenSSL)
homeoffice 2:       OPNsense 21.10.2 (amd64/OpenSSL)

[mimugmail]

route-based IPsec is known to be unstable with dynamic IPs and FQDNs.
If all systems are OPNsense you really should go the way with OpenVPN instead of IPsec

[BusinessTux]

Thanks.

The DynDNS-Names I use only for Admin-OpenVPN-Connection.

The IPsec connection to the mainofficerouter will be connected from both homeoffice routers via static IP (like to se in screenshot from homeoffice 2 router)

The mainofficerouter is set to respond only in IKE Phase 1

[mimugmail]

Why do you tick "Dynamic Gateway"? This is only needed for respond-only.

You really should start at the beginning:

On both sites "default", not respond or start, only use IPs instead of names, use PSK instead of certs.
If this work activate one by another.

[BusinessTux]

Why do you tick "Dynamic Gateway"? This is only needed for respond-only.

Yes, you're right. This is from one of the many attempts.

You really should start at the beginning:

On both sites "default", not respond or start, only use IPs instead of names, use PSK instead of certs.
If this work activate one by another.

I will do and report.

Thanks

[BusinessTux]

Today I created a new IPsec tunnel only with PSK.

[BusinessTux]

But the result is the same 

[BusinessTux]

Routes are online

Code: [Select]

root@mainofficerouter:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            62.156.244.25      UGS      pppoe1
8.8.8.8            62.156.244.25      UGHS     pppoe1
9.9.9.9            192.168.2.1        UGHS       igb3
10.0.1.0/24        100.64.21.2        UGS      ipsec1
10.0.2.0/24        100.64.21.2        UGS      ipsec1
10.0.5.0/24        100.64.21.2        UGS      ipsec1
10.1.1.0/24        link#3             U          igb0
10.1.1.1           link#3             UHS         lo0
10.1.2.0/24        link#12            U      lagg0_vl
10.1.2.1           link#12            UHS         lo0
10.1.3.0/24        link#13            U      lagg0_vl
10.1.3.1           link#13            UHS         lo0
10.1.4.0/24        link#14            U      lagg0_vl
10.1.4.1           link#14            UHS         lo0
10.1.6.0/24        link#15            U      lagg0_vl
10.1.6.1           link#15            UHS         lo0
10.1.12.0/24       100.64.22.2        UGS      ipsec2
10.1.21.0/24       100.64.21.2        UGS      ipsec1
10.1.21.253        100.64.21.2        UGHS     ipsec1
10.1.22.0/24       100.64.22.2        UGS      ipsec2
10.1.22.2          100.64.22.2        UGHS     ipsec2
10.1.62.0/24       100.64.22.2        UGS      ipsec2
62.156.244.25      link#20            UH       pppoe1
80.153.119.52      link#20            UHS         lo0
100.64.11.0/24     100.64.11.2        UGS      ovpns1
100.64.11.1        link#18            UHS         lo0
100.64.11.2        link#18            UH       ovpns1
100.64.21.1        link#21            UHS         lo0
100.64.21.2        ipsec1             UHS      ipsec1
100.64.22.1        link#19            UHS         lo0
100.64.22.2        ipsec2             UHS      ipsec2
127.0.0.1          link#8             UH          lo0
192.168.2.0/24     link#6             U          igb3
192.168.2.1        3c:ec:ef:89:35:87  UHS        igb3
192.168.2.201      link#6             UHS         lo0
217.237.149.205    62.156.244.25      UGHS     pppoe1
217.237.151.51     62.156.244.25      UGHS     pppoe1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::200:ff:fe00:0%pppoe1    UG       pppoe1
::1                               link#8                        UH          lo0
2003:a:77f:f6bc::/64              link#20                       U        pppoe1
2003:a:77f:f6bc:42a6:b7ff:fe3c:f8cd link#20                     UHS         lo0
fe80::%ixl1/64                    link#2                        U          ixl1
fe80::42a6:b7ff:fe3c:f8cd%ixl1    link#2                        UHS         lo0
fe80::%igb0/64                    link#3                        U          igb0
fe80::3eec:efff:fe89:3584%igb0    link#3                        UHS         lo0
fe80::%igb2/64                    link#5                        U          igb2
fe80::3eec:efff:fe89:3586%igb2    link#5                        UHS         lo0
fe80::%igb3/64                    link#6                        U          igb3
fe80::3eec:efff:fe89:3587%igb3    link#6                        UHS         lo0
fe80::%lo0/64                     link#8                        U           lo0
fe80::1%lo0                       link#8                        UHS         lo0
fe80::%lagg0/64                   link#11                       U         lagg0
fe80::42a6:b7ff:fe3c:f8cc%lagg0   link#11                       UHS         lo0
fe80::%lagg0_vlan1120/64          link#12                       U      lagg0_vl
fe80::42a6:b7ff:fe3c:f8cc%lagg0_vlan1120 link#12                UHS         lo0
fe80::%lagg0_vlan1130/64          link#13                       U      lagg0_vl
fe80::42a6:b7ff:fe3c:f8cc%lagg0_vlan1130 link#13                UHS         lo0
fe80::%lagg0_vlan1140/64          link#14                       U      lagg0_vl
fe80::42a6:b7ff:fe3c:f8cc%lagg0_vlan1140 link#14                UHS         lo0
fe80::%lagg0_vlan1160/64          link#15                       U      lagg0_vl
fe80::42a6:b7ff:fe3c:f8cc%lagg0_vlan1160 link#15                UHS         lo0
fe80::%igb2_vlan7/64              link#16                       U      igb2_vla
fe80::3eec:efff:fe89:3586%igb2_vlan7 link#16                    UHS         lo0
fe80::%ixl1_vlan7/64              link#17                       U      ixl1_vla
fe80::42a6:b7ff:fe3c:f8cd%ixl1_vlan7 link#17                    UHS         lo0
fe80::42a6:b7ff:fe3c:f8cd%ovpns1  link#18                       UHS         lo0
fe80::%ipsec2/64                  link#19                       U        ipsec2
fe80::42a6:b7ff:fe3c:f8cd%ipsec2  link#19                       UHS         lo0
fe80::%pppoe1/64                  link#20                       U        pppoe1
fe80::3eec:efff:fe89:3584%pppoe1  link#20                       UHS         lo0
fe80::42a6:b7ff:fe3c:f8cd%pppoe1  link#20                       UHS         lo0
fe80::%ipsec1/64                  link#21                       U        ipsec1
fe80::42a6:b7ff:fe3c:f8cd%ipsec1  link#21                       UHS         lo0


Code: [Select]

root@homeoffice2router:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            62.155.241.177     UGS      pppoe0
8.8.8.8            62.155.241.177     UGHS     pppoe0
10.1.1.0/24        100.64.22.1        UGS      ipsec1
10.1.2.0/24        100.64.22.1        UGS      ipsec1
10.1.2.2           100.64.22.1        UGHS     ipsec1
10.1.12.0/24       link#2             U          igb1
10.1.12.1          link#2             UHS         lo0
10.1.22.0/24       link#12            U      igb1_vla
10.1.22.1          link#12            UHS         lo0
10.1.32.0/24       link#13            U      igb1_vla
10.1.32.1          link#13            UHS         lo0
10.1.62.0/24       link#14            U      igb1_vla
10.1.62.1          link#14            UHS         lo0
62.155.241.177     link#17            UH       pppoe0
79.207.107.190     link#17            UHS         lo0
100.64.12.0/24     100.64.12.2        UGS      ovpns1
100.64.12.1        link#16            UHS         lo0
100.64.12.2        link#16            UH       ovpns1
100.64.22.1        ipsec1             UHS      ipsec1
100.64.22.2        link#18            UHS         lo0
127.0.0.1          link#8             UH          lo0
217.237.150.115    62.155.241.177     UGHS     pppoe0
217.237.151.205    62.155.241.177     UGHS     pppoe0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
default                           fe80::231:46ff:fe06:6f83%pppoe0 UG     pppoe0
::1                               link#8                        UH          lo0
2001:4860:4860::8888              fe80::231:46ff:fe06:6f83%pppoe0 UGHS   pppoe0
2003:e2:af2f:3801::/64            link#2                        U          igb1
2003:e2:af2f:3801:de58:bcff:fee0:38cb link#2                    UHS         lo0
2003:e2:af2f:3802::/64            link#12                       U      igb1_vla
2003:e2:af2f:3802:de58:bcff:fee0:38cb link#12                   UHS         lo0
2003:e2:af2f:3803::/64            link#13                       U      igb1_vla
2003:e2:af2f:3803:de58:bcff:fee0:38cb link#13                   UHS         lo0
2003:e2:af2f:3806::/64            link#14                       U      igb1_vla
2003:e2:af2f:3806:de58:bcff:fee0:38cb link#14                   UHS         lo0
2003:e2:afff:2f85::/64            link#17                       U        pppoe0
2003:e2:afff:2f85:de58:bcff:fee0:38ca link#17                   UHS         lo0
fe80::%igb1/64                    link#2                        U          igb1
fe80::de58:bcff:fee0:38cb%igb1    link#2                        UHS         lo0
fe80::%igb5/64                    link#6                        U          igb5
fe80::de58:bcff:fee0:38cf%igb5    link#6                        UHS         lo0
fe80::%lo0/64                     link#8                        U           lo0
fe80::1%lo0                       link#8                        UHS         lo0
fe80::%lagg0/64                   link#11                       U         lagg0
fe80::de58:bcff:fee0:38cc%lagg0   link#11                       UHS         lo0
fe80::%igb1_vlan1122/64           link#12                       U      igb1_vla
fe80::de58:bcff:fee0:38cb%igb1_vlan1122 link#12                 UHS         lo0
fe80::%igb1_vlan1132/64           link#13                       U      igb1_vla
fe80::de58:bcff:fee0:38cb%igb1_vlan1132 link#13                 UHS         lo0
fe80::%igb1_vlan1162/64           link#14                       U      igb1_vla
fe80::de58:bcff:fee0:38cb%igb1_vlan1162 link#14                 UHS         lo0
fe80::%igb5_vlan7/64              link#15                       U      igb5_vla
fe80::de58:bcff:fee0:38cf%igb5_vlan7 link#15                    UHS         lo0
fe80::de58:bcff:fee0:38ca%ovpns1  link#16                       UHS         lo0
fe80::%pppoe0/64                  link#17                       U        pppoe0
fe80::de58:bcff:fee0:38ca%pppoe0  link#17                       UHS         lo0
fe80::de58:bcff:fee0:38cb%pppoe0  link#17                       UHS         lo0
fe80::%ipsec1/64                  link#18                       U        ipsec1
fe80::de58:bcff:fee0:38ca%ipsec1  link#18                       UHS         lo0

Ping says network is down.

Code: [Select]

###ifconfig
ipsec1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1400
        tunnel inet 80.153.119.52 --> 91.5.102.162
        inet6 fe80::42a6:b7ff:fe3c:f8cd%ipsec1 prefixlen 64 scopeid 0x15
        inet 100.64.21.1 --> 100.64.21.2 netmask 0xfffffffc
        groups: ipsec
        reqid: 1
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
ipsec2: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1400
        inet 100.64.22.1 --> 100.64.22.2 netmask 0xfffffffc
        inet6 fe80::42a6:b7ff:fe3c:f8cd%ipsec2 prefixlen 64 tentative scopeid 0x13
        groups: ipsec
        reqid: 2
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

### Ping Homeoffice 1
root@mainofficerouter:~ # ping 100.64.21.2
PING 100.64.21.2 (100.64.21.2): 56 data bytes
64 bytes from 100.64.21.2: icmp_seq=0 ttl=64 time=40.066 ms
64 bytes from 100.64.21.2: icmp_seq=1 ttl=64 time=40.157 ms
^C
--- 100.64.21.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 40.066/40.112/40.157/0.046 ms
### Ping Homeoffice 2
root@mainofficerouter:~ # ping 100.64.22.2
PING 100.64.22.2 (100.64.22.2): 56 data bytes
ping: sendto: Network is down
ping: sendto: Network is down
ping: sendto: Network is down
^C
--- 100.64.22.2 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss

[BusinessTux]

After a lot of testing my thanks go to the professional services from m.a.x. it. It was a pleasure and workshop together with great results.

After all, ipsec with dynmic wan ips work only with workarounds like Monit. After i activated and used statice ips vpn tunnel and reconnect works flawlessly. In the homeoffice with dynamic ip Monit reconnects the vpn tunnel after disconnects.