IPSEC and Carp?

[dragon2611]

Have 3 Opnsense firewalls connecting to a RouterOS device.

2 of them are a pair, the other is standalone (Different Network)

On the pair in HA/CARP the connection will drop and not re-establish (no phase2) unless I bounce the Primary of the Pair.

Peer is 0.0.0.0 with a Identifier set due to the remote end being a dynamic IP.

This worked when it was just a single firewall so I suspect the issue is around CARP/the VIP.

Nat rules are set so 500/4500 get's natted to the VIP going out and the VIP is set as the IP to use in the IPSEC settings, tried changing the identifier on the Opnsense end from "Interface Address" to manually set and then put in the VIP address.

Tried flushing the SA's on the RouterOS side and restarting strongswan on opnsense but it doesn't seem to help.

Code: [Select]

Aug 13 13:05:41 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:41 charon: 11[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
Aug 13 13:05:41 charon: 11[IKE] sending cert request for "C=GB, ST=here, L=ssd, O=ssd, E=here@here.local, CN=internal-ca"
Aug 13 13:05:41 charon: 11[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:05:41 charon: 11[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:05:41 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:41 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:30 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:30 charon: 11[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:30 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:30 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:25 charon: 11[JOB] deleting half open IKE_SA with 81.108.xxx.xxx after timeout
Aug 13 13:05:25 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:25 charon: 11[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:25 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:25 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:20 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:20 charon: 11[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:20 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:20 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:18 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:18 charon: 11[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
Aug 13 13:05:18 charon: 11[IKE] sending cert request for "C=GB, ST=here, L=ssd, O=ssd, E=here@here.local, CN=internal-ca"
Aug 13 13:05:18 charon: 11[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:05:18 charon: 11[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:05:18 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:18 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:10 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:10 charon: 11[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:10 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:10 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:05 charon: 11[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:05 charon: 11[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:05 charon: 11[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:05 charon: 11[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:05:03 charon: 13[JOB] deleting half open IKE_SA with 81.108.xxx.xxx after timeout
Aug 13 13:05:00 charon: 13[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:05:00 charon: 13[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:05:00 charon: 13[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:05:00 charon: 13[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:04:55 charon: 13[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:04:55 charon: 13[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
Aug 13 13:04:55 charon: 13[IKE] sending cert request for "C=GB, ST=here, L=ssd, O=ssd, E=here@here.local, CN=internal-ca"
Aug 13 13:04:55 charon: 13[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:04:55 charon: 13[IKE] 81.108.xxx.xxx is initiating an IKE_SA
Aug 13 13:04:55 charon: 13[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Aug 13 13:04:55 charon: 13[NET] received packet: from 81.108.xxx.xxx[4500] to 78.xxx.xxx.xxx[4500] (680 bytes)
Aug 13 13:04:45 charon: 13[NET] sending packet: from 78.xxx.xxx.xxx[4500] to 81.108.xxx.xxx[4500] (721 bytes)
Aug 13 13:04:45 charon: 13[IKE] received retransmit of request with ID 0, retransmitting response
Aug 13 13:04:45 charon: 13[ENC] parsed IKE_SA_INIT request 0 [ N(NATD_D_IP) N(NATD_S_IP) No KE SA ]
Edit:

Tried disabling NAT-T, no difference.

[dragon2611]

Anyone any ideas? 

The standalone OpnSense on the other server seems to be behaving itself, but this pair is being a right pain for IPSEC.

I was really hoping to be able to use IPSEC with the VIP, reconnecting during failover works it's just the sessions never stay up on whichever opnsense instance is the active one, the phase2's drop out and will never re-establish.

[mimugmail]

Can you post the log from the other side? You really use certificates? IKEv1 or v2?

[dragon2611]

IKEv2 and a PSK, so not sure why it's going on about certs 

I can provide logs from the Mikrotik end but I'll need to reconfigure it back to using the VIP first, that said it doesn't even seem stable to the Primary firewalls real IP.

[dragon2611]

Hmm Looks like if I switch to IKEv1 one of the ends is trying to use the real IP for the firewall in NAT-T and not the VIP.

There is config for the real IP but it's disabled both ends.

Code: [Select]

Aug 28 11:39:36 charon: 09[NET] sending packet: from 78.xxx.xxx254[500] to 81.xxx.xxx.53[500] (716 bytes)
Aug 28 11:39:36 charon: 09[ENC] generating ID_PROT response 0 [ KE No NAT-D NAT-D ]
Aug 28 11:39:36 charon: 09[ENC] parsed ID_PROT request 0 [ KE No NAT-D NAT-D ]
Aug 28 11:39:36 charon: 09[NET] received packet: from 81.xxx.xxx.53[500] to 78.xxx.xxx254[500] (708 bytes)
Aug 28 11:39:33 charon: 09[NET] sending packet: from 78.xxx.xxx254[500] to 81.xxx.xxx.53[500] (140 bytes)
Aug 28 11:39:33 charon: 09[ENC] generating ID_PROT response 0 [ SA V V V ]
Aug 28 11:39:33 charon: 09[IKE] 81.xxx.xxx.53 is initiating a Main Mode IKE_SA
Aug 28 11:39:33 charon: 09[IKE] 81.xxx.xxx.53 is initiating a Main Mode IKE_SA
Aug 28 11:39:33 charon: 09[IKE] received DPD vendor ID
Aug 28 11:39:33 charon: 09[IKE] received Cisco Unity vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-00 vendor ID
Aug 28 11:39:33 charon: 09[ENC] received unknown vendor ID: 16:f6:ca:16:e4:a4:06:6d:83:82:1a:0f:0a:ea:a8:62
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-02 vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-04 vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-05 vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-06 vendor ID
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Aug 28 11:39:33 charon: 07[NET] sending packet: from 78.xxx.xxx.250[4500] to 81.xxx.xxx.53[4500] (140 bytes)
Aug 28 11:39:33 charon: 09[IKE] received draft-ietf-ipsec-nat-t-ike-08 vendor ID
Aug 28 11:39:33 charon: 07[ENC] generating INFORMATIONAL_V1 request 2426471521 [ HASH N(AUTH_FAILED) ]
Aug 28 11:39:33 charon: 09[IKE] received NAT-T (RFC 3947) vendor ID
Aug 28 11:39:33 charon: 07[IKE] no peer config found
Aug 28 11:39:33 charon: 09[ENC] parsed ID_PROT request 0 [ SA V V V V V V V V V V V V V ]
Aug 28 11:39:33 charon: 07[CFG] looking for pre-shared key peer configs matching 78.xxx.xxx250...81.xxx.xxx.53[remoteID@domain]
Aug 28 11:39:33 charon: 07[ENC] parsed ID_PROT request 0 [ ID HASH ]
Aug 28 11:39:33 charon: 09[NET] received packet: from 81.xxx.xxx.53[500] to 78.xxx.xxx.254[500] (348 bytes)
Aug 28 11:39:33 charon: 07[NET] received packet: from 81.xxx.xxx.53[4500] to 78.xxx.xxx.250[4500] (124 bytes)
Aug 28 11:39:30 charon: 07[NET] sending packet: from 78.xxx.xxx.254[500] to 81.xxx.xxx.53[500] (716 bytes)
Aug 28 11:39:30 charon: 07[ENC] generating ID_PROT response 0 [ KE No NAT-D NAT-D ]
Aug 28 11:39:30 charon: 07[ENC] parsed ID_PROT request 0 [ KE No NAT-D NAT-D ]
Aug 28 11:39:30 charon: 07[NET] received packet: from 81.xxx.xxx.53[500] to 78.xxx.xxx.254[500] (708 bytes)
Aug 28 11:39:27 charon: 12[NET] sending packet: from 78.xxx.xxx.254[500] to 81.xxx.xxx.53[500] (140 bytes)
Aug 28 11:39:27 charon: 12[ENC] generating ID_PROT response 0 [ SA V V V ]
Aug 28 11:39:27 charon: 12[IKE] 81.xxx.xxx.53 is initiating a Main Mode IKE_SA
Aug 28 11:39:27 charon: 12[IKE] 81.xxx.xxx.53 is initiating a Main Mode IKE_SA
Aug 28 11:39:27 charon: 12[IKE] received DPD vendor ID
Aug 28 11:39:27 charon: 12[IKE] received Cisco Unity vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-00 vendor ID
Aug 28 11:39:27 charon: 12[ENC] received unknown vendor ID: 16:f6:ca:16:e4:a4:06:6d:83:82:1a:0f:0a:ea:a8:62
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-02 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-04 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-05 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-06 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received draft-ietf-ipsec-nat-t-ike-08 vendor ID
Aug 28 11:39:27 charon: 12[IKE] received NAT-T (RFC 3947) vendor ID
Regarding the Mikrotik end I'll have to either setup a syslog server or stick a memory card in as I don't want to log to the internal flash (would be a lot of writes)

[dragon2611]

I think the problem happens if the Mikrotik tries to use NAT-T/Port 4500 during IKE as when I forced it onto 500udp it's  been behaving itself

I also have a static nat rule to ensure anything going out of the Opnsense firewalls gets natted to the VIP and doesn't come from the real IP but I'm not sure if that's needed.