IPsec IKEv1 Phase 2 rejected by FortiGate (PFS enabled) - Quick Mode sent withou

Started by garroz, June 15, 2026, 04:52:56 PM

Previous topic - Next topic
Print
Go Down Pages1 2
User actions
June 15, 2026, 04:52:56 PM
Hi everyone,
I'm trying to establish a site-to-site IPsec VPN between OPNsense 26.1.9 and a FortiGate.
The IKE Phase 1 comes up successfully, but Phase 2 is always rejected by the FortiGate.
FortiGate configuration
Phase 1

Code Select
IKE Version: IKEv1
Mode: Main
Authentication: PSK
Proposal: AES128-SHA256
DH Group: 5 (MODP1536)
Lifetime: 86400
NAT-T: enabled

Phase 2

Code Select
PFS: enabled
DH Groups: 14,5
Proposals:
- AES128-SHA1
- AES256-SHA1
- AES128-SHA256
- AES256-SHA256
- AES128-GCM
- AES256-GCM
Lifetime: 43200

Traffic selectors:

Code Select
Local:  10.202.159.192/26
Remote: 10.200.0.0/14

There is also a second Phase2 on the FortiGate:

Code Select
Local:  10.202.159.192/26
Remote: 10.224.0.0/11

OPNsense configuration
Phase 1

Code Select
IKEv1
Main Mode
PSK
AES128-SHA256
DH Group 5

Phase 2

Code Select
ESP: AES128-SHA256
PFS Group: 5 (also tested with 14)
Local TS: 10.202.159.192/26
Remote TS: 10.200.0.0/14

Generated swanctl.conf:

Code Select
children {
    fae426f8-xxxx {
        esp_proposals = aes128-sha256-modp1536
        local_ts = 10.202.159.192/26
        remote_ts = 10.200.0.0/14
    }
}

What works
IKE Phase 1 is established successfully.
StrongSwan log:

Code Select
selected proposal:
IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1536
IKE_SA established

Problem
Immediately after IKE_SA establishment, OPNsense sends:

Code Select
generating QUICK_MODE request [ HASH SA No KE ID ID ]

The FortiGate immediately responds with:

Code Select
received (24576) notify

and rejects Phase 2.
Full log excerpt:

Code Select
IKE_SA established
generating QUICK_MODE request [ HASH SA No KE ID ID ]
received (24576) notify

Packet capture
The Quick Mode exchange is visible:

Code Select
phase 2/others I oakley-quick[E]

followed immediately by an encrypted Informational packet from the FortiGate.
Question
Even though OPNsense generates:

Code Select
esp_proposals = aes128-sha256-modp1536

the Quick Mode log still shows:

Code Select
[ HASH SA No KE ID ID ]

which appears to indicate that no PFS/DH exchange is being included in Phase 2.
Is this expected behavior in StrongSwan/OPNsense for IKEv1?
Could this indicate that the Phase2 PFS group is not actually being applied, despite the generated proposal containing
Code Select
modp1536?
Has anyone seen FortiGate reject Quick Mode with Fortinet vendor notify
Code Select
(24576) under similar circumstances?
Any suggestions on additional debugging steps would be appreciated.
Thanks!
June 15, 2026, 04:54:55 PM #1
Why not IKEv2?


Cheers,
Franco
June 15, 2026, 05:08:15 PM #2
Ciao Franco,
the client on his fortinet has these parameters and I have to respect them

June 15, 2026, 05:28:00 PM #3
Been a long time since I had a Fortinet, but your settings look ok when I compare to some ancient configuration I still have in my backups.  It has to be some other mismatch with the traffic selector or behaviour setting.


Cheers,
Franco
June 15, 2026, 05:45:31 PM #4
The FortiGate has two Phase2 definitions under the same Phase1:

10.202.159.192/26 <-> 10.200.0.0/14
10.202.159.192/26 <-> 10.224.0.0/11

OPNsense currently has only the first Child SA configured.

Could FortiGate reject Quick Mode if it expects both selectors to be configured on the peer?

Also, OPNsense generates start_action=start while FortiGate has auto-negotiate disable.
June 15, 2026, 05:50:37 PM #5
Strongswan logs if the traffic selectors are unacceptable.

For IPsec in general its a good idea if both peers are configured exactly the same, especially for IKEv1 which is pain.

Create both children.
Hardware:
DEC740
June 15, 2026, 06:08:58 PM #6
But as far as I remember you need separate phase 1 for the Fortigate so it should accept the first one already.


Cheers,
Franco
June 16, 2026, 09:05:55 AM #7
Do you think this is correct?
(attachments)
June 16, 2026, 09:13:25 AM #8
Better create two separate children so they are two separate SAs.

Child1:
10.202.159.192/26 <-> 10.200.0.0/14

Child2:
10.202.159.192/26 <-> 10.224.0.0/11
Hardware:
DEC740
June 16, 2026, 09:48:19 AM #9
And again: separate phase 1 too ;)
June 16, 2026, 10:17:52 AM #10
I separated the phase2:
Code Select
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="127"] 00[DMN] Starting IKE charon daemon (strongSwan 6.0.7, FreeBSD 14.3-RELEASE-p15, amd64)
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="128"] 00[LIB] providers loaded by OpenSSL: default legacy
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="129"] 00[CFG] using '/sbin/resolvconf' to install DNS servers
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="130"] 00[CFG] loaded 0 RADIUS server configurations
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="131"] 00[LIB] loaded plugins: charon blowfish md4 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pgp dnskey sshkey pem openssl pkcs8 xcbc cmac kdf gcm ml drbg curl attr kernel-pfkey kernel-pfroute resolve socket-default vici updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam whitelist addrblock counters
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="132"] 00[JOB] spawning 16 worker threads
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="133"] 13[CFG] loaded certificate 'CN=internal-ca, C=IT,
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="134"] 13[CFG] loaded certificate 'CN=internal-ca, C=IT,
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="135"] 13[CFG] loaded IKE shared key with id 'ike-4355b512-aaf6-48d9-a7cb-966dc1585874' for: 'LOCAL IP', 'REMOTE IP'
<30>1 2026-06-16T09:55:24+02:00 FwSede3.local charon 54247 - [meta sequenceId="136"] 11[CFG] added vici connection: 72a30ffe-dc67-45d4-a40f-e0b13fa528cb
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="137"] 12[NET] <1> received packet: from REMOTE IP[500] to LOCAL IP[500] (292 bytes)
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="138"] 12[ENC] <1> parsed ID_PROT request 0 [ SA V V V V V V V V V V ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="139"] 12[IKE] <1> received NAT-T (RFC 3947) vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="140"] 12[IKE] <1> received draft-ietf-ipsec-nat-t-ike-03 vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="141"] 12[IKE] <1> received draft-ietf-ipsec-nat-t-ike-02 vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="142"] 12[IKE] <1> received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="143"] 12[ENC] <1> received unknown vendor ID: 16:f6:ca:16:e4:a4:06:6d:83:82:1a:0f:0a:ea:a8:62
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="144"] 12[IKE] <1> received draft-ietf-ipsec-nat-t-ike-00 vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="145"] 12[IKE] <1> received DPD vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="146"] 12[IKE] <1> received FRAGMENTATION vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="147"] 12[IKE] <1> received FRAGMENTATION vendor ID
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="148"] 12[ENC] <1> received unknown vendor ID: 82:99:03:17:57:a3:60:82:c6:a6:21:de:00:00:00:00
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="149"] 12[IKE] <1> REMOTE IP is initiating a Main Mode IKE_SA
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="150"] 12[CFG] <1> selected proposal: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_1536
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="151"] 12[ENC] <1> generating ID_PROT response 0 [ SA V V V V ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="152"] 12[NET] <1> sending packet: from LOCAL IP[500] to REMOTE IP[500] (164 bytes)
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="153"] 12[NET] <1> received packet: from REMOTE IP[500] to LOCAL IP[500] (316 bytes)
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="154"] 12[ENC] <1> parsed ID_PROT request 0 [ KE No NAT-D NAT-D ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="155"] 12[ENC] <1> generating ID_PROT response 0 [ KE No NAT-D NAT-D ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="156"] 12[NET] <1> sending packet: from LOCAL IP[500] to REMOTE IP[500] (332 bytes)
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="157"] 12[NET] <1> received packet: from REMOTE IP[500] to LOCAL IP[500] (108 bytes)
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="158"] 12[ENC] <1> parsed ID_PROT request 0 [ ID HASH N(INITIAL_CONTACT) ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="159"] 12[CFG] <1> looking for pre-shared key peer configs matching LOCAL IP...REMOTE IP[REMOTE IP]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="160"] 12[CFG] <1> selected peer config "72a30ffe-dc67-45d4-a40f-e0b13fa528cb"
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="161"] 12[IKE] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> IKE_SA 72a30ffe-dc67-45d4-a40f-e0b13fa528cb[1] established between LOCAL IP[LOCAL IP]...REMOTE IP[REMOTE IP]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="162"] 12[IKE] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> scheduling rekeying in 41980s
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="163"] 12[IKE] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> scheduling reauthentication in 78665s
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="164"] 12[IKE] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> maximum IKE_SA lifetime 50620s
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="165"] 12[ENC] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> generating ID_PROT response 0 [ ID HASH ]
<30>1 2026-06-16T09:55:27+02:00 FwSede3.local charon 54247 - [meta sequenceId="166"] 12[NET] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> sending packet: from LOCAL IP[500] to REMOTE IP[500] (92 bytes)
<30>1 2026-06-16T09:55:32+02:00 FwSede3.local charon 54247 - [meta sequenceId="167"] 12[NET] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> received packet: from REMOTE IP[500] to LOCAL IP[500] (108 bytes)
<30>1 2026-06-16T09:55:32+02:00 FwSede3.local charon 54247 - [meta sequenceId="168"] 12[ENC] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> parsed INFORMATIONAL_V1 request 1034562287 [ HASH N(DPD) ]
<30>1 2026-06-16T09:55:32+02:00 FwSede3.local charon 54247 - [meta sequenceId="169"] 12[ENC] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> generating INFORMATIONAL_V1 request 4249451368 [ HASH N(DPD_ACK) ]
<30>1 2026-06-16T09:55:32+02:00 FwSede3.local charon 54247 - [meta sequenceId="170"] 12[NET] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> sending packet: from LOCAL IP[500] to REMOTE IP[500] (108 bytes)
<30>1 2026-06-16T09:55:42+02:00 FwSede3.local charon 54247 - [meta sequenceId="171"] 12[NET] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> received packet: from REMOTE IP[500] to LOCAL IP[500] (108 bytes)
<30>1 2026-06-16T09:55:42+02:00 FwSede3.local charon 54247 - [meta sequenceId="172"] 12[ENC] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> parsed INFORMATIONAL_V1 request 313032658 [ HASH N(DPD) ]
<30>1 2026-06-16T09:55:42+02:00 FwSede3.local charon 54247 - [meta sequenceId="173"] 12[ENC] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> generating INFORMATIONAL_V1 request 2708051821 [ HASH N(DPD_ACK) ]
<30>1 2026-06-16T09:55:42+02:00 FwSede3.local charon 54247 - [meta sequenceId="174"] 12[NET] <72a30ffe-dc67-45d4-a40f-e0b13fa528cb|1> sending packet: from LOCAL IP[500] to REMOTE IP[500] (108 bytes)

What do you mean by separating phase1? How should I do it?
June 16, 2026, 10:27:49 AM #11
clone phase 1, set second phase 2 there.

As far as I remember the Fortinet will only connect you to one phase 2 per phase 1.


Cheers,
Franco
June 16, 2026, 11:03:49 AM #12
I cloned the phase 1s, but only one of them came up.
Not even chatgpt was able to advise me.
I don't want to set up a pfsense just for this purpose, but I think it's the only way :(
June 16, 2026, 11:39:05 AM #13
The thing is that StrongSwan is not different either way.

What's the issue on the second one now? And if you reverse the traffic selectors on the phase 2 is the first one still working or does it switch to the second one as working (indicating the second one has a wrong config)?


Cheers,
Franco
June 16, 2026, 12:02:11 PM #14
During Phase 2, OPNsense sends a valid QUICK_MODE request:

Code Select
generating QUICK_MODE request [ HASH SA No KE ID ID ]

but the FortiGate never responds with a QUICK_MODE reply, NO_PROPOSAL_CHOSEN, or INVALID_ID_INFORMATION.
Instead, it only sends DPD packets and a vendor-specific notify (24576), while OPNsense retransmits the QUICK_MODE request until timeout.
The latest logs suggest that:
  • Phase 1 is fully functional
  • Connectivity between peers is fine
  • The failure occurs only during Phase 2 negotiation
Could this indicate a Traffic Selector / Proxy-ID mismatch or a FortiGate-specific Phase 2 policy issue? 
Print
Go Up Pages1 2
User actions