OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: Andreas_ on September 01, 2020, 03:52:20 pm

Title: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Andreas_ on September 01, 2020, 03:52:20 pm
We have an opnsense installation (CARP pair), running on 20.1.3 until recently, with 4 ipsec peers (2xIKEv1, 2xIKEv2) and some 20 tunnels defined. This used to run flawlessly, until I upgraded those machines to 20.7.1. Since then, the tunnels will stop working after a while, until a new connect is forced on the tunnel. Strangely, all logging looks normal on both sides of the tunnel, even when the tunnel traffic has stalled (still IKE/ISAKMP traffic, but no more ESP packets)

The situation is a little different between peers, and sometimes there are stable phases for one peer, getting bad again after a while, but none is 100% fine. It will take some seconds to some minutes until the tunnels stall; more traffic seems to speed up the failure.

I reinstalled one firewall with 20.1, and now we have stable performance again. The backup machine is in maintenance mode and still 20.7.1 (with syslog-ng fixed).

When reviewing the updates that happened between 20.1.3 and 20.7.1, strongswan was upgraded from 4.8.2 to 4.8.4 (in April), and the kernel from 11.2 to 12.1. Since IKE/ISAKMP traffic seems normal, I'd suspect some issue in the kernel/pf, but I'm out of clues how to narrow down the reason further.

Any thoughts on this?
Regards,
Andreas
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 01, 2020, 04:22:29 pm
route-based or policy-based IPsec?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Andreas_ on September 01, 2020, 04:47:03 pm
All Tunnel IPV4 policy-based.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 01, 2020, 07:17:40 pm
This is strange, only opn to opn or any logs when it stops sending traffic?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Andreas_ on September 02, 2020, 09:45:05 am
No hint in the logs on either side of the tunnel (already elevated some log levels)
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 02, 2020, 10:49:27 am
And when you do a packet capture

a) do you see packets in ipsec / enc0 interface?
b) do you see encrypted ESP packets on WAN interface leaving?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Andreas_ on September 02, 2020, 11:47:12 am
I did a packet capture on WAN, no more ESP packets visible.
Didn't capture on other interfaces.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Andreas_ on September 04, 2020, 07:10:44 pm
I did an after-hour test, after upgrading the fw to 20.7.2.

The tunnel traffic still stalls after a while (it did so after about 100MB inbound traffic).

When pinging a remote host, I see ICMP on enc0 entering the tunnel, a corresponding outgoing ESP packet on wan, but no returning packet; there's still communication on port 500, with no anomalies (afaics) in the log.

Switching back to the downgraded fw, same config (synced from the 20.7 machine) works flawlessly.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 04, 2020, 07:18:46 pm
Can you tick Disable Mobike?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 08, 2020, 04:18:03 pm
I seem to be facing a similar issue. After upgrading from 20.1.4 to 20.7.2 IPsec phase 2 tunnels will randomly stall (IKEv1, mode tunnel IPv4). Only restarting strongswan seems to fix this issue (temporarely).

I should add that NOT ALL tunnels will stall AT ONCE. It seems to start with some tunnels, and other tunnels will follow after some time.

From my perspective the tunnels look perfectly fine:

- "ipsec statusall" shows all tunnels are in working condition and established
- tcpdump shows incoming/outgoing traffic with correct SPI IDs
- "setkey -DP" looks good too

I plan to test again with an older version of strongswan (5.8.3, which was included in 20.1.4). I also plan to test with strongswan 5.9. But this will take some time, because I need to build these packages for FreeBSD 12.1 first.

I've also came across this bug report:
https://wiki.strongswan.org/issues/2315
And I've wondered if the mentioned workaround that was introduced in strongswan 5.8.3 could be related.
EDIT: This was the version that was included in 20.1.4, so it is probably unrelated.

Quote
The tunnel traffic still stalls after a while (it did so after about 100MB inbound traffic).

Unfortunately, this is not the case here. I was able to transfer hundreds MB but it did not cause the traffic to stall.


Regards
- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 08, 2020, 04:19:43 pm
Quote from: fraenki on September 08, 2020, 04:18:03 pm

I plan to test again with an older version of strongswan (the one that was included in 20.1.4; need to find out the version number).

This wont work since ABI changed.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 08, 2020, 04:23:30 pm
Quote from: mimugmail on September 08, 2020, 04:19:43 pm
This wont work since ABI changed.

I will build the package for FreeBSD 12.1 manually :)

Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 08, 2020, 05:06:09 pm
After talking to mimugmail I've switched the IPsec tunnel mode from IKEv1 to IKEv2. Let's see if this changes anything.

EDIT: Even with IKEv2 the traffic stalled, but this time strongswan recognized the error and restarted the tunnel connection:

Code: [Select]
Sep  8 17:03:05 charon[62985]: 05[IKE] <con5|1> giving up after 10 path probings
Sep  8 17:03:05 charon[62985]: 05[IKE] <con5|1> restarting CHILD_SA con5

EDIT 2: With IKEv2 the issue is better reproducable to me and now I can confirm that the traffic stalls after transferring ~200-400 MB.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 08, 2020, 05:36:34 pm
Quote
I plan to test again with an older version of strongswan (5.8.3, which was included in 20.1.4). I also plan to test with strongswan 5.9. But this will take some time, because I need to build these packages for FreeBSD 12.1 first.

I've manually built packages for strongswan 5.8.3 and 5.9.0 on FreeBSD and tested them on OPNsense 20.7.2. Unfortunately this did not improve the situation. It's probably a change in FreeBSD 12.x that causes this issue.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 09, 2020, 09:35:12 am
I need some screenshots of Phase1 and Phase2.
My first test with IKEv2 I had over night 57075 pings transmitted and only 32 lost.

I'll now switch to IKEv1
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 09, 2020, 09:56:33 am
Quote from: mimugmail on September 09, 2020, 09:35:12 am
I need some screenshots of Phase1 and Phase2.

Sure, but nothing special there, I guess:
(I've removed sensitive data like remote host, psk, identifiers.)
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 09, 2020, 11:45:49 am
@Andreas_: Could you please provide screenshots of your configuration too? Maybe we find similarities...

@mimugmai: Maybe you could also share your test configuration...
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 09, 2020, 12:11:04 pm
I flipped from v2 to v1 and after 16k pings only 12 losses.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 09, 2020, 12:11:55 pm
I'm using the same values as you, but no DPD and no auto-ping.
Can you try to adjust as mine ?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 09, 2020, 03:13:41 pm
15 losses in 20k pings .. I'll now enable DPD
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 09, 2020, 03:58:13 pm
I tend to assume that this is not really a IPsec bug. It could be related to the hardware platform. My firewalls are running on A2SDi-2C-HLN4F with a Intel C3338 CPU and Intel C3000 NIC chip. This NIC chip is driven by the ix driver on FreeBSD.

@Andreas_ Which hardware are you using, especially the NIC chip/driver? And could you share a screenshot of the Interfaces -> Settings page?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 09, 2020, 10:40:52 pm
Quote from: fraenki on September 09, 2020, 03:58:13 pm
I tend to assume that this is not really a IPsec bug. It could be related to the hardware platform. My firewalls are running on A2SDi-2C-HLN4F with a Intel C3338 CPU and Intel C3000 NIC chip. This NIC chip is driven by the ix driver on FreeBSD.

I've tested again with most defensive interface settings, unfortunately no change:

Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on September 10, 2020, 09:33:53 am
Doesn't seem to be generic indeed.

On my side it's a virtualbox VM and an openstack VM on the other side. I put 1GB over the tunnel and it's still up with your settings.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Cerberus on September 18, 2020, 05:19:32 pm
I had massive IPSEC problems after updating to 20.7. My current ipsec connections:

kvm opnsense  <ikev2 or ikev1> kvm opnsense: connection is dead after few minutes, no traffic or error, ipsec still shown as "up"
kvm opnsense <ikev1> sophos utm on hw: Works most times, sometimes down and have to be restarted, dpd not working
kvm opnsense <ikev2> azure gateway: opnsense connection is dead after few minutes, no traffic or error, ipsec still shown as "up"

For the first and third scenario i switched to Zerotier for now and connection works fine, slower than ipsec but acceptable for my use case.

I had plans to replace the Sophos UTM with OPNsense, but this issue is a blocker for me.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on September 26, 2020, 11:10:14 pm
Quote from: fraenki on September 09, 2020, 03:58:13 pm
I tend to assume that this is not really a IPsec bug. It could be related to the hardware platform. My firewalls are running on A2SDi-2C-HLN4F with a Intel C3338 CPU and Intel C3000 NIC chip. This NIC chip is driven by the ix driver on FreeBSD.

@Andreas_ Which hardware are you using, especially the NIC chip/driver? And could you share a screenshot of the Interfaces -> Settings page?

I am running OPNsense on comparable hardware (A2SDi-4C-HLN4F).

I've updated to OPNsense 20.7.3 from 20.1, today. My first IPsec tests do not show any incorrect behaviour. I've pushed nearly 8 GB through the IPsec tunnel in each direction.

Let's see how the results of a long term test will look like.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 28, 2020, 02:46:06 pm
Quote from: Cerberus on September 18, 2020, 05:19:32 pm
I had massive IPSEC problems after updating to 20.7.

As reported by mimugmail this cannot be reproduced, so we should focus on making this reproducable. For this I ask you to provide the following information:

* configuration of Phase 1 and Phase 2
* your configuration in "Interfaces: Settings"
* your configuration in "Firewall: Settings: Advanced"
* your configuration in "System: Settings: Tunables"

Thanks!

- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on September 28, 2020, 02:49:01 pm
Quote from: glasi on September 26, 2020, 11:10:14 pm
I am running OPNsense on comparable hardware (A2SDi-4C-HLN4F).
I've updated to OPNsense 20.7.3 from 20.1, today. My first IPsec tests do not show any incorrect behaviour. I've pushed nearly 8 GB through the IPsec tunnel in each direction.

Thanks for smashing my theory. ;-) (In all seriousness, I appreciate your feedback!)
For comparison sake, would you mind to provide the data I've requested in my previous post?

Thanks!

- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Ricardo on September 29, 2020, 08:46:41 pm
As I am also planning to build some new FW with ipsec ESP,  I am eagerly waiting if this will be a blocker for me as well.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on September 29, 2020, 09:08:29 pm
IPsec settings...
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on September 29, 2020, 09:10:39 pm
Interfaces: Settings
Firewall: Settings: Advanced
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on September 29, 2020, 09:11:11 pm
System: Settings: Tunables
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on October 07, 2020, 12:50:57 pm
Quote from: glasi on September 29, 2020, 09:08:29 pm
IPsec settings...

Thanks for sharing your settings. Another mix of different settings, nothing that looks suspicious to me.
The only thing that we have in common are Intel NICs. Does it make any change if you disable "VLAN Hardware Filtering" (and reboot)?

It would be interesting to know the NIC brand/chipset of the other users...


Regards
- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on October 11, 2020, 12:31:15 am
Doesn't make any difference on my system.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on October 13, 2020, 10:18:27 am
Hello,

that is my first post at this forum and i found this topic while searching to the described problem. I don't know your short hardware description, but i run opnsense at Intel hw (scope7 1510*). If i can do something to help just tell me. I am not familiar with opnsense yet. I just started replacing one of our bintec routers.

Regards, Proctor


* https://www.landitec.com/products/open-source-appliance-solutions/scope7-open-source-appliances/scope7-1510-detail/
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on October 14, 2020, 11:20:25 pm
Enough people have reported this problem, so I've created a bug report:
https://github.com/opnsense/core/issues/4415

If you are able to contribute substantial information, please add it to the bug report. But all other discussions and comments should continue here in the forums :)
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on October 15, 2020, 06:20:59 am
Quote from: proctor on October 13, 2020, 10:18:27 am
Hello,

that is my first post at this forum and i found this topic while searching to the described problem. I don't know your short hardware description, but i run opnsense at Intel hw (scope7 1510*). If i can do something to help just tell me. I am not familiar with opnsense yet. I just started replacing one of our bintec routers.

Regards, Proctor


* https://www.landitec.com/products/open-source-appliance-solutions/scope7-open-source-appliances/scope7-1510-detail/

Can you give a more detailed explanation? This error depends on 20.7 to 20.7 on Hardware only. You talk about replacing one bintec
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on October 15, 2020, 12:27:36 pm
Hello,

I used bintec routers for about 20 years, but the models i need are out of stock. So i started to try/use opnsense as replacement this year. - Therefore no much experience in opnsense.

I set up an opnsense box with version 20.1. x at july and upgraded to version 20.7.x  about 2 weeks ago. After that i had to struggle with broken tunnels.


Configuration attached.

Regards,
Proctor
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on October 15, 2020, 04:02:40 pm
Can you open a new thread, it's different than this one
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on October 19, 2020, 01:04:12 pm
Quote from: proctor on October 13, 2020, 10:18:27 am
Hello,

that is my first post at this forum and i found this topic while searching to the described problem. I don't know your short hardware description, but i run opnsense at Intel hw (scope7 1510*). If i can do something to help just tell me. I am not familiar with opnsense yet. I just started replacing one of our bintec routers.

Regards, Proctor


* https://www.landitec.com/products/open-source-appliance-solutions/scope7-open-source-appliances/scope7-1510-detail/

The guys from Landitec sent me a 1510 and I added it to my VPN mesh, pushed 8GB without a problem.
Is there a chance to add your live system to my lab? It would just be ping from LAN IP to LAN IP for testing.

https://github.com/opnsense/core/issues/4415#issuecomment-712056820
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on October 21, 2020, 08:57:36 am
I downgraded the live system a couple of days ago (no issues since). I am going to setup an additional device for testing / reproducing the issue for myself. So maybe you could use this one too (should be ready not later than tomorrow).
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on October 23, 2020, 09:05:58 am
I think, I could reproduce the issue in my test setup with two OPNsense devices (version 20.7.3). The culprit seems to me a vlan at the lan interface. Without a vlan I could not reproduce the stalling. @mimugmail - if helpful I could give you access to my testing environment (or connect it to yours).
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on October 23, 2020, 09:15:35 am
Just as I wrote the last reply the tunnel stalled. So please forget the vlan part. But I still have a setup to reproduce at least my issue.

- tunnel seems ok in status overview
- gateway monitoring shows offline
- no traffic (ping) through the tunnel
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on October 23, 2020, 11:19:59 am
Please DM me your phone number ..
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on October 23, 2020, 12:18:56 pm
Quote from: proctor on October 23, 2020, 09:05:58 am
Without a vlan I could not reproduce the stalling.

Oh, that's interesting.
@mimugmail, are you using VLANs in your test setup?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on October 29, 2020, 11:50:47 am
Quote from: proctor on October 23, 2020, 09:15:35 am
So please forget the vlan part.

No problems with vlan in my setup...

Code: [Select]
                                                                        LACP
+----------------------+    IPsec          +----------------------+     Trunk         +----------------------+
| OPNsense 20.7.4      |    Tunnel         | OPNsense 20.7.4      |     VLAN          | Cisco SG250-18       |
| Intel Atom C3558     |-------------------| Intel Atom C3558     |-------------------| Switch               |
| 8 GB RAM             |    IPv4           | 8 GB RAM             |     2x 1 Gb/s     |                      |
+----------------------+    policy based   +----------------------+                   +----------------------+
                                                                                           /          \
                                                                                          /            \
                                                                                  1 Gb/s /              \ 1 Gb/s
                                                                                        /                \
                                                                                       /                  \
                                                                     +----------------------+      +----------------------+
                                                                     | File server          |      | Client               |
                                                                     | VLAN 10              |      | VLAN 70              |
                                                                     |                      |      |                      |
                                                                     +----------------------+      +----------------------+

Even VLAN on the WAN connection does not seem to be a deal breaker. My provider requires VLAN 7 on the link interface for the WAN PPPoE connection.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on November 03, 2020, 09:33:09 am
I use route-based IPsec in my setup, but maybe the issue could also depend on the update too. We tested two devices, one with 20.1.9 (A) and one with 20.7.3 (B). Each device was initially configured with that version. For verification we used a third non OPNsense based device (C).

For testing we used 2 tunnels:
A --> B
A --> C

The setup ran with permanent ping from B|C --> A a couple of days with no issue. After updating A to 20.7.4 both tunnels are showing the issue. Roughly every 2 hours the tunnels start to run fine (that is the lifetime for phase 2) for some time (see attached scrennshot).
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on December 01, 2020, 01:20:27 pm
Some additional information to this issue.

Now I have a simple working configuration with version 20.7.3 (without update and without configuration import), where it seems to be possible to reproduce at least a similar kind of issue.

IPsec tunnel with route-based ESP (no gateway defined for the tunnel-ip) - ping runs over 48 h with 0,02% loss. After defining a gateway (far gateway with gateway monitoring) it took less then an hour for the first break. After 4 h i have 2% of lost pings.

Any ideas are welcome.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: marcquark on December 02, 2020, 09:45:22 am
started seeing this yesterday too, ran fine for weeks before that. is it worth testing this on stock FreeBSD/HBSD at this point, possibly on multiple versions?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Cerberus on December 02, 2020, 12:09:27 pm
I had this last week two times on one of my OPNsense installs (Fresh 20.7). IPv6 only IKEv1 between Sophos UTM and OPNSense. Phase 1 and 2 was still active but no traffic. After restarting IPSEC service on OPNsense the traffic started flowing again.

Next time that thing stall, i try to get as much information as possible out of that thing.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: mimugmail on December 02, 2020, 03:50:26 pm
Please don't mix your problems with the one fraenki has posted.

Fraenki's problem only happens when running on 20.7 and only after upgrade. If reverted to 20.1 it works again.

If you (both) encountered a timeout, stall, whatevery, please open a new thread and post as many details as you have and dont hang on this one (only if you can reproduce that it works perfect with 20.1).
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on December 13, 2020, 09:45:04 pm
If anyone is still affected by IPsec instability, please test the following:

Change the following setting...
System: Settings: Miscellaneous -> Hardware acceleration
...from "AES-NI CPU-based" to "none" and save the change. Be sure to reboot the firewall afterwards.

Please report back.


Thanks
- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: juan.syad on December 20, 2020, 09:06:35 pm
Quote from: fraenki on December 13, 2020, 09:45:04 pm
If anyone is still affected by IPsec instability, please test the following:

Change the following setting...
System: Settings: Miscellaneous -> Hardware acceleration
...from "AES-NI CPU-based" to "none" and save the change. Be sure to reboot the firewall afterwards.

Please report back.


Thanks
- Frank

Thanks alot Frank, that did the trick and the tunnel is finally stable again.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: tomiboy on January 31, 2021, 02:17:11 pm
Quote from: juan.syad on December 20, 2020, 09:06:35 pm
Quote from: fraenki on December 13, 2020, 09:45:04 pm
If anyone is still affected by IPsec instability, please test the following:
Please report back.

Hello Frank, we have the exact same problem with a new installed 21.1. Disable hardware acceleration doesnt help us. We tried to run the vm with e1000 card instead a vmxnet3 Vmware card, nothing helps.

The setup works properly with EAP-Radius and W10 ikeV2 Clients, but after transmitting 200 - 250 Mbyte Data the Tunnel stalled.

Any Ideas?

Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Ricardo on February 07, 2021, 01:50:39 pm
Quote from: fraenki on December 13, 2020, 09:45:04 pm
If anyone is still affected by IPsec instability, please test the following:

Change the following setting...
System: Settings: Miscellaneous -> Hardware acceleration
...from "AES-NI CPU-based" to "none" and save the change. Be sure to reboot the firewall afterwards.

Please report back.


Thanks
- Frank

Hi Frank,

what did you make believe this was the fault of the AESNI acceleration?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on February 07, 2021, 02:02:20 pm
Quote from: Ricardo on February 07, 2021, 01:50:39 pm
what did you make believe this was the fault of the AESNI acceleration?

Extensive testing. It fixes the issue for me, it's 100% reproducable.
If it does not fix the issue for you, then you're likely affected by a different issue.


Regards
- Frank
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: tomiboy on February 07, 2021, 07:30:09 pm
Problem could be fixed! The fault was the activation of PFS. The Windows 10 client does not receive this setting, if not appropriately set via Powershell. This then led to exactly this error pattern.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Ricardo on February 07, 2021, 10:19:16 pm
Quote from: tomiboy on February 07, 2021, 07:30:09 pm
Problem could be fixed! The fault was the activation of PFS. The Windows 10 client does not receive this setting, if not appropriately set via Powershell. This then led to exactly this error pattern.

Can you share some details how you figured this out, and what was the resolution?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: tomiboy on February 07, 2021, 10:35:46 pm
The Windows 10 IPSec client has not activated PFS by default.

I had activated PFS under "VPN: IPsec: Mobile Clients -> Phase 2 PFS Group". Windows 10 silently establishes a connection without errors. The connection dies after approx. 200-300 MB of data has been transferred.

To solve this, the connection must be created via Powershell and, for example, the correct PFS parameters must be transferred. This is not possible in the GUI.

PS C:\> Add-VpnConnection -Name "Contoso" -ServerAddress 176.16.1.2 -TunnelType "Ikev2"
PS C:\> Set-VpnConnectionIPsecConfiguration -ConnectionName "Contoso" -AuthenticationTransformConstants None -CipherTransformConstants AES256 -EncryptionMethod AES256 -IntegrityCheckMethod SHA384 -PfsGroup ECP384 -DHGroup ECP384 -PassThru -Force
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: atoll on February 15, 2021, 10:27:37 pm
Quote from: fraenki on December 13, 2020, 09:45:04 pm
If anyone is still affected by IPsec instability, please test the following:

Change the following setting...
System: Settings: Miscellaneous -> Hardware acceleration
...from "AES-NI CPU-based" to "none" and save the change. Be sure to reboot the firewall afterwards.

Please report back.


Thanks
- Frank

Hi Frank,

disabling AES-NI worked for me, too.

IPsecv2 EAP-MS-Chapv2, Scope7 1510 Fiber, OPNsense 21.1

Just one little problem remains: With hardware acceleration, the VPN gives me about 500 Mbit/s. (for about 3,5 seconds, the the packets stop flowing, measured locally via iperf3)

Without, its about 60Mbit/s.

In a production environment, that's a serious problem.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Cerberus on March 14, 2021, 09:26:28 pm
I found something in the pfsense forums about issues with aes-ni and sha256 hw acceleration, their workaround for now is using qat (which opnsense dont have and requires certain hardware), disable aes-ni, not using sha-256 hash or switch to aes-gcm without the need for a hash. Any of the last three solutions help solving the issue for me.
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: proctor on March 15, 2021, 11:39:46 am
Thanks for the hint, I will give it a try (just changed and rebooted).
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: glasi on May 01, 2021, 12:33:43 pm
Sill no problems on my end with AES-NI and SHA256.

Have you ever tried AES-GCM instead of AES?
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: franco on May 02, 2021, 10:09:11 pm
Might be https://cgit.freebsd.org/src/commit/?id=62e32cf9140e6c13663dcd69ec3b3c7ca4579782 just a couple of days old.


Cheers,
Franco
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: Gilad on May 06, 2021, 12:25:29 am
Hi, I have a similar problem with OPNsense 21.1.5 running on DEC850 (AMD EPYC 3201). IPsec VPN "Road Warrior" to an iOS device, with the following settings: AES-256, SHA256, DH-14 and ESP.

I can connect successfully, and the VPN tunnel works for 10-20 seconds, but then just dies. I've tried different combinations of encryption and hash, with the same results.

Is the only option currently to disable the AES-NI accelaration?

Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: jfranken on May 11, 2021, 12:02:24 pm
On our OPNsense 21.1.4/DEC3850 we were experiencing several hanging ipsec ikev2 associations per day until I disabled aesni.

Four weeks ago, I changed the phase 1 and 2 algorithms from CBC (aes256-sha256-modp2048!) to GCM (aes256gcm16-sha256-modp2048!) and re-enabled aesni. Since then, not a single hitch, same with 21.1.5.

Check
grep -e " ike =" -e " esp =" /usr/local/etc/ipsec.conf
to test if you got them all.

Regards
Johannes Franken
Title: Re: IPSEC traffic stalling after 20.7.1 upgrade
Post by: fraenki on May 27, 2021, 12:22:50 pm
This issue will be fixed in today's release of OPNsense 21.1.6 (about to be released in the upcoming hours).

If it does not solve your issue, then you're most likely experiencing a different issue.
In that case I'd suggest to report a new issue on GitHub.


Regards
- Frank