Strange IPSEC VTI (ROUTED MODE) performance

[danderson]

So I have a S2S VTI vpn, 200M down / 10 up connection.

UDP I get full speed both directions, TCP i get full 10 up, but like 1M down.

I have tried with my shaping and queues on and off, no difference. Dont know when the issue started, but it was not like this prior to 21.7.4.   Currently on 21.7.6

IKEV2/IPSEC encryption set to
Phase 1 - 256 bit AES-GCM with 128 bit ICV + SHA512 + DH Group 21
Phase 2 - aes256gcm16 + + 21 (NIST EC 521 bits)

I am using / testing RSS, and have tried with it on and off. my nic is type is intel EM0 (Intel 82583V)


-----

root@OXNUNIFI001:~# iperf3 -c 192.168.1.245 -i 1 -t 30 -V -b 200M -u
iperf 3.6
Linux OXNUNIFI001 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64
Control connection MSS 1348
Setting UDP block size to 1348
Time: Fri, 03 Dec 2021 21:18:46 GMT
Connecting to host 192.168.1.245, port 5201
      Cookie: np7wpu6zehkvi7znfaavwe72sipwwpq3nflm
[  5] local 10.80.203.53 port 54837 connected to 192.168.1.245 port 5201
Starting Test: protocol: UDP, 1 streams, 1348 byte blocks, omitting 0 seconds, 30 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  23.8 MBytes   200 Mbits/sec  18533
[  5]   1.00-2.00   sec  23.8 MBytes   200 Mbits/sec  18545
[  5]   2.00-3.00   sec  23.8 MBytes   200 Mbits/sec  18546
[  5]   3.00-4.00   sec  23.8 MBytes   200 Mbits/sec  18545
[  5]   4.00-5.00   sec  23.8 MBytes   200 Mbits/sec  18547
[  5]   5.00-6.00   sec  23.8 MBytes   200 Mbits/sec  18546
[  5]   6.00-7.00   sec  23.8 MBytes   200 Mbits/sec  18546
[  5]   7.00-8.00   sec  23.8 MBytes   200 Mbits/sec  18546
[  5]   8.00-9.00   sec  23.8 MBytes   200 Mbits/sec  18546
[  5]   9.00-10.00  sec  23.8 MBytes   200 Mbits/sec  18545
[  5]  10.00-11.00  sec  23.8 MBytes   200 Mbits/sec  18547
[  5]  11.00-12.00  sec  23.8 MBytes   200 Mbits/sec  18545
[  5]  12.00-13.00  sec  23.8 MBytes   200 Mbits/sec  18547
[  5]  13.00-14.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  14.00-15.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  15.00-16.00  sec  23.8 MBytes   200 Mbits/sec  18545
[  5]  16.00-17.00  sec  23.8 MBytes   200 Mbits/sec  18547
[  5]  17.00-18.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  18.00-19.00  sec  23.8 MBytes   200 Mbits/sec  18545
[  5]  19.00-20.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  20.00-21.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  21.00-22.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  22.00-23.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  23.00-24.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  24.00-25.00  sec  23.8 MBytes   200 Mbits/sec  18547
[  5]  25.00-26.00  sec  23.8 MBytes   200 Mbits/sec  18545
[  5]  26.00-27.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  27.00-28.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  28.00-29.00  sec  23.8 MBytes   200 Mbits/sec  18546
[  5]  29.00-30.00  sec  23.8 MBytes   200 Mbits/sec  18547
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.00  sec   715 MBytes   200 Mbits/sec  0.000 ms  0/556366 (0%)  sender
[  5]   0.00-30.00  sec   669 MBytes   187 Mbits/sec  0.070 ms  36439/556361 (6.5%)  receiver
CPU Utilization: local/sender 14.1% (3.2%u/10.9%s), remote/receiver 18.2% (2.5%u/15.7%s)

iperf Done.



root@OXNUNIFI001:~# iperf3 -c 192.168.1.245 -i 1 -t 30 -V -b 200M
iperf 3.6
Linux OXNUNIFI001 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64
Control connection MSS 1348
Time: Fri, 03 Dec 2021 21:19:30 GMT
Connecting to host 192.168.1.245, port 5201
      Cookie: fortwut5v3yy4xzlwgovxzwaz274xvzoybvj
      TCP MSS: 1348 (default)
[  5] local 10.80.203.53 port 50624 connected to 192.168.1.245 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 30 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   149 KBytes  1.22 Mbits/sec    2   10.5 KBytes
[  5]   1.00-2.00   sec  48.7 KBytes   399 Kbits/sec    4   3.95 KBytes
[  5]   2.00-3.00   sec  39.5 KBytes   324 Kbits/sec    2   5.27 KBytes
[  5]   3.00-4.00   sec  38.2 KBytes   313 Kbits/sec    1   5.27 KBytes
[  5]   4.00-5.00   sec  40.8 KBytes   334 Kbits/sec    2   5.27 KBytes
[  5]   5.00-6.00   sec  46.1 KBytes   377 Kbits/sec    3   1.32 KBytes
[  5]   6.00-7.00   sec  36.9 KBytes   302 Kbits/sec    0   6.58 KBytes
[  5]   7.00-8.00   sec  46.1 KBytes   377 Kbits/sec    2   7.90 KBytes
[  5]   8.00-9.00   sec  43.4 KBytes   356 Kbits/sec    1   9.21 KBytes
[  5]   9.00-10.00  sec  47.4 KBytes   388 Kbits/sec    3   3.95 KBytes
[  5]  10.00-11.00  sec  38.2 KBytes   313 Kbits/sec    2   3.95 KBytes
[  5]  11.00-12.00  sec  43.4 KBytes   356 Kbits/sec    1   5.27 KBytes
[  5]  12.00-13.00  sec  42.1 KBytes   345 Kbits/sec    0   9.21 KBytes
[  5]  13.00-14.00  sec  46.1 KBytes   377 Kbits/sec    5   5.27 KBytes
[  5]  14.00-15.00  sec  42.1 KBytes   345 Kbits/sec    2   6.58 KBytes
[  5]  15.00-16.00  sec  38.2 KBytes   313 Kbits/sec    2   6.58 KBytes
[  5]  16.00-17.00  sec  48.7 KBytes   399 Kbits/sec    4   2.63 KBytes
[  5]  17.00-18.00  sec  0.00 Bytes  0.00 bits/sec    4   3.95 KBytes
[  5]  18.00-19.00  sec  39.5 KBytes   324 Kbits/sec    2   3.95 KBytes
[  5]  19.00-20.00  sec  39.5 KBytes   324 Kbits/sec    2   2.63 KBytes
[  5]  20.00-21.00  sec  0.00 Bytes  0.00 bits/sec    1   3.95 KBytes
[  5]  21.00-22.00  sec  75.0 KBytes   615 Kbits/sec    1   3.95 KBytes
[  5]  22.00-23.00  sec  0.00 Bytes  0.00 bits/sec    2   3.95 KBytes
[  5]  23.00-24.00  sec  38.2 KBytes   313 Kbits/sec    1   2.63 KBytes
[  5]  24.00-25.00  sec  36.9 KBytes   302 Kbits/sec    2   2.63 KBytes
[  5]  25.00-26.00  sec  40.8 KBytes   334 Kbits/sec    0   7.90 KBytes
[  5]  26.00-27.00  sec  50.0 KBytes   410 Kbits/sec    2   7.90 KBytes
[  5]  27.00-28.00  sec  38.2 KBytes   313 Kbits/sec    3   3.95 KBytes
[  5]  28.00-29.00  sec  0.00 Bytes  0.00 bits/sec    3   3.95 KBytes
[  5]  29.00-30.00  sec  36.9 KBytes   302 Kbits/sec    1   5.27 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  1.20 MBytes   336 Kbits/sec   60             sender
[  5]   0.00-30.00  sec  1.16 MBytes   324 Kbits/sec                  receiver
CPU Utilization: local/sender 2.0% (0.6%u/1.5%s), remote/receiver 0.2% (0.2%u/0.1%s)
iperf Done.

[mimugmail]

Maybe fragmentation?
You can try Interfaces : LAN : MSS -> 1300

[danderson]

I changed the MSS on the Cisco ASA (far side of tunnel) from 1360 to 1300 and that solved the issue. so not an opnsense issue, but yes a fragment issue.

thx for the direction to look mimugmail