IPsec + Traffic Shaper = Slow web interface

[camouflageX]

Hello people,

I have an unusual issue regarding the Traffic Shaper and IPsec connections: We have three branches connected with OPNsense boxes over small Internet links (about 4 Mbit/s). When I open the web interface of these boxes and the packets go through the IPsec VPN, then the website loads very slowly (about 15 secs). When I disable Traffic Shaping, then this is not the case. Every other data going through the traffic shaper is always fine.

Now I created a small test scenario. For testing purposes created a simple OPN setup with two VirtualBox VMs:
1. Hostname: OPNsense1
OPNsense version: 18.1.9
LAN: 192.168.56.2/24
WAN: 10.0.0.1/24

2. Hostname: OPNsense2
OPNsense version: 18.1.9
LAN: 192.168.57.2/24
WAN: 10.0.0.2/24


Firewall:
Firewall disabled for testing purposes.


IPsec:
These are the IPsec settings on OPNsense2 (192.168.57.0/24 -> 192.168.56.0/24). Settings on OPNsense1 are similar to this.

Code: [Select]

Type Remote Gateway Mode Phase 1 Proposal Authentication Description
IPv4 IKEv2 WAN 10.0.0.1 AES (128 bits) + AESXCBC + DH Group 19 (256 bit elliptic curve) Mutual PSK 2 -> 1

Code: [Select]

Type Local Subnet Remote Subnet Encryption Protocols Authenticity Protocols PFS
ESP IPv4 tunnel LAN 192.168.56.0/24 AES (auto), Blowfish (auto), 3DES, CAST128 AES-XCBC off

Traffic Shaper:
In Traffic Shaper I created a simple upload shaper. All other settings at default.

Pipes:

Code: [Select]

Enabled Bandwidth Metric Mask Description
[X] 11000 kbit/s - pipe-up
Queues:

Code: [Select]

Enabled Pipe Weight Description
[X] pipe-up 100 queue-up
Rules:

Code: [Select]

# Interface Protocol Source Destination Target Description
1 IPsec ip 192.168.57.0/24 192.168.56.0/24 queue-up rule-up

Routes:
On Windows I added a new route, so that all packets destined at OPNsense2 go to OPNsense1 and through the IPsec VPN.

Code: [Select]

ROUTE ADD 192.168.57.2 MASK 255.255.255.255 192.168.56.2
A packet would go this way:
PC (192.168.56.1) -> OPNsense1 (192.168.56.2) -> IPsec VPN -> OPNsense2 (192.168.57.2)


Testing:
When I set the upload pipe to 11000 kbit/s or below and open the web interface of OPNsense2 on my PC, the web sites opens really slowly. It takes about 15 seconds until it is loaded completely. Ping times are always below 1 ms.
When I change the bandwidth of the upload pipe to 12000 kbit/s, the website opens in about 2 seconds.


What could be the cause?  Is this a bug?


Thanks for any feedback.

[camouflageX]

Still happening in 18.1.10.. 

[mimugmail]

Try a tcpdump on FW1 Interface enc0 from beginning till end and upload it. There must be a reason for this ...

[franco]

Sounds like you are shaping your web interface along with the IPsec?


Cheers,
Franco

[camouflageX]

Hello and thank you very much for your replies!

There is only one traffic shaping rule set to IPsec on OPNsense2. There is no rule for the WAN interface and no traffic shaping on OPNsense1 at all.

Code: [Select]

OPNsense2 # ipfw -a list
[...]
60000    0       0 return ip from any to any
60001 1207 1476628 queue 10000 ip from 192.168.57.0/24 to 192.168.56.0/24 via enc0 // enc0: queue-up
65533 4798 1861553 allow ip from any to any
65534    0       0 deny ip from any to any
65535    0       0 allow ip from any to any

I created some packet captures. One where the traffic shaping bandwidth was set to 11440 kbit/s (slow loading of web GUI) and one with 11450 kbit/s bandwidth (fast loading).

When I try to open these files in Wireshark, I get the following error message:

The capture file appears to be damaged or corrupted.
The file has 679044193-byte packet, bigger than the maximum of 262144.

Weird, isn't it?

[mimugmail]

I can only imagine some kinde of fragmentation. I also have sites with IPSec and Shaper in it, no problem reaching the UI. Can you take the dump from you live setup please?

[camouflageX]

OK, I just captures some traffic from our firewall in production and there are no corrupted packets. But as you can see there are many TCP Retransmission from the OPNsense firewall (10.3.34.1) to my desktop (192.168.241.15) ...

Any idea how this could happen?

Packet capture: https://ufile.io/4n1qd

[camouflageX]

I did some more testing... traffic is still very slow. I did a quick test using iperf3:

Traffic Shaper Pipe Bandwidth 11400 kbps

Code: [Select]

root@OPNsense2:~ # ipfw pipe show
10000:  11.400 Mbit/s    0 ms burst 0
q141072  50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail
 sched 75536 type FIFO flags 0x0 0 buckets 0 active

Code: [Select]

root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[  5] local 192.168.57.2 port 61934 connected to 192.168.56.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  72.2 KBytes   591 Kbits/sec    3   23.6 KBytes
[  5]   1.00-2.00   sec  13.5 KBytes   110 Kbits/sec    5   37.0 KBytes
[  5]   2.00-3.00   sec  13.5 KBytes   110 Kbits/sec    5   50.5 KBytes
[  5]   3.00-4.00   sec  21.5 KBytes   176 Kbits/sec    5   64.0 KBytes
[  5]   4.00-5.00   sec  21.5 KBytes   176 Kbits/sec    5   77.4 KBytes
[  5]   5.00-6.00   sec  29.5 KBytes   241 Kbits/sec    5   90.9 KBytes
[  5]   6.00-7.00   sec  29.5 KBytes   241 Kbits/sec    5    104 KBytes
[  5]   7.00-8.00   sec  21.5 KBytes   176 Kbits/sec    5    118 KBytes
[  5]   8.00-9.00   sec  29.5 KBytes   241 Kbits/sec    5    131 KBytes
[  5]   9.00-10.00  sec  29.5 KBytes   241 Kbits/sec    5    145 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   281 KBytes   231 Kbits/sec   48             sender
[  5]   0.00-10.00  sec   129 KBytes   106 Kbits/sec                  receiver

iperf Done.
Traffic Shaper Pipe Bandwidth 11500 kbps

Code: [Select]

root@OPNsense2:~ # ipfw pipe show
10000:  11.500 Mbit/s    0 ms burst 0
q141072  50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail
 sched 75536 ty

Code: [Select]

root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[  5] local 192.168.57.2 port 23220 connected to 192.168.56.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  99.2 KBytes   812 Kbits/sec   12   40.4 KBytes
[  5]   1.00-2.00   sec  45.7 KBytes   374 Kbits/sec   15   67.9 KBytes
[  5]   2.00-3.00   sec  55.0 KBytes   450 Kbits/sec   12   91.3 KBytes
[  5]   3.00-4.00   sec  95.2 KBytes   779 Kbits/sec   18    130 KBytes
[  5]   4.00-5.00   sec   113 KBytes   923 Kbits/sec   20    176 KBytes
[  5]   5.00-6.00   sec  71.1 KBytes   582 Kbits/sec   15    205 KBytes
[  5]   6.00-7.00   sec  52.5 KBytes   430 Kbits/sec   18    209 KBytes
[  5]   7.00-8.00   sec  48.5 KBytes   398 Kbits/sec   16    209 KBytes
[  5]   8.00-9.00   sec  39.1 KBytes   320 Kbits/sec   14    209 KBytes
[  5]   9.00-10.00  sec  33.7 KBytes   276 Kbits/sec   12    209 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   652 KBytes   534 Kbits/sec  152             sender
[  5]   0.00-10.00  sec   437 KBytes   358 Kbits/sec                  receiver

iperf Done.
Traffic Shaping disabled

Code: [Select]

root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[  5] local 192.168.57.2 port 54607 connected to 192.168.56.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  9.37 MBytes  78.5 Mbits/sec  140   70.2 KBytes
[  5]   1.00-2.00   sec  8.74 MBytes  73.0 Mbits/sec   29   61.9 KBytes
[  5]   2.00-3.00   sec  8.86 MBytes  74.6 Mbits/sec   38   78.9 KBytes
[  5]   3.00-4.00   sec  8.88 MBytes  74.5 Mbits/sec   41   67.4 KBytes
[  5]   4.00-5.00   sec  8.80 MBytes  73.7 Mbits/sec   19   67.2 KBytes
[  5]   5.00-6.00   sec  8.73 MBytes  73.2 Mbits/sec   27   47.5 KBytes
[  5]   6.00-7.00   sec  8.75 MBytes  73.5 Mbits/sec   12   84.3 KBytes
[  5]   7.00-8.00   sec  8.85 MBytes  74.2 Mbits/sec   43   70.2 KBytes
[  5]   8.00-9.00   sec  8.94 MBytes  74.9 Mbits/sec   27   56.2 KBytes
[  5]   9.00-10.00  sec  8.83 MBytes  74.2 Mbits/sec   50   63.2 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  88.7 MBytes  74.4 Mbits/sec  426             sender
[  5]   0.00-10.00  sec  88.6 MBytes  74.3 Mbits/sec                  receiver

I also added a new packet capture. This time there are not more corruptions: https://ufile.io/iwfpx

Could you please test web GUI over IPsec and traffic shaping with bandwidth <11400 kbps with your box?

[mimugmail]

Hm, I have to setup a test env .. takes some time.

You really get only 75Mbit without a shaper? Sounds to me like a totally underpowered machine.

[camouflageX]

That would be great!

It is inside a VirtualBox VM with a slow PCnet-PCI II. Also the traffic goes over an encrypted IPsec tunnel to the other VM.

[camouflageX]

Hey, just want to ask if you already had the time to create a simple test setup? It is still happening with 18.1.11.

[mimugmail]

Hopefully tomorrow, can you ping me in IRC tomorrow so I dont forget it?

[mimugmail]

My Lab:

10.0.1.1 --- FW-A 10.55.1.1 --- 10.55.1.2 --- FW-B --- 10.0.2.1

No issues when I surf from 10.0.1.10 (Client A) to 10.0.2.1.

Following screenshots of VPN setup on FW-A (FW-B same of course) and the TS config.