OPNsense Forum

Archive => 18.1 Legacy Series => Topic started by: camouflageX on June 11, 2018, 03:34:41 pm

Title: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 11, 2018, 03:34:41 pm: Hello people,

I have an unusual issue regarding the Traffic Shaper and IPsec connections: We have three branches connected with OPNsense boxes over small Internet links (about 4 Mbit/s). When I open the web interface of these boxes and the packets go through the IPsec VPN, then the website loads very slowly (about 15 secs). When I disable Traffic Shaping, then this is not the case. Every other data going through the traffic shaper is always fine.

Now I created a small test scenario. For testing purposes created a simple OPN setup with two VirtualBox VMs:
1. Hostname: OPNsense1
OPNsense version: 18.1.9
LAN: 192.168.56.2/24
WAN: 10.0.0.1/24

2. Hostname: OPNsense2
OPNsense version: 18.1.9
LAN: 192.168.57.2/24
WAN: 10.0.0.2/24

Firewall:
Firewall disabled for testing purposes.

IPsec:
These are the IPsec settings on OPNsense2 (192.168.57.0/24 -> 192.168.56.0/24). Settings on OPNsense1 are similar to this.
Code: [Select]
Type Remote Gateway Mode Phase 1 Proposal Authentication Description IPv4 IKEv2 WAN 10.0.0.1 AES (128 bits) + AESXCBC + DH Group 19 (256 bit elliptic curve) Mutual PSK 2 -> 1
Code: [Select]
Type Local Subnet Remote Subnet Encryption Protocols Authenticity Protocols PFS ESP IPv4 tunnel LAN 192.168.56.0/24 AES (auto), Blowfish (auto), 3DES, CAST128 AES-XCBC off

Traffic Shaper:
In Traffic Shaper I created a simple upload shaper. All other settings at default.

Pipes:
Code: [Select]
Enabled Bandwidth Metric Mask Description [X] 11000 kbit/s - pipe-up
Queues:
Code: [Select]
Enabled Pipe Weight Description [X] pipe-up 100 queue-up
Rules:
Code: [Select]
# Interface Protocol Source Destination Target Description 1 IPsec ip 192.168.57.0/24 192.168.56.0/24 queue-up rule-up

Routes:
On Windows I added a new route, so that all packets destined at OPNsense2 go to OPNsense1 and through the IPsec VPN.
Code: [Select]
ROUTE ADD 192.168.57.2 MASK 255.255.255.255 192.168.56.2
A packet would go this way:
PC (192.168.56.1) -> OPNsense1 (192.168.56.2) -> IPsec VPN -> OPNsense2 (192.168.57.2)

Testing:
When I set the upload pipe to 11000 kbit/s or below and open the web interface of OPNsense2 on my PC, the web sites opens really slowly. It takes about 15 seconds until it is loaded completely. Ping times are always below 1 ms.
When I change the bandwidth of the upload pipe to 12000 kbit/s, the website opens in about 2 seconds.

What could be the cause? Is this a bug?

Thanks for any feedback.
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 22, 2018, 09:08:48 am: Still happening in 18.1.10.. :-\
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: mimugmail on June 22, 2018, 09:46:03 am: Try a tcpdump on FW1 Interface enc0 from beginning till end and upload it. There must be a reason for this ...
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: franco on June 22, 2018, 11:25:20 am: Sounds like you are shaping your web interface along with the IPsec?

Cheers,
Franco
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 22, 2018, 01:08:17 pm: Hello and thank you very much for your replies!

There is only one traffic shaping rule set to IPsec on OPNsense2. There is no rule for the WAN interface and no traffic shaping on OPNsense1 at all.

Code: [Select]
OPNsense2 # ipfw -a list [...] 60000 0 0 return ip from any to any 60001 1207 1476628 queue 10000 ip from 192.168.57.0/24 to 192.168.56.0/24 via enc0 // enc0: queue-up 65533 4798 1861553 allow ip from any to any 65534 0 0 deny ip from any to any 65535 0 0 allow ip from any to any

I created some packet captures. One where the traffic shaping bandwidth was set to 11440 kbit/s (slow loading of web GUI) and one with 11450 kbit/s bandwidth (fast loading).

When I try to open these files in Wireshark, I get the following error message:
Quote
The capture file appears to be damaged or corrupted.
The file has 679044193-byte packet, bigger than the maximum of 262144.

Weird, isn't it?
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: mimugmail on June 22, 2018, 01:36:31 pm: I can only imagine some kinde of fragmentation. I also have sites with IPSec and Shaper in it, no problem reaching the UI. Can you take the dump from you live setup please?
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 22, 2018, 04:42:33 pm: OK, I just captures some traffic from our firewall in production and there are no corrupted packets. But as you can see there are many TCP Retransmission from the OPNsense firewall (10.3.34.1) to my desktop (192.168.241.15) ...

Any idea how this could happen?

Packet capture: https://ufile.io/4n1qd
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 29, 2018, 02:22:15 pm: I did some more testing... traffic is still very slow. I did a quick test using iperf3:

Traffic Shaper Pipe Bandwidth 11400 kbps
Code: [Select]
root@OPNsense2:~ # ipfw pipe show 10000: 11.400 Mbit/s 0 ms burst 0 q141072 50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail sched 75536 type FIFO flags 0x0 0 buckets 0 active

Code: [Select]
root@OPNsense2:~ # iperf3 -c 192.168.56.1 Connecting to host 192.168.56.1, port 5201 [ 5] local 192.168.57.2 port 61934 connected to 192.168.56.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 72.2 KBytes 591 Kbits/sec 3 23.6 KBytes [ 5] 1.00-2.00 sec 13.5 KBytes 110 Kbits/sec 5 37.0 KBytes [ 5] 2.00-3.00 sec 13.5 KBytes 110 Kbits/sec 5 50.5 KBytes [ 5] 3.00-4.00 sec 21.5 KBytes 176 Kbits/sec 5 64.0 KBytes [ 5] 4.00-5.00 sec 21.5 KBytes 176 Kbits/sec 5 77.4 KBytes [ 5] 5.00-6.00 sec 29.5 KBytes 241 Kbits/sec 5 90.9 KBytes [ 5] 6.00-7.00 sec 29.5 KBytes 241 Kbits/sec 5 104 KBytes [ 5] 7.00-8.00 sec 21.5 KBytes 176 Kbits/sec 5 118 KBytes [ 5] 8.00-9.00 sec 29.5 KBytes 241 Kbits/sec 5 131 KBytes [ 5] 9.00-10.00 sec 29.5 KBytes 241 Kbits/sec 5 145 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 281 KBytes 231 Kbits/sec 48 sender [ 5] 0.00-10.00 sec 129 KBytes 106 Kbits/sec receiver iperf Done.
Traffic Shaper Pipe Bandwidth 11500 kbps
Code: [Select]
root@OPNsense2:~ # ipfw pipe show 10000: 11.500 Mbit/s 0 ms burst 0 q141072 50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail sched 75536 ty
Code: [Select]
root@OPNsense2:~ # iperf3 -c 192.168.56.1 Connecting to host 192.168.56.1, port 5201 [ 5] local 192.168.57.2 port 23220 connected to 192.168.56.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 99.2 KBytes 812 Kbits/sec 12 40.4 KBytes [ 5] 1.00-2.00 sec 45.7 KBytes 374 Kbits/sec 15 67.9 KBytes [ 5] 2.00-3.00 sec 55.0 KBytes 450 Kbits/sec 12 91.3 KBytes [ 5] 3.00-4.00 sec 95.2 KBytes 779 Kbits/sec 18 130 KBytes [ 5] 4.00-5.00 sec 113 KBytes 923 Kbits/sec 20 176 KBytes [ 5] 5.00-6.00 sec 71.1 KBytes 582 Kbits/sec 15 205 KBytes [ 5] 6.00-7.00 sec 52.5 KBytes 430 Kbits/sec 18 209 KBytes [ 5] 7.00-8.00 sec 48.5 KBytes 398 Kbits/sec 16 209 KBytes [ 5] 8.00-9.00 sec 39.1 KBytes 320 Kbits/sec 14 209 KBytes [ 5] 9.00-10.00 sec 33.7 KBytes 276 Kbits/sec 12 209 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 652 KBytes 534 Kbits/sec 152 sender [ 5] 0.00-10.00 sec 437 KBytes 358 Kbits/sec receiver iperf Done.
Traffic Shaping disabled
Code: [Select]
root@OPNsense2:~ # iperf3 -c 192.168.56.1 Connecting to host 192.168.56.1, port 5201 [ 5] local 192.168.57.2 port 54607 connected to 192.168.56.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 9.37 MBytes 78.5 Mbits/sec 140 70.2 KBytes [ 5] 1.00-2.00 sec 8.74 MBytes 73.0 Mbits/sec 29 61.9 KBytes [ 5] 2.00-3.00 sec 8.86 MBytes 74.6 Mbits/sec 38 78.9 KBytes [ 5] 3.00-4.00 sec 8.88 MBytes 74.5 Mbits/sec 41 67.4 KBytes [ 5] 4.00-5.00 sec 8.80 MBytes 73.7 Mbits/sec 19 67.2 KBytes [ 5] 5.00-6.00 sec 8.73 MBytes 73.2 Mbits/sec 27 47.5 KBytes [ 5] 6.00-7.00 sec 8.75 MBytes 73.5 Mbits/sec 12 84.3 KBytes [ 5] 7.00-8.00 sec 8.85 MBytes 74.2 Mbits/sec 43 70.2 KBytes [ 5] 8.00-9.00 sec 8.94 MBytes 74.9 Mbits/sec 27 56.2 KBytes [ 5] 9.00-10.00 sec 8.83 MBytes 74.2 Mbits/sec 50 63.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 88.7 MBytes 74.4 Mbits/sec 426 sender [ 5] 0.00-10.00 sec 88.6 MBytes 74.3 Mbits/sec receiver
I also added a new packet capture. This time there are not more corruptions: https://ufile.io/iwfpx

Could you please test web GUI over IPsec and traffic shaping with bandwidth <11400 kbps with your box?
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: mimugmail on June 29, 2018, 02:39:51 pm: Hm, I have to setup a test env .. takes some time.

You really get only 75Mbit without a shaper? Sounds to me like a totally underpowered machine.
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on June 29, 2018, 02:50:24 pm: That would be great!

It is inside a VirtualBox VM with a slow PCnet-PCI II. Also the traffic goes over an encrypted IPsec tunnel to the other VM.
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: camouflageX on July 05, 2018, 07:22:13 am: Hey, just want to ask if you already had the time to create a simple test setup? It is still happening with 18.1.11.
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: mimugmail on July 05, 2018, 07:34:41 am: Hopefully tomorrow, can you ping me in IRC tomorrow so I dont forget it? :)
Title: Re: IPsec + Traffic Shaper = Slow web interface
Post by: mimugmail on July 20, 2018, 08:12:18 am: My Lab:

10.0.1.1 --- FW-A 10.55.1.1 --- 10.55.1.2 --- FW-B --- 10.0.2.1

No issues when I surf from 10.0.1.10 (Client A) to 10.0.2.1.

Following screenshots of VPN setup on FW-A (FW-B same of course) and the TS config.