Hello people,
I have an unusual issue regarding the Traffic Shaper and IPsec connections: We have three branches connected with OPNsense boxes over small Internet links (about 4 Mbit/s). When I open the web interface of these boxes and the packets go through the IPsec VPN, then the website loads very slowly (about 15 secs). When I disable Traffic Shaping, then this is not the case. Every other data going through the traffic shaper is always fine.
Now I created a small test scenario. For testing purposes created a simple OPN setup with two VirtualBox VMs:
1. Hostname: OPNsense1
OPNsense version: 18.1.9
LAN: 192.168.56.2/24
WAN: 10.0.0.1/24
2. Hostname: OPNsense2
OPNsense version: 18.1.9
LAN: 192.168.57.2/24
WAN: 10.0.0.2/24
Firewall:
Firewall disabled for testing purposes.
IPsec:
These are the IPsec settings on OPNsense2 (192.168.57.0/24 -> 192.168.56.0/24). Settings on OPNsense1 are similar to this.
Type Remote Gateway Mode Phase 1 Proposal Authentication Description
IPv4 IKEv2 WAN 10.0.0.1 AES (128 bits) + AESXCBC + DH Group 19 (256 bit elliptic curve) Mutual PSK 2 -> 1
Type Local Subnet Remote Subnet Encryption Protocols Authenticity Protocols PFS
ESP IPv4 tunnel LAN 192.168.56.0/24 AES (auto), Blowfish (auto), 3DES, CAST128 AES-XCBC off
Traffic Shaper:
In Traffic Shaper I created a simple upload shaper. All other settings at default.
Pipes:
Enabled Bandwidth Metric Mask Description
[X] 11000 kbit/s - pipe-up
Queues:
Enabled Pipe Weight Description
[X] pipe-up 100 queue-up
Rules:
# Interface Protocol Source Destination Target Description
1 IPsec ip 192.168.57.0/24 192.168.56.0/24 queue-up rule-up
Routes:
On Windows I added a new route, so that all packets destined at OPNsense2 go to OPNsense1 and through the IPsec VPN.
ROUTE ADD 192.168.57.2 MASK 255.255.255.255 192.168.56.2
A packet would go this way:
PC (192.168.56.1) -> OPNsense1 (192.168.56.2) -> IPsec VPN -> OPNsense2 (192.168.57.2)
Testing:
When I set the upload pipe to 11000 kbit/s or below and open the web interface of OPNsense2 on my PC, the web sites opens really slowly. It takes about 15 seconds until it is loaded completely. Ping times are always below 1 ms.
When I change the bandwidth of the upload pipe to 12000 kbit/s, the website opens in about 2 seconds.
What could be the cause? Is this a bug?
Thanks for any feedback.
Still happening in 18.1.10.. :-\
Try a tcpdump on FW1 Interface enc0 from beginning till end and upload it. There must be a reason for this ...
Sounds like you are shaping your web interface along with the IPsec?
Cheers,
Franco
Hello and thank you very much for your replies!
There is only one traffic shaping rule set to IPsec on OPNsense2. There is no rule for the WAN interface and no traffic shaping on OPNsense1 at all.
OPNsense2 # ipfw -a list
[...]
60000 0 0 return ip from any to any
60001 1207 1476628 queue 10000 ip from 192.168.57.0/24 to 192.168.56.0/24 via enc0 // enc0: queue-up
65533 4798 1861553 allow ip from any to any
65534 0 0 deny ip from any to any
65535 0 0 allow ip from any to anyI created some packet captures. One where the traffic shaping bandwidth was set to 11440 kbit/s (slow loading of web GUI) and one with 11450 kbit/s bandwidth (fast loading).
When I try to open these files in Wireshark, I get the following error message:
QuoteThe capture file appears to be damaged or corrupted.
The file has 679044193-byte packet, bigger than the maximum of 262144.
Weird, isn't it?
I can only imagine some kinde of fragmentation. I also have sites with IPSec and Shaper in it, no problem reaching the UI. Can you take the dump from you live setup please?
OK, I just captures some traffic from our firewall in production and there are no corrupted packets. But as you can see there are many TCP Retransmission from the OPNsense firewall (10.3.34.1) to my desktop (192.168.241.15) ...
Any idea how this could happen?
Packet capture: https://ufile.io/4n1qd
I did some more testing... traffic is still very slow. I did a quick test using iperf3:
Traffic Shaper Pipe Bandwidth 11400 kbps
root@OPNsense2:~ # ipfw pipe show
10000: 11.400 Mbit/s 0 ms burst 0
q141072 50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail
sched 75536 type FIFO flags 0x0 0 buckets 0 active
root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[ 5] local 192.168.57.2 port 61934 connected to 192.168.56.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 72.2 KBytes 591 Kbits/sec 3 23.6 KBytes
[ 5] 1.00-2.00 sec 13.5 KBytes 110 Kbits/sec 5 37.0 KBytes
[ 5] 2.00-3.00 sec 13.5 KBytes 110 Kbits/sec 5 50.5 KBytes
[ 5] 3.00-4.00 sec 21.5 KBytes 176 Kbits/sec 5 64.0 KBytes
[ 5] 4.00-5.00 sec 21.5 KBytes 176 Kbits/sec 5 77.4 KBytes
[ 5] 5.00-6.00 sec 29.5 KBytes 241 Kbits/sec 5 90.9 KBytes
[ 5] 6.00-7.00 sec 29.5 KBytes 241 Kbits/sec 5 104 KBytes
[ 5] 7.00-8.00 sec 21.5 KBytes 176 Kbits/sec 5 118 KBytes
[ 5] 8.00-9.00 sec 29.5 KBytes 241 Kbits/sec 5 131 KBytes
[ 5] 9.00-10.00 sec 29.5 KBytes 241 Kbits/sec 5 145 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 281 KBytes 231 Kbits/sec 48 sender
[ 5] 0.00-10.00 sec 129 KBytes 106 Kbits/sec receiver
iperf Done.
Traffic Shaper Pipe Bandwidth 11500 kbps
root@OPNsense2:~ # ipfw pipe show
10000: 11.500 Mbit/s 0 ms burst 0
q141072 50 sl. 0 flows (1 buckets) sched 75536 weight 0 lmax 0 pri 0 droptail
sched 75536 ty
root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[ 5] local 192.168.57.2 port 23220 connected to 192.168.56.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 99.2 KBytes 812 Kbits/sec 12 40.4 KBytes
[ 5] 1.00-2.00 sec 45.7 KBytes 374 Kbits/sec 15 67.9 KBytes
[ 5] 2.00-3.00 sec 55.0 KBytes 450 Kbits/sec 12 91.3 KBytes
[ 5] 3.00-4.00 sec 95.2 KBytes 779 Kbits/sec 18 130 KBytes
[ 5] 4.00-5.00 sec 113 KBytes 923 Kbits/sec 20 176 KBytes
[ 5] 5.00-6.00 sec 71.1 KBytes 582 Kbits/sec 15 205 KBytes
[ 5] 6.00-7.00 sec 52.5 KBytes 430 Kbits/sec 18 209 KBytes
[ 5] 7.00-8.00 sec 48.5 KBytes 398 Kbits/sec 16 209 KBytes
[ 5] 8.00-9.00 sec 39.1 KBytes 320 Kbits/sec 14 209 KBytes
[ 5] 9.00-10.00 sec 33.7 KBytes 276 Kbits/sec 12 209 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 652 KBytes 534 Kbits/sec 152 sender
[ 5] 0.00-10.00 sec 437 KBytes 358 Kbits/sec receiver
iperf Done.
Traffic Shaping disabled
root@OPNsense2:~ # iperf3 -c 192.168.56.1
Connecting to host 192.168.56.1, port 5201
[ 5] local 192.168.57.2 port 54607 connected to 192.168.56.1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 9.37 MBytes 78.5 Mbits/sec 140 70.2 KBytes
[ 5] 1.00-2.00 sec 8.74 MBytes 73.0 Mbits/sec 29 61.9 KBytes
[ 5] 2.00-3.00 sec 8.86 MBytes 74.6 Mbits/sec 38 78.9 KBytes
[ 5] 3.00-4.00 sec 8.88 MBytes 74.5 Mbits/sec 41 67.4 KBytes
[ 5] 4.00-5.00 sec 8.80 MBytes 73.7 Mbits/sec 19 67.2 KBytes
[ 5] 5.00-6.00 sec 8.73 MBytes 73.2 Mbits/sec 27 47.5 KBytes
[ 5] 6.00-7.00 sec 8.75 MBytes 73.5 Mbits/sec 12 84.3 KBytes
[ 5] 7.00-8.00 sec 8.85 MBytes 74.2 Mbits/sec 43 70.2 KBytes
[ 5] 8.00-9.00 sec 8.94 MBytes 74.9 Mbits/sec 27 56.2 KBytes
[ 5] 9.00-10.00 sec 8.83 MBytes 74.2 Mbits/sec 50 63.2 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 88.7 MBytes 74.4 Mbits/sec 426 sender
[ 5] 0.00-10.00 sec 88.6 MBytes 74.3 Mbits/sec receiver
I also added a new packet capture. This time there are not more corruptions: https://ufile.io/iwfpx
Could you please test web GUI over IPsec and traffic shaping with bandwidth <11400 kbps with your box?
Hm, I have to setup a test env .. takes some time.
You really get only 75Mbit without a shaper? Sounds to me like a totally underpowered machine.
That would be great!
It is inside a VirtualBox VM with a slow PCnet-PCI II. Also the traffic goes over an encrypted IPsec tunnel to the other VM.
Hey, just want to ask if you already had the time to create a simple test setup? It is still happening with 18.1.11.
Hopefully tomorrow, can you ping me in IRC tomorrow so I dont forget it? :)
My Lab:
10.0.1.1 --- FW-A 10.55.1.1 --- 10.55.1.2 --- FW-B --- 10.0.2.1
No issues when I surf from 10.0.1.10 (Client A) to 10.0.2.1.
Following screenshots of VPN setup on FW-A (FW-B same of course) and the TS config.