OPNsense Forum
Archive => 17.7 Legacy Series => Topic started by: fraenki on August 29, 2017, 03:45:36 pm
-
Hi,
I have deployed a new OPNsense cluster that shows abysmal IPsec performance:
- traffic over IPsec: ~1-2 Mbps
- traffic without IPsec: full speed
SSH file transfers will start at ~25 Mbps, but will immediately drop to 3 Mbps and drop even further within a few seconds.
HTTPS file transfers may even stall completely (this being our main issue). Other connections with "large" data transfers will also abort/stall.
There's no significant load shown in "top" when utilizing the IPsec tunnel.
Tunnel config is pretty old fashioned: AES256/SHA256/DH Group 2 (same for Phase 1+2)
Enable/disable HW offloading does not make any difference.
Hardware is a Intel x5-Z8350 SOC with a Realtek NIC (UP board).
I've seen a lot similar reports for pfSense:
https://superuser.com/questions/570049/pfsense-firewall-blocking-some-outbound-web-packets-large-http-downloads-just
https://forum.pfsense.org/index.php?topic=74159.msg405436
https://forum.pfsense.org/index.php?topic=123823.msg683776
(Just google for "pfsense ipsec speed"...)
We have some other OPNsense clusters that don't show this issue.
FWIW, this is the only location with a PPPoE router. I've tested the same PPPoE router (Zyxel VMG1312-B30A) at another location with no issues. So I don't think it's the router that causes this issue.
Any ideas?
Thanks
- Frank
-
FWIW, I've already tested with MTU 1300 and MSS 1300 on the WAN interface, but this didn't change anything.
-
If you have 25Mbps and a throttle to 1-2Mbps it's mostly packetloss (line, nic, driver etc) and a suboptimal windows size. Packetloss would also slow down IPSec, so I'd go for problems on the line or the nic.
-
If you have 25Mbps and a throttle to 1-2Mbps it's mostly packetloss (line, nic, driver etc) and a suboptimal windows size. Packetloss would also slow down IPSec, so I'd go for problems on the line or the nic.
Please note that the throttle only occurs for traffic that goes through the IPsec tunnel. When sending traffic to the same host without IPsec it easily reaches full speed.
-
Oh, ok, now I've read the complete thread :)
Can you try MSS to 1000 on IPSEC or LAN interface?
I only use the router/modems from Zyxel in bridge mode, perhaps they have some sort or IPSEC replay detection which is enabled?
-
Can you try MSS to 1000 on IPSEC or LAN interface?
I've set both MTU and MSS to 1000. Doesn't make a difference. :(
I only use the router/modems from Zyxel in bridge mode, perhaps they have some sort or IPSEC replay detection which is enabled?
I'm pretty sure it's not the router, I've tested the Zyxel router at home before sending it to the remote location. I was able to use IPsec at full speed with this router and another OPNsense firewall. (BTW the Zyxel router replaced a LANCOM router which showed the same IPsec performance issue.)
I *guess* it's a OPNsense configuration issue, or a general networking issue. I've read so many similar reports regarding pfSense, but wasn't able to find a solution yet. :(
I've captured a TCP dump (on OPNsense) while copying a large file over SSH. I think it doesn't look too bad, right? (see attachment)
- Frank
-
MTU should be higher than MSS. Try 1200 MTU and 1000 MSS. Also show a complete capture (first 5 seconds but wth 3way handshake) inside the tunnel.
Do you use some QoS outside of the tunnel to guarantee traffic to IPSEC? Then the reordering could also throttle the tunnel.
-
MTU should be higher than MSS. Try 1200 MTU and 1000 MSS.
OPNsense already reduces the MSS by 40. If I configure a MSS of 1000, OPNsense will set it to 960. Would this be sufficient or should I try 1200MTU/1000MSS nonetheless?
Also show a complete capture (first 5 seconds but wth 3way handshake) inside the tunnel.
I've captured the HTTPS connection, because it stalls/breaks very quickly and the dump is rather small:
(sorry, can't paste it in this forum due to post size limits)
TCP capture on the "bad" OPNsense at the remote location (behind a PPPoE router):
http://paste.debian.net/plainh/565326dd
TCP capture on the "good" OPNsense at the other location:
http://paste.debian.net/plainh/b8df834c
And >1 minute after the HTTPS connection has died, I'm seing the following log entries on the "good" firewall:
00:02:42.558779 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [P.], seq 3004363110:3004363875, ack 423027980, win 1264, length 765
00:00:00.000056 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [F.], seq 765, ack 1, win 1264, length 0
00:00:00.093263 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [F.], seq 765, ack 1, win 1264, length 0
00:00:00.247810 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.248096 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.247992 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.247945 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.248026 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.247961 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765
00:00:00.248023 rule 1..16777216/0(match): block out on enc0: HTTP_SERVER.443 > HTTP_CLIENT.58101: Flags [FP.], seq 0:765, ack 1, win 1264, length 765(this is from another test, so the local port and seq numbers will not match anything)
Does this tell us why the HTTPS connection suddenly stalls/breaks?
Do you use some QoS outside of the tunnel to guarantee traffic to IPSEC? Then the reordering could also throttle the tunnel.
No QoS, it's a (supposedly) simple setup: PPPoE router <-> VLAN switch <-> OPNsense.
The VLAN switch is simple and stupidly cheap, have it at another location too.
Thanks
- Frank
-
In the 3way handshake the MSS is still 1460, so your setting doesn't work.
Also the windows size of 256 is way too small .. it should grow to 64kb when there's no loss.
A pcap file (you know my private Mail) would be better to trace.
-
In the 3way handshake the MSS is still 1460, so your setting doesn't work.
Also the windows size of 256 is way too small .. it should grow to 64kb when there's no loss.
That's because the TCP dump was taken from the virtual IPsec interface "enc0". On OPNsense the MTU for this interface cannot be changed. However, I've set an MTU1200/MSS1000 on the WAN interface, but it didn't change anything.
A pcap file (you know my private Mail) would be better to trace.
Will do, thanks for your help :)
- Frank
-
The problem is that ESP is not TCP, so on the WAN side it wont be reduced since it don't sees any TCP packets.
But I'm quite sure it's something with the line because the window size is really too small.
-
The problem is that ESP is not TCP, so on the WAN side it wont be reduced since it don't sees any TCP packets.
That's true, but in this case the OPNsense firewall is behind NAT and thus it's all UDP 4500. On the WAN interface the packets seem to be really small...
10:42:25.923159 IP GOOD_OPNSENSE.4500 > BAD_OPNSENSE.4500: UDP-encap: ESP(spi=0xc63f6a01,seq=0x6ea), length 104
10:42:25.923444 IP BAD_OPNSENSE.4500 > GOOD_OPNSENSE.4500: UDP-encap: ESP(spi=0xce9f75ee,seq=0xe95), length 248
10:42:25.923812 IP BAD_OPNSENSE.4500 > GOOD_OPNSENSE.4500: UDP-encap: ESP(spi=0xce9f75ee,seq=0xe96), length 376
10:42:25.924155 IP BAD_OPNSENSE.4500 > GOOD_OPNSENSE.4500: UDP-encap: ESP(spi=0xce9f75ee,seq=0xe97), length 376
But I'm quite sure it's something with the line because the window size is really too small.
Hmm, interesting... I'm just wondering: How would a line issue only affect IPsec and no other connections?
Thanks
- Frank
-
A quick update...
* tested with IKEv2 instead of IKEv1
* tested various MTU/MSS combinations (on WAN and all other interfaces, except enc0)
* in Firewall->Settings->Advanced tested the option "Disable reply-to"
* double-checked that no feature on the Switch causes this
Still no luck. Any idea?
Thanks
- Frank
-
I am currently testing IPSec performance (Release 17.7). I am using the AES-NI driver and the achievable performance is around 450 Mbps. This corresponds about to what is stated in the appliance shop:
https://www.applianceshop.eu/opnsense-quad-core-gen3-10gb-ssd.html#product-attribute-specs-table
Has anyone achieved higher performance ? What is the limiting factor for the poor performance ? I would assume the more cores a cpu has, the higher the throughput. The cpu load on the dashboard shows only 15%
Cheers
Peter
-
Hi Peter,
welcome to the forums!
Has anyone achieved higher performance ? What is the limiting factor for the poor performance ? I would assume the more cores a cpu has, the higher the throughput. The cpu load on the dashboard shows only 15%
I think this is off-topic... This topic is about solving a very specific IPsec performance issue, not about comparing IPsec performance in general.
Regards
- Frank
-
Turns out it was some sort of regression in the early 17.7.x series.
After upgrading to 17.7.7 the issue disappeared.
- Frank
-
I find that hard to believe. What version was this tested against? 17.7 exactly?
-
I find that hard to believe. What version was this tested against? 17.7 exactly?
Me too, but I couldn't find a better explanation. :)
The boxes were running the ancient 17.7.1_2 before upgrading to 17.7.7.
I've already diffed the config.xml backups, but there was no difference.
Regards
- Frank
-
Hi all
In the last days, we want to exchange our Cisco ASA's with opnsense boxes, but we run in this issue. Our old configuration is a ASA5520 <-> ASA5540 with IPSec tunnel, the new are two opnsense boxes with these CPU's "AMD GX-412TC SOC (4 cores)" <-> "Intel(R) Xeon(R) CPU E3-1280 V2 @ 3.60GHz (8 cores)", but i don't think this is the problem, also with an IPSec tunnel.
On the location A is a VM who get a nightly backup from a VM in the location "B". Both are RHEL 7 VM's and we just switch the gateway from the ASAs to the opnSense Boxes. Routers (Cisco and Draytek), Lines, Switches on both locations are untouched.
The VM in the location B have also a public IP on the opnsense FW. So if we try to download a file from the web server or to a scp via the IPSec tunnel, we just get s hand full of KB/s (scp says "--stalled--"), via the public IP full speed. At the beginning the transfer is fast for a short moment, then it comes down very fast. But not all the time sometimes, it goes a bit longer fast before "stalled".
The tunnel on the ASA side was IPSecV1, on opnsense IPSecV2, both ESP etc.
gruss ivo
-
Hi all
In the last days, we want to exchange our Cisco ASA's with opnsense boxes, but we run in this issue. Our old configuration is a ASA5520 <-> ASA5540 with IPSec tunnel, the new are two opnsense boxes with these CPU's "AMD GX-412TC SOC (4 cores)" <-> "Intel(R) Xeon(R) CPU E3-1280 V2 @ 3.60GHz (8 cores)", but i don't think this is the problem, also with an IPSec tunnel.
On the location A is a VM who get a nightly backup from a VM in the location "B". Both are RHEL 7 VM's and we just switch the gateway from the ASAs to the opnSense Boxes. Routers (Cisco and Draytek), Lines, Switches on both locations are untouched.
The VM in the location B have also a public IP on the opnsense FW. So if we try to download a file from the web server or to a scp via the IPSec tunnel, we just get s hand full of KB/s (scp says "--stalled--"), via the public IP full speed. At the beginning the transfer is fast for a short moment, then it comes down very fast. But not all the time sometimes, it goes a bit longer fast before "stalled".
The tunnel on the ASA side was IPSecV1, on opnsense IPSecV2, both ESP etc.
gruss ivo
This sounds like a classic MTU issue. I wrote some hints regarding MTU in this thread, just try them.
-
Salü mimugmail
Yes, it was also my first intend and I tried some modifications, but I never touched the MSS field. Thank you to give me the kick to go through the thread again. It's solved now, here is how:
I try with ping, what is the biggest size where I got a reply:
[root@linux01 ~]# ping -s 1395 -M do 198.18.8.48
PING 198.18.8.48 (198.18.8.48) 1395(1423) bytes of data.
^C
--- 198.18.8.48 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4000ms
[root@linux01 ~]#
[root@linux01 ~]# ping -s 1394 -M do 198.18.8.48
PING 198.18.8.48 (198.18.8.48) 1394(1422) bytes of data.
1402 bytes from 198.18.8.48: icmp_seq=1 ttl=62 time=15.2 ms
1402 bytes from 198.18.8.48: icmp_seq=2 ttl=62 time=15.0 ms
1402 bytes from 198.18.8.48: icmp_seq=3 ttl=62 time=14.3 ms
^C
--- 198.18.8.48 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.316/14.856/15.203/0.386 ms
[root@linux01 ~]#
I set the MSS value "1422" from the "good" ping on the LAN interface on the opnsense and the performance was fine. I try to set it only on location A and it works, but this location is connected via DSL.
I also had a look on the Cisco ASA, there was no settings for MSS, so the default value should be active:
"By default the ASA sets the TCP MSS option in the SYN packets to 1380." (from Cisco).
gruss ivo
-
You can double check the behavior with a simple tcpdump on the OPNsense on the LAN interface.
Do a telnet to an open port behind the ASA network and you'll see the MSS size in the 3way handshake.
You client sets mss 1460 (LAN)
06:11:46.357659 IP 192.168.19.30.55704 > 192.168.169.5.445: Flags [SEW], seq 972300290, win 29200, options [mss 1460,sackOK,TS val 151542601 ecr 0,nop,wscale 5], length 0
The server replies with 1460.
06:11:46.357910 IP 192.168.169.5.445 > 192.168.19.30.55704: Flags [S.E], seq 62474480, ack 972300291, win 8192, options [mss 1460,nop,wscale 8,sackOK,TS val 3002803536 ecr 151542601], length 0
If the ASA modifies this value you'll see a decremented number.
-
Cisco ASA 55XX by default always transparently fix MSS to 1380 (norm MSS=1460)
on all transiting TCP connections on all interfaces and all RA/S2S VPNs.
This default setting is not visible in config.
But visible via ASDM:
(https://c.radikal.ru/c22/1801/02/c20dea36d543.jpg)
-
I have some experiments with OpenVPN,
and i see something like ASA:
OpenVPN also defaultly transparently fixing MSS of all transiting TCP-traffic,
but fixing to MSS=1410.
-
Today i see by own eyes: opnsense reduces MSS by 100 (to 1360).
Traffic capture on webserser side:
22:14:40.305764 30:e4:db:xx:xx:xx > 52:54:00:00:xx:xx, ethertype IPv4 (0x0800), length 66: opn-s-ip.16825 > web-server-ip.80: Flags [ S ], seq 4280183093, win 8192, options [mss 1360,nop,wscale 2,nop,nop,sackOK], length 0
22:14:40.305855 52:54:00:00:xx:xx > 30:e4:db:xx:xx:xx, ethertype IPv4 (0x0800), length 66: web-server-ip.80 > opn-s-ip.16825: Flags [ S. ], seq 3520754639, ack 4280183094, win 5840, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
-
Hi all
We have also some DSL <-> Fiber VPN's (both side connected by pppoe via a bridge/converter to the provider) and we need to reduce the MSS to 1380 on the LAN side. Without, RDP is useless.
gruss ivo