Openconnect throughput

[mimugmail]

You really should test with OPN .. FreeBSD doesnt support metrics, and Interface renaming in Openconnect is only available since 0.12 ;) I'm also on the Openconnect mailing list :)

[Jeroen1000]

I got my Qotom mini-PC today so I'll be testing pretty soon time permitting. I got it working as I want in Ubuntu but let's what I can do with BSD:-)!

[Jeroen1000]

So I'm back from holiday so it is time for an update.

Step 1:
- I've installed Opnsense on my Qotom mini-pc.
- I can surf the Internet using a standard home NAT setup

Topology:

Code Select Expand

Qotom WAN (igb0 DHCP client) ---BRIDGED_---> Mikrotik routerboard WAN ---> cable modem 
Qotom LAN (igb1 192.168.200.250/24)  ------> Netgear switch ----> LAN clients

The Mikrotik is invisible for the Qotom box because of the bridge. So the Opnsense WAN igb0 has a public IP on it.

Step 2: openconnect

- Plugin is installed
- VPN is up

Code Select Expand

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.65.X.X      UGS      ocvpn0
10.65.0.0/16   10.65.X.X       UGS      ocvpn0
10.65.X.X       link#9        UH       ocvpn0
- Outbond NAT rule to masquerade traffic to the VPN interface ocvpn0 has been made and is hit

Code Select Expand

root@OPNsense:~ # pfctl -v -s nat
No ALTQ support in kernel
ALTQ related functions disabled
nat log on ocvpn inet from 192.168.200.0/24 to any -> (ocvpn:0) port 1024:65535 round-robin
  [ Evaluations: 468       Packets: 709       Bytes: 106489      States: 0     ]
  [ Inserted: uid 0 pid 2448 State Creations: 62    ]

So far I can ping a random host from the Opnsense box itself with the traffic going over the vpn as intended.
Hosts on the LAN, however, are unable to do this. So there's something still iffy with either routing or firewall.
Once I find what's wrong, I'll focus on getting multiple vpn instances running and routing traffic over those based on source IP and/or TOS marking.

[mimugmail]

Can you post a screenshot of your NAT rule?
For me this setup works fine ...

[Jeroen1000]

Screenshot in attach. It appears ICMP is working fine contrary to what I said in my previous post.  But HTTPS and HTTP are not working hmm. Although that doesn't make sense to me just yet.

[mimugmail]

Transparent Proxy?

[Jeroen1000]

I'll probably have to sniff the traffic to see what is happening. This should not be a proxy issue. VPN is working fine if the tunnel is setup on a Linux box so it has to be Opnsense specific. But just to rule out my Mikrotik router, I'll plug the WAN port directly into the cable modem.

[Jeroen1000]

Transparent Proxy?

I believe it might be the MTU. It is set to 1322 for the vpn interface.

Code Select Expand

ocvpn0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1322
So I started a tunnel myself via the shell

Code Select Expand

tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1322
It has the same MTU. When the ICMP goes payload goes beyond 1294 bytes (+ 8 for ICMP + 20 for IP header makes 1322) I get this message

Code Select Expand

LZS decompression failed: File too large


So the max. bytes I can get through for ICMP is 1294 (+ 20 for IP header + 8 for ICMP makes 1322). Any higher and I can see this error:

Code Select Expand

LZS decompression failed: File too large

On my MAC the MTU for the interface is 1340 so maybe that is what the server is using and I'm dealing with some MTU mismatch...

[mimugmail]

Then you have to lower the MSS in LANfor TCP and rwgarding ICMP just dont enable DF bit

[Jeroen1000]

But it is DTLS (udp) so tcp mss probably does not apply?
I sniffed the connection in the meanwhile. The TCP 3-way handshake completes with destination host 213.239.154.31. But then it seems my 192.168.200.50 source host receives a packet it was not expecting. So it sends a duplicate ack to tell this to 213.239.154.31. This happens twice before 213.239.154.31 closes the connection.

So there is most likely something that is causing packet loss (MTU still being my prime suspect). I did this test with the Opnsense box directly connected to the cable modem. All I need to do is find the cause now ::)

[Jeroen1000]

Alright, I'm getting somewhere near the cause.

I manually loaded the NAT rule via console and manually started a tunnel with this option

Code Select Expand

--no-deflate

This at least caused this error to go away

Code Select Expand

LZS decompression failed: File too large. Now I am able to browse the web. The speed of the tunnel has taken quite a large hit because compression is now completely disabled.

MTU related debug info with compression disabled:

Code Select Expand

X-DTLS-CipherSuite: PSK-NEGOTIATE
X-CSTP-Base-MTU: 1406
X-CSTP-MTU: 1340
DTLS option X-DTLS-DPD : 90
DTLS option X-DTLS-Port : 22
DTLS option X-DTLS-Rekey-Time : 172838
DTLS option X-DTLS-Rekey-Method : ssl
DTLS MTU reduced to 1322
Established DTLS connection (using OpenSSL). Ciphersuite PSK-AES256-CBC-SHA.
Initiating IPv4 MTU detection (min=661, max=1322)
No change in MTU after detection (was 1322)

So the issue lies with the LZS decompression. Does this give you an idea as to how this can be fixed?

[mimugmail]

If you find a config option to disable I can integrate it

[Jeroen1000]

The option I mentioned disables it but it's not a good idea as the loss of speed is very severe. I'll leave a line on the mailing list to ask what the cause might be.

[Jeroen1000]

If you find a config option to disable I can integrate it

I can confirm now there is an MTU-mismatch. On Opnsense I have 1322 bytes but the remote side has 1340. There are workarounds but I still need to test them. And also this might be a bug that needs fixing but I'm asking this on the openconnect mailing list. Also I believe 1322 is incorrect. I see no reason why it settles on that?

May I ask what your interface MTU is when you connect to your server? It says what it negotiates too of you start it with -v for more debug info.

[mimugmail]

ocvpn0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1406
        options=80000<LINKSTATE>
        inet6 fe80::a00:27ff:febf:2658%ocvpn0 prefixlen 64 scopeid 0x9
        inet 10.24.69.165 --> 10.24.69.165  netmask 0xffffffff
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: tun ocvpn
        Opened by PID 81777

This is the value my ASA assigns to the client ...