Slow FTPS upload

[Clumpton]

Hi,
Bit of a noob here so please be gentle.

OPNsense 24.7.10_2-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15

Running on an Intel N100 mini PC with 4x i266-V, 8GB DDR4, 250GB NVMe Gen 3x4.
igc0 connected to a 2.5Gb/s fibre modem/router in bridge mode (French Freebox Ultra)
igc1-3 bridged for LAN

I am connecting to ftp servers on a couple of remote dedicated servers (Scaleway), both running Windows Server in an ESXi VM. One server is Win 2012 IIS, the other Filezilla under Win 2019. Both servers have 1000Mb/s connections.
Both servers have a range of 1000 open ports defined and opened in Windows FW.

FTP and FTPS downloads from either server are rapid, generally rising to 100 MB/s.
Uploads to the servers are dire, maxing out at 5 MB/s, however I can do  several concurrent uploads to a combined limit of about 25MB/s.

Behaviour is the same whatever the client software (SmartFTP, Filezilla, lftp).
iperf3 shows symmetrical normal speeds between servers and home PC
Ookla shows symmetrical normal speeds on the client and servers.
Debian cli ftp shows the same poor upload speeds.

Here is a typical example, connection from local Debian to distant Filezilla on Windows 2019:

Code Select Expand

lftp Me@www.xxx.yyy.zzz:/> put "The Flint Street Nativity (1999)-1.mp4"
723355399 bytes transferred in 152 seconds (4.54 MiB/s)
lftp Me@www.xxx.yyy.zzz:/> !rm "The Flint Street Nativity (1999)-1.mp4"
lftp Me@www.xxx.yyy.zzz:/> get "The Flint Street Nativity (1999)-1.mp4"
723355399 bytes transferred in 6 seconds (108.66 MiB/s)

Here are the corresponding Filezilla server log entries, nothing exceptional as far as I can see:
UPLOAD

Code Select Expand

<Date/Time> Info [Type] Message
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] FEAT
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 211-Features:
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 211 End
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] AUTH TLS
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 234 Using authentication type TLS.
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] OPTS UTF8 ON
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 202 UTF8 mode is always enabled. No need to send this command
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] USER Me
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Response] 331 Please, specify the password.
<11/12/2024 16:49:12> FTP Session 41 <IP Addr hidden> [Command] PASS ****
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> [Response] 230 Login successful.
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Command] PBSZ 0
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Response] 200 PBSZ=0
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Command] PROT P
<11/12/2024 16:49:13> FTP Session 41 <IP Addr hidden> Me [Response] 200 Protection level set to P
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] TYPE I
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 200 Type set to I
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] PASV
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 227 Entering Passive Mode (nnn,nnn,nnn,nnn,30,216)
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Command] STOR The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:49:17> FTP Session 41 <IP Addr hidden> Me [Response] 150 Starting data transfer.
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Response] 226 Operation successful
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Command] MFMT 20241209201305 The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:51:22> FTP Session 41 <IP Addr hidden> Me [Response] 213 modify=20241209201305.000; /The Flint Street Nativity (1999)-1.mp4


DOWNLOAD

Code Select Expand

<Date/Time> Info [Type] Message
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] FEAT
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 211-Features:
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 211 End
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] AUTH TLS
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 234 Using authentication type TLS.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] OPTS UTF8 ON
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 202 UTF8 mode is always enabled. No need to send this command
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] USER Me
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 331 Please, specify the password.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Command] PASS ****
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> [Response] 230 Login successful.
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PBSZ 0
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 PBSZ=0
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PROT P
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 Protection level set to P
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] TYPE I
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 200 Type set to I
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] SIZE The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 213 723355399
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] MDTM The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 213 20241209201305.000
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] PASV
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 227 Entering Passive Mode (nnn,nnn,nnn,nnn,28,212)
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Command] RETR The Flint Street Nativity (1999)-1.mp4
<11/12/2024 16:42:37> FTP Session 40 <IP Addr hidden> Me [Response] 150 Starting data transfer.
<11/12/2024 16:42:45> FTP Session 40 <IP Addr hidden> Me [Response] 226 Operation successful


I am fairly sure (but not 100%) that the Windows servers or ESXi are not at fault since 'normal' speed tests seem to show good symmetric transfers.
Similarly FTP clients on Windows and Debian seem to show the same problem, but speed tests are good.
ntttcp test is also good.
The only elements left are OPNSense or my ISP's modem/router. I'm fairly sure that if the modem was  throttling FTPS uploads we would have heard of it by now.
Looking at OPNSense's cpu usage during transfers gives no clues, it's about 8% for upload and download.
I am not at all at ease with firewall rules and monitoring. I've reached my diagnostic limits here. Has anybody come across a similar problem or have an idea of what to look at next, please?
Many thanks in advance.

[Patrick M. Hausen]

FTP and FTPS are horrible antique protocols that are supposed to die. Seriously. Can you use SFTP instead?

Long version:

FTP and FTPS open a so called control connection from the client to the server and then for each file transfer but also e.g. the output of a dir command a separate data connection. By default this one goes from the server back to the client from server port 20 to a random port on the client. (which one is negotiated in the control connection)

That makes it notoriously difficult to firewall and NAT.

If your client is behind a NAT gateway, e.g. your OPNsense, but the server is not (!) then try so called passive mode. This reverses the direction in which the data connection is established so it plays well with NAT on the client side.

If both are behind NAT, you are kind of screwed  ;)

If you can use neither passive mode nor SFTP (which runs over a single SSH connection) you can also use a proxy for FTP(S). I have no experience configuring one and whether it plays nicely with FTPS, which is encrypted, so you cannot "snoop" on those dynamically negotiated ports.

HTH,
Patrick

[Clumpton]

Thanks for the reply.
I'll try SFTP, but the servers are used by friends for whom anything more complex than a user name and password is a bit frightening. However if SFTP solves the problem I'll drag them into the 21st century.

[Clumpton]

UPDATE
Using OpenSSH on the server and WinSCP on the client SFTP gives 48MB/s download (about 50% of FTPS) and 6-10MB/s upload (an improvement, but far from what it should be).

[meyergru]

When both of those protocols are slow upstream, it seems like the BDP is high on your connection. You could use traffic shaping or try disabling "Optimize connection buffer size" there. This is explained here.

[Clumpton]

When both of those protocols are slow upstream, it seems like the BDP is high on your connection. You could use traffic shaping or try disabling "Optimize connection buffer size" there. This is explained here.

Thanks for these pointers. Changing the encryption algo was the only one that made a difference though, doubled upload throughput by moving to Blowfish. This could mean that there is a problem on the client side. I will look into traffic shaping, but I'm not hopeful.