Weird SSH performance with NAT vs HAProxy

[sarbian]

I have a performance problem that I do not understand.

On my home network I have a Linux server with an SSH server that listens on port 1222 to serve file with sftp (with cipher setup for speed).
On OPNsense I configured a NAT that redirect the port 1222 to the Linux server port 1222. I also configured a TCP HAProxy that listens on port 1223 and redirect the traffic to the same Linux server 1222.

If I download a file using the NAT on 1222 I get terrible performance (1.1MB/s).
If I download the same file using the TCP HAProxy on 1223 I get the expected performance (~40MB/s).

Performance with other protocols and NAT is fine. An iperf going through the NAT give the same ~40MB/s as HAProxy. The CPU is an Intel J3160 (Quad core).