Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

[meyergru]

I have a question about HAproxy SSL performance with large downloads:

Using a NAT port forward to an internal HTTPS nginx server, I get full wire speed i.e. (45 MByte/s) from the outside, but using HAproxy following this tutorial, I am limited to download speeds of ~4-5 MByte/s.

I have tried to use HTTP instead of HTTPs for the backend server connection to no avail.

I suspected that the SSL termination would lead to an interrupt of the streaming flow in that each packet must be acknowledged over the whole chain before the next one can get requested, so I have somewhat optimized my throughput by a factor of 2 via this tuning:

Code: [Select]

    tune.h2.max-frame-size 4048576
    tune.pipesize 4048576
    tune.h2.initial-window-size       1048576
    tune.h2.be.initial-window-size    1048576
    tune.h2.fe.initial-window-size    1048576

But it seems that the HTTPS termination within HAproxy itself is the culprit. I found some other source that suggest there might be a problem with only some ISPs showing that behaviour when SSL traffic is terminated by HAproxy.

Does everybody else see this or may my ISP really be the culprit?

[oLeDfrEeZe]

Yes, I can confirm this, I also have performance issues with HAProxy and Plex. When I use NAT port forward then I get full speed, but when I use HAProxy then my transfer rate is also limited to only ~1Mb/s

[TheHellSite]

I have a question about HAproxy SSL performance with large downloads:

Using a NAT port forward to an internal HTTPS nginx server, I get full wire speed i.e. (45 MByte/s) from the outside, but using HAproxy following this tutorial, I am limited to download speeds of ~4-5 MByte/s.

I have tried to use HTTP instead of HTTPs for the backend server connection to no avail.

I suspected that the SSL termination would lead to an interrupt of the streaming flow in that each packet must be acknowledged over the whole chain before the next one can get requested, so I have somewhat optimized my throughput by a factor of 2 via this tuning:

Code: [Select]

    tune.h2.max-frame-size 4048576
    tune.pipesize 4048576
    tune.h2.initial-window-size       1048576
    tune.h2.be.initial-window-size    1048576
    tune.h2.fe.initial-window-size    1048576

But it seems that the HTTPS termination within HAproxy itself is the culprit. I found some other source that suggest there might be a problem with only some ISPs showing that behaviour when SSL traffic is terminated by HAproxy.

Does everybody else see this or may my ISP really be the culprit?

No issues on my side here.

VDSL 100/40 MBit/s on HAProxy side.
VDSL 250/40 MBit/s on Client side.

Client side downloads files from HAProxy side using File Browser with max speed of 40 MBit/s.
This is using a direct connection between two parties and no peering service (f.e. Cloudflare) involved.

[oLeDfrEeZe]

That's strange... Do you have OPNsense version 24.1.6 installed?

[TheHellSite]

That's strange... Do you have OPNsense version 24.1.6 installed?

Yes.

[meyergru]

I also use direct peering, no tunnels or anything. FWIW, the issue does not happen when I access the port from inside, i.e. without any delays. So it is not a general bottleneck. I see it happen only when I access the service from outside, with different counterparts.

I also tried a packet trace, and my TCP window size gets reduced to < 400 bytes when the problem occurs. From what I have read about HAproxy, it controls TCP directly in order to stream the data between frontend and backend, be able to fail-over (hence "HA") and things like that. Over the years, there have been discussions on subtle problems like this over and over. I get the impression that HAproxy is a very complex piece of software.

I have tried many parameters to no avail, but it seems fruitless considering that there are so many tuning parameters for both frontend and backend (i.e. buffers sizes and such). My connections seem to use HTTP/1.1, so it seems it is not a HTTP/2 or QUIC issue.

Other mentions of the problem also suggested that this is not a FreeBSD or OpnSense problem, since it occurs on Linux as well.

[oLeDfrEeZe]

I also use direct peering, no tunnels or anything. FWIW, the issue does not happen when I access the port from inside, i.e. without any delays. So it is not a general bottleneck. I see it happen only when I access the service from outside, with different counterparts.

Completely the same for me. I've set up an opnsense with haproxy before and with this tutorial, which ran for months without any problems. Now I have reinstalled the opnsense and suddenly have this problem.

Greetings

[meyergru]

Wow. Seems the other reports are correct in stating that the problem surfaces only with some ISPs:

My own OpnSense is on M-Net with a 1000/400 GPON connection and it exhibits the problem (i.e. 40 MBits upstream over HAproxy).
I just tested with Deutsche Glasfaser (400/200 GPON) and got the full expected 200 Mbit/s upstream on an otherwise similar setup.

For me, I have only found this slowness a short while ago. I am wondering if this could be caused by the haproxy 2.6 -> 2.8 upgrade?

P.S.: Well. I just tried to downgrade haproxy to 2.6, only to find that one of its prerequisites is OpenSSL 1.1.1. Mind you, now we are at OpenSSL 3.0 in OpnSense, so I could not downgrade.

Googling around, I found this: https://www.mail-archive.com/haproxy@formilux.org/msg43600.html, and I quote:

  - OpenSSL: version 3.1 is now supported. It's less slow than 3.0 but still
    significantly slower than 1.1.1, but might be usable for most users with
    a low enough traffic.

However, this seems to be a problem only for high-load websites with many clients. Also, this would not explain why the problem surfaces only with certain ISPs.

[Underpay6703]

Has anyone tried this HAProxy setup in combination with CARP?
The short of my problem is that my Backup node refuses to start the HAProxy service after syncing the settings to it. The error I receive when trying to start it from the shell is a mere "Haproxy failed to start" without additional information.

Clearing the configuration from the Backup will make HAProxy run again, I thought of meticulously adding each setting one by one as it is on the Master until I see where it fails, but it's become quite the config.

Are there things that should be configured differently per system with HAProxy on CARP?
I should note that all other (CARP) functions works as intended.

EDIT: Fixed it,
Running haproxy -d -f /usr/local/etc/haproxy.conf on the shell revealed that my SNI couldn't listen on port 80 because my GUI still allowed connections to HTTP through port 80.
I disabled HTTP GUI in System: Settings: Administration and checking "Disable web GUI redirect rule"

[meyergru]

I found out something more about the slowness with certain ISPs: Trying out Caddy instead of HAproxy, it gave the same speed, which got me thinking.

Then, I put a big file into /usr/local/www, opened up my OpnSense interface for direct external access and guess what? Same (slow) speed as Caddy or HAproxy. So it must be something in the FreeBSD TCP parameters that is the culprit, because if the TCP connection is controlled directly by a Linux client via a port forward, speed is OK.

I will dig into this later...

[oLeDfrEeZe]

@meyergru

Thanks for the effort, I wish I could help you more.

[meyergru]

I found it: I had traffic shaping activated.

Once I disabled that, everything works as expected. I still do not get why this happens only with traffic originating from the OpnSense machine itself, but whatever. If others are experiencing this, it might be caused by external traffic shaping by your provider which may also explain why it sometimes happens and sometimes not.

I also found that it was indeed HTTP/2 traffic and that enlarging the H2 initial window sizes helps a bit.

[dMopp]

Thanks for the great tutorial.

Is there a way to exclude the HTTPS force for specific Backends? (Based on the tutorial here). Background: For HomeAssistant and stupid IOT devices, i need to have my HA instance reachable over http, too (with a different domain at least so i can firewall it a lot )

[MadTizz]

Hey there and thank you so, so much for this great tutorial! It gave me exactly what I needed!

Yet there is a reason why I'm quoting this particular post.

Configuration made basing on your tutorial was working flawlessly on version 23.7.1 (os-haproxy 4.0, haproxy26 2.6.14), but after update to 23.7.2 and haproxy26 2.6.15 HAProxy service was failing to start.

I followed sorano's suggestion to not use virtual ip and bingo! That was it (it took me hours to find out where the issue is, as there were no message in logs - just a startup failure of HAProxy).

Maybe it would be good to add adnotation or a second way to configure HTTPS_frontend?

I can confirm that it works flawlessly with dynamic WAN ip.

Once again thank you very much and @sorano too

Cheers
Paweł

I did not see a reply to this so I will post my experience.

I think I had the same issue: HAProxy fail to start whenever I enabled HTTP and HTTPS_frontend with private IP config.
Turns out that I had selected "Other" in the Private IP type config instead of IP Alias as per the tutorial.

As always the tutorial is correct and working.

[dMopp]

Thanks for the great tutorial.

Is there a way to exclude the HTTPS force for specific Backends? (Based on the tutorial here). Background: For HomeAssistant and stupid IOT devices, i need to have my HA instance reachable over http, too (with a different domain at least so i can firewall it a lot )

Ping