Network performance and latency issues with haproxy

[deviantintegral]

First, I'm quite impressed by the HAProxy and Let's Encrypt plugins, and how they work together.

I'm running opnsense on an APU2 board, and I'm noticing:

• Slow transfer rates with high CPU usage
• Latency spikes for unrelated network traffic when HAProxy is under load

For example, the backend web server (a basic Apache setup) can saturate the 1Gb network connection with no issues. If I do a straight HTTP proxy through HAProxy in opnsense (no SSL offloading), performance caps out at about 200Mbit. I expected closer to 300 or 400 Mbits. During a transfer, the HAProxy process is using close to 100% of a CPU core (the board has 4 cores) with 50% system CPU usage. If I ping a host during the speed test from opnsense, latency goes up significantly for all connections going through the firewall.

So far, I noticed that at http://www.haproxy.org it's mentioned that pf causes quite a performance hit compared to Linux, but it's not clear how recent that note is.

Any ideas on how to solve the latency issues? I can live with slower transfer performance, but lag spikes will be a deal breaker for me.

[fraenki]

Could you post the contents of /usr/local/etc/haproxy.conf? This might help to find the root cause for the high CPU load.

Although I doubt it's something in haproxy.conf, because 50% system CPU usage indicate that it's a different issue.


Regards
- Frank

[deviantintegral]

Here it is. redacted_http_test is the no-ssl backend I've been testing with.

Code Select Expand

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    # NOTE: Could be a security issue, but required for some feature.
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket level admin
    nbproc                      1
    nbthread                    1
    tune.ssl.default-dh-param   1024
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    # WARNING: pass through options below this line
    mode http
    option httplog

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Frontend: redacted_ssl ()
frontend redacted_ssl
    bind *:443 name *:443 ssl  crt-list /tmp/haproxy/ssl/5cc4fcfb4d50d1.96664841.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Frontend: letsencrypt ()
frontend letsencrypt
    bind *:80 name *:80
    mode http
    option http-keep-alive
    default_backend acme_challenge_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: redacted_http_test ()
frontend redacted_http_test
    bind *:8080 name *:8080
    mode http
    option http-keep-alive
    default_backend server1_apache
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server acme_challenge_host 127.0.0.1:43580

# Backend: server1_apache ()
backend server1_apache
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server server1_apache backend.lan:80

# Backend: backend_grafana ()
backend backend_grafana
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server backend_grafana grafana.lan:80