OPNsense Forum

Archive => 19.1 Legacy Series => Topic started by: deviantintegral on April 28, 2019, 05:44:21 pm

Title: Network performance and latency issues with haproxy
Post by: deviantintegral on April 28, 2019, 05:44:21 pm
First, I'm quite impressed by the HAProxy and Let's Encrypt plugins, and how they work together.

I'm running opnsense on an APU2 board, and I'm noticing:


For example, the backend web server (a basic Apache setup) can saturate the 1Gb network connection with no issues. If I do a straight HTTP proxy through HAProxy in opnsense (no SSL offloading), performance caps out at about 200Mbit. I expected closer to 300 or 400 Mbits. During a transfer, the HAProxy process is using close to 100% of a CPU core (the board has 4 cores) with 50% system CPU usage. If I ping a host during the speed test from opnsense, latency goes up significantly for all connections going through the firewall.

So far, I noticed that at http://www.haproxy.org it's mentioned that pf causes quite a performance hit compared to Linux, but it's not clear how recent that note is.

Any ideas on how to solve the latency issues? I can live with slower transfer performance, but lag spikes will be a deal breaker for me.
Title: Re: Network performance and latency issues with haproxy
Post by: fraenki on April 30, 2019, 01:23:12 pm
Could you post the contents of /usr/local/etc/haproxy.conf? This might help to find the root cause for the high CPU load.

Although I doubt it's something in haproxy.conf, because 50% system CPU usage indicate that it's a different issue.


Regards
- Frank
Title: Re: Network performance and latency issues with haproxy
Post by: deviantintegral on April 30, 2019, 10:47:30 pm
Here it is. redacted_http_test is the no-ssl backend I've been testing with.

Code: [Select]
#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    # NOTE: Could be a security issue, but required for some feature.
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket level admin
    nbproc                      1
    nbthread                    1
    tune.ssl.default-dh-param   1024
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log /var/run/log local0

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    # WARNING: pass through options below this line
    mode http
    option httplog

# autogenerated entries for ACLs

# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Frontend: redacted_ssl ()
frontend redacted_ssl
    bind *:443 name *:443 ssl  crt-list /tmp/haproxy/ssl/5cc4fcfb4d50d1.96664841.certlist
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Frontend: letsencrypt ()
frontend letsencrypt
    bind *:80 name *:80
    mode http
    option http-keep-alive
    default_backend acme_challenge_backend
    # tuning options
    timeout client 30s

    # logging options

# Frontend: redacted_http_test ()
frontend redacted_http_test
    bind *:8080 name *:8080
    mode http
    option http-keep-alive
    default_backend server1_apache
    option forwardfor
    # tuning options
    timeout client 30s

    # logging options
    # ACL: host_redacted
    acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted

    # ACTION: map_redacted
    use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server acme_challenge_host 127.0.0.1:43580

# Backend: server1_apache ()
backend server1_apache
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server server1_apache backend.lan:80

# Backend: backend_grafana ()
backend backend_grafana
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse never
    server backend_grafana grafana.lan:80