First, I'm quite impressed by the HAProxy and Let's Encrypt plugins, and how they work together.
I'm running opnsense on an APU2 board, and I'm noticing:
- Slow transfer rates with high CPU usage
- Latency spikes for unrelated network traffic when HAProxy is under load
For example, the backend web server (a basic Apache setup) can saturate the 1Gb network connection with no issues. If I do a straight HTTP proxy through HAProxy in opnsense (no SSL offloading), performance caps out at about 200Mbit. I expected closer to 300 or 400 Mbits. During a transfer, the HAProxy process is using close to 100% of a CPU core (the board has 4 cores) with 50% system CPU usage. If I ping a host during the speed test from opnsense, latency goes up significantly for all connections going through the firewall.
So far, I noticed that at http://www.haproxy.org it's mentioned that pf causes quite a performance hit compared to Linux, but it's not clear how recent that note is.
Any ideas on how to solve the latency issues? I can live with slower transfer performance, but lag spikes will be a deal breaker for me.
Could you post the contents of /usr/local/etc/haproxy.conf? This might help to find the root cause for the high CPU load.
Although I doubt it's something in haproxy.conf, because 50% system CPU usage indicate that it's a different issue.
Regards
- Frank
Here it is. redacted_http_test is the no-ssl backend I've been testing with.
#
# Automatically generated configuration.
# Do not edit this file manually.
#
global
# NOTE: Could be a security issue, but required for some feature.
uid 80
gid 80
chroot /var/haproxy
daemon
stats socket /var/run/haproxy.socket level admin
nbproc 1
nbthread 1
tune.ssl.default-dh-param 1024
spread-checks 0
tune.chksize 16384
tune.bufsize 16384
tune.lua.maxmem 0
log /var/run/log local0
defaults
log global
option redispatch -1
timeout client 30s
timeout connect 30s
timeout server 30s
retries 3
# WARNING: pass through options below this line
mode http
option httplog
# autogenerated entries for ACLs
# autogenerated entries for config in backends/frontends
# autogenerated entries for stats
# Frontend: redacted_ssl ()
frontend redacted_ssl
bind *:443 name *:443 ssl crt-list /tmp/haproxy/ssl/5cc4fcfb4d50d1.96664841.certlist
mode http
option http-keep-alive
option forwardfor
# tuning options
timeout client 30s
# logging options
# ACL: host_redacted
acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted
# ACTION: map_redacted
use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346
# Frontend: letsencrypt ()
frontend letsencrypt
bind *:80 name *:80
mode http
option http-keep-alive
default_backend acme_challenge_backend
# tuning options
timeout client 30s
# logging options
# Frontend: redacted_http_test ()
frontend redacted_http_test
bind *:8080 name *:8080
mode http
option http-keep-alive
default_backend server1_apache
option forwardfor
# tuning options
timeout client 30s
# logging options
# ACL: host_redacted
acl acl_5cc4fe5883d959.03367346 hdr_end(host) -i redacted
# ACTION: map_redacted
use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/5cc509feaa78f3.47022982.txt)] if acl_5cc4fe5883d959.03367346
# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m
stick on src
# tuning options
timeout connect 30s
timeout server 30s
http-reuse never
server acme_challenge_host 127.0.0.1:43580
# Backend: server1_apache ()
backend server1_apache
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m
stick on src
# tuning options
timeout connect 30s
timeout server 30s
http-reuse never
server server1_apache backend.lan:80
# Backend: backend_grafana ()
backend backend_grafana
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m
stick on src
# tuning options
timeout connect 30s
timeout server 30s
http-reuse never
server backend_grafana grafana.lan:80