HAproxy not starting after upgrading firmware

[hv-tech]

Hi Forum,

If I was more technical I wouldn't post, but after upgrading from 22.7.7 to 22.7.8, since I've upgraded to 22.7.9 without fixing the problem. Here is the output for when I manually try to start the service;

root@ctgwfw01:~ # service haproxy restart
haproxy not running? (check /var/run/haproxy.pid).
Starting haproxy.
[ALERT]    (21092) : Starting frontend External-Pub: cannot bind socket (Can't assign requested address) [72.10.1.x:443]
[ALERT]    (21092) : Starting frontend External-Pub: cannot bind socket (Can't assign requested address) [72.10.2.x:443]
[ALERT]    (21092) : [/usr/local/sbin/haproxy.main()] Some protocols failed to start their listeners! Exiting.
/usr/local/etc/rc.d/haproxy: WARNING: failed to start haproxy
root@ctgwfw01:~ #


Note I have a PPPoe IP from my ISP. So each time I reboot, it seems to be binding to the old IP that is no longer being used. I am not sure if there is a cache I need to wipe out?


Any help would be great
Thanks

[hv-tech]

Anyone has any advise as to what I can do to fix this?

[cookiemonster]

how are you binding to the wan ip? Normally you would bind to 0.0.0.0:port if you have dynamic public ips. 127.0.0.1:port is also a possibility.

[hv-tech]

So I am using DDNS/Cloudflare and am binding to those DNS entries that are pointing to my PPPoe address assigned by the ISP which always worked flawlessly until the upgrade.

[cookiemonster]

I mean how are you binding it in your haproxy configuration in OPN.

[hv-tech]

Here is my config.

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin expose-fd listeners
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         172.16.10.6:514 local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats


# Resolver: HV-DNS
resolvers 60d520816d7b32.78243365
    nameserver 8.8.8.8:53 8.8.8.8:53
    parse-resolv-conf
    resolve_retries 3
    timeout resolve 1s
    timeout retry 1s



# Frontend: External-Pub ()
frontend External-Pub
    bind ctlgmon01.hvnoclabs.com:443 name ctlgmon01.hvnoclabs.com:443 ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6121ccbe699ab8.48952667.certlist
    bind ctauth02.hvnoclabs.com:443 name ctauth02.hvnoclabs.com:443 ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6121ccbe699ab8.48952667.certlist
    bind ctitools01.hvnoclabs.com:443 name ctitools01.hvnoclabs.com:443 ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6121ccbe699ab8.48952667.certlist
    bind ctlgmon02.hvnoclabs.com:443 name ctlgmon02.hvnoclabs.com:443 ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6121ccbe699ab8.48952667.certlist
    bind ctcoms01.hvnoclabs.com:443 name ctcoms01.hvnoclabs.com:443 ssl prefer-client-ciphers ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256 ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/6121ccbe699ab8.48952667.certlist
    mode http
    option http-keep-alive
    # tuning options
    timeout client 30s
    # stickiness
    stick-table type ip size 50k expire 30m 
    tcp-request connection track-sc0 src
    # logging options
    option httplog
    # ACL: Netbox
    acl acl_60dea475186677.51330295 hdr(host) -i ctitools01.hvnoclabs.com
    # ACL: Graylog
    acl acl_61208941d9bf35.04710772 hdr(host) -i ctlgmon01.hvnoclabs.com
    # ACL: Keycloak
    acl acl_61209978a36e65.49477166 hdr(host) -i ctauth02.hvnoclabs.com
    # ACL: Mattermost
    acl acl_612d2c6c0e9208.90351294 hdr(host) -i ctcoms01.hvnoclabs.com

    # ACTION: Netbox
    use_backend External-Netbox if acl_60dea475186677.51330295
    # ACTION: Graylog
    use_backend External-Graylog if acl_61208941d9bf35.04710772
    # ACTION: Keycloak
    use_backend External-Keycloak if acl_61209978a36e65.49477166
    # ACTION: Zabbix
    # NOTE: actions with no ACLs/conditions will always match
    use_backend External-Zabbix
    # ACTION: Mattermost
    use_backend External-Mattermost if acl_612d2c6c0e9208.90351294

# Backend: External-Netbox (Pool to Internet)
backend External-Netbox
    option log-health-checks
    # health check: Monitoring Profile
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctitools01 172.16.10.11:80 check inter 2s

# Backend: External-Graylog (Pool to Internet)
backend External-Graylog
    option log-health-checks
    # health check: Monitoring Profile
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctlgmon01 172.16.10.8:443 check inter 2s  ssl verify none

# Backend: External-Keycloak (Pool to Internet)
backend External-Keycloak
    option log-health-checks
    # health check: Monitoring Profile
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctauth02 172.16.10.25:443 check inter 2s  ssl alpn h2,http/1.1 verify none

# Backend: External-Zabbix (Pool to Internet)
backend External-Zabbix
    option log-health-checks
    # health check: Monitoring Profile
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctlgmon02 172.16.10.9:80 check inter 2s

# Backend: External-Mattermost (Pool to Internet)
backend External-Mattermost
    option log-health-checks
    # health check: Monitoring Profile
    option httpchk
    http-check send meth OPTIONS uri / ver HTTP/1.0
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctcoms01 172.16.10.24:80 check inter 2s

# Backend: External-ctcoms01 (Pool to Internet)
backend External-ctcoms01
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server ctcoms01 172.16.10.75:443




listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

[cookiemonster]

I thought so. You are binding your front end to a name and that needs to be resolved, and after a reboot the ip might have changed. So I think your suspicion is correct, it's in a loop: your public ip has changed, the public dns record hasn't been updated, haproxy queries the name and gets the old one back.
A long delay might be an unworkable workaround but I can't explain why it was working before the upgrades. All versions of OPN wouldn't deal with this.
I would investigate the option needed to bind to your interface ip, as I said earlier 0.0.0.0:port "should" work but you'll need to test.

[hv-tech]

You are 100% correct, I guess I didn't understand before. So removing all other entries and adding a external binding of 0.0.0.0/24 worked. Thanks so much for the help!