OPNsense Forum

Archive => 21.7 Legacy Series => Topic started by: Rene1 on October 31, 2021, 03:23:21 PM

Title: [solved] HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list'
Post by: Rene1 on October 31, 2021, 03:23:21 PM
[solved]

Hello,

HAProxy won't start after upgrade OPNsense to 21.7 today.

# service haproxy restart
haproxy not running? (check /var/run/haproxy.pid).
[ALERT] 303/151127 (81180) : parsing [/usr/local/etc/haproxy.conf:46] : 'bind x.x.x.x:8448' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/151127 (81180) : parsing [/usr/local/etc/haproxy.conf:47] : 'bind x.x.x.x:443' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/151127 (81180) : Error(s) found in configuration file : /usr/local/etc/haproxy.conf
[ALERT] 303/151127 (81180) : Fatal errors found in configuration.
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy

# ls /tmp/haproxy/ssl/
it's empty


HAProxy stopped working after upgrade and reboot.  Letsencrypt certificate with acme client is still valid.
Didn't change the config in the last weeks.

Can you please help me?

Thanks,
Rene

Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: benyamin on October 31, 2021, 03:32:17 PM
Hi Rene,

This topic (https://forum.opnsense.org/index.php?topic=25337.0) might be related.

The fix is documented here (https://github.com/opnsense/plugins/issues/2616).

Namely:
opnsense-patch -c plugins 31b82cd 18cd9f6

FWIW, it was merged into master here (https://github.com/opnsense/plugins/commit/b1953fc71206e94b945acec46fd9304f1c38f9ff).

As such, this should work too:

opnsense-patch -c plugins b1953fc

EDIT: Just don't run both. See Fright's post below:

Quote from: Fright on October 31, 2021, 08:01:27 PM
you reversed 'opnsense-patch -c plugins 31b82cd 18cd9f6' by applying # opnsense-patch -c plugins b1953fc  ;)
just run
opnsense-patch -c plugins 31b82cd 18cd9f6
again

HTH,
Ben
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: benyamin on October 31, 2021, 03:44:42 PM
Just in case you aren't aware opnsense-patch is executed from the console shell (option 8 - at the console or via SSH session).
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 05:31:20 PM
Hi Ben,

Thanks, 2 patches applied via ssh.

No change in behaviour.

Trying to change anything in the config, doesn't work too. Last change 1month ago.
# ls -al /usr/local/etc/haproxy.conf
-rw-r--r--  1 root  wheel  4642 Sep 30 11:25 /usr/local/etc/haproxy.conf

https://x.x.x.x/ui/haproxy/export   config diff shows changes, which are not written into /usr/local/etc/haproxy.conf

# service haproxy start
[ALERT] 303/172454 (44675) : parsing [/usr/local/etc/haproxy.conf:46] : 'bind x.x.x.x:8448' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/172454 (44675) : parsing [/usr/local/etc/haproxy.conf:47] : 'bind x.x.x.x:443' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/172454 (44675) : Error(s) found in configuration file : /usr/local/etc/haproxy.conf
[ALERT] 303/172454 (44675) : Fatal errors found in configuration.
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy


Thanks,
Rene
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 05:53:25 PM
hi
Are there any errors thrown when you apply the haproxy configuration in gui?
Any errors in the backend log when applied??
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 06:17:08 PM
Hi Fright,

I've never configured https://x.x.x.x/ui/haproxy/maintenance.
But when I select anything there, it's loading forever.
Screenshot is attached. Doesn't matter if Firefox or Safari is used.


The haproxy config file, which worked before:




#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbproc                      1
    nbthread                    1
    hard-stop-after             60s
    tune.ssl.default-dh-param   1024
    spread-checks               0
    tune.chksize                16384
    tune.bufsize                16384
    tune.lua.maxmem             0
    log x.x.x.15:514 local0 info
    ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

defaults
    log     global
    option redispatch -1
    maxconn 20
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: abc.xyz.efg (extern an x.x.x.x WAN opnsense)
frontend abc.xyz.efg
    bind x.x.x.222:8448 name x.x.x.222:8448 ssl ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/5f786192aa1931.32444913.certlist
    bind x.x.x.222:443 name x.x.x.222:443 ssl ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/5f786192aa1931.32444913.certlist
    mode http
    option http-keep-alive
    default_backend backendPool_MATRIX
    # remove quotes from persistence cookie
    http-request replace-header Cookie '^(.*?; )?(SRVCOOKIE=)"([^;"]*)"(;.*)?$' \1\2\3\4
    option forwardfor
    # tuning options
    maxconn 99
    timeout client 30s
    timeout http-request 1m

    # logging options
    option log-separate-errors
    option httplog
    option socket-stats

# Frontend: abc.xyz.efg_letsencrypt (extern an x.x.x.222:80 WAN opnsense 4 letsencrypt)
frontend abc.xyz.efg_letsencrypt
    bind x.x.x.222:80 name x.x.x.222:80
    mode http
    option http-keep-alive
    option forwardfor
    # tuning options
    maxconn 99
    timeout client 30s
    timeout http-request 1m

    # logging options
    option log-separate-errors
    option httplog
    option socket-stats
    # ACL: find_acme_challenge
    acl acl_5f786a15533763.99172881 path_beg -i /.well-known/acme-challenge/

    # ACTION: redirect_acme_challenges
    use_backend acme_challenge_backend if acl_5f786a15533763.99172881

# Backend: backendPool_MATRIX (zeigt auf alle Matrix Server im Backend)
backend backendPool_MATRIX
    # health checking is DISABLED
    mode http
    balance source
    cookie SRVCOOKIE prefix
    # tuning options
    timeout connect 30s
    timeout server 30s
    server matrix.abc.local x.x.x.22:8448 ssl alpn h2,http/1.1 verify none resolve-prefer ipv4 cookie 5f7865de6d0ed749902345

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m 
    stick on src
    # tuning options
    timeout connect 30s
    timeout server 30s
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580 resolve-prefer ipv4

listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# statistics are DISABLED




Thanks,
Rene
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 07:02:04 PM
can be related (templates reloads when you visit this page)
any errors in System: Log Files: Backend after that?
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 07:35:39 PM
Hi Fright,

unable to start the HAProxy service via GUI.
Tried this via ssh  service haproxy start
here a the log entries from Log Files -> Backend

I don't know what </usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py> is trying, but when HAproxy start fails, it's always logging:    ...'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory...

letsencrypt certs are valid, just refreshed after the letsencrypt intermediate change end of September.

HAProxy was running fine until 2pm today. Stopped serving after opnsense update & reboot.


Thanks for your help! 
Rene


Date
Process
Line
2021-10-31T19:28:13   configd.py[80385]   [35d7aded-7bdf-4e3f-a45f-4c5000f00b5a] Show log   
2021-10-31T19:28:07   configd.py[80385]   [7de597f1-dca2-486e-bd16-37185d7701c6] request pf current overall table record count and table-entries limit   
2021-10-31T19:28:04   configd.py[80385]   [6745bdc6-195b-47a2-9e40-b0f4908359d1] Show log   
2021-10-31T19:28:00   configd.py[80385]   [f0f98d92-1693-4fae-b49b-fc482f07ad60] rsyncing certs to matrix   
2021-10-31T19:27:47   configd.py[80385]   [6e4e18f9-ab6b-46ee-bab3-3dfa734b8753] Show log   
2021-10-31T19:27:00   configd.py[80385]   [d8199760-9350-4dc4-bee6-b5997914293f] rsyncing certs to matrix   
2021-10-31T19:26:07   configd.py[80385]   [14b53d8f-4304-42bc-b546-cf1aa2822bb9] request pf current overall table record count and table-entries limit   
2021-10-31T19:26:00   configd.py[80385]   [1df24f88-39c2-44b4-8fac-1116340e4fa4] rsyncing certs to matrix   
2021-10-31T19:25:26   configd.py[80385]   [381242fc-5948-4396-a738-c003b9d12b99] Show log   
2021-10-31T19:25:14   configd.py[80385]   [089899d8-7a21-46eb-90af-053f0ef74625] requesting haproxy status   
2021-10-31T19:25:09   configd.py[80385]   [2d7d8553-25dd-41ce-b062-da16cf32e5c4] requesting haproxy status   
2021-10-31T19:25:01   configd.py[80385]   [9a4fd2f0-e132-48a2-8139-cce3214c0150] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 478, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1.   
2021-10-31T19:25:00   configd.py[80385]   [a7ce6c55-c6ba-48a3-8e3d-ed343a65ae81] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/socketCommand.py show-servers --output bootstrap --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1. at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 478, in execute subprocess.check_call(script_command, env=self.config_environment, shell=True, File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/socketCommand.py show-servers --output bootstrap --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1.   
2021-10-31T19:25:00   configd.py[80385]   [9a4fd2f0-e132-48a2-8139-cce3214c0150] Show certificate diff list   
2021-10-31T19:25:00   configd.py[80385]   [99f9e40d-dd2d-4b0e-a769-4d15f669988e] Inline action failed with OPNsense/HAProxy OPNsense/HAProxy/haproxy.conf expected token ',', got 'integer' at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/template.py", line 247, in _generate j2_page = self._j2_env.get_template(template_filename) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 997, in get_template return self._load_template(name, globals) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 958, in _load_template template = self.loader.load(self, name, self.make_globals(globals)) File "/usr/local/lib/python3.8/site-packages/jinja2/loaders.py", line 137, in load code = environment.compile(source, name, filename) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 757, in compile self.handle_exception(source=source_hint) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception raise rewrite_traceback_stack(source=source) File "/usr/local/opnsense/service/modules/../templates/OPNsense/HAProxy/haproxy.conf", line 136, in template {% set acl_enabled = '0' %} jinja2.exceptions.TemplateSyntaxError: expected token ',', got 'integer' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 506, in execute return ph_inline_actions.execute(self, inline_act_parameters) File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute filenames = tmpl.generate(parameters) File "/usr/local/opnsense/service/modules/template.py", line 341, in generate raise render_exception File "/usr/local/opnsense/service/modules/template.py", line 332, in generate for filename in self._generate(template_name, create_directory): File "/us   
2021-10-31T19:25:00   configd.py[80385]   [a7ce6c55-c6ba-48a3-8e3d-ed343a65ae81] show server status list   
2021-10-31T19:25:00   configd.py[80385]   [a723fbf5-9564-441b-a684-7cc92953f1ae] Inline action failed with OPNsense/HAProxy OPNsense/HAProxy/haproxy.conf expected token ',', got 'integer' at Traceback (most recent call last): File "/usr/local/opnsense/service/modules/template.py", line 247, in _generate j2_page = self._j2_env.get_template(template_filename) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 997, in get_template return self._load_template(name, globals) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 958, in _load_template template = self.loader.load(self, name, self.make_globals(globals)) File "/usr/local/lib/python3.8/site-packages/jinja2/loaders.py", line 137, in load code = environment.compile(source, name, filename) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 757, in compile self.handle_exception(source=source_hint) File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception raise rewrite_traceback_stack(source=source) File "/usr/local/opnsense/service/modules/../templates/OPNsense/HAProxy/haproxy.conf", line 136, in template {% set acl_enabled = '0' %} jinja2.exceptions.TemplateSyntaxError: expected token ',', got 'integer' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/opnsense/service/modules/processhandler.py", line 506, in execute return ph_inline_actions.execute(self, inline_act_parameters) File "/usr/local/opnsense/service/modules/ph_inline_actions.py", line 51, in execute filenames = tmpl.generate(parameters) File "/usr/local/opnsense/service/modules/template.py", line 341, in generate raise render_exception File "/usr/local/opnsense/service/modules/template.py", line 332, in generate for filename in self._generate(template_name, create_directory): File "/us   
2021-10-31T19:25:00   configd.py[80385]   [82fb7b54-06a8-4c14-9333-e86724372bf4] rsyncing certs to matrix   
2021-10-31T19:25:00   configd.py[80385]   generate template container OPNsense/HAProxy   
2021-10-31T19:25:00   configd.py[80385]   generate template container OPNsense/HAProxy   
2021-10-31T19:24:59   configd.py[80385]   [a723fbf5-9564-441b-a684-7cc92953f1ae] generate template OPNsense/HAProxy   
2021-10-31T19:24:59   configd.py[80385]   [99f9e40d-dd2d-4b0e-a769-4d15f669988e] generate template OPNsense/HAProxy   
2021-10-31T19:24:58   configd.py[80385]   [6752389e-7929-4691-bd04-74db80a42fa0] requesting haproxy statistics   
2021-10-31T19:24:58   configd.py[80385]   [9040b850-bb0e-48cd-b911-037851d4e24c] requesting haproxy statistics   
2021-10-31T19:24:58   configd.py[80385]   [efda7f86-4f7d-4528-9362-e42b7410bc39] requesting haproxy statistics   
2021-10-31T19:24:26   configd.py[80385]   [3950c3c5-2d22-4331-9dbd-d532411180be] Show log   
2021-10-31T19:24:14   configd.py[80385]   [7c0951f4-402a-47f4-b5af-0b7a22ac25c3] requesting haproxy status   
2021-10-31T19:24:06   configd.py[80385]   [fc69ddb2-6fd4-4a6e-9bea-e6f3c9787963] request pf current overall table record count and table-entries limit   
2021-10-31T19:24:00   configd.py[80385]   [b1487d67-84b8-4776-b513-b2cd06aa48dc] rsyncing certs to matrix   
2021-10-31T19:23:00   configd.py[80385]   [6f28862e-7444-4e3f-a89a-edcd4f35bbe7] rsyncing certs to matrix   
2021-10-31T19:22:06   configd.py[80385]   [37aab7ee-d323-421c-849d-981babe6eb54] request pf current overall table record count and table-entries limit
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 07:40:49 PM
QuoteFile "/usr/local/opnsense/service/modules/../templates/OPNsense/HAProxy/haproxy.conf", line 136, in template {% set acl_enabled = '0' %}
patch not applied?
opnsense-patch -c plugins 31b82cd 18cd9f6

Quotebut when HAproxy start fails, it's always logging:    ...'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist'
yes. it cant work if templates not working
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 07:54:47 PM


Hi Fright,

the output of both commands are still in my ssh session:

# opnsense-patch -c plugins 31b82cd 18cd9f6
Fetched 31b82cd via https://github.com/opnsense/plugins
Fetched 18cd9f6 via https://github.com/opnsense/plugins
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 31b82cdd2e43724c3e1873821e961469b99433db Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:32:48 +0300
|Subject: [PATCH] typo
|
|---
| .../opnsense/service/templates/OPNsense/HAProxy/haproxy.conf    | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|index 7d11f3df99..d6e0624ef3 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/haproxy.conf using Plan A...
Hunk #1 succeeded at 131.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 18cd9f647799b061371e9c0eabbeefd5da12c78b Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:34:26 +0300
|Subject: [PATCH] indentfirst->first
|
|https://jinja.palletsprojects.com/en/3.0.x/templates/#jinja-filters.indent
|---
| .../opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml   | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|index fd766e0025..2f7d7a4eec 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml using Plan A...
Hunk #1 succeeded at 53.
done
All patches have been applied successfully.  Have a nice day.
#



# opnsense-patch -c plugins b1953fc
Fetched b1953fc via https://github.com/opnsense/plugins
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 31b82cdd2e43724c3e1873821e961469b99433db Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:32:48 +0300
|Subject: [PATCH 1/2] typo
|
|---
| .../opnsense/service/templates/OPNsense/HAProxy/haproxy.conf    | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|index 7d11f3df99..d6e0624ef3 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/haproxy.conf using Plan A...
Reversed (or previously applied) patch detected!  Assuming -R.Hunk #1 succeeded at 131.
Hmm...  The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|
|From 18cd9f647799b061371e9c0eabbeefd5da12c78b Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:34:26 +0300
|Subject: [PATCH 2/2] indentfirst->first
|
|https://jinja.palletsprojects.com/en/3.0.x/templates/#jinja-filters.indent
|---
| .../opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml   | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|index fd766e0025..2f7d7a4eec 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml using Plan A...
Reversed (or previously applied) patch detected!  Assuming -R.Hunk #1 succeeded at 53.
done
All patches have been applied successfully.  Have a nice day.


Thanks,
Rene
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 07:55:41 PM
Versions   OPNsense 21.7.4-amd64
FreeBSD 12.1-RELEASE-p20-HBSD
OpenSSL 1.1.1l 24 Aug 2021
CPU type   Intel(R) Celeron(R) CPU J3160 @ 1.60GHz (4 cores)
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 08:01:27 PM
you reversed 'opnsense-patch -c plugins 31b82cd 18cd9f6' by applying # opnsense-patch -c plugins b1953fc  ;)
just run
opnsense-patch -c plugins 31b82cd 18cd9f6
again
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 08:14:53 PM

Hi Fright,


# opnsense-patch -c plugins 31b82cd 18cd9f6
Found local copy of 31b82cd, skipping fetch.
Found local copy of 18cd9f6, skipping fetch.
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 31b82cdd2e43724c3e1873821e961469b99433db Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:32:48 +0300
|Subject: [PATCH] typo
|
|---
| .../opnsense/service/templates/OPNsense/HAProxy/haproxy.conf    | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|index 7d11f3df99..d6e0624ef3 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/haproxy.conf
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/haproxy.conf using Plan A...
Hunk #1 succeeded at 131.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 18cd9f647799b061371e9c0eabbeefd5da12c78b Mon Sep 17 00:00:00 2001
|From: kulikov-a <36099472+kulikov-a@users.noreply.github.com>
|Date: Thu, 28 Oct 2021 20:34:26 +0300
|Subject: [PATCH] indentfirst->first
|
|https://jinja.palletsprojects.com/en/3.0.x/templates/#jinja-filters.indent
|---
| .../opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml   | 2 +-
| 1 file changed, 1 insertion(+), 1 deletion(-)
|
|diff --git a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|index fd766e0025..2f7d7a4eec 100644
|--- a/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
|+++ b/net/haproxy/src/opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml
--------------------------
Patching file opnsense/service/templates/OPNsense/HAProxy/sslCerts.yaml using Plan A...
Hunk #1 succeeded at 53.
done
All patches have been applied successfully.  Have a nice day.


# service haproxy start
[ALERT] 303/200545 (60557) : parsing [/usr/local/etc/haproxy.conf:46] : 'bind x.x.x.222:8448' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/200545 (60557) : parsing [/usr/local/etc/haproxy.conf:47] : 'bind x.x.x.222:443' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/200545 (60557) : Error(s) found in configuration file : /usr/local/etc/haproxy.conf
[ALERT] 303/200545 (60557) : Fatal errors found in configuration.
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy


Now, there is no entry in the Backend Log about HAProxy
...
2021-10-31T20:08:14   configd.py[80385]   [5d9fa72c-8768-4771-88a6-ced27154973a] request pf current overall table record count and table-entries limit   
2021-10-31T20:08:10   configd.py[80385]   [9583790e-65f7-45ca-adab-c37f7bc1cd9a] Show log   
2021-10-31T20:08:00   configd.py[80385]   [42a22034-7f06-4949-b9bc-dc29c15b3f6c] rsyncing certs to matrix
...



haproxy -c -f /usr/local/etc/haproxy.conf

[NOTICE] 303/201000 (21827) : haproxy version is 2.2.17-dd94a25
[ALERT] 303/201000 (21827) : parsing [/usr/local/etc/haproxy.conf:46] : 'bind x.x.x.222:8448' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/201000 (21827) : parsing [/usr/local/etc/haproxy.conf:47] : 'bind x.x.x.222:443' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/201000 (21827) : Error(s) found in configuration file : /usr/local/etc/haproxy.conf
[ALERT] 303/201000 (21827) : Fatal errors found in configuration.


The same message via GUI HAProxy - Settings - Test syntax


HAProxy - Config Export - Config Diff shows:
--- /usr/local/etc/haproxy.conf   2021-09-30 11:25:27.265579000 +0200
+++ /usr/local/etc/haproxy.conf.staging   2021-10-31 20:10:34.197483000 +0100
@@ -11,6 +11,7 @@
     stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
     nbproc                      1
     nbthread                    1
+    hard-stop-after             60s
     tune.ssl.default-dh-param   1024
     spread-checks               0
     tune.chksize                16384
@@ -91,7 +92,7 @@
     # tuning options
     timeout connect 30s
     timeout server 30s
-    server matrix.bci.local x.x.x.22:8448 ssl alpn h2,http/1.1 verify none cookie 5f7865de6d0ed749902345
+    server matrix.bci.local x.x.x.22:8448 ssl alpn h2,http/1.1 verify none resolve-prefer ipv4 cookie 5f7865de6d0ed749902345

# Backend: acme_challenge_backend (Added by Let's Encrypt plugin)
backend acme_challenge_backend
@@ -105,7 +106,7 @@
     timeout connect 30s
     timeout server 30s
     http-reuse safe
-    server acme_challenge_host 127.0.0.1:43580
+    server acme_challenge_host 127.0.0.1:43580 resolve-prefer ipv4

listen local_statistics
     bind            127.0.0.1:8822



Thanks,
Rene

Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 08:20:56 PM
Quote# service haproxy start
templates not applied yet
try this one

# configctl haproxy start
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 08:36:02 PM
# configctl haproxy start
Error (1)


Log files - backend log
2021-10-31T20:31:55   configd.py[80385]   [c23a223e-7710-4dcc-acd3-1036453abe6f] Show log   
2021-10-31T20:31:46   configd.py[80385]   [443c78c0-bc13-46e0-8074-b9cc2239c202] returned exit status 1   
2021-10-31T20:31:45   configd.py[80385]   [a7066062-0a1e-4f30-a8cf-4c300257bf03] request pf current overall table record count and table-entries limit   
2021-10-31T20:31:45   configd.py[80385]   [f149fe45-da60-42ef-89d6-f94a53a9074b] request pf current overall table record count and table-entries limit   
2021-10-31T20:31:44   configd.py[80385]   [92679197-8689-47bd-872a-0e2d8f385063] request pf current overall table record count and table-entries limit   
2021-10-31T20:31:43   configd.py[80385]   [8fc214dc-9cf7-44b9-9390-5560e1b6ace3] request pf current overall table record count and table-entries limit   
2021-10-31T20:31:43   configd.py[80385]   [443c78c0-bc13-46e0-8074-b9cc2239c202] starting haproxy   
2021-10-31T20:31:20   configd.py[80385]   [cf4aa895-3fd6-4e95-b76b-35bcc7655272] Show log


# service haproxy start
[ALERT] 303/202857 (76508) : parsing [/usr/local/etc/haproxy.conf:46] : 'bind x.x.x.222:8448' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/202857 (76508) : parsing [/usr/local/etc/haproxy.conf:47] : 'bind x.x.x.222:443' : 'crt-list' : cannot open file '/tmp/haproxy/ssl/5f786192aa1931.32444913.certlist' : No such file or directory
[ALERT] 303/202857 (76508) : Error(s) found in configuration file : /usr/local/etc/haproxy.conf
[ALERT] 303/202857 (76508) : Fatal errors found in configuration.
/usr/local/etc/rc.d/haproxy: WARNING: failed precmd routine for haproxy

# ls -al /usr/local/etc/haproxy.conf
-rw-r--r--  1 root  wheel  4642 Sep 30 11:25 /usr/local/etc/haproxy.conf



Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 08:42:54 PM
I got a message in our syslog server

[27490d40-3a1f-4558-9928-2bdd21e29407] Script action failed with Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1. at Traceback (most recent call last):   File /usr/local/opnsense/service/modules/processhandler.py, line 478, in execute     subprocess.check_call(script_command, env=self.config_environment, shell=True,   File /usr/local/lib/python3.8/subprocess.py, line 364, in check_call     raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '/usr/local/opnsense/scripts/OPNsense/HAProxy/syncCerts.py actions --output bootgrid --page-rows '10' --page '1' --search '' --sort-col '' --sort-dir ''' returned non-zero exit status 1.



Line 478 is marked bold

try:
                    with tempfile.NamedTemporaryFile() as error_stream:
                        with tempfile.NamedTemporaryFile() as output_stream:
                            subprocess.check_call(script_command, env=self.config_environment, shell=True,
                                                  stdout=output_stream, stderr=error_stream)
                            output_stream.seek(0)
                            error_stream.seek(0)
                            script_output = output_stream.read()
                            script_error_output = error_stream.read()
                            if len(script_error_output) > 0:
                                syslog_error('[%s] Script action stderr returned "%s"' %(
                                    message_uuid, script_error_output.strip()[:255]
                                ))
                            return script_output.decode()
                except Exception as script_exception:
                    syslog_error('[%s] Script action failed with %s at %s' % (
                        message_uuid, script_exception, traceback.format_exc()
                    ))
                    return 'Execute error'
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on October 31, 2021, 08:50:17 PM
ah. this command also does not reapply templates. sorry then
what if in gui:
Services: HAProxy: Settings: ->Real Servers->Real Servers --> Apply ?
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 09:12:46 PM
Hi Fright,

attached

Thanks,
Rene
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Rene1 on October 31, 2021, 10:58:48 PM
Hi,

Thanks for your support.

Now it's working again. I reinforced fetching a new letsencrypt certificate before it will expire in Dec.
And voila the directory /tmp/haproxy/ssl/ isn't empty anymore.

Rene
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: benyamin on November 01, 2021, 01:54:38 AM
Quote from: Fright on October 31, 2021, 08:01:27 PM
you reversed 'opnsense-patch -c plugins 31b82cd 18cd9f6' by applying # opnsense-patch -c plugins b1953fc  ;)
just run
opnsense-patch -c plugins 31b82cd 18cd9f6
again

I'll edit my post to say one or the other not both... 8^d
Title: Re: HAProxy won't start after upgrade opnsense to OPNsense 21.7. crt-list' : cannot
Post by: Fright on November 01, 2021, 07:13:44 AM
@Rene1
glad to know )
it will be great if you mark the topic as [SOLVED]
Title: Re: [solved] HAProxy won't start after upgrade opnsense to OPNsense21.7.crt-list
Post by: Rene1 on November 01, 2021, 07:34:37 AM
 [solved]