Messages

@granute I am a lighttpd developer and would be interested in seeing the `truss -a -f -D -s 1024 -p "<lighttpd_pid>"` output of the lighttpd process.
https://man.freebsd.org/cgi/man.cgi?truss

Back in Jan, there was https://forum.opnsense.org/index.php?topic=44391.msg226740#msg226740 and I posted some steps to troubleshoot, so you might take a look at that thread.  However, that thread concluded that the CPU issue was due to processing interrupts.

Back in 2021, there was a post about an issue in lighttpd that caused high CPU use, and it was fixed in lighttpd 1.4.59 (released Nov 2020)
https://forum.opnsense.org/index.php?topic=21210.msg101803#msg101803

Reply #2, #6, and #9 showed htop with the lighttpd process having taking a large amount of CPU time, so I wonder if that is contributing.

On the off chance that there is some interaction with openssl KTLS on your large system, which might also have TLS acceleration hardware used by openssl drivers, please *test* with KTLS disabled in lighttpd, as lighttpd mod_openssl uses KTLS by default on Linux and FreeBSD, when available.  lighttpd.conf: `ssl.openssl.ssl-conf-cmd += ("Options" => "-KTLS")`, or you can *test* disabling KTLS system-wide on FreeBSD `sysctl kern.ipc.tls.enable=0`

Something must be wrong for lighttpd to be using that much CPU and your system not to be under heavy request load.  Is there any information in the lighttpd error log?  Is there a high request load visible in the lighttpd access log?  Can you get an strace or truss of the lighttpd process and share it?  That CPU usage is aberrant.  I am a lighttpd developer but I do not have an opnsense test system.  If you have a debugger installed on the system, a few stack traces might also be useful.  `echo bt full | gdb -p $lighttpd_pid`

OCSP stapling has been available in lighttpd since lighttpd 1.4.56, released Nov 2020, which is 4 years ago.

Looks like you filed https://github.com/opnsense/core/issues/8084 with that info, too.

lighttpd provides a script to help retrieve OCSP stapling info from OCSP responders.
https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/doc/scripts/cert-staple.sh

FYI: Let's Encrypt is shutting down its OCSP responders 6 Aug 2025.  No more OCSP stapling from Let's Encrypt after then.
https://letsencrypt.org/2024/12/05/ending-ocsp/

> "(/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/h2.c.612) h2: xx.xx.xx.xx sent too many RST_STREAM too quickly"
> That error is emitted after triggering the detection of activity which looks like the HTTP/2 rapid reset attack (CVE-2023-44487).

I am the lighttpd developer who wrote this code.  lighttpd closes the connection if the request pattern looks like HTTP/2 rapid reset attack (CVE-2023-44487).

My comment about adjusting the client is for the benefit of valid clients.  If the behavior of a valid client looks like an attack on the server, then the client should be modified to avoid attacking the server, or appearing to attack the server.

Yes, lighttpd can run as non-root.

For binding to privileged ports, lighttpd can be run from inetd/xinetd, or lighttpd can accept systemd socket activation (or something else privileged setting the simple environment variables and providing the listening ports on already-open file descriptors when starting lighttpd).  Another option is a small daemon I wrote over a decade ago: https://github.com/gstrauss/bsock which can run on just about any unix-like system which supports passing file descriptors over unix domain sockets.

PHP scripts are potentially more vulnerable than lighttpd, and the PHP would still need to be able to run privileged commands.  As an interim step, you could have lighttpd start up the PHP fastcgi server as root via a setuid script, but it would be better to have the PHP run as non-root, too, and be permitted to run specific commands with privileges by another set of scripts which checks the arguments before running the commands as root.

This thread reminds me of race conditions in the PHP and shell scripts that I pointed out in https://github.com/opnsense/core/issues/6351 (make sure to show comments which might be hidden by github due to the long conversation thread)

I am a professional programmer (lighttpd developer) and made suggestions how to eliminate the race conditions I identified, and some -- but not all -- changes were implemented.  It might be useful if some one else with scripting and process management experience would review that code again and provide feedback to Franco, as my posts in https://github.com/opnsense/core/issues/6351 were not enough.

FYI: "opnsense-patch 724f8494d6" gets you this workaround to the issue:
https://github.com/opnsense/core/commit/724f8494d64074e0384b9f593e699daf2d84d604

"(/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/h2.c.612) h2: xx.xx.xx.xx sent too many RST_STREAM too quickly"
That error is emitted after triggering the detection of activity which looks like the HTTP/2 rapid reset attack (CVE-2023-44487).

Maybe the client side of the GUI should manage widget requests through a queue to better limit the number of concurrent requests?

> "Error   lighttpd   (/usr/obj/usr/ports/www/lighttpd/work/lighttpd-1.4.76/src/mod_openssl.c.3510) SSL: 1 error:0A00009C:SSL routines::http request (10.67.6.102)

That suggests that you made an HTTP (clear-text ) request to a port on which lighttpd was configured to expect HTTPS (TLS).

Or maybe you misconfigured your lighttpd.conf to have lighttpd expect TLS on port 80, so when you made an HTTP (clear-text) request to port 80, it failed as above.

Glad you identified the actual culprit.

FYI: various gateway protocols (e.g. CGI, FastCGI, SCGI, ...) require CONTENT_LENGTH.  If lighttpd is configured to stream the request body using a non-zero value, e.g.

Code Select Expand

server.stream-request-body = 1, then lighttpd will return "411 Length Required" for requests containing a request body which do not provide Content-Length in the request headers (when lighttpd is required to provide CONTENT_LENGTH to backend protocols).

However, if request body streaming is disabled in lighttpd.conf (the default if not specified)

Code Select Expand

server.stream-request-body = 0, then lighttpd will collect the request body -- by decoding Transfer-Encoding: chunked (HTTP/1.1) or collecting DATA frames (HTTP/2) -- and lighttpd will set CONTENT_LENGTH for the backends.

Therefore, a workaround is to set

Code Select Expand

server.stream-request-body = 0 if intermediaries impolitely modify requests to remove Content-Length.

Test and pre-flight the lighttpd config by manually running:

Code Select Expand

lighttpd -f /etc/lighttpd/lighttpd.conf -tt

Just a follow-up that this was fixed in lighttpd-1.4.59 and is in OPNsense 21.1.1 (released Feb 10)
https://opnsense.org/opnsense-21-1-1-released/

[do not]

@mgsteve do you have the cert files that caused the error for you?  Can you share the exact files?  (Please do not share the private key.  If it is part of the ssl.pemfile or ssl.ca-file, then please remove the contents between "-----BEGIN PRIVATE KEY-----" and "-----END PRIVATE KEY-----" (or similar) before posting, but otherwise leave those lines as-is so that I can see the format of the file.)

I am a lighttpd developer and have been unable to reproduce the issue.  I tried different combinations of stray char at end of file, and differences in line endings ("\n" and "\r\n"), but lighttpd 1.4.59 with openssl 1.1.1i handles them just fine.  lighttpd uses openssl routines to read certificate files, so I don't think this is an issue with lighttpd, but if lighttpd can detect the issue, maybe lighttpd can issue better error trace.

Maybe the issue you were having was related to https://forum.opnsense.org/index.php?topic=20325.0 ?

> So no more ssl.ca-file, right?

Correct: if the full certificate chain is provided in ssl.pemfile, then ssl.ca-file is no longer needed for that purpose (building the certificate chain).

ssl.ca-file is still used for client certificate verification -- if that is enabled with ssl.verifyclient.* -- and might contain the certificates for the CA which issues the client certificates.

@Franco, lighttpd 1.4.56 migrates from old openssl interfaces (<= openssl 1.0.1) to newer openssl certificate callback interfaces, and lighttpd 1.4.56 tries to build the certificate chain when one is not provided.  The best solution is to provide the certificate chain:

Code Select Expand


    ssl.privkey= "/etc/lighttpd/certs/www.example.com/privkey.pem"
    ssl.pemfile= "/etc/lighttpd/certs/www.example.com/fullchain.pem"


I am a lighttpd developer and am looking into the "building cert chain" issue, which was only recently reported in https://redmine.lighttpd.net/boards/2/topics/9620

If you are aware of other issues with lighttpd, please post to https://redmine.lighttpd.net/boards/2/ so that lighttpd developers are aware of the issue, and you can link to forum.opnsense.org.