Nginx SSL termination lag

[leprasmurf]

Hi all.  I've got Nginx working as an SSL terminating load balancer with the acme plugin for certificates.  I've also got the CA cert installed locally from before the acme integration was working for me.

I keep running into lag issues when accessing the web gui through nginx.  CSS and JS files either time out or finish after a long delay (60 seconds+).

Request delay (sanitized):

    GET /ui/js/bootstrap-select.min.js?v=7762533db81d1dfa HTTP/1.1
    Host: firewall.my.domain.com
    Connection: keep-alive
    Pragma: no-cache
    Cache-Control: no-cache
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
    DNT: 1
    Accept: */*
    Sec-Fetch-Site: same-origin
    Sec-Fetch-Mode: no-cors
    Sec-Fetch-Dest: script
    Referer: https://firewall.my.domain.com/ui/acmeclient
    Accept-Encoding: gzip, deflate, br
    Accept-Language: en-US,en;q=0.9
    Cookie: remoteClock=1612442552000; localClock=1609764275495; cookie_test=8f418ff1294353c6bea9cefbd02ee806; PHPSESSID=2fdb4670a1e4244a7506758e4a1aaa8c

Response headers:

    HTTP/1.1 200 OK
    Server: nginx
    Date: Tue, 05 Jan 2021 14:33:49 GMT
    Content-Type: text/javascript
    Content-Length: 13004
    Connection: keep-alive
    Vary: Accept-Encoding
    Content-Encoding: gzip
    Last-Modified: Thu, 22 Oct 2020 07:55:36 GMT
    ETag: "2420155145"
    Accept-Ranges: bytes
    Expires: Thu, 07 Jan 2021 16:33:49 GMT
    Cache-Control: max-age=180000

This 13kb js file took a full minute to transfer whereas the largest of the files (SourceSansPro-Regular.woff @ 119kb) took 53ms.

I'm struggling to identify the source of this bottleneck, but it makes accessing the web ui through this ssl endpoint painfully slow.

If I access the HTTP side via port 8080 everything is as responsive as expected.  Response headers (sanitized):

    GET /ui/js/bootstrap-select.min.js?v=7762533db81d1dfa HTTP/1.1
    Host: firewall.my.domain.com:8080
    Connection: keep-alive
    Pragma: no-cache
    Cache-Control: no-cache
    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
    DNT: 1
    Accept: */*
    Referer: http://firewall.my.domain.com:8080/services_dhcp.php?if=opt2
    Accept-Encoding: gzip, deflate
    Accept-Language: en-US,en;q=0.9
    Cookie: PHPSESSID=2fdb4670a1e4244a7506758e4a1aaa8c; remoteClock=1612535979000;     localClock=1609857583008

Response header:

    HTTP/1.1 200 OK
    Vary: Accept-Encoding
    Content-Encoding: gzip
    Last-Modified: Thu, 22 Oct 2020 07:55:36 GMT
    ETag: "2420155145"
    Content-Type: text/javascript
    Accept-Ranges: bytes
    Expires: Thu, 07 Jan 2021 16:40:12 GMT
    Cache-Control: max-age=180000
    Content-Length: 13004
    Date: Tue, 05 Jan 2021 14:40:12 GMT
    Server: OPNsense

This time the asset took 38ms to download.

Any guidance or thoughts would be appreciated.

Kind regards,
Tim

[fabian]

For me, one minute sounds like a timeout.

What nginx probably does differently is that it by default uses HTTP 1.0 in the backend while browsers use 1.1 or 2 if available. This may cause more TCP connections to be opened since 1.0 closes them by default after use.
The nginx plugin has an advanced option to switch to 1.1 for websockets. You may give that a try.

That is the only thing I can guess because this sounds more like a network timeout. BTW: AFAIK the timeout can be reduced as well.

[Fright]

I could not reproduce this behavior with any settings (but did not try to switch GUI from https to http)
Any clues in traffic statistic or error logs?

[leprasmurf]

Thank you for the replies, sorry it took me so long to reply.  I've been experimenting with the settings and configuration but I've been unable to isolate the issue.

The timeout in the nginx config has been reduced to 15 seconds but resources continue to stall for up to a minute before loading.  I'm worried I might have some bad routing or something.

I'll post more as I find it.