Nextcloud AiO behind HAProxy

[Maginos]

@cookiemonster and @viragomann:

I found, that for the health monitoring in haproxy, the server is marked as down, maybe there is a problem.
I also attached the entries in the logs of haproxy and the settings for the health checks.

Maybe you can comment on that.

EDIT: I was able to get rid of the issue with health monitoring. Now the server is displayed as "UP" in the health monitoring. Unfortunately, the error 503 still persists.

EDIT 2: When I try to access the NC via mobile phone, I get the message "An SSL error has occured. A safe connection to the server can not be established". The certificate is the right one. What could be the problem?

[viragomann]

503 indicates, that HAproxy doesn't get a response from the backend.

If you check the status is it even shown as online?

[Maginos]

Yes, it is shown now as "UP" (see screenshot).

BUT: When I try to access the NC via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG

The website can not be displayed, because the authenticity of the recieved data can not be verified".


Does that help?

[cookiemonster]

We're still guessing your setup.
- Is your AIO thinguie using SSL/TLS ie using a certificate? Is it a signed by a major CA or is it self signed?
- You seem to be using a non-standard port of 11000. You put that in your server entry in HAProxy, right?
- Try from the command line ON OPN:

Code Select Expand

$ wget http://192.168.5.158
--2025-03-15 22:14:29--  http://192.168.5.158/
Connecting to 192.168.5.158:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.5.158/login [following]
--2025-03-15 22:14:29--  https://192.168.5.158/login
Connecting to 192.168.5.158:443... failed: Connection refused.You see in my attempt for showing you, I am attempting the connection to the port I have set (default 80 http). It connects. Then follows to move the connection and then login. This is not important. What is important is that there was a successful connection.
- Where are you trying to connect from? Very important.

[Maginos]

- Is your AIO thinguie using SSL/TLS ie using a certificate? Is it a signed by a major CA or is it self signed?

I have a certificate from Lets Encrypt, that works for other services perfectly fine.

- You seem to be using a non-standard port of 11000. You put that in your server entry in HAProxy, right?

11000 IS the standard port for Nextcloud AiO, specifically the Apache port. And yes, I configured that in haproxy.

- Try from the command line ON OPN:

Code Select Expand

$ wget http://192.168.5.158
--2025-03-15 22:14:29--  http://192.168.5.158/
Connecting to 192.168.5.158:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.5.158/login [following]
--2025-03-15 22:14:29--  https://192.168.5.158/login
Connecting to 192.168.5.158:443... failed: Connection refused.You see in my attempt for showing you, I am attempting the connection to the port I have set (default 80 http). It connects. Then follows to move the connection and then login. This is not important. What is important is that there was a successful connection.
- Where are you trying to connect from? Very important.

So I entered the following command in the shell of my OPNSense and got this output:

Code Select Expand

root@OPNsense:~ # wget http://10.50.1.2:11000
--2025-03-16 09:14:14--  http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 09:14:14--  https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 09:14:18 ERROR 503: Service Unavailable.
Explanation:
10.50.1.2 is my docker host (located in another network than all other devices at home)
10.20.1.1 is my OPNSense

So in the third last line it says "... connected", but then the 503 error occurs. What can we derive from this?
Since it seems to be important: I configured a DNS Override in Unbound, that querys to subdomain.domain.tld are redirected to 10.20.1.1 (IP of haproxy), so that queries from my home network do not leave the OPNSense and are redirected directly to the haproxy and then to the docker host.

- Where are you trying to connect from? Very important.

When I try to connect from outside my home network, I get the "An SSL error has occured. A safe connection to the server can not be established" error.

When I try to access the NC from inside my LAN network via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG"

I hope, that clarifies it.

[viragomann]

I have a certificate from Lets Encrypt, that works for other services perfectly fine.

I'm wondering, who you got an LE certificate behind HAproxy.

So in the third last line it says "... connected", but then the 503 error occurs. What can we derive from this?
Since it seems to be important: I configured a DNS Override in Unbound, that querys to subdomain.domain.tld are redirected to 10.20.1.1 (IP of haproxy), so that queries from my home network do not leave the OPNSense and are redirected directly to the haproxy and then to the docker host.

Nextcloud obviously redirects the request to https://subdomain.domain.tld/login, since it is configured to use this domain.
Seems normal to me as far.

Can you configure the AIO to accept simply the IP as well?
You will have to use the "--no-check-certificate" option in wget then, in case it requires SSL.

Is the AIO even aware of the reverse proxy?

When I try to access the NC from inside my LAN network via https://internal_ip:11000

This would require, that the internal IP is included in the SSL certificate, which I'm in doubt.

[Maginos]

I'm wondering, who you got an LE certificate behind HAproxy.

No, for behind the proxy I have the default certificate from Nextcloud.

Can you configure the AIO to accept simply the IP as well?
You will have to use the "--no-check-certificate" option in wget then, in case it requires SSL.

I added 10.20.1.1 to the trusted proxy section in the config.php, restarted all containers and executed the command again. This is the output:

Code Select Expand

root@OPNSense:~ # wget --no-check-certificate http://10.50.1.2:11000
--2025-03-16 20:37:07--  http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 20:37:07--  https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 20:37:10 ERROR 503: Service Unavailable.
Seems to be the same output as before.

Is the AIO even aware of the reverse proxy?

Yes it is, the IP of the proxy is also located in the trusted proxy section of the config.php.

This would require, that the internal IP is included in the SSL certificate, which I'm in doubt.

When the AiO is "freshly" set up, it is possible to access the AiO NC interface via the internal IP. There is no issue with the certificate.


Ok, I found something: In the "Real Server" section, the box for "SSL" was checked. Since the connection from the proxy to the AiO is via http, not via https, I think checking the SSL box is wrong.
After unchecking it, I get a "400 bad request" error and not the 503 error any more. When I execute the wget command, the last two lines change to the "new" error.

In the logs of Nextcloud I don't find anything that matches in time.

[viragomann]

I added 10.20.1.1 to the trusted proxy section in the config.php

You have to add it to the trusted domains.

[cookiemonster]

This is what I mean. We don't know so far if you were communicating haproxy and Nextcloud via https. You have only now said it is http. I'm not guessing anymore. Draw it with the endpoints shown and which protocols. It would be useful for you too. Until then when you accurately describe the setup I won't be inclined to chime in.
p.s. if you are getting http errors (400, 503, etc.) that means the network communication is happening successfully. The reverse proxy is reaching the webserver. You want to have a look not only on nextcloud's logs but also and more importantly at this point, your webserver logs. Apache is hhtp-access and http-error (normally). But you could be using a different webserver. Depends on the AIO thinguie setup.

[Maginos]

You have to add it to the trusted domains.

Tried it, did unfortunately not help.

This is what I mean. We don't know so far if you were communicating haproxy and Nextcloud via https. You have only now said it is http. I'm not guessing anymore. Draw it with the endpoints shown and which protocols. It would be useful for you too. Until then when you accurately describe the setup I won't be inclined to chime in.
p.s. if you are getting http errors (400, 503, etc.) that means the network communication is happening successfully. The reverse proxy is reaching the webserver. You want to have a look not only on nextcloud's logs but also and more importantly at this point, your webserver logs. Apache is hhtp-access and http-error (normally). But you could be using a different webserver. Depends on the AIO thinguie setup.

I'm sorry, I made a mistake. I thought the proxy communicates via https, but http is correct. I checked the nginx proxy manager config on the github page of the NC AiO and now I'm 100% sure, that it communicates via http.

So the connection from the client to the haproxy is via https, the connection from haproxy to the NC AiO is via http.
Here are the logs of the Apache web server:

Code Select Expand

docker logs nextcloud-aio-apache
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Connection to nextcloud-aio-nextcloud (172.19.0.7) 9000 port [tcp/*] succeeded!
{"level":"info","ts":1742231207.8322253,"msg":"using config from file","file":"/tmp/Caddyfile"}
{"level":"info","ts":1742231207.8334265,"msg":"adapted config to JSON","adapter":"caddyfile"}
[Mon Mar 17 18:06:47.844311 2025] [mpm_event:notice] [pid 46:tid 46] AH00489: Apache/2.4.63 (Unix) configured -- resuming normal operations
[Mon Mar 17 18:06:47.844343 2025] [core:notice] [pid 46:tid 46] AH00094: Command line: '/usr/local/apache2/bin/httpd -D FOREGROUND'

I also checked the logs of all other nextcloud containers and that one of OPNSense and I found nothing suspicious. The only thing I found was this error message:

Code Select Expand

Error connection: Server error: `GET https://subdomain.domain.tld/onlyoffice/healthcheck` resulted in a `503 Service Unavailable` response:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
I then disabled OnlyOffice via a occ command and tried to access the NC afterwards, but that didn't help.

Are these information enough or do you need more?

[viragomann]

You have to add it to the trusted domains.

Tried it, did unfortunately not help.

This should just enable to access it using the container IP from internal for testing purposes.

[Maginos]

You have to add it to the trusted domains.

Tried it, did unfortunately not help.

This should just enable to access it using the container IP from internal for testing purposes.

When I enter http://10:50.1.2:11000, I get redirected to https://subdomain.domain.tld and there I get the error 400 bad request.

[viragomann]

I don't know, what redirects the request then. Possibly the web server settings.

If I add the IP of my Nextcloud VM to the trusted domain, I can connect to it using the IP. The browser is complaining about an invalid SSL certificate, however, but if I add an exception, I can connect.

If the IP is missing in the trusted domains, it also doesn't redirect me to somewhere else, but complains that the domain is not trusted.

Did you even obey the instructions at "Reverse Proxy Documentation?

[Maginos]

I deactivated the DNS Override in OPNSense, but that didn't help as well...

You added the IP of your docker host to the trusted domain section? Ok.

Yes I followed that documentation. Maybe I should add, that the AiO Container worked for my former Sophos UTM. Initially, I set up the AiO Container with a Sophos UTM Firewall and everything worked perfectly fine. Now I switched to OPNSense and I get the error 400.

Edit: I tried to prevent the redirect by commenting out the overwriteprotocol, overwritehost and overwrite.cli.url, but that didn't help.

[viragomann]

You added the IP of your docker host to the trusted domain section?

I'm actually running a Nextcloud simply in a VM on Apache web server, no AIO, no docker. But it runs behind HAproxy on OPNsense.

Initially, I set up the AiO Container with a Sophos UTM Firewall and everything worked perfectly fine

Behind a reverse proxy? Presumably not. And that's the point here, I guess. Why should this be an issue with OPNsense??

Now I switched to OPNSense and I get the error 400.

400? We were troubleshooting 503, as you stated before.

Maybe any details to see in the HAproxy log?

Edit: I tried to prevent the redirect by commenting out the overwriteprotocol, overwritehost and overwrite.cli.url, but that didn't help.

Yes, at least the override host must be removed, when accessing via IP.