Hi guys,
I have a question concerning hosting a Nextcloud AiO behind HAProxy.
On the github page of the Nextcloud AiO I found this (https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md) guide.
I tried to set it up in OPNSense, but unfortunately, the error "503 Service unavailable" occurs.
Now I would like to ask, if you could check my configuration and help me to set it up correctly.
For this, I put my current configuration in the annex.
If you need more information, just ask.
Thank you for your help.
Maginos
I actually host my own instance of Nextcloud and proxy it out with OPNSense's HAProxy. I'll say this much; I wouldn't even bother trying to tweak Nextcloud's proxy settings for any reason. I've learned that it tends to make things a bit messy. If all you need is hosting capabilities so you can access it from outside of your home network, setting it up in OPNSense/HAProxy should be a breeze. I certainly wouldn't be hosting anything via HTTP (80) even for redirection purposes, but that's just me. From a cybersecurity analyst's perspective, I'd lock that down to exclusively use HTTPS and use an ACME-generated certificate. Do you know for a fact that you are calling the correct address/sub-domain when trying to resolve to your service? I just can't tell when all you've posted is your Public Service and not your condition, rule, backend config, etc.
post your haproxy config here by adding the picture to the post, not as an attachment. I for one won't be clicking on it.
I have just resolved the last gnarly bit of my setup. My nextcloud is in a freebsd jail on another host to OPN. Haproxy on OPN does the reverse proxy and terminates the TLS and I am leaving nextcloud on http internally.
Hi guys and thank you for your replies.
@cookiemonster: Since I was not able to paste images directly in the post, I made another PDF with further screenshots of my config. I can understand, that you don't trust my PDF, so how can I show you the screenshots in a way, that works for you?
@petrij98: I just want the NC to be accessible from outside my home network. I have an LE certificate for the subdomain and yes, I am sure, that I call the correct subdomain. I even checked with DNSchecker, that public DNS resolvers point to my public IPv4 for that domain.
Instead of a PDF just attach a PNG or JPEG and it will be shown right in the forum view.
@Patrick M. Hausen thank you for the tip.
So here are the screenshots for the backend pool, rule and condition.
And here are bigger screenshots for the backend pool
This is only the setup of the haproxy backend so it is very hard to know where there might be a problem. Please know that I do not do docker, have no interest whatsoever in it. If anything I want to run is docker-only, I move on looking for an alternative that does have "old style" application configurations.
So now let's establish the overall setup. By the way mine is very largely based on this haproxy-on-opn-tutorial-by-thehellsite (https://forum.opnsense.org/index.php?topic=23339.0).
Take a look at the picture to figure out if you are on the same setup and if different, please explain it. But you can follow that as a basis for an uncomplicated setup: create real server (your nextcloud), create a backend containing that server, create a front end WITH A TLS CERT, create firewall rules to allow the front end to be accessed. Here your haproxy --> backend server can be http or https but if you do https, then you have to deal with those certs separately. Makes sense?
Hmm... The only thing I can really think of is just making sure your backend, condition, and rule are all named differently (i.e. nextcloud-back, nextcloud-condition, nextcloud-rule). It wouldn't surprise me if HAProxy completely craps out because of something as insignificant as that. Also, did you make sure to change the port you access OPNSense on? If it's still 443, that could explain what's happening.
It doesn't. The names of rules and conditions can be totally random and still be used in evaluations. They get issued an internal identifier like acl acl_67ced2068d01c7.64292703 . Useful is the "test syntax". Bad settings will be shown with errror. Warnings and Notices but all ending with a "configuration is valid" will not crash it.
Quote from: cookiemonster on March 14, 2025, 02:59:15 PMBy the way mine is very largely based on this haproxy-on-opn-tutorial-by-thehellsite (https://forum.opnsense.org/index.php?topic=23339.0).
Take a look at the picture to figure out if you are on the same setup and if different, please explain it. But you can follow that as a basis for an uncomplicated setup: create real server (your nextcloud), create a backend containing that server, create a front end WITH A TLS CERT, create firewall rules to allow the front end to be accessed. Here your haproxy --> backend server can be http or https but if you do https, then you have to deal with those certs separately. Makes sense?
Thank you for the link, I will compare configurations. I followed the overall setup guides for other services, so NC is not the first service I want to access from outside. Other services work fine for 95% of the time.
@petrij98: The port of OPNSense was changed. As I wrote above, I have several other services, which are accessible through HAProxy.
Quote from: cookiemonster on March 14, 2025, 02:59:15 PMPlease know that I do not do docker, have no interest whatsoever in it. If anything I want to run is docker-only, I move on looking for an alternative that does have "old style" application configurations.
So now let's establish the overall setup. By the way mine is very largely based on this haproxy-on-opn-tutorial-by-thehellsite (https://forum.opnsense.org/index.php?topic=23339.0).
Take a look at the picture to figure out if you are on the same setup and if different, please explain it. But you can follow that as a basis for an uncomplicated setup: create real server (your nextcloud), create a backend containing that server, create a front end WITH A TLS CERT, create firewall rules to allow the front end to be accessed. Here your haproxy --> backend server can be http or https but if you do https, then you have to deal with those certs separately. Makes sense?
So you set up NC with a LAMP Stack, I guess? Unfortunately this doesn't help me, because I already have a running NC instance from a Linuxserver Docker Container. This instance works, but the AiO Setup does not.
I set up haproxy with following videos:
So my configuration differs in several ways from the setup of the link you posted above.
And now I'm not sure, if my configuration is wrong/disadvantageous.
Hi. I've not watched these but from the titles they seem to be how-to setup haproxy on OPNSense.
My config is simple but to simplify it a bit more: I have a server on the LAN that hosts nextcloud. It has its own ip address in the LAN space 192.168.5.158. On it is a kind of LAMP stack yes, apache as the webserver, mysql database and redis. Nothing other than the nextcloud requirements except the data, database and configs are on zfs datasets.
The important part for your question is that on OPN's haproxy, all I do is reverse proxy the connections to that ip.
So in your case to reverse proxy from haproxy to your nextcloud installation, you create your backend and real server. Your pictures there show the backend, looks right although I have added the required:
acl carddav-endpoint path_beg /.well-known/carddav
http-request set-path /remote.php/dav if carddav-endpointas passthrough options. Note this is something you might want but is not why you would get an http 503, that you are dealing with.
you need to revise your "real server" on haproxy settings. It needs to be the ip of the webserver of nextcloud.
Quote from: Maginos on March 14, 2025, 09:50:13 AMSo here are the screenshots for the backend pool, rule and condition.
You showed a rule though, however, where is it used?
Also your condition doesn't show the value. And is there any reason for configuring a "host stars with" condition? For a single hostname, I's rather use a "host is" condition.
Quote from: cookiemonster on March 14, 2025, 10:37:28 PMyou need to revise your "real server" on haproxy settings. It needs to be the ip of the webserver of nextcloud.
Yes, I understand that. In this case, it is the IP address of my docker host. And also port 11000 is correct. Maybe it has something to do with SSL, I'm not sure. The "SSL" box is checked, "Verify SSL" is not. Should be right. I have the same setting for the linuxserver container on the same docker host, so this should be ok. Do you have any other ideas, where the problem could be?
Quote from: viragomann on March 14, 2025, 10:55:41 PMYou showed a rule though, however, where is it used?
Also your condition doesn't show the value. And is there any reason for configuring a "host stars with" condition? For a single hostname, I's rather use a "host is" condition.
The rule is used at the "Public Service".
Yes, the condition shows no value, because there's my domain written and I don't want make it public.
I changed the "host starts with" setting to "host matches", but that didn't change a thing, unfortunately. I still get the error 503.
Do you have any other ideas?
@cookiemonster and @viragomann:
I found, that for the health monitoring in haproxy, the server is marked as down, maybe there is a problem.
I also attached the entries in the logs of haproxy and the settings for the health checks.
Maybe you can comment on that.
EDIT: I was able to get rid of the issue with health monitoring. Now the server is displayed as "UP" in the health monitoring. Unfortunately, the error 503 still persists.
EDIT 2: When I try to access the NC via mobile phone, I get the message "An SSL error has occured. A safe connection to the server can not be established". The certificate is the right one. What could be the problem?
503 indicates, that HAproxy doesn't get a response from the backend.
If you check the status is it even shown as online?
Yes, it is shown now as "UP" (see screenshot).
BUT: When I try to access the NC via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG
The website can not be displayed, because the authenticity of the recieved data can not be verified".
Does that help?
We're still guessing your setup.
- Is your AIO thinguie using SSL/TLS ie using a certificate? Is it a signed by a major CA or is it self signed?
- You seem to be using a non-standard port of 11000. You put that in your server entry in HAProxy, right?
- Try from the command line ON OPN:
$ wget http://192.168.5.158
--2025-03-15 22:14:29-- http://192.168.5.158/
Connecting to 192.168.5.158:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.5.158/login [following]
--2025-03-15 22:14:29-- https://192.168.5.158/login
Connecting to 192.168.5.158:443... failed: Connection refused.You see in my attempt for showing you, I am attempting the connection to the port I have set (default 80 http). It connects. Then follows to move the connection and then login. This is not important. What is important is that there was a successful connection.
- Where are you trying to connect from? Very important.
Quote from: cookiemonster on March 15, 2025, 11:19:19 PM- Is your AIO thinguie using SSL/TLS ie using a certificate? Is it a signed by a major CA or is it self signed?
I have a certificate from Lets Encrypt, that works for other services perfectly fine.
Quote from: cookiemonster on March 15, 2025, 11:19:19 PM- You seem to be using a non-standard port of 11000. You put that in your server entry in HAProxy, right?
11000 IS the standard port for Nextcloud AiO, specifically the Apache port. And yes, I configured that in haproxy.
Quote from: cookiemonster on March 15, 2025, 11:19:19 PM- Try from the command line ON OPN:
$ wget http://192.168.5.158
--2025-03-15 22:14:29-- http://192.168.5.158/
Connecting to 192.168.5.158:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.5.158/login [following]
--2025-03-15 22:14:29-- https://192.168.5.158/login
Connecting to 192.168.5.158:443... failed: Connection refused.You see in my attempt for showing you, I am attempting the connection to the port I have set (default 80 http). It connects. Then follows to move the connection and then login. This is not important. What is important is that there was a successful connection.
- Where are you trying to connect from? Very important.
So I entered the following command in the shell of my OPNSense and got this output:
root@OPNsense:~ # wget http://10.50.1.2:11000
--2025-03-16 09:14:14-- http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 09:14:14-- https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 09:14:18 ERROR 503: Service Unavailable.Explanation:
10.50.1.2 is my docker host (located in another network than all other devices at home)
10.20.1.1 is my OPNSense
So in the third last line it says "... connected", but then the 503 error occurs. What can we derive from this?
Since it seems to be important: I configured a DNS Override in Unbound, that querys to subdomain.domain.tld are redirected to 10.20.1.1 (IP of haproxy), so that queries from my home network do not leave the OPNSense and are redirected directly to the haproxy and then to the docker host.
Quote from: cookiemonster on March 15, 2025, 11:19:19 PM- Where are you trying to connect from? Very important.
When I try to connect from outside my home network, I get the "An SSL error has occured. A safe connection to the server can not be established" error.
When I try to access the NC from inside my LAN network via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG"
I hope, that clarifies it.
Quote from: Maginos on March 16, 2025, 09:34:43 AMI have a certificate from Lets Encrypt, that works for other services perfectly fine.
I'm wondering, who you got an LE certificate behind HAproxy.
Quote from: Maginos on March 16, 2025, 09:34:43 AMSo in the third last line it says "... connected", but then the 503 error occurs. What can we derive from this?
Since it seems to be important: I configured a DNS Override in Unbound, that querys to subdomain.domain.tld are redirected to 10.20.1.1 (IP of haproxy), so that queries from my home network do not leave the OPNSense and are redirected directly to the haproxy and then to the docker host.
Nextcloud obviously redirects the request to https://subdomain.domain.tld/login, since it is configured to use this domain.
Seems normal to me as far.
Can you configure the AIO to accept simply the IP as well?
You will have to use the "--no-check-certificate" option in wget then, in case it requires SSL.
Is the AIO even aware of the reverse proxy?
Quote from: Maginos on March 16, 2025, 09:34:43 AMWhen I try to access the NC from inside my LAN network via https://internal_ip:11000
This would require, that the internal IP is included in the SSL certificate, which I'm in doubt.
Quote from: viragomann on March 16, 2025, 06:00:11 PMI'm wondering, who you got an LE certificate behind HAproxy.
No, for behind the proxy I have the default certificate from Nextcloud.
Quote from: viragomann on March 16, 2025, 06:00:11 PMCan you configure the AIO to accept simply the IP as well?
You will have to use the "--no-check-certificate" option in wget then, in case it requires SSL.
I added 10.20.1.1 to the trusted proxy section in the config.php, restarted all containers and executed the command again. This is the output:
root@OPNSense:~ # wget --no-check-certificate http://10.50.1.2:11000
--2025-03-16 20:37:07-- http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 20:37:07-- https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 20:37:10 ERROR 503: Service Unavailable.Seems to be the same output as before.
Quote from: viragomann on March 16, 2025, 06:00:11 PMIs the AIO even aware of the reverse proxy?
Yes it is, the IP of the proxy is also located in the trusted proxy section of the config.php.
Quote from: viragomann on March 16, 2025, 06:00:11 PMThis would require, that the internal IP is included in the SSL certificate, which I'm in doubt.
When the AiO is "freshly" set up, it is possible to access the AiO NC interface via the internal IP. There is no issue with the certificate.
Ok, I found something: In the "Real Server" section, the box for "SSL" was checked. Since the connection from the proxy to the AiO is via http, not via https, I think checking the SSL box is wrong.
After unchecking it, I get a "400 bad request" error and not the 503 error any more. When I execute the wget command, the last two lines change to the "new" error.
In the logs of Nextcloud I don't find anything that matches in time.
Quote from: Maginos on March 16, 2025, 08:58:51 PMI added 10.20.1.1 to the trusted proxy section in the config.php
You have to add it to the trusted domains.
This is what I mean. We don't know so far if you were communicating haproxy and Nextcloud via https. You have only now said it is http. I'm not guessing anymore. Draw it with the endpoints shown and which protocols. It would be useful for you too. Until then when you accurately describe the setup I won't be inclined to chime in.
p.s. if you are getting http errors (400, 503, etc.) that means the network communication is happening successfully. The reverse proxy is reaching the webserver. You want to have a look not only on nextcloud's logs but also and more importantly at this point, your webserver logs. Apache is hhtp-access and http-error (normally). But you could be using a different webserver. Depends on the AIO thinguie setup.
Quote from: viragomann on March 16, 2025, 10:49:53 PMYou have to add it to the trusted domains.
Tried it, did unfortunately not help.
Quote from: cookiemonster on March 16, 2025, 10:56:54 PMThis is what I mean. We don't know so far if you were communicating haproxy and Nextcloud via https. You have only now said it is http. I'm not guessing anymore. Draw it with the endpoints shown and which protocols. It would be useful for you too. Until then when you accurately describe the setup I won't be inclined to chime in.
p.s. if you are getting http errors (400, 503, etc.) that means the network communication is happening successfully. The reverse proxy is reaching the webserver. You want to have a look not only on nextcloud's logs but also and more importantly at this point, your webserver logs. Apache is hhtp-access and http-error (normally). But you could be using a different webserver. Depends on the AIO thinguie setup.
I'm sorry, I made a mistake. I thought the proxy communicates via https, but http is correct. I checked the nginx proxy manager config on the github page (https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md) of the NC AiO and now I'm 100% sure, that it communicates via http.
So the connection from the client to the haproxy is via https, the connection from haproxy to the NC AiO is via http.
Here are the logs of the Apache web server:
docker logs nextcloud-aio-apache
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Connection to nextcloud-aio-nextcloud (172.19.0.7) 9000 port [tcp/*] succeeded!
{"level":"info","ts":1742231207.8322253,"msg":"using config from file","file":"/tmp/Caddyfile"}
{"level":"info","ts":1742231207.8334265,"msg":"adapted config to JSON","adapter":"caddyfile"}
[Mon Mar 17 18:06:47.844311 2025] [mpm_event:notice] [pid 46:tid 46] AH00489: Apache/2.4.63 (Unix) configured -- resuming normal operations
[Mon Mar 17 18:06:47.844343 2025] [core:notice] [pid 46:tid 46] AH00094: Command line: '/usr/local/apache2/bin/httpd -D FOREGROUND'
I also checked the logs of all other nextcloud containers and that one of OPNSense and I found nothing suspicious. The only thing I found was this error message:
Error connection: Server error: `GET https://subdomain.domain.tld/onlyoffice/healthcheck` resulted in a `503 Service Unavailable` response:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>I then disabled OnlyOffice via a occ command and tried to access the NC afterwards, but that didn't help.
Are these information enough or do you need more?
Quote from: Maginos on March 17, 2025, 06:26:47 PMQuoteYou have to add it to the trusted domains.
Tried it, did unfortunately not help.
This should just enable to access it using the container IP from internal for testing purposes.
Quote from: viragomann on March 17, 2025, 06:33:02 PMQuote from: Maginos on March 17, 2025, 06:26:47 PMQuoteYou have to add it to the trusted domains.
Tried it, did unfortunately not help.
This should just enable to access it using the container IP from internal for testing purposes.
When I enter http://10:50.1.2:11000, I get redirected to https://subdomain.domain.tld and there I get the error 400 bad request.
I don't know, what redirects the request then. Possibly the web server settings.
If I add the IP of my Nextcloud VM to the trusted domain, I can connect to it using the IP. The browser is complaining about an invalid SSL certificate, however, but if I add an exception, I can connect.
If the IP is missing in the trusted domains, it also doesn't redirect me to somewhere else, but complains that the domain is not trusted.
Did you even obey the instructions at "Reverse Proxy Documentation (https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md)?
I deactivated the DNS Override in OPNSense, but that didn't help as well...
You added the IP of your docker host to the trusted domain section? Ok.
Yes I followed that documentation. Maybe I should add, that the AiO Container worked for my former Sophos UTM. Initially, I set up the AiO Container with a Sophos UTM Firewall and everything worked perfectly fine. Now I switched to OPNSense and I get the error 400.
Edit: I tried to prevent the redirect by commenting out the overwriteprotocol, overwritehost and overwrite.cli.url, but that didn't help.
Quote from: Maginos on March 17, 2025, 07:21:04 PMYou added the IP of your docker host to the trusted domain section?
I'm actually running a Nextcloud simply in a VM on Apache web server, no AIO, no docker. But it runs behind HAproxy on OPNsense.
Quote from: Maginos on March 17, 2025, 07:21:04 PMInitially, I set up the AiO Container with a Sophos UTM Firewall and everything worked perfectly fine
Behind a reverse proxy? Presumably not. And that's the point here, I guess. Why should this be an issue with OPNsense??
Quote from: Maginos on March 17, 2025, 07:21:04 PMNow I switched to OPNSense and I get the error 400.
400? We were troubleshooting 503, as you stated before.
Maybe any details to see in the HAproxy log?
Quote from: Maginos on March 17, 2025, 07:21:04 PMEdit: I tried to prevent the redirect by commenting out the overwriteprotocol, overwritehost and overwrite.cli.url, but that didn't help.
Yes, at least the override host must be removed, when accessing via IP.