Messages

Hi sy,

I followed the "Have feedback?" form and sent the logs there.

I hope, they can help me. Fingers crossed.

Hi all,

I have a question concerning Zenarmor periodicals:


Every Day at 4:03 am I get the following message from Zabbix:


Problem: Lack of avail. memory on server OPNsense 2
Problem started at 04:03:13 on 2026.02.08
Problem name: Lack of avail. memory on server OPNsense 2
Host: OPNsense
Severity: Warning
Operational data: 3.45 GB
Original problem ID: 79101

The value at Operational data varies, it can be 450 MB or less.

My Opnsense (version 25.7_11 so far, I just upgraded atm to 26.1) has a total amount of 16 GB of RAM, which is plenty for only Zenarmor running and no Suricata.

I looked up in /var/log/system/latest.log and found NO Out-Of-Memory entry there, so I assume, the OOM error from Zabbix comes from the ARC of my ZFS array. I have two SSDs, which are configured in Z1.

The message from Zabbix I get after 3-4 days and then daily always at 4 am in the morning.

I looked up my cronjobs and only Zenarmor periodicals are running at that time.

After some research I found, that the file /usr/local/datastore/sqlite/conn_all.sqlite keeps growing in size up to some GB.

When this file reaches a certain size, I get the OOM error in zabbix. That's the reason why I see the error in Zabbix after some days.

If I reset the database of Zenarmor via GUI manually, the conn_all.sqlite file is reduced in size and the zabbix error message does not occur for some days.

After that it occurs again and I have to reset the database manually again.

In future, I would like to avoid this.

On a second OPNSense machine, everything works perfectly.

I have following settings on the OPNSense:

- Sqlite
- Reporting Period 2 days
- Memory Disk Size 300 MB
- Tracefs Partition Size 100 MB

Logs:

- Log Level Debug
- Rotation 1 Day
- Retire 3 Days


I already uninstalled and reinstalled Zenarmor (restore from backup), but that didn't help.

So my assumption is, that the database is not properly cleaned up by Zenarmor periodicals cronjob and therefore the conn_all.sqlite file keeps growing.

What can I do in this situation?


Thank you for your help.

Maginos

I deactivated the DNS Override in OPNSense, but that didn't help as well...

You added the IP of your docker host to the trusted domain section? Ok.

Yes I followed that documentation. Maybe I should add, that the AiO Container worked for my former Sophos UTM. Initially, I set up the AiO Container with a Sophos UTM Firewall and everything worked perfectly fine. Now I switched to OPNSense and I get the error 400.

Edit: I tried to prevent the redirect by commenting out the overwriteprotocol, overwritehost and overwrite.cli.url, but that didn't help.

You have to add it to the trusted domains.

Tried it, did unfortunately not help.

This should just enable to access it using the container IP from internal for testing purposes.

When I enter http://10:50.1.2:11000, I get redirected to https://subdomain.domain.tld and there I get the error 400 bad request.

You have to add it to the trusted domains.

Tried it, did unfortunately not help.

This is what I mean. We don't know so far if you were communicating haproxy and Nextcloud via https. You have only now said it is http. I'm not guessing anymore. Draw it with the endpoints shown and which protocols. It would be useful for you too. Until then when you accurately describe the setup I won't be inclined to chime in.
p.s. if you are getting http errors (400, 503, etc.) that means the network communication is happening successfully. The reverse proxy is reaching the webserver. You want to have a look not only on nextcloud's logs but also and more importantly at this point, your webserver logs. Apache is hhtp-access and http-error (normally). But you could be using a different webserver. Depends on the AIO thinguie setup.

I'm sorry, I made a mistake. I thought the proxy communicates via https, but http is correct. I checked the nginx proxy manager config on the github page of the NC AiO and now I'm 100% sure, that it communicates via http.

So the connection from the client to the haproxy is via https, the connection from haproxy to the NC AiO is via http.
Here are the logs of the Apache web server:

Code Select Expand

docker logs nextcloud-aio-apache
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Waiting for Nextcloud to start...
Connection to nextcloud-aio-nextcloud (172.19.0.7) 9000 port [tcp/*] succeeded!
{"level":"info","ts":1742231207.8322253,"msg":"using config from file","file":"/tmp/Caddyfile"}
{"level":"info","ts":1742231207.8334265,"msg":"adapted config to JSON","adapter":"caddyfile"}
[Mon Mar 17 18:06:47.844311 2025] [mpm_event:notice] [pid 46:tid 46] AH00489: Apache/2.4.63 (Unix) configured -- resuming normal operations
[Mon Mar 17 18:06:47.844343 2025] [core:notice] [pid 46:tid 46] AH00094: Command line: '/usr/local/apache2/bin/httpd -D FOREGROUND'

I also checked the logs of all other nextcloud containers and that one of OPNSense and I found nothing suspicious. The only thing I found was this error message:

Code Select Expand

Error connection: Server error: `GET https://subdomain.domain.tld/onlyoffice/healthcheck` resulted in a `503 Service Unavailable` response:
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
I then disabled OnlyOffice via a occ command and tried to access the NC afterwards, but that didn't help.

Are these information enough or do you need more?

I'm wondering, who you got an LE certificate behind HAproxy.

No, for behind the proxy I have the default certificate from Nextcloud.

Can you configure the AIO to accept simply the IP as well?
You will have to use the "--no-check-certificate" option in wget then, in case it requires SSL.

I added 10.20.1.1 to the trusted proxy section in the config.php, restarted all containers and executed the command again. This is the output:

Code Select Expand

root@OPNSense:~ # wget --no-check-certificate http://10.50.1.2:11000
--2025-03-16 20:37:07--  http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 20:37:07--  https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 20:37:10 ERROR 503: Service Unavailable.
Seems to be the same output as before.

Is the AIO even aware of the reverse proxy?

Yes it is, the IP of the proxy is also located in the trusted proxy section of the config.php.

This would require, that the internal IP is included in the SSL certificate, which I'm in doubt.

When the AiO is "freshly" set up, it is possible to access the AiO NC interface via the internal IP. There is no issue with the certificate.


Ok, I found something: In the "Real Server" section, the box for "SSL" was checked. Since the connection from the proxy to the AiO is via http, not via https, I think checking the SSL box is wrong.
After unchecking it, I get a "400 bad request" error and not the 503 error any more. When I execute the wget command, the last two lines change to the "new" error.

In the logs of Nextcloud I don't find anything that matches in time.

- Is your AIO thinguie using SSL/TLS ie using a certificate? Is it a signed by a major CA or is it self signed?

I have a certificate from Lets Encrypt, that works for other services perfectly fine.

- You seem to be using a non-standard port of 11000. You put that in your server entry in HAProxy, right?

11000 IS the standard port for Nextcloud AiO, specifically the Apache port. And yes, I configured that in haproxy.

- Try from the command line ON OPN:

Code Select Expand

$ wget http://192.168.5.158
--2025-03-15 22:14:29--  http://192.168.5.158/
Connecting to 192.168.5.158:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://192.168.5.158/login [following]
--2025-03-15 22:14:29--  https://192.168.5.158/login
Connecting to 192.168.5.158:443... failed: Connection refused.You see in my attempt for showing you, I am attempting the connection to the port I have set (default 80 http). It connects. Then follows to move the connection and then login. This is not important. What is important is that there was a successful connection.
- Where are you trying to connect from? Very important.

So I entered the following command in the shell of my OPNSense and got this output:

Code Select Expand

root@OPNsense:~ # wget http://10.50.1.2:11000
--2025-03-16 09:14:14--  http://10.50.1.2:11000/
Connecting to 10.50.1.2:11000... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://subdomain.domain.tld/login [following]
--2025-03-16 09:14:14--  https://subdomain.domain.tld/login
Resolving subdomain.domain.tld (subdomain.domain.tld)... 10.20.1.1
Connecting to subdomain.domain.tld (subdomain.domain.tld)|10.20.1.1|:443... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2025-03-16 09:14:18 ERROR 503: Service Unavailable.
Explanation:
10.50.1.2 is my docker host (located in another network than all other devices at home)
10.20.1.1 is my OPNSense

So in the third last line it says "... connected", but then the 503 error occurs. What can we derive from this?
Since it seems to be important: I configured a DNS Override in Unbound, that querys to subdomain.domain.tld are redirected to 10.20.1.1 (IP of haproxy), so that queries from my home network do not leave the OPNSense and are redirected directly to the haproxy and then to the docker host.

- Where are you trying to connect from? Very important.

When I try to connect from outside my home network, I get the "An SSL error has occured. A safe connection to the server can not be established" error.

When I try to access the NC from inside my LAN network via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG"

I hope, that clarifies it.

Yes, it is shown now as "UP" (see screenshot).

BUT: When I try to access the NC via https://internal_ip:11000, I get a SSL error that says: "SSL got an entry, which exceeded the maximum permitted length. Errorcode: SSL_ERROR_RX_RECORD_TOO_LONG

The website can not be displayed, because the authenticity of the recieved data can not be verified".


Does that help?

@cookiemonster and @viragomann:

I found, that for the health monitoring in haproxy, the server is marked as down, maybe there is a problem.
I also attached the entries in the logs of haproxy and the settings for the health checks.

Maybe you can comment on that.

EDIT: I was able to get rid of the issue with health monitoring. Now the server is displayed as "UP" in the health monitoring. Unfortunately, the error 503 still persists.

EDIT 2: When I try to access the NC via mobile phone, I get the message "An SSL error has occured. A safe connection to the server can not be established". The certificate is the right one. What could be the problem?

you need to revise your "real server" on haproxy settings. It needs to be the ip of the webserver of nextcloud.

Yes, I understand that. In this case, it is the IP address of my docker host. And also port 11000 is correct. Maybe it has something to do with SSL, I'm not sure. The "SSL" box is checked, "Verify SSL" is not. Should be right. I have the same setting for the linuxserver container on the same docker host, so this should be ok. Do you have any other ideas, where the problem could be?

You showed a rule though, however, where is it used?

Also your condition doesn't show the value. And is there any reason for configuring a "host stars with" condition? For a single hostname, I's rather use a "host is" condition.

The rule is used at the "Public Service".

Yes, the condition shows no value, because there's my domain written and I don't want make it public.

I changed the "host starts with" setting to "host matches", but that didn't change a thing, unfortunately. I still get the error 503.

Do you have any other ideas?

Please know that I do not do docker, have no interest whatsoever in it. If anything I want to run is docker-only, I move on looking for an alternative that does have "old style" application configurations.
So now let's establish the overall setup. By the way mine is very largely based on this haproxy-on-opn-tutorial-by-thehellsite.
Take a look at the picture to figure out if you are on the same setup and if different, please explain it. But you can follow that as a basis for an uncomplicated setup: create real server (your nextcloud), create a backend containing that server, create a front end WITH A TLS CERT, create firewall rules to allow the front end to be accessed. Here your haproxy --> backend server can be http or https but if you do https, then you have to deal with those certs separately. Makes sense?

So you set up NC with a LAMP Stack, I guess? Unfortunately this doesn't help me, because I already have a running NC instance from a Linuxserver Docker Container. This instance works, but the AiO Setup does not.

I set up haproxy with following videos:

So my configuration differs in several ways from the setup of the link you posted above.

And now I'm not sure, if my configuration is wrong/disadvantageous.

By the way mine is very largely based on this haproxy-on-opn-tutorial-by-thehellsite.
Take a look at the picture to figure out if you are on the same setup and if different, please explain it. But you can follow that as a basis for an uncomplicated setup: create real server (your nextcloud), create a backend containing that server, create a front end WITH A TLS CERT, create firewall rules to allow the front end to be accessed. Here your haproxy --> backend server can be http or https but if you do https, then you have to deal with those certs separately. Makes sense?

Thank you for the link, I will compare configurations. I followed the overall setup guides for other services, so NC is not the first service I want to access from outside. Other services work fine for 95% of the time.


@petrij98: The port of OPNSense was changed. As I wrote above, I have several other services, which are accessible through HAProxy.

And here are bigger screenshots for the backend pool

@Patrick M. Hausen thank you for the tip.
So here are the screenshots for the backend pool, rule and condition.

Hi guys and thank you for your replies.

@cookiemonster: Since I was not able to paste images directly in the post, I made another PDF with further screenshots of my config. I can understand, that you don't trust my PDF, so how can I show you the screenshots in a way, that works for you?

@petrij98: I just want the NC to be accessible from outside my home network. I have an LE certificate for the subdomain and yes, I am sure, that I call the correct subdomain. I even checked with DNSchecker, that public DNS resolvers point to my public IPv4 for that domain.