MBUF Exhaust when Using NGINX-ReverseProxy for NextCloud-Publish.

[itngo]

Hi,

maybe someone can say something about this.

We publish Nextcloud with NGINX. Response-Buffering and Request-Buffering is already disabled.

We can upload and download files and use Webfrontend in Nextcloud flawlessly.

However when we download a larger file from Nextcloud we have massive MBUF-Exhaust and the opnsense freezes at least.

Already put kern.ipc.nmbclusters to 2000000 but they still get exhausted by a simple download. We also see that even setting kern.ipc.nmbclusters to 6 or 12 millions does not help, as opnsense will never use more than about 2.2 millions even there is still plenty of RAM available. And the freeze or stop traffic flow at all.

It looks like NGINX is loading the 2.x Gigabyte file directly to the MBUFs while client has only limited bandwidth and can not get the data fast enough to release the MBUFs again.....

We already tried with shaping in opnsense and also with request-limits in NGINX, but both have no real effect. The Downloads slow down, but the "upstream" saturation still goes 900Mbit and more and kills the opnsense within 2 or 3 minutes....

Anyone has an idea where to start?

[itngo]

Also See https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093027.html as this looks like a comparable issue....

[meyergru]

If it is, then it points to sendfile... have you turned that off?

[itngo]

If it is, then it points to sendfile... have you turned that off?

Sendfile is disabled in HTTP Server and also General Settings.

[Fright]

hi
actually the value of kern.ipc.maxsockbuf was assigned as the culprit (https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093041.html)
and would try the other way around with the sendfile enabled.
is proxy_buffering enabled?

[itngo]

hi
actually the value of kern.ipc.maxsockbuf was assigned as the culprit (https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093041.html)
and would try the other way around with the sendfile enabled.
is proxy_buffering enabled?

Maybe, meanwhile we can confirm this does not only happen when using Webserver published by NGINX.
We have running Proxmox and today wanted to restore a Backup from Proxmox Backup Server which is in the Cloud.

So traffic needed to flow through the opnsense where one end is the PBS in our Datacenter and the other and, the PVE is at customer-site. Connected with IPSEC. When we start to download a Backup-File the same problem appears.
MBUF gets exhausted in minutes until the system comes to full freeze....

[itngo]

hi
actually the value of kern.ipc.maxsockbuf was assigned as the culprit (https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093041.html)
and would try the other way around with the sendfile enabled.
is proxy_buffering enabled?

Yep! That's it! PHEW.. thank you... we had kern.ipc.maxsockbuf= 614400000
after returning it to default "4262144" problem is gone....

Reminder: Don't follow every dumb tuning guide and never think again about what you have done to your Firewall....

[itngo]

hi
actually the value of kern.ipc.maxsockbuf was assigned as the culprit (https://lists.freebsd.org/pipermail/freebsd-stable/2021-February/093041.html)
and would try the other way around with the sendfile enabled.
is proxy_buffering enabled?

Yep! That's it! PHEW.. thank you... we had kern.ipc.maxsockbuf= 614400000
after returning it to default "4262144" problem is gone....

Reminder: Don't follow every dumb tuning guide and never think again about what you have done to your Firewall....

New problem... now UNBOUND is complaining about Bufferspace.... 

2023-04-18T15:55:01   Error   unbound   [3702:5] error: can't create socket: No buffer space available

[Fright]

may be some customs in unbound (so-rcvbuf)?

[itngo]

may be some customs in unbound (so-rcvbuf)?

I bet it is something like that.... can you give me a hint where to look? so-rcvbuf does not give much where I can find this setting in opnsense? Is it a tunable?

[Fright]

no gui for custom options anymore
https://docs.opnsense.org/manual/unbound.html#advanced-configurations

[itngo]

Ok, we did not change any settings there. Is it necessary to set something there if UNBOUND is still on default settings?

no gui for custom options anymore
https://docs.opnsense.org/manual/unbound.html#advanced-configurations

[Fright]

Is it necessary to set something there if UNBOUND is still on default settings?

in that case no
but "can't create socket: No buffer space available" looks like a sysctl tuning consequences for me (values not adjusted adjusted one to another)? (or the system was not rebooted)

[itngo]

Is it necessary to set something there if UNBOUND is still on default settings?

in that case no
but "can't create socket: No buffer space available" looks like a sysctl tuning consequences for me (values not adjusted adjusted one to another)? (or the system was not rebooted)

Yeah, I think you are right, we messed around with these tunables to much.
Is there a list which are the default Setup-Default-TunableSettings? Or can I just delete all tunables and they get reset to default?

[itngo]

Small Update. We created Shaper Pipes, Rules and Queues do limit bandwidth to the backend-server which is hosting the Nextcloud.

This prevents the MBUF-Underrun. It is not the solution, more a workaround to prevent the system from exhaust and crash.