"ping: sendto: No buffer space available" under heavy load using pfatt bypass

[neel]

I have Minisforum MS-01 running OPNsense 24.7. This is connected to Frontier FiberOptic (ex-AT&T Fiber) using 802.1X with the AT&T Fiber bypass here: https://github.com/MonkWho/pfatt

My connection runs 6 Tor relays where each instance has 8000+ TCP sessions open at once.

I have a problem: when running my server for more than a couple hours, the WAN goes down and I get an "ping: sendto: No buffer space available" issue. Rebooting brings back the connection until this happens again. If I use the Minisforum i226 NICs, it happens within 1-2 hours. Using a Dell Broadcom BCM5720 NIC, it lasts longer but goes down around 12 hours.

Is there a way to solve this?

When I had the issues, I didn't have any tunables but I'm going to try this in /boot/loader.conf.local:

Code Select Expand

kern.ipc.maxsockbuf=68194304
kern.ipc.nmbclusters=32311168
kern.ipc.nmbjumbop=16155584

[Patrick M. Hausen]

Use System > Settings > Tunables instead of loader.conf.local if you want the tunables to take effect.

[neel]

That's what I'm actually doing now, using the tunables section in the WebUI.

[neel]

I've found the real culprit (hopefully), apparently logging while using pfatt causes issues, that or the buffers were too small.

Tor relays open a lot of TCP connections and trying to log each session while running bypass for some reason kills my internet. pfSense+ does have the advantage of a built-in AT&T bypass but CE lacks it and Netgate no longer gives homelab licenses.

I've had 24 hours of uptime since I disabled logging while allowing ~150-250 Mbps concurrent Tor traffic.

[neel]

Nevermind, it crashed after about 30 hours. I'm trying something else now: the obscure net.graph.maxdgram and net.graph.recvspace tunables. For some reason they were very small and while it works for most bypass applications crashes with Tor relays.

This forum post had some tunables and got me thinking, it must be netgraph buffers: https://freebsd-questions.freebsd.narkive.com/lPixO7df/ping-sendto-no-buffer-space-available