after upgrade from OPNsense 25.1.12 to 25.7 / 25.7.1, observing frequent frr OSPF Error logs, immediately after OSPF neighbors are in Full state
Prior to the upgrade to 25.7 / 25.7.1 these errors are not observed.
Just as well i have ESXi VM snapshot, considering rolling back....to 25.1.12....
At bit of googling found the below....
Explanation of the Discrepancy:
This discrepancy occurs because the operating system has a maximum limit for the SO_SNDBUF size that can be set for a socket. Even if an application requests a larger buffer, the kernel will cap it at its configured maximum. This limit is typically controlled by a sysctl variable on FreeBSD, such as kern.ipc.maxsockbuf.
so i increased the tunable kern.ipc.maxsockbuf from the default value, to 33554432 (32Mbytes), and re-started frr and has resolved the issue....
Anybody know if any downsides by increasing kern.ipc.maxsockbuf to 32Mbytes ?
While I do not know much about OSPF, I have looked up a lot of tuning elements for OPNSense and FreeBSD as mine runs on older metal host and I'm doing 10G intranet.
https://calomel.org/freebsd_network_tuning.html
This website ^ has details about many tuning elements, but they use a different reference of values for kern.ipc.maxsockbuf and so while I do not think there is a limitation/drawback to increasing the value (much the opposite it seems), using a logical value seems wise.
For my router, I have the value set at: 614400000
Seems yours is already working, but who knows if there is odd grouping/read/writes to that space due to a unqiue value - that said, how this values comes to be seems pretty odd to me, and so I didn't do all the work necessary to evaluate your "33554432" - just wanted to share a resource I found that has helped lower buffer bloat and latency.
The specific part from their website that seems important:
# speed: 1 Gbit maxsockbuf: 2MB wscale: 6 in-flight: 2^6*65KB = 4MB (default)
# speed: 2 Gbit maxsockbuf: 4MB wscale: 7 in-flight: 2^7*65KB = 8MB
# speed: 10 Gbit maxsockbuf: 16MB wscale: 9 in-flight: 2^9*65KB = 32MB
# speed: 40 Gbit maxsockbuf: 150MB wscale: 12 in-flight: 2^12*65KB = 260MB
# speed: 100 Gbit maxsockbuf: 600MB wscale: 14 in-flight: 2^14*65KB = 1064MB
#
#kern.ipc.maxsockbuf=2097152 # (wscale 6 ; default)
#kern.ipc.maxsockbuf=4194304 # (wscale 7)
kern.ipc.maxsockbuf=16777216 # (wscale 9)
#kern.ipc.maxsockbuf=157286400 # (wscale 12)
#kern.ipc.maxsockbuf=614400000 # (wscale 14)
i've been monitoring mbuf and memory utilization, and so far so good.
I suspect the new version of frr, decided that as vmxnet3 (vmx) interface's runs @10Gbps, it decided for optimization reasons, to request 32Mbytes of SO_SNDBUF to FreeBSD.....which FreeBSD denied and decided to limit SO_SNDBUF to 2Mbytes....Looks like FreeBSD are a bit behind the optimization times.....
It's a bit interesting that frr responded to this FreeBSD limitation as an error, rather than a warning...I might ping the frr project to get their view on the matter, see what they have to say....
I also run frr 10.3 and 10.4 on a number (dozens) of other production linux (Debian, Ubuntu and Rocky) hosts/routers, running a mix of BFD. OSPF and BGP, with as little as 1Gbyte RAM, also with vmxnet3 interfaces @10Gbps, and frr is not complaining....and performance is just superb...
To be fair, FreeBSD ports has had a package message regarding the proper fix:
https://github.com/opnsense/ports/blob/ec41f23e1fecedb87f4671fc554389db176835b4/net/frr10/pkg-message#L10-L14
Cheers,
Franco
I have tested that earlier and for some reason since frr10 you need to specify double the amount to satisfy requirements:
https://github.com/opnsense/plugins/issues/4840#issuecomment-3126806779