Wan connection fails after upgrading to 26.7.3

[sber]

After upgrading to 26.7.3 WAN connection failed on my DEC750, installed hotfixes to 26.7.3_8 but this didn't solve the problem. Serial console shows:

[fib_algo] inet.0 setup_fd_instance: radix4_lockless algo instance setup failed, failures=64
[fib_algo] inet.0 (radix4_lockless#30) rebuild_fd_flm: table rebuild failed
[fib_algo] inet.0 (radix4_lockless#30) rebuild_fd: sync rebuild failed.

Reinstalled 26.7 and imported configuration, system is up and running for now.

[newsense]

igb or igc interfaces ?

[RamSense]

I can confirm the same radix4_lockless problem on a DEC850v2 running OPNsense 26.7.3_8.

In my case WAN itself remained available, but IPv4 routing to a WireGuard interface was broken after restoring the configuration onto a replacement SSD. The WireGuard tunnel had a healthy handshake and IPv6 through the same tunnel worked normally.

The kernel log contains repeated errors:

[fib_algo] inet.0 setup_fd_instance: radix4_lockless algo instance setup failed
[fib_algo] inet.0 (radix4_lockless#43) rebuild_fd_flm: table rebuild failed
[fib_algo] inet.0 (radix4_lockless#43) rebuild_fd: sync rebuild failed

The failure counter reached at least 42.

The interesting part is that the routing table itself looked completely correct:

route -n get 10.10.10.2
  interface: wg0

netstat -rWn4:
10.10.10.2  link#11  UHS  Nhop#16  wg0

netstat -4onW:
Nhop#16:
IFA      10.10.10.1
Gateway  wg0/resolve
Netif    wg0

Despite this, normal IPv4 traffic to 10.10.10.2 was incorrectly sent towards pppoe0.

A normal:

ping -c 1 10.10.10.2

failed, while:

ping -r -S 10.10.10.1 -c 1 10.10.10.2

worked immediately.

IPv6 to the same WireGuard peer also worked normally.

The decisive test was changing only the IPv4 FIB algorithm:

sysctl net.route.algo.inet.algo=radix4

Immediately afterwards the normal IPv4 ping worked through wg0 and full IPv4 Internet access through WireGuard was restored.

Changing nothing else was required.

Current workaround:

System > Settings > Tunables

net.route.algo.inet.algo = radix4

So this appears to be a radix4_lockless FIB rebuild/lookup failure rather than a WireGuard, NAT or firewall configuration problem.

Hardware: DEC850v2
OPNsense: 26.7.3_8
WAN: PPPoE
IPv4 default algorithm before workaround: radix4_lockless
Available algorithms: bsearch4, radix4_lockless, radix4

I have now also filed this as OPNsense src issue #323, including the full diagnostic results and the reproducible radix4_lockless → radix4 A/B test.

GitHub: https://github.com/opnsense/src/issues/323

Hopefully this helps confirm that the radix4_lockless rebuild errors can cause different IPv4 routing failures depending on the system/configuration.

[newsense]

There weren't any igc updates for 26.7.3 afaik...

Are you sure there weren't any radix issues when booting the previous kernel ?

[RamSense]

That's a fair point.

I cannot be sure that the previous kernel was free of radix4_lockless errors, because unfortunately I did not save the dmesg output from the previous installation before replacing the SSD. The SSD had reached a very high level of wear after substantial disk writes, which I am investigating separately with Zenarmor because of the amount of write activity generated by its logging. Warning for other users here.

In fact, I think I may have seen similar fib_algo/radix messages before, while WireGuard and normal IPv4 routing were still working correctly.

So I don't want to claim that 26.7.3 necessarily introduced the radix issue, nor that this is related to an igc change.

It is quite possible that the radix4_lockless problem was already present but latent/non-impacting, and that the clean installation/config restore exposed it during a fresh FIB rebuild.

What I can reproduce reliably on the current 26.7.3_8 installation is:

- radix4_lockless repeatedly reports setup/rebuild failures
- the routing table and nexthops still appear correct
- actual IPv4 lookup to wg0 is wrong
- IPv6 is unaffected
- changing only net.route.algo.inet.algo from radix4_lockless to radix4 immediately restores both the normal IPv4 route to the WireGuard peer and full IPv4 Internet access through WireGuard

So the exact version where the radix errors first appeared is unknown. The reproducible part is the radix4_lockless failure and the radix4 workaround on the current installation.

[emmitt]

I just wanted to confirm that there is another error in the same hardware (Deciso 750) / software (26.7.3_8) combination.

Unfortunately, my friend has already performed a clean install and restored the configuration data, so I can't provide any further information.

[sber]

There weren't any igc updates for 26.7.3 afaik...

Are you sure there weren't any radix issues when booting the previous kernel ?

I'm back at 26.7.2_2; same situation as before upgrading and see no radix issues in the logs. I will make a snapshot and upgrade again today and hopefully i'm able to get some more information to share.