Wan connection fails after upgrading to 26.7.3

[sber]

After upgrading to 26.7.3 WAN connection failed on my DEC750, installed hotfixes to 26.7.3_8 but this didn't solve the problem. Serial console shows:

[fib_algo] inet.0 setup_fd_instance: radix4_lockless algo instance setup failed, failures=64
[fib_algo] inet.0 (radix4_lockless#30) rebuild_fd_flm: table rebuild failed
[fib_algo] inet.0 (radix4_lockless#30) rebuild_fd: sync rebuild failed.

Reinstalled 26.7 and imported configuration, system is up and running for now.

[newsense]

igb or igc interfaces ?

[RamSense]

I can confirm the same radix4_lockless problem on a DEC850v2 running OPNsense 26.7.3_8.

In my case WAN itself remained available, but IPv4 routing to a WireGuard interface was broken after restoring the configuration onto a replacement SSD. The WireGuard tunnel had a healthy handshake and IPv6 through the same tunnel worked normally.

The kernel log contains repeated errors:

[fib_algo] inet.0 setup_fd_instance: radix4_lockless algo instance setup failed
[fib_algo] inet.0 (radix4_lockless#43) rebuild_fd_flm: table rebuild failed
[fib_algo] inet.0 (radix4_lockless#43) rebuild_fd: sync rebuild failed

The failure counter reached at least 42.

The interesting part is that the routing table itself looked completely correct:

route -n get 10.10.10.2
  interface: wg0

netstat -rWn4:
10.10.10.2  link#11  UHS  Nhop#16  wg0

netstat -4onW:
Nhop#16:
IFA      10.10.10.1
Gateway  wg0/resolve
Netif    wg0

Despite this, normal IPv4 traffic to 10.10.10.2 was incorrectly sent towards pppoe0.

A normal:

ping -c 1 10.10.10.2

failed, while:

ping -r -S 10.10.10.1 -c 1 10.10.10.2

worked immediately.

IPv6 to the same WireGuard peer also worked normally.

The decisive test was changing only the IPv4 FIB algorithm:

sysctl net.route.algo.inet.algo=radix4

Immediately afterwards the normal IPv4 ping worked through wg0 and full IPv4 Internet access through WireGuard was restored.

Changing nothing else was required.

Current workaround:

System > Settings > Tunables

net.route.algo.inet.algo = radix4

So this appears to be a radix4_lockless FIB rebuild/lookup failure rather than a WireGuard, NAT or firewall configuration problem.

Hardware: DEC850v2
OPNsense: 26.7.3_8
WAN: PPPoE
IPv4 default algorithm before workaround: radix4_lockless
Available algorithms: bsearch4, radix4_lockless, radix4

I have now also filed this as OPNsense src issue #323, including the full diagnostic results and the reproducible radix4_lockless → radix4 A/B test.

GitHub: https://github.com/opnsense/src/issues/323

Hopefully this helps confirm that the radix4_lockless rebuild errors can cause different IPv4 routing failures depending on the system/configuration.

[newsense]

There weren't any igc updates for 26.7.3 afaik...

Are you sure there weren't any radix issues when booting the previous kernel ?

[RamSense]

That's a fair point.

I cannot be sure that the previous kernel was free of radix4_lockless errors, because unfortunately I did not save the dmesg output from the previous installation before replacing the SSD. The SSD had reached a very high level of wear after substantial disk writes, which I am investigating separately with Zenarmor because of the amount of write activity generated by its logging. Warning for other users here.

In fact, I think I may have seen similar fib_algo/radix messages before, while WireGuard and normal IPv4 routing were still working correctly.

So I don't want to claim that 26.7.3 necessarily introduced the radix issue, nor that this is related to an igc change.

It is quite possible that the radix4_lockless problem was already present but latent/non-impacting, and that the clean installation/config restore exposed it during a fresh FIB rebuild.

What I can reproduce reliably on the current 26.7.3_8 installation is:

- radix4_lockless repeatedly reports setup/rebuild failures
- the routing table and nexthops still appear correct
- actual IPv4 lookup to wg0 is wrong
- IPv6 is unaffected
- changing only net.route.algo.inet.algo from radix4_lockless to radix4 immediately restores both the normal IPv4 route to the WireGuard peer and full IPv4 Internet access through WireGuard

So the exact version where the radix errors first appeared is unknown. The reproducible part is the radix4_lockless failure and the radix4 workaround on the current installation.

[emmitt]

I just wanted to confirm that there is another error in the same hardware (Deciso 750) / software (26.7.3_8) combination.

Unfortunately, my friend has already performed a clean install and restored the configuration data, so I can't provide any further information.

[sber]

There weren't any igc updates for 26.7.3 afaik...

Are you sure there weren't any radix issues when booting the previous kernel ?

I'm back at 26.7.2_2; same situation as before upgrading and see no radix issues in the logs. I will make a snapshot and upgrade again today and hopefully i'm able to get some more information to share.

[sber]

After upgrading to 26.7.3_8 the WAN interface was up but there was no route to the internet, and the radix issues were back in the log. In System -> Gateways -> Group there is a WAN Backup with only a Tier 1 gateway defined. As a test, I added the same gateway as Tier 2 and connectivity to the internet was restored. I've used a backup WAN long ago when switching providers, and removed the interface after migration, but it seems I didn't clean up the gateway group at the time. No radix issues in the log anymore. I will remove the group since I'm not using it anymore.

[RamSense]

Thanks for the update. Your gateway-group finding prompted me to check my configuration as well.

I also found an old unused gateway group. It contained only `WAN_DHCP6` as Tier 2 and was not referenced by any firewall rule, so I removed the group through the GUI and applied the configuration.

However, this does not appear to be the root cause on my system.

After removing the unused gateway group, I kept my existing `radix4` workaround active and then tried to switch the IPv4 FIB algorithm live back to `radix4_lockless`:

sysctl net.route.algo.inet.algo=radix4_lockless

This failed immediately:

net.route.algo.inet.algo: radix4
sysctl: net.route.algo.inet.algo=radix4_lockless: Invalid argument

At the same time the kernel logged a new failure:

[fib_algo] inet.0 setup_fd_instance: radix4_lockless algo instance setup failed, failures=0

The active algorithm therefore remained `radix4`.

My WireGuard IPv4 route and connectivity remained correct during the test, but this was because the switch to `radix4_lockless` never succeeded:

route to: 10.10.10.2
fib: 0
interface: wg0

So in my case, removing the old unused gateway group did not make `radix4_lockless` usable again.

For now I therefore still need:

net.route.algo.inet.algo=radix4

as the workaround.

This suggests that an incomplete or stale gateway group may be a trigger in some configurations, but it does not appear to be the sole cause of the `radix4_lockless` setup/rebuild failure.

I have not rebooted after removing the group; this was a live test while keeping the persistent `radix4` tunable in place.

DEC850v2 / OPNsense 26.7.3_8 / PPPoE / WireGuard