OSPFv3 fails in 22.1.3 - seems to be PPPoE related?

[jchnnz]

I'm coming back to OPNsense after switching to pfSense for a couple of years due not being able to get more than ~1.2Gbps throughput (whereas with pfSense I'm able to get ~4Gbps). Looks like 22.1/FreeBSD 13 has fixed the root cause as in my testing I'm now able to push ~6Gbps, which is awesome. Platform is ESXi 7.0U3 with VMXNET3 interfaces.

There's only one thing holding up the cutover - OSPFv3 seems to break (and all IPv6 routes end up expiring) when I shutdown pfSense and bring up the PPPoE connection on OPNsense. It really has me stumped.

Of note, OSPFv3 is working fine with pfSense and OSPF (i.e. for IPv4) works fine across both pfSense and OPNsense. The version of FRR is also the same across both platforms (7.5.1_3). Socket buffers are also set to 16777216. The last version of OPNsense I used was 20.1 (IIRC) and OSPFv3 was working fine back then.

For the cutover I'm doing the following (and there's a diagram of the network layout that hopefully helps):

• pfSense and OPNsense running side by side, albeit with different interface addresses and the OPNsense WAN interface is set to a test VLAN at the VM level to prevent establishing a PPPoE connection
• All four neighbours see each other and are sharing route information - both OSPF and OSPFv3 is working correctly. I can ping all the OPNsense IPv4 and IPv6 interface address from the 3.0/24 and 4.0/24 networks. Life is good.
• Shutdown pfSense
• Change the VLAN for the WAN interface on OPNsense to allow the PPPoE session to be established
• Reconfigure OSPF on OPNsense to advertise the default gateway
• This restarts FRR. OPSF routes are populated as expected, but OSPFv3 never comes back and eventually the IPv6 routes on the downstream devices expire. I can still ping the OPNsense IPv6 addresses from each subnet, so the interfaces themselves are still up

The logs don't show anything useful, even at debug level. I've performed the following troubleshooting

• Reversed the order of reconfiguring OSPF to advertise the default gateway and establishing the PPPoE session (i.e. reconfigure OSPF first, then change the VLAN to allow the session to establish)
• Completely removed the OSPF and OSPFv3 configs and reconfigured from scratch, both before and after establishing the PPPoE session
• Created a multitude of firewall rules for the OSPF protocol. Floating rules, interface rules, any/any rules for the OSPF protocol, rules containing the multicast and link-local ranges, rules targeting the interface address - there are no blocks and when OSPFv3 is working I see matches and passes
• Rebooted too many times to count

I believe it's related to PPPoE for the following two reasons:

• If I simply shutdown pfSense and don't establish the PPPoE session on OPNsense, OSPFv3 continues to operate normally (albeit with no WAN connection)
• If disconnect the PPPoE session on OPNsense and power on pfSense, OSPFv3 on OPNsense springs back to life

Config is below:

Code: [Select]

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname core-fw1
log syslog
!
interface vmx1
 ip ospf area 0.0.0.0
 ip ospf priority 200
 ipv6 ospf6 network broadcast
 ipv6 ospf6 priority 200
!
router ospf
 ospf router-id 192.168.2.254
 redistribute kernel
 redistribute connected
 redistribute static
 default-information originate always
!
router ospf6
 ospf6 router-id 192.168.2.254
 redistribute kernel
 redistribute connected
 redistribute static
 interface vmx1 area 0.0.0.0
!
line vty
!
end

Is there something I've missed? Something else I should try? Or is this a bug?

[mimugmail]

PLease remove area config in interface section and add areas to network tab.

[jchnnz]

There's no Networks tab for OSPFv3. Only Interfaces.

• I removed the interface configuration from both OSPF and OSPFv3 and created an entry under OSFP -> Networks. OSPF (IPv4) is configured, but that has always worked. OSPFv3 (IPv6) now has no config and stops completely in either scenario
• Created entries under OSPF -> Networks and OSPFv3 -> Interfaces. No change - same outcome as described in my original post

[jchnnz]

I've done some further testing...

- I replicated the config from pfSense through vtysh rather than the GUI. Not all commands stuck in the running config, possibly because they're default values. The config seems to survive a restart of routing, but doesn't seem to survive a reboot.

- If I leave pppoe0 disconnected, I can restart routing through the GUI as often as I want and both OSPF (IPv4) and OSPFv3 (IPv6) work as expected. Within 15 or so seconds all the neighbours communicate and start sharing routes. Internal IPv6 endpoints become reachable. Can't quite say indefinitely, but everything kept working for 20mins which seems a sufficiently long test

- If I then establish a pppoe session, both OSPF (IPv4) and OSPFv3 (IPv6) continue to work. However, after approx 10 mins IPv6 routes drop out of the routing tables of OPNsense and the downstream devices. OSPFv3 becomes dead in the water

- Simply disconnecting the pppoe session and restarting routing has no effect. To restore OSPFv3 I need remove the OSPFv3 config (no router ospf6 in vtysh), write out the config, restart routing, recreate the config, write out the config, and then restart routing.

- If routing is restarted from the GUI while a pppoe session is active OSPFv3 will never recover

- If OPNsense is rebooted and prevented from establishing a pppoe session, OSPFv3 starts up and operates normally

- If OPNsense is rebooted normally, OSPFv3 never start operating properly

There seems to be some sort of bug or conflict between pppoe and ospf6d...