Topics

1

22.1 Legacy Series / OSPFv3 fails in 22.1.3 - seems to be PPPoE related?

« on: March 25, 2022, 08:01:46 am »

I'm coming back to OPNsense after switching to pfSense for a couple of years due not being able to get more than ~1.2Gbps throughput (whereas with pfSense I'm able to get ~4Gbps). Looks like 22.1/FreeBSD 13 has fixed the root cause as in my testing I'm now able to push ~6Gbps, which is awesome. Platform is ESXi 7.0U3 with VMXNET3 interfaces.

There's only one thing holding up the cutover - OSPFv3 seems to break (and all IPv6 routes end up expiring) when I shutdown pfSense and bring up the PPPoE connection on OPNsense. It really has me stumped.

Of note, OSPFv3 is working fine with pfSense and OSPF (i.e. for IPv4) works fine across both pfSense and OPNsense. The version of FRR is also the same across both platforms (7.5.1_3). Socket buffers are also set to 16777216. The last version of OPNsense I used was 20.1 (IIRC) and OSPFv3 was working fine back then.

For the cutover I'm doing the following (and there's a diagram of the network layout that hopefully helps):

• pfSense and OPNsense running side by side, albeit with different interface addresses and the OPNsense WAN interface is set to a test VLAN at the VM level to prevent establishing a PPPoE connection
• All four neighbours see each other and are sharing route information - both OSPF and OSPFv3 is working correctly. I can ping all the OPNsense IPv4 and IPv6 interface address from the 3.0/24 and 4.0/24 networks. Life is good.
• Shutdown pfSense
• Change the VLAN for the WAN interface on OPNsense to allow the PPPoE session to be established
• Reconfigure OSPF on OPNsense to advertise the default gateway
• This restarts FRR. OPSF routes are populated as expected, but OSPFv3 never comes back and eventually the IPv6 routes on the downstream devices expire. I can still ping the OPNsense IPv6 addresses from each subnet, so the interfaces themselves are still up

The logs don't show anything useful, even at debug level. I've performed the following troubleshooting

• Reversed the order of reconfiguring OSPF to advertise the default gateway and establishing the PPPoE session (i.e. reconfigure OSPF first, then change the VLAN to allow the session to establish)
• Completely removed the OSPF and OSPFv3 configs and reconfigured from scratch, both before and after establishing the PPPoE session
• Created a multitude of firewall rules for the OSPF protocol. Floating rules, interface rules, any/any rules for the OSPF protocol, rules containing the multicast and link-local ranges, rules targeting the interface address - there are no blocks and when OSPFv3 is working I see matches and passes
• Rebooted too many times to count

I believe it's related to PPPoE for the following two reasons:

• If I simply shutdown pfSense and don't establish the PPPoE session on OPNsense, OSPFv3 continues to operate normally (albeit with no WAN connection)
• If disconnect the PPPoE session on OPNsense and power on pfSense, OSPFv3 on OPNsense springs back to life

Config is below:

Code: [Select]

Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname core-fw1
log syslog
!
interface vmx1
 ip ospf area 0.0.0.0
 ip ospf priority 200
 ipv6 ospf6 network broadcast
 ipv6 ospf6 priority 200
!
router ospf
 ospf router-id 192.168.2.254
 redistribute kernel
 redistribute connected
 redistribute static
 default-information originate always
!
router ospf6
 ospf6 router-id 192.168.2.254
 redistribute kernel
 redistribute connected
 redistribute static
 interface vmx1 area 0.0.0.0
!
line vty
!
end

Is there something I've missed? Something else I should try? Or is this a bug?