Messages

[2406:xxxx:xxxx:b170::1/64fe80::250:56ff:feaa:bb70]

I've also reproduced this. The problem is in the dhcrelay6 implementation

The relevant OPNsense interface has both:

2406:xxxx:xxxx:b170::1/64
fe80::250:56ff:feaa:bb70

but the Relay-Forward sent to the DHCPv6 server contains:

Link-address: fe80::250:56ff:feaa:bb70

My Windows DHCP server receives the Relay-Forward but can't associate that link-local address with the 2406:xxxx:xxxx:b170::/64 scope, so it sends no reply. For comparison, a Dell switch relaying to the same DHCPv6 server correctly uses its client facing GUA as link-address and Windows immediately returns a Relay-Reply.

I vibe reviewed the dhcrelay source code and dhcrelay6.c explicitly does:

Code Select Expand

dsr->dsr_linkaddr = intf->linklocal;
In other words, when building the Relay-Forward dhcrelay6 explicitly populates link-address from the interfaces link-local IPv6 address. That matches exactly what appears in the packet capture.

dispatch.c only records that a non-link-local IPv6 address exists:

Code Select Expand

intf->gipv6 = 1;
The actual GUA/ULA isn't stored, so relay6_pushrelaymsg() has no GUA/ULA available to use for link-address.

I've raised a bug here: https://github.com/opnsense/core/issues/10686

This also looks related to #10623, where Relay-Replies are sent to the wrong VLAN. Different symptom, but likely wrapped up in the same underlying link-address handling.

In any case, I can confirm from packet captures that the endpoint sends a correctly formatted Solicit, OPNsense receives it and generates a Relay-Forward, and that Relay-Forward arrives at the DHCPv6 server successfully over a properly routed, GUA based IPv6 network. The relay and server are on different routed subnets, and the outer IPv6 packet is delivered correctly. I can also rule out the DHCPv6 server itself. A separate Dell switch relays DHCPv6 for other subnets to the same Windows DHCP server successfully. The problem is the embedded DHCPv6 link-address in the OPNsense Relay-Forward. OPNsense uses the client-facing interface's link-local address instead of its GUA, so the DHCP server can't associate the request with the correct scope.

TL;DR: DHCPv6 Solicit is valid → OPNsense receives it → DHCPv6 Relay-Forward is generated → DHCP server receives it → DHCP server itself works with another relay → embedded link-address supplied by OPNsense in the Relay-Forward is wrong.

I'm seeing the same thing on 22.1.4_1 (among other IPv6 related issues). But in my case at least the loss of IPv6 on the pppoe WAN interface seems to be conditional. Unfortunately the steps  to restore (other than a reboot) work intermittently at best.

I use pppoe and have a /56 PD, using the basic settings of "Request only an IPv6 prefix", "Prefix delegation size" of 56, and "Use IPv4 connectivity"

I've also applied patch 1a5dfc932f8.

• When pppoe is established at boot, the WAN interface overview shows both the IPv6 PD and the IPv6 gateway addresses. IPv6 is working and seems to be stable
• If the pppoe interface is reloaded, only the IPv6 gateway address is shown. The PD has disappeared. IPv6 is not working
• If the pppoe interface is disconnected/connected, IPv6 PD and the IPv6 gateway addresses are shown, but 9 times out of 10 after about 4-5 pings the IPv6 connection drops and the PD has also disappeared from the WAN interface overview. IPv6 works for only a few seconds
  
• Doing a disconnect/connect a few times in quick succession sometimes brings IPv6 back. Sometimes doing a disconnect and then a reload also brings it back. But then doing just a reload will always break it again.

Removed the patch (reverted to a snapshot I took before applying) and the behaviour is the same, except that occasionally IPv6 will randomly work even when the PD address isn't shown in the overview...

I've done some further testing...

- I replicated the config from pfSense through vtysh rather than the GUI. Not all commands stuck in the running config, possibly because they're default values. The config seems to survive a restart of routing, but doesn't seem to survive a reboot.

- If I leave pppoe0 disconnected, I can restart routing through the GUI as often as I want and both OSPF (IPv4) and OSPFv3 (IPv6) work as expected. Within 15 or so seconds all the neighbours communicate and start sharing routes. Internal IPv6 endpoints become reachable. Can't quite say indefinitely, but everything kept working for 20mins which seems a sufficiently long test

- If I then establish a pppoe session, both OSPF (IPv4) and OSPFv3 (IPv6) continue to work. However, after approx 10 mins IPv6 routes drop out of the routing tables of OPNsense and the downstream devices. OSPFv3 becomes dead in the water

- Simply disconnecting the pppoe session and restarting routing has no effect. To restore OSPFv3 I need remove the OSPFv3 config (no router ospf6 in vtysh), write out the config, restart routing, recreate the config, write out the config, and then restart routing.

- If routing is restarted from the GUI while a pppoe session is active OSPFv3 will never recover

- If OPNsense is rebooted and prevented from establishing a pppoe session, OSPFv3 starts up and operates normally

- If OPNsense is rebooted normally, OSPFv3 never start operating properly

There seems to be some sort of bug or conflict between pppoe and ospf6d...

There's no Networks tab for OSPFv3. Only Interfaces.

• I removed the interface configuration from both OSPF and OSPFv3 and created an entry under OSFP -> Networks. OSPF (IPv4) is configured, but that has always worked. OSPFv3 (IPv6) now has no config and stops completely in either scenario
• Created entries under OSPF -> Networks and OSPFv3 -> Interfaces. No change - same outcome as described in my original post

I'm coming back to OPNsense after switching to pfSense for a couple of years due not being able to get more than ~1.2Gbps throughput (whereas with pfSense I'm able to get ~4Gbps). Looks like 22.1/FreeBSD 13 has fixed the root cause as in my testing I'm now able to push ~6Gbps, which is awesome. Platform is ESXi 7.0U3 with VMXNET3 interfaces.

There's only one thing holding up the cutover - OSPFv3 seems to break (and all IPv6 routes end up expiring) when I shutdown pfSense and bring up the PPPoE connection on OPNsense. It really has me stumped.

Of note, OSPFv3 is working fine with pfSense and OSPF (i.e. for IPv4) works fine across both pfSense and OPNsense. The version of FRR is also the same across both platforms (7.5.1_3). Socket buffers are also set to 16777216. The last version of OPNsense I used was 20.1 (IIRC) and OSPFv3 was working fine back then.

For the cutover I'm doing the following (and there's a diagram of the network layout that hopefully helps):

• pfSense and OPNsense running side by side, albeit with different interface addresses and the OPNsense WAN interface is set to a test VLAN at the VM level to prevent establishing a PPPoE connection
• All four neighbours see each other and are sharing route information - both OSPF and OSPFv3 is working correctly. I can ping all the OPNsense IPv4 and IPv6 interface address from the 3.0/24 and 4.0/24 networks. Life is good.
• Shutdown pfSense
• Change the VLAN for the WAN interface on OPNsense to allow the PPPoE session to be established
• Reconfigure OSPF on OPNsense to advertise the default gateway
• This restarts FRR. OPSF routes are populated as expected, but OSPFv3 never comes back and eventually the IPv6 routes on the downstream devices expire. I can still ping the OPNsense IPv6 addresses from each subnet, so the interfaces themselves are still up

The logs don't show anything useful, even at debug level. I've performed the following troubleshooting

• Reversed the order of reconfiguring OSPF to advertise the default gateway and establishing the PPPoE session (i.e. reconfigure OSPF first, then change the VLAN to allow the session to establish)
• Completely removed the OSPF and OSPFv3 configs and reconfigured from scratch, both before and after establishing the PPPoE session
• Created a multitude of firewall rules for the OSPF protocol. Floating rules, interface rules, any/any rules for the OSPF protocol, rules containing the multicast and link-local ranges, rules targeting the interface address - there are no blocks and when OSPFv3 is working I see matches and passes
• Rebooted too many times to count

I believe it's related to PPPoE for the following two reasons:

• If I simply shutdown pfSense and don't establish the PPPoE session on OPNsense, OSPFv3 continues to operate normally (albeit with no WAN connection)
• If disconnect the PPPoE session on OPNsense and power on pfSense, OSPFv3 on OPNsense springs back to life

Config is below:

Code Select Expand


Current configuration:
!
frr version 7.5.1
frr defaults traditional
hostname core-fw1
log syslog
!
interface vmx1
ip ospf area 0.0.0.0
ip ospf priority 200
ipv6 ospf6 network broadcast
ipv6 ospf6 priority 200
!
router ospf
ospf router-id 192.168.2.254
redistribute kernel
redistribute connected
redistribute static
default-information originate always
!
router ospf6
ospf6 router-id 192.168.2.254
redistribute kernel
redistribute connected
redistribute static
interface vmx1 area 0.0.0.0
!
line vty
!
end


Is there something I've missed? Something else I should try? Or is this a bug?