BIND plus ipv6 prefix delegation results in failure to listen

[seacycle]

OPNsense 24.1.4-amd64, using BIND for DNS (unbound is disabled).  BIND is configured to listen on :: for IPv6, yielding this in the BIND configuration:

    listen-on-v6 port 53 { any; };

WAN side v6 address is assigned via DHCP6 with prefix delegation. The LAN interfaces are assigned from the prefix delegation (Track Interface).

The LAN side router advertisements default to advertising the interface's global unicast address, derived from the DHCP prefix delegation, as the name server address.

The problem: Each time the WAN side DHCP6 client refreshes the WAN address and prefix delegation, it also refreshes the LAN addresses tied to the prefix delegation. This is fine, the prefix is the same, the addresses are the same. But every time this happens, BIND stops listening on the prefix delegated LAN side addresses. It takes a manual restart of BIND to start listening again.

Is there a way to automatically kick BIND to re-evaluate its listening addresses when this happens?

[seacycle]

Actually, it looks like BIND does get a kick when the WAN side dhcp6c refreshes, but it errors out on listening to the global unicast addresses, for example:

23-Mar-2024 14:51:28.795 network: info: listening on IPv6 interface vlan0.1.4, <redacted-v6-address>#53
23-Mar-2024 14:51:28.795 network: error: creating IPv6 interface vlan0.1.4 failed; interface ignored

And so on through all the interfaces with dhcp6c derived addresses.

Maybe a race condition? Manually restarting BIND moments later, everything works.

[seacycle]

So when this happens, dhcp6c is manipulating the global unicast addresses on the interfaces tracking the prefix delegation, and while this is happening, named is trying to adapt to the addresses as they come and go.  After dhcp6c is done, named is left in a state where every minute, it tries to listen on the these addresses, and gets an error. A minute later, tries again, fails again and so on until I restart named.

While in the state, I did a syscall ktrace on named and see this pattern:

Code: [Select]

18016 isc-net-0000 CALL  socket(PF_INET6,0x2<SOCK_DGRAM>,IPPROTO_IP)
 18016 isc-net-0000 RET   socket 58/0x3a
 18016 isc-net-0000 CALL  setsockopt(0x3a,IPPROTO_IPV6,IPV6_DONTFRAG,0x8370b11d4,0x4)
 18016 isc-net-0000 RET   setsockopt 0
 18016 isc-net-0000 CALL  setsockopt(0x3a,IPPROTO_IPV6,IPV6_V6ONLY,0x8370b11dc,0x4)
 18016 isc-net-0000 RET   setsockopt 0
 18016 isc-net-0000 CALL  setsockopt(0x3a,SOL_SOCKET,SO_REUSEPORT,0x8370b11dc,0x4)
 18016 isc-net-0000 RET   setsockopt 0
 18016 isc-net-0000 CALL  setsockopt(0x3a,SOL_SOCKET,SO_REUSEPORT_LB,0x8370b11dc,0x4)
 18016 isc-net-0000 RET   setsockopt 0
 18016 isc-net-0000 CALL  write(0xd,0x827490f45,0x1)
 18016 isc-net-0000 RET   write 1
 18016 isc-net-0001 RET   kevent 1
 18016 isc-net-0000 CALL  socket(PF_INET6,0x2<SOCK_DGRAM>,IPPROTO_IP)
 18016 isc-net-0001 CALL  read(0xc,0x839d78990,0x400)
 18016 isc-net-0001 RET   read 1
 18016 isc-net-0001 CALL  setsockopt(0x3a,IPPROTO_IPV6,IPV6_USE_MIN_MTU,0x839d7885c,0x4)
 18016 isc-net-0001 RET   setsockopt 0
 18016 isc-net-0001 CALL  ioctl(0x3a,FIONBIO,0x839d7880c)
 18016 isc-net-0001 RET   ioctl 0
 18016 isc-net-0001 CALL  setsockopt(0x3a,SOL_SOCKET,SO_REUSEPORT,0x839d7884c,0x4)
 18016 isc-net-0001 RET   setsockopt 0
 18016 isc-net-0001 CALL  getpeername(0x3a,0x839d78798,0x839d7875c)
 18016 isc-net-0001 RET   getpeername -1 errno 57 Socket is not connected
 18016 isc-net-0001 CALL  setsockopt(0x3a,IPPROTO_IPV6,IPV6_V6ONLY,0x839d787fc,0x4)
 18016 isc-net-0001 RET   setsockopt 0
 18016 isc-net-0001 CALL  bind(0x3a,0x849924d30,0x1c)
 18016 isc-net-0001 RET   bind -1 errno 13 Permission denied
 18016 isc-net-0001 CALL  _umtx_op(0x84a1706b0,0x8<UMTX_OP_CV_WAIT>,0,0x84a170690,0)

And not long after, the socket 0x3a gets closed.  Why would permission be denied?

[seacycle]

Oh, this is interesting: https://kb.isc.org/docs/aa-00621

Normally binding to a reserved port on FreeBSD requires the process to be be running as root. For most uses this is not a problem as named binds to port 53 before changing user id; however, if you are running in a environment where interface addresses are changing this can be a issue. FreeBSD has a kernel module, mac-portacl, that will allow a non-privileged user to bind to specified ports.

Update: confirmed, this works. It is still not optimal, because named stops listening when dhcp6c is fiddling with the addresses, the initial listen fails (maybe because the addresses are waiting for DAD to complete and not yet bindable?) and they only get added a minute later.

My workaround is a statically configured ULA address alias on lo0 that I put in the router advertisements for DNS.

Update 2: confirmed that duplicate address detection is at the root of named failing to initially bind to the "new" addresses when dhcp6c fiddles with the interface addresses.  With DAD disabled, named is successful binding to the addresses on the first try.

So in summary, there are two problems here:

1. named as configured in the plugin to listen on all addresses will track when addresses change, BUT when they do change, named is unable to bind to the updated addresses because it dropped root privileges after the initial scan-and-bind pass at service start. This can be addressed by the method described in https://kb.isc.org/docs/aa-00621, or by restarting named.
2. When detecting new IPv6 addresses, named will initially fail to listen due a race condition between named attempting to listen and duplicate address detection. Unfortunately, this is common because dhcp6c as configured will periodically remove and re-add a prefix-delegation derived address even when the address is unchanged. This can be addressed by disabling duplicate address detection, or by restarting named.

It is not clear to me what the best solution is to these in the broader OPNsense context.

An alternate no-code-change workaround is to configure the bind plugin to only listen on statically assigned addresses.

cc: @franco, should I write this up as a bind plugin bug?