Hello,
Similar to a few other post I could find here (https://forum.opnsense.org/index.php?topic=46359.0) or in the KEA DHCP mailing-list (https://lists.isc.org/pipermail/kea-users/2023-June/004054.html), I do sometimes get the HA_LEASE_UPDATE_CONFLICT message in the KEA DHCP logs.
Eventually, this leads to KEA DHCP terminating HA (based on the max-rejected-lease-updates (default 10))
I noticed that it usually happens after the primary node gets rebooted (after an update for example) or when I "Enter Persistent CARP Maintenance Mode" on the primary node and then eventually get out of it.
As I was looking at the KEA DHCP configuration files to try and find a clue as to why it may happen, I noticed that the "kea-dhcp4.conf" configuration file content had all it's slashes ('/') escaped => '\/'
I wonder what is the reason for that? I thought from what I read that the only thing that needed to be escaped in the KEA configuration files are the commas (',')
Also looking around on KEA configuration file examples, I did not notice anybody else escaping the slashes in their configuration files.
Other than that, I did no see anything particular that could explain the issue I am facing.
Below the logs on the primary (hot) when the issue happens:
2025-04-15T20:51:47-04:00 Warning kea-dhcp4 WARN [kea-dhcp4.ha-hooks.0x395ec216600] HA_LEASE_UPDATE_CONFLICT OPNsense-primary: lease update [hwtype=1 xx:xx:xx:xx:2b:25], cid=[no info], tid=0x5418305d sent to OPNsense-backup (http://10.99.0.252:8001/) returned conflict status code: ResourceBusy: IP address:10.90.0.54 could not be updated. (error code 4)
and the corresponding log on the backup (standby):
2025-04-15T20:51:47-04:00 Warning kea-dhcp4 WARN [kea-dhcp4.lease-cmds-hooks.0x38dd92616d00] LEASE_CMDS_UPDATE4_CONFLICT lease4-update command failed due to conflict (parameters: { "expire": 1744766507, "force-create": true, "fqdn-fwd": false, "fqdn-rev": false, "hostname": "REDACTED", "hw-address": "xx:xx:xx:xx:2b:25", "ip-address": "10.90.0.54", "origin": "ha-partner", "state": 0, "subnet-id": 6, "valid-lft": 1800 }, reason: ResourceBusy: IP address:10.90.0.54 could not be updated.)
I redacted part of the MAC address and the hostname.
Eventually, after enough of those warning, eventually it leads to termination:
On the primary (hot):
2025-04-15T20:51:47-04:00 Error kea-dhcp4 ERROR [kea-dhcp4.ha-hooks.0x395ec216600] HA_TERMINATED HA OPNsense-primary: service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
2025-04-15T20:51:47-04:00 Error kea-dhcp4 ERROR [kea-dhcp4.ha-hooks.0x395ec216600] HA_LEASE_UPDATE_REJECTS_CAUSED_TERMINATION OPNsense-primary: too many rejected lease updates cause the HA service to terminate
and on the backup (standby):
2025-04-15T20:51:51-04:00 Error kea-dhcp4 ERROR [kea-dhcp4.ha-hooks.0x38dd92615f00] HA_TERMINATED HA OPNsense-backup: service terminated due to an unrecoverable condition. Check previous error message(s), address the problem and restart!
Running OPNsense 25.1.5_5-amd64 at the time of writing
You can either wait for 25.1.6 next week or install OPNsense in a couple VMs - switch to development train and check for updates - then start playing with Dnsmasq DNS & DHCP - which is where probably most people will end up when ISC goes away.
I don't see much development done on KEA github, and the mere fact the OPNsense team decided to invest the time and effort into Dnsmasq DHCP after the initial KEA integration should be a good indication on where things are. Feature wise there's a ton more stuff you'll be able to do with Dnsmasq DHCP compared to what was possible in KEA - right out of the gate.
Bottom line, sure you have an issue. If it would have been up to the OPNsense team I'm pretty sure it would have been solved a while ago.
Given the new perspective I tried to depict above it's probably best to cut your losses and start preparing for better times than hoping magic will suddenly happen overnight in KEA land.
There is a ton of stuff one can do with Kea? https://kea.readthedocs.io/en/latest/arm/hooks.html
https://github.com/opnsense/core/issues/7475
Quote from: newsense on April 16, 2025, 05:11:50 AMstart playing with Dnsmasq DNS & DHCP - which is where probably most people will end up when ISC goes away.
[...]
Feature wise there's a ton more stuff you'll be able to do with Dnsmasq DHCP compared to what was possible in KEA - right out of the gate.
[...]
it's probably best to cut your losses and start preparing for better times than hoping magic will suddenly happen overnight in KEA land.
Thanks for your reply, a quick question though:
will the new dnsmasq DHCP implementation support HA out of the box?Both ISC and KEA do support HA today and according to KEA's doc (https://kb.isc.org/docs/kea-ha-quickstart-guide#how-does-hotstandby-mode-ha-work), KEA's HA implementation "should be" much better.
So if dnsmasq DHCP implementation supports HA, then I will very likely go that route, but if it does not, what's the plan?
Fine, twist my hand...
A picture is worth 1000 words anyway