KeaDHCP dynamic DHCP question

[stauf]

Well good to know.  I was going to try to turn on Automatic Discovery and see if I could find a rogue ARP request on my network causing OpnSense to mis-behave like this, but, for now, I think I am good just leaving it off.  I, personally, don't really Automatic Discovery and it appears it does more harm than good at the moment.  Thanks for checking into it.

[FrankAusNRW]

Same issue here w/ OPNsense 26.1.4. ICS-DHCP is not installed anymore and DNSmask DHCP is disabled, so only KEA is running.
The DHCP range is going from .200-.249. All IP adresses in the DHCP range AND all unused IP addresses outside the DHCP range are blocked, even for inactive clients w/ fixed leases.
Effectively there is no chance for a client to obtain an IP address at the moment.
This is causing some trouble.

Is there a workaround or a fix in the near future?
If not, I need to get back to ICS or DNSmask DHCP for the time beeing.

The installation was a fresh ISO 26.1.2 installation.

[Monviech (Cedrik)]

I assume it would make sense to check the KEA logs then why it assigns these leases?

[stauf]

Frank,

For me, there seemed to be two workarounds and my "fix" was to disable Automatic Discovery.

1. Disable Automatic Discovery and wait.
2. If you need a solution faster, you can follow the instructions earlier in this thread.  SSH in (you may have to enable SSH), get into the shell and then go to /var/db/kea/.  There should be at least 1 .csv file in here.  Stop KeaDHCP, edit these files removing the incorrect entries.  This may take a few minutes to ensure you do it correctly as the files don't appear to be in order and there may be multiple (I had 2).  Restart KeaDHCP.  For me, I did this live.  I was stuck with all my addresses "in-use" so there didn't seem much harm to me doing it live.  Didn't appear to make anything worse.  Even with Automatic Discovery disabled, I did have to do this a couple times, but now, a couple days later, I see no erroneous entries in my cache.

As a side note, I did experience Automatic Discovery seemingly turning itself on once.  I had disabled and applied but then noticed it was on again sometime later.  I was never able to reproduce this so not sure what caused it.  I will just caution you to double check a few times that it is disabled, just in case.

[FrankAusNRW]

I've read the recommendations earlier and disabled "Automatic Discovery". In a first instance this doesn't help.
Maintaining the csv file in /var/db/kea/ was helping, but an hour later all IP addresses have been "in use" again.

The proxmox comment is also the same here. I'm running 2 proxmox nodes, and 2 container are the root cause for utilizing the IP addresses, identified by the MAX addresses in the "Automatic Discovery" log when this was still activated. I coudn't figure out what is different at the two containers compared to the oder ones, apart from the fact, that the interfaces had only IPv6 addresses assigned. But I'm not using nor deploying IPv6 adresses at all by KEA. And no other DHCP server is running.  - weired!

I'm also not using VLAN - yet.

[Monviech (Cedrik)]

FYIY, the csv files do not always represent the current in-memory state of the leases in the KEA daemon itself.

I'm going to address this to make the GUI reflect the correct state more closely by polling the control socket.

https://github.com/opnsense/core/pull/10019


To reinstate something: Automatic Discovery and KEA are two completely different daemons that listen for completely different protocols.

Automatic Discovery -> Listens passively for NDP and ARP messages
KEA -> Listens/responds for DHCPv4/v6 messages

Now please stop all the guessing as there is no overlap in these protocols.

[stauf]

Thanks, that makes total sense based off what I saw.  I disabled Automatic Discovery and cleaned up the csv files, but that did not completely fix my problem.  Some of the erroneous entries were still showing up in the table.  To be fair, once the problem was "mostly" solved, that was enough for me at the time and I didn't pay super close attention to every detail.  All I can say is the next day, all of these 86400 second lifetime, MAC-less and hostname-less entries were totally gone from the KeaDHCP Leases table and, so far, have not come back.

Frank, are you saying you are getting buildup of what appear to be erroneous entries in your KeaDHCP Leases table and you have verified that Automatic Discovery is disabled?  I'm a bit confused by your statements.  You say you aren't using v6, but have v6 addresses assigned?  If you have KeaDHCPv6 disabled, these may just be auto assigned v6 addresses by Proxmox or your containers running on Proxmox.

I have not observed my KeaDHCP pool get used up while Automatic Discovery is Disabled nor was I scrutinizing the logs.  If you have logs referring to Automatic Discovery, sounds to me like you might still have it enabled.

[meyergru]

I may have found the answer - and the observed problem is most probably uncorrelated to the host discovery service.

Matter-of fact, Kea uses two files to register dynamic leases: kea-dhcp4.leases.csv and kea-dhcp4.leases.csv.2. One of them is a journal that is sometimes flushed to the other file: https://kea.readthedocs.io/en/latest/arm/dhcp4-srv.html#why-is-lease-file-cleanup-necessary

Also note, that unlike with ISC DHCP, Kea registers static reservatione in these files. If you then change something in the reservation, say, for instance, you change a MAC because you exchange a VM for another on your Proxmox host after an upgrade and want to keep its IP or if you only move a client to another VLAN, the lease file will still hold the old entry. Matter-of-fact, that was exactly what I did before the problem that I described before occured again.

When the newly created client with the static reservation now requests an IP via DHCP, there will be a conflict with the message I showed above.

What is more, is that this seems to trigger a cascading effect in Kea which seems to be a bug and is also discussed here for the "other product":

https://forum.netgate.com/topic/198115/changing-the-mac-address-on-a-kea-static-lease-does-not-work/4

At least one of you who observed the same problem also uses Proxmox, so it is kind of likely that something like this occured.

Thus, for the time being, it seems advisable to stop Kea, manually delete conflicting (or all) leases from both lease files and start Kea again when changes to static reservations are made.

[FrankAusNRW]

I just did it a bid more radical, after i've identified the two container on proxmox:
1. I've stopped KEA
2. I've renamed both KEA csv-files, no other files exist in that folder (just to keep them)
3. I gave the two proxmox container a new MAC-address (both hat multiple IPv4 addresses assigned to just 1 (!) MAC-address
4. I've maintained the new MAC-addresses in KEA as fix leases for the two proxmox container
5. I've fired KEA up again, immediately a new file "kea-leases4.csv" was created
6. I've fired up the two proxmox container
7. The fix leases are correctly assigned to the proxmox container

Finally, no weired assignments w/ 86400 seconds and no MAC address and no name appear in the csv-file

[stauf]

Out of curiosity, why was Automatic Discovery defaulted to enabled?  While I understand this feature could be valuable to some people, most products I have been involved with, new features default to disabled.  This allows for easier upgrade/downgrade to/from versions of sw that support/don't support the feature and essentially assures users that new features won't break what they currently have working.  I know there were warnings about Automatic Discovery and shame on me for not diving into this more actively, I guess I am just curious at a higher level that this specific feature.  Is there a case-by-case discussion on new features deciding to default them to enabled or disabled?

I can certainly imagine some features aren't as clear cut at this one to have them enabled or disabled, but this one seems pretty clear as it is just another daemon sitting there listening for ARPs/NDP messages.  Just seems odd to me that it would be enabled by default.