Trying to fully understand an ISC DHCP behavior

[EricPerl]

I'm in the process of swapping out my existing prosumer router over to OPNsense.
I'm also trying to do this in stages (I might post about the end-to-end experience) so as to minimize potential disruptions. So far, I haven't messed up (at least not visibly).
I've inserted an OPNsense (virtualized if it matters) in my existing network and made use of VLANs to segregate traffic from my current setup. 1 VLAN for the OPN WAN, another for the native OPN LAN. So far so good.

I then did the following:

• tested the new setup with a test machine (Ubuntu), got an expected IP in the default 192.168.1.0/24 network.
• brought that machine back in a TEST VLAN handled by my current router, got proper IP right away.
• updated my current router to forget that TEST VLAN and created the same VLAN in OPN, including DHCP and FW rules

That's when things got weird. I gained internet connectivity right away but then tried to renew the DHCP lease (just ran dhclient).
The IP looked correct client side but still showed up in ISC lease list with the old 192 IP & offline!
There was also a correct entry in the OPN TEST VLAN but it showed expired but online!!!

Per the attached Log, I concluded that:

• the client still knew about its old 192. IP and requested it.
• Somehow, although the IP was not in the network range, ISC ACKed
• Sending the packet failed (either on the OPN side because of the bogus IP, or not picked up by the client because its current IP was already a 10. IP).

Is this the correct interpretation?

Reading a bit about the linux DHCP client, I cleaned up the leases file and removed all mentions of the 192 IP.
I also cleaned up some of the leases on the OPN sense side, and I ended up where I was expecting.

I'm a little anxious about moving forward, although I don't expect many clients to have stale IPs (they'll have other OSes too).
Did I do something wrong in my dry run?

Note that in the meantime, ISC has NACKed the .2 address seen in the log and issued a new one .3 (for the same MAC and hostname)...
[Update:] That Ubuntu test client now has both .2 and .3 IP active... One of them assigned to enp0s25:avahi
I have no clue why I'm getting one of these zeroconf IP when the machine already gets IP via DHCP...
[Update2:] I may have caused that last bit by setting up OPNsense in the local domain.
I guess I need another dry run with another VLAN...

[bimbar]

DHCPD shouldn't be able to hand out IPs that it is not configured to.

However I'm not entirely clear on your setup, so I can't really say if there's a justification for that behaviour or not.

[EricPerl]

In this case, per the log, the client requested 192.168.1.102 (an IP held by that machine, issued by ISC in the default subnet) and seemed to ACK it even though it was over an interface corresponding to 10.254.254.0/24.
But sending that ACK appears to have failed.

How I got there:
* I had that machine in OPN's default subnet (192.168.1.0/24).
* I moved the machine back to the TEST VLAN of my current LAN.
IOW, per ifconfig, its IP was 10.254.254.2 offered by the DHCP server of my current network/router.
Unbeknownst to me at the time, Ubuntu still knew about the 192.168.1.102 (client side leases file).
* I moved the handling of the TEST VLAN from my current router to OPN.

That's when I got that output. Plenty more requests for that 102 IP earlier.
But at some point (I was trying to clean the client side leases file in the meantime), a fresh discovery happened.
It's difficult to correlate client and server side, but I think that's the period when I saw 2 leases for the same MAC:
10.254.254.2 expired online (not visible in the UI by default so I likely missed its initial appearance)
192.168.1.102 active offline

Note that I did this as a dry run for a router migration (tp-link to OPNsense).
I have both running concurrently on the same physical network, isolated with VLANs.
When I say I moved the machine from one network to another, I'm merely switching a VLAN port profile in the UI of my network management system (Omada).

That was yesterday.
Today I migrated the GUEST VLAN with a machine connected wirelessly (I have 1 WLAN per VLAN).
That went more smoothly (apart from one stupid mistake). OTOH, I had wiped the leases file on the test machine...

I'll try to repro again tomorrow while I migrate my IOT network.
It's all wireless so it should be fine. I'll add a wired test machine for good measure.

Final note: it appears that the output of ifconfig might not be reliable when IP changes under the covers.
This said, I'd rather test in a situation closer to migration.
But if I have to unplug/replug network cables, so be it...