Messages

I wonder how big the problem you describe actually is because you have the first report. There are lots of big HA setups out there that run this just fine it seems even if there is the potential for a race condition in lease assignment.

Well, perhaps I wasn't fully understood because I didn't provide enough context for the issues that led me to the explanations I've been sharing since my first post. I apologize in advance for any misunderstanding.

We recently migrated from ISC DHCP to Kea. To perform this migration, I read the OPNsense documentation, searched this forum for any questions other colleagues might have had regarding this transition, and watched some online videos to ensure everything was correct. Like with any service migration, I kept a close eye on the logs over the past few weeks to ensure everything was running smoothly. From the beginning, I noticed the DHCPSRV_MULTIPLE_RAW_SOCKETS_PER_IFACE warning, but since it was just a warning, I let the service run, assuming it might be normal for the way OPNsense configures it.

After running Kea for a few days, I received a report of an IP conflict—something we hadn't experienced in years. Naturally, I looked into the Kea logs to understand what was happening. In most cases, we see DHCP packet duplication where Kea hands out the same IP, which only impacts the lease time. However, under certain conditions that I can't completely pinpoint (but I suspect are related to the timing of how threads handle duplicated packets), we can observe a single thread offering one IP for the first duplicated packet, and then offering a different IP for the second duplicated packet. Here is an example:

Code Select Expand

2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x2da78ea76008] DHCP4_PACKET_SEND [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: trying to send packet DHCPOFFER (type 2) from 10.y.y.2:67 to 10.y.y.89:68 on interface vlan04
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x2da78ea76008] DHCP4_LEASE_OFFER [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: lease 10.y.y.89 will be offered
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x2da78ea76808] DHCP4_PACKET_SEND [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: trying to send packet DHCPOFFER (type 2) from 10.y.y.2:67 to 10.y.y.88:68 on interface vlan04
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x2da78ea76808] DHCP4_LEASE_OFFER [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: lease 10.y.y.88 will be offered
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x2da78ea76008] DHCP4_PACKET_RECEIVED [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan04
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x2da78ea76808] DHCP4_PACKET_RECEIVED [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan04
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x2da78ea76808] DHCP4_QUERY_LABEL received query: [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573
2026-05-01T15:10:31-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x2da78ea76008] DHCP4_QUERY_LABEL received query: [hwtype=1 00:1e:e5:xx:xx:9d], cid=[01:00:1e:e5:xx:xx:9d], tid=0x5573
In other situations, since I have 4 threads available, each thread grabs one of these packet copies at the exact same time and offers up to 4 different IPs to the exact same computer:

Code Select Expand

2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x4aae83ac9008] DHCP4_LEASE_OFFER [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: lease 10.y.y.67 will be offered
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83ad0008] DHCP4_PACKET_SEND [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: trying to send packet DHCPOFFER (type 2) from 10.y.y.2:67 to 10.y.y.66:68 on interface vlan02
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x4aae83ad0008] DHCP4_LEASE_OFFER [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: lease 10.y.y.66 will be offered
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83ac9008] DHCP4_PACKET_RECEIVED [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan02
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83ad0008] DHCP4_PACKET_RECEIVED [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan02
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x4aae83ac9008] DHCP4_QUERY_LABEL received query: [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6
2026-04-23T09:58:15-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x4aae83ad0008] DHCP4_QUERY_LABEL received query: [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83ac9808] DHCP4_PACKET_SEND [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: trying to send packet DHCPOFFER (type 2) from 10.y.y.2:67 to 10.y.y.65:68 on interface vlan02
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x4aae83ac9808] DHCP4_LEASE_OFFER [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: lease 10.y.y.65 will be offered
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83a7b808] DHCP4_PACKET_SEND [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: trying to send packet DHCPOFFER (type 2) from 10.y.y.2:67 to 10.y.y.64:68 on interface vlan02
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.leases.0x4aae83a7b808] DHCP4_LEASE_OFFER [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: lease 10.y.y.64 will be offered
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83ac9808] DHCP4_PACKET_RECEIVED [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan02
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.packets.0x4aae83a7b808] DHCP4_PACKET_RECEIVED [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6: DHCPDISCOVER (type 1) received from 0.0.0.0 to 255.255.255.255 on interface vlan02
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x4aae83a7b808] DHCP4_QUERY_LABEL received query: [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6
2026-04-23T09:58:12-03:00 Informational kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x4aae83ac9808] DHCP4_QUERY_LABEL received query: [hwtype=1 90:09:df:xx:xx:50], cid=[01:90:09:df:xx:xx:50], tid=0x9a6360f6
I have a Wi-Fi network with controllers from different vendors, and one of them has flagged the detection of clients with conflicting IPs on different interfaces/VLANs in its logs, as shown in image below.

You cannot view this attachment.

I have been deep-diving into this issue over the last few weeks, and the only logical reason I can find for these IP conflicts is this packet duplication. I can't explain the exact internal mechanics behind it (maybe Kea ends up offering the same IP to two different devices, or tries to use an IP already assigned to someone else due to the race condition). I have also been monitoring leases to see if there is any premature exhaustion of the available IP pool, but that hasn't happened so far.

Since an IP conflict directly affects the client device, it's usually hard to get end-users to actively report it to the network admin. I was lucky that a coworker reported it to me and that one of our Wi-Fi controllers also logs this specific event. Additionally, processing these duplicated packets wastes CPU cycles, which could become a problem during peak hours.

I came to this forum because I genuinely couldn't find anyone else reporting this, so I might be the first to post about it. However, that doesn't invalidate the fact that there is something negatively impacting my setup. I am confident I read all the configuration recommendations from the official OPNsense and Kea documentation, and I believe I configured it correctly, but there is always a chance I missed something. Since my very first post, my intention has been to present the issue and see if anyone could point out a configuration flaw on my end. My goal here is simply to seek help and collaborate with the community to resolve this issue.

Finally, to be fully transparent as requested: I have indeed been using AI assistance (Gemini) to help me with my research, log parsing, and to structure my technical analyses. Furthermore, since English is not my native language, I use it to translate and properly adapt what I write into English.

[all sockets on this link will receive this message and multiple responses will be sent to the client.]

Thank you for the detailed explanation and for the GitHub references. I completely understand the design priority: startup resilience is paramount, and a service that fails to start because an IP is missing is indeed a major stability risk.

It seems we are looking at a side effect of that resilience in HA/CARP environments. While binding to the interface name ensures Kea always starts, the Kea 3.0.2 documentation (Section 9.2.4) explains why this leads to the duplication I'm seeing:

'Caution should be taken when configuring the server to open multiple raw sockets on the interface with several IPv4 addresses assigned... all sockets on this link will receive this message and multiple responses will be sent to the client. ... the configuration with multiple IPv4 addresses assigned should not be used when the directly connected clients are operating on that link.'

This matches my findings: even with thread-pool-size: 1, Kea processes the same request twice because it has two active BPF sockets (one for the Physical IP and one for the CARP). This leads to lease inconsistencies (potential IP conflicts) and log exhaustion.

To help improve the Kea implementation for complex HA setups without risking current startup stability, could we perhaps consider a 'middle ground' solution for Kea on OPNsense? Would it be possible to add an optional checkbox in the GUI for 'Strict HA Binding'? This would allow advanced users to manually enable the 'interface/address' notation. This way, the default behavior remains 'safe' for everyone, but administrators of HA clusters could opt-in to the binding method recommended by the Kea manual to prevent sequential duplication.

In my view, offering an 'opt-in' method for address-specific binding would greatly enhance Kea's reliability for production-grade HA environments while respecting OPNsense's startup stability goals. I'm happy to perform more tests if needed!

Update Note:

Section 9.2.4 of the Kea manual provides built-in parameters to handle startup resilience:

The service-sockets-require-all option makes Kea require all sockets to be successfully bound. If any opening fails, Kea interrupts the initialization and exits with a non-zero status. (Default is false).

"The port can be unavailable only temporary. In this case, retrying the opening may resolve the problem. Kea provides two options to specify the retrying: service-sockets-max-retries and service-sockets-retry-wait-time."

By setting service-sockets-max-retries to a non-zero value and keeping service-sockets-require-all as false, Kea will retry opening missing sockets for a defined period and will not fail to start even if some sockets remain unopened.

[The Sequential Duplication Proof (Single Thread)]

I am following up on my Kea issue in a High Availability (HA/CARP) cluster. After extensive debugging and manual configuration testing, I've identified a fundamental integration issue between Kea's Raw Socket implementation and FreeBSD's BPF behavior in the presence of CARP VIPs.

The Sequential Duplication Proof (Single Thread)

To rule out race conditions between multiple threads, I manually edited kea-dhcp4.conf. I set "thread-pool-size": 1 to force a single-threaded execution. Despite this, the log shows that the exact same thread (0x2bb5ec85c008) processes the same Transaction ID (tid=0x4ca805b) twice in a row, sequentially, within the same millisecond.

Log Evidence:

Code Select Expand

2026-04-29T11:32:25-03:00 kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x2bb5ec85c008] DHCP4_QUERY_LABEL received query: [hwtype=1 86:eb:0d:xx:xx:af], tid=0x4ca805b
2026-04-29T11:32:25-03:00 kea-dhcp4 INFO [kea-dhcp4.packets.0x2bb5ec85c008] DHCP4_PACKET_SEND [tid=0x4ca805b]: trying to send packet DHCPACK from 10.x.x.2:67 to 10.x.x.32:68 on interface vlan0151

-- (Immediately followed by the exact same process by the SAME thread) --

2026-04-29T11:32:25-03:00 kea-dhcp4 INFO [kea-dhcp4.dhcp4.0x2bb5ec85c008] DHCP4_QUERY_LABEL received query: [hwtype=1 86:eb:0d:xx:xx:af], tid=0x4ca805b
2026-04-29T11:32:25-03:00 kea-dhcp4 INFO [kea-dhcp4.packets.0x2bb5ec85c008] DHCP4_PACKET_SEND [tid=0x4ca805b]: trying to send packet DHCPACK from 10.x.x.2:67 to 10.x.x.32:68 on interface vlan0151
"Deny Service Binding" is ignored by Raw Sockets

In an attempt to stop Kea from "hearing double", I enabled the "Deny service binding" option on the CARP VIP configuration. However, technical analysis via sockstat confirms that this has no effect on Kea when using Raw Sockets.

Sockstat output:

Code Select Expand

root kea-dhcp4 58683 54 udp4 10.x.x.2:67 *:* # Physical IP Socket
root kea-dhcp4 58683 56 udp4 10.x.x.1:67 *:* # CARP VIP Socket (Should have been denied)
The Definitive Root Cause (Kea Documentation)

I found the explanation in the official Kea 3.0.2 Documentation (Section 9.2.4 - Interface Configuration). It explicitly warns about this exact behavior:

"Caution should be taken when configuring the server to open multiple raw sockets on the interface with several IPv4 addresses assigned. If the directly connected client sends the message to the broadcast address, all sockets on this link will receive this message and multiple responses will be sent to the client."

This confirms that by configuring Kea using only the interface name (e.g., vlan04), OPNsense triggers this "double-hearing" behavior because Kea detects both the physical IP and the CARP VIP. 

The Documented Solution vs. Kea Implementation on OPNSense

The Kea documentation provides a clear fix: 

"To use a single address on such an interface, the 'interface-name/address' notation should be used."

Currently, OPNSense only generates the configuration listing the interface names:

Code Select Expand

"interfaces": [ "vlan01", "vlan02" ]
To be stable in HA/CARP environments, the service should implement (or allow) the notation:

Code Select Expand

"interfaces": [ "vlan01/10.x.x.2", "vlan02/10.x.y.2" ] (binding only to the Physical IP).
Feature Requests for OPNsense Developers

Based on the evidence above and the official Kea documentation, I would like to propose the following enhancements to the Kea implementation on OPNSense to better support CARP/HA deployments on FreeBSD:

User-configurable Thread Pool Size:
Exposing the thread-pool-size parameter in the Web GUI would allow administrators to limit Kea to a single thread when necessary. While Kea is highly performant, in environments where BPF duplication occurs, forcing a single thread (size: 1) effectively mitigates simultaneous race conditions where different threads might offer conflicting leases for the same client request.

Implementation of Explicit Interface Binding (interface/address):
According to Kea documentation (Section 9.2.4), the current method of binding to the interface name alone causes every open Raw/BPF socket to receive a copy of broadcast packets. I suggest updating the plugin logic to allow (or automatically implement) the "interface-name/address" notation (e.g., "vlan01/10.x.x.2"). Binding specifically to the Physical IP of the node, while still defining the CARP VIP as the gateway in the subnet options, appears to be the only documented way to prevent the sequential double-processing of packets in a CARP setup while maintaining the mandatory Raw Socket mode for FreeBSD.

These adjustments would make the Kea integration significantly more robust for High Availability clusters, preventing log exhaustion and potential IP conflicts.

[Environment Setup:]

Hello community,

I am experiencing a persistent issue with the Kea DHCP implementation on an OPNsense HA Cluster. When using the default configuration (Raw Sockets + Multi-threading), the Kea service seems to "hear double" on interfaces where CARP is active, leading to duplicated logs and, more critically, conflicting IP offers to the same client.

Environment Setup:

Version: OPNsense 25.7.11_9-amd64.
Setup: Two nodes in HA (Master: Physical / Backup: Virtual).
Networking: High Availability with CARP acting as gateways across multiple VLANs.
DHCP: Kea DHCP configured in HA (Hot-Standby).

The Problem:
When Kea is configured with the default Socket Type: Raw, the service seems to "double-process" incoming broadcast packets on interfaces that have both a physical IP and a CARP VIP.

Because Kea uses multi-threading by default (4 threads), two (or more) different threads capture the same DHCPDISCOVER or DHCPREQUEST at the exact same millisecond. This creates a race condition: each thread checks the lease database, sees an available IP, and sends a separate DHCPOFFER or DHCPACK. In some instances, they even offer different IPs to the same client for the same transaction.

Evidence (Logs):
Notice the different thread IDs (e.g., ...9808 and ...a008) processing the same Transaction ID (tid=0xcb0e85d2) at the exact same millisecond:

Code Select Expand

2026-04-29T08:46:41-03:00 kea-dhcp4 INFO [kea-dhcp4.packets.0x3fb7b6ac9808] DHCP4_PACKET_RECEIVED [hwtype=1 74:86:e2:xx:xx:43], cid=[01:74:86:e2:xx:xx:43], tid=0xcb0e85d2: DHCPREQUEST received from 0.0.0.0 to 255.255.255.255 on interface vlan031
2026-04-29T08:46:41-03:00 kea-dhcp4 INFO [kea-dhcp4.packets.0x3fb7b6aca008] DHCP4_PACKET_RECEIVED [hwtype=1 74:86:e2:xx:xx:43], cid=[01:74:86:e2:xx:xx:43], tid=0xcb0e85d2: DHCPREQUEST received from 0.0.0.0 to 255.255.255.255 on interface vlan031

2026-04-29T08:46:41-03:00 kea-dhcp4 INFO [kea-dhcp4.leases.0x3fb7b6ac9808] DHCP4_LEASE_OFFER [hwtype=1 74:86:e2:xx:xx:43], tid=0xcb0e85d2: lease 10.x.x.11 will be offered
2026-04-29T08:46:41-03:00 kea-dhcp4 INFO [kea-dhcp4.leases.0x3fb7b6aca008] DHCP4_LEASE_OFFER [hwtype=1 74:86:e2:xx:xx:43], tid=0xcb0e85d2: lease 10.x.x.10 will be offered
Troubleshooting already performed:

Network Mask Check: Validated that there are no subnet overlaps on CARP VIPs.

Socket Type Change: * When switching to Socket Type: UDP, the duplication stops and logs become clean. However, many clients (especially those initiating discovery from 0.0.0.0) fail to receive IPs in UDP mode.

Analysis:

Binding: sockstat -4 -l shows Kea binding to both the Physical IP (e.g., 10.x.x.2:67) and the CARP VIP (e.g., 10.x.x.1:67).

Code Select Expand

USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
0        kea-dhcp4  47445 18  udp4   10.x.x.2:67           *:*
0        kea-dhcp4  47445 20  udp4   10.x.x.1:67           *:*
It appears that with Raw Sockets, the BPF (Berkeley Packet Filter) delivers the packet to all listening threads. Since both the Physical IP and the CARP VIP are on the same interface, Kea seems to be binding in a way that triggers this double processing.

Questions:

Persistent Thread Limit: Is there a way to limit Kea threads to 1 through the GUI or a tunable?

Deny Service Binding: I noticed the "Deny service binding" option in the CARP settings.

If I enable this for the CARP, would it prevent Kea from listening on the Virtual IP, thus solving the duplication while keeping Raw Sockets?

What are the side effects for other services like ntpd or unbound if they rely on the CARP to serve clients?

Best Practice: Is this a known limitation of Kea on FreeBSD when CARP is involved, and what is the recommended way to use Kea in an HA setup without duplicate processing?

Any insights would be greatly appreciated!