Messages

A much belated update after much stress & chaos -- but a solution! You were correct Passeri, this is NOT a Kea related issue at all.

There were a few problems, but I believe the two big ones were:

1. VLAN #1 being treated differently by different systems.  Some have it as a default, some have it as a normal VLAN. The equipment I started on treated it normal so I made the mistake of relying on it. Won't do that again!

2. It turns out that the MS-01 uses vPro -- an IPMI that uses a shared network port (unlike iDrac) with the host OS. Further, a subset of vPro (conveniently including the MS-01) does not turn off entirely even when you specifically tell it to turn off, and continues to have certain parts running which happen to include some sort of network management which interferes with certain packets (such as DHCP packets) before the OS has a chance to see them.

So the packets were being sent to the hardware just fine, but the OS never saw them. As soon as I swapped to ports outside of vPro's scope it all worked fine!

Hopefully others will find this information useful, it sure wasn't easy to find or diagnose (until I knew exactly what to look for).

Update: I found out that part of the problem is that when I introduced the new router I also introduced a new switch.  It turns out that part of the problem is that the two switches (old & new) handle VLAN 1 differently, which is leading packets to be tagged to incorrect VLANs on incorrect ports leading to unexpected behavior.  Since VLAN 1 is where my management ports are, I suspect that this is a major factor in the problems I'm seeing on that network.

While I don't know that this explains all the problems I'm running into, I do believe it explains enough that it warrants investigation.  This "investigation" so to speak, is a complete rebuild of my network.  This will obviously take some time, and I will report back when done -- hopefully with success.

I am happy that it seems (at least for now) that the problem is not Kea, or OPNSense at all.

"coatmaker618, this may seem pedantic but your answers are not actually clarifying things."  Nah, that's a pretty important aspect of answering questions, given that I'm not running for office.  Also, I appreciate your breakdown of assumptions.

What I really should've clarified is that I'm not sure I'm understanding what you were asking for exactly.


1. Check your client configuration is (if possible) pointed at your DHCP server
I'm not sure what configurations you want me to check when I set network info to "automatic". Doesn't that, by definition mean there are no configs?

2. In Kea reserve any address so long as it is outside your pool, using the client's MAC address
I appreciate you saying outside your pool instead of above here.  I have several reserved/statically assigned IPs in Kea that work fine on other interfaces.  In fact this is my preferred method of IP assignment.  This includes the desktop, it already has a statically reserved IP in Kea on the LAN interface.

3. Ask the client to renew its lease
I admit I mostly do this by unplugging a cable rather than using release/refresh/renew commands, I assume that is sufficient (overkill) if I wait a few seconds?  if not I can look up the commands, but given how messy things are right now, I do lean towards simple/overkill.

4. Show exactly what was the client configuration before you tried
Again, not sure what client configuration you would like.

5. Show /var/log/kea/latest.log where there is any record of the client's MAC or IP.
At the moment let us assume there is not -- since as you said my logs don't match yours.  What does this mean, especially on the LAN?

An interesting note:
I ran into an issue a few days ago where I lost internet connection -- TL;DR that ended up being Verizon's fault but I was obviously suspicious & troubleshooting since things currently aren't stable.

During the course of troubleshooting I brought in my laptop (Linux) and noted that the laptop also does not get a DHCP lease on the LAN but also works fine with static IP. However laptop does get an IP on other network interfaces which interestingly the windows desktop does not.  I don't know what logs you want from where, but let me know and I'll be happy to share.

This does make me think that there's something wrong with desktop, but I think there's also something wrong with LAN although it may NOT be Kea (I do not know what else it could be, gateway maybe as that's the other piece of info you provide in a static IP?).

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing? Is communication from LAN client to server verified as happening, e.g. with a ping?

What does the log show in that situation? Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.

The situation is somewhat confused for me because I do not have a consistent case of basic client configuration with an associated log. You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing?
Not sure I understand the questions. I thought the gateway came with DHCP assignment from the DHCP server?

Is communication from LAN client to server verified as happening, e.g. with a ping?
Again, not sure what you mean here. Without the client having a valid IP on the subnet how can I ping?
That said, yes if the client is set to DHCP it gets nothing.

What does the log show in that situation?
The OPNSense log or client log?  I posted the OPNSense log here https://forum.opnsense.org/index.php?topic=49321.msg250177#msg250177  only filtered by MAC, so it should have everything with the desktop.  If there's another filter or something else you want me to post, just let me know.

Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.
My pool is 100-199, so everything below 100 is actually reserved.

The situation is somewhat confused for me.
Me too!

You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.
I have every firewall rule log enabled. I've had too many issues in the past where things weren't obvious and it's cause a firewall rule was allowing/blocking it unexpectedly.

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing? Is communication from LAN client to server verified as happening, e.g. with a ping?

What does the log show in that situation? Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.

The situation is somewhat confused for me because I do not have a consistent case of basic client configuration with an associated log. You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing?
Not sure I understand the questions. I thought the gateway came with DHCP assignment from the DHCP server?

Is communication from LAN client to server verified as happening, e.g. with a ping?
Again, not sure what you mean here. Without the client having a valid IP on the subnet how can I ping?
That said, yes if the client is set to DHCP it gets nothing.

What does the log show in that situation?
The OPNSense log or client log?  I posted the OPNSense log here https://forum.opnsense.org/index.php?topic=49321.msg250177#msg250177  only filtered by MAC, so it should have everything with the desktop.  If there's another filter or something else you want me to post, just let me know.

Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.
My pool is 100-199, so everything below 100 is actually reserved.

The situation is somewhat confused for me.
Me too!

You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.
I have every firewall rule log enabled. I've had too many issues where things weren't obvious and it's cause a firewall rule was allowing/blocking it unexpectedly.

Update: The rest of the interfaces (everything but LAN) are now working.  I forgot that by default all network traffic on those interfaces is blocked (which is a good default).

So now it's ONLY the LAN interface that's acting up.

Note: I did check the LAN, it has default allow all rules.  So sadly it's not the same problem there too.

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

From the server side, I guess nothing because I was jumping back and forth a lot with this.

Was there something more specific you wanted me to check?

Ok, just making sure it wasn't a problem. 

Static assignment config is attached.

I don't think it's a client issue because everything works with the old OPNSense router.  It's just old hardware and has some quirks built up after years of use, a fresh start seemed to be in order.

Ahhh, I can explain. So the 192.168.x.y is a format I'm using.  The x represents the subnet, easy enough. The y is 3 for the router since there's a longterm goal of using this router in a HA/failover setup.  I did setup CARP on each interface to be the .1 address but I turned that off days ago as it adds more complexity to troubleshooting.

But that's why you're seeing a strange number choice. I can turn CARP back on (or reboot yet again) if that would help (eg: if something is looking for .1 -- it shouldn't be a problem since this is the only router so it's always master/main on the CARP interface). But I hope that helps explain the strange IPs you're seeing (.3 for a router instead of .1).

Note that this is the same on 192.168.1.y & 192.168.10.y

Also, is 'implied gateway' just because a.b.c.1 is the started gateway or is it stated somewhere in the log/settings? I didn't see it, but I sure could be looking right at it and missing it.

The total logfile is a bit long (a little over 5k lines, but I did a search for the MAC of my desktop as well as the MAC of a server getting a static assignment successfully via DHCP) so you can see the results of each.  I guess I've been restarting the server a lot while debugging!

Per the command request:
root@OPNsense02:~ # sockstat -ln | egrep -ai 'user|:67'
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
0        kea-dhcp4  73038 15  udp4   192.168.1.3:67        *:*
0        kea-dhcp4  73038 17  udp4   192.168.2.3:67        *:*
0        kea-dhcp4  73038 19  udp4   192.168.3.3:67        *:*
0        kea-dhcp4  73038 21  udp4   192.168.10.3:67       *:*
0        ntpd       32097 23  udp6   fe80::5a47:caff:fe79:6752%igc0:123 *:*

Here's what I can piece together. I've disabled a bunch of entries just for the sake of testing but it's still problematic.

Is there any easy way to export those? I only ask as screenshots are kind of tough with the low filesize limit.

Heh. Not that I know of. I'm not an image-editing wizard, and I have bad eyes to boot. But it's tough to speculate without your config. I didn't see anything in the log that stood out.

Hah, fair enough. Turns out it may be a red herring after all!

I just tried the old ISC DHCP server on the LAN interface and the desktop is STILL not getting a DHCP address.  I've used that DHCP server enough to be reasonably comfortable with it, so I think it's pretty unlikely I did anything wrong there. Besides, now I have two DHCP servers not working!

So I'm thinking it must be something common, as in not the DHCP server itself but some other router setting? I'm at a loss, but I can give you the list of VLANs if that'll help, or (probably easier) delete most of them and re-add them once I have this working.

Just a quick update, I got the widget to correctly display the gateway.  While restarting the system did not fix that issue, deleting and re-adding the widget did.  I suspect it was a cached list somewhere in the widget itself where the widget was querying only the gateways it knew rather than querying the

Still no luck on the DHCP server not providing IP addresses.

I have a new OPNSense install that I am setting up, and one of the new things I'm doing is using Kea as the DHCP server instead of the (apparently now defunct per https://docs.opnsense.org/manual/isc.html#isc-dhcp) ISC.

So far I really like Kea from a GUI perspective, it's much more straightforward and clear than ISC so I'd prefer to keep using it.  However I am not getting DHCP assigned on my LAN.  Interestingly, I am seemingly getting DHCP addresses on the VLANs (at least from a preliminary look).  I know everything is setup correctly as setting a static IP on my desktop (on the LAN) works perfectly.

To confuse things further I went to Kea's logs and did a few searches which seem to indicate that it is seeing a DHCP request from my desktop and trying to issue a DHCP lease to it -- at least per my reading of the logs.  I've attached the results of a search of the desktop MAC (which is not getting an IP via DHCP).

It turns out that while setting up I did unintentionally activate dnsmasq, but that has been stopped and OPNSense has been rebooted so I hope that's now just a red herring.

Post your interface assignments and IPs, Kea settings and subnets.

Is there any easy way to export those? I only ask as screenshots are kind of tough with the low filesize limit.

I have a new OPNSense install that I am setting up, and one of the new things I'm doing is using Kea as the DHCP server instead of the (apparently now defunct per https://docs.opnsense.org/manual/isc.html#isc-dhcp) ISC.

So far I really like Kea from a GUI perspective, it's much more straightforward and clear than ISC so I'd prefer to keep using it.  However I am not getting DHCP assigned on my LAN.  Interestingly, I am seemingly getting DHCP addresses on the VLANs (at least from a preliminary look).  I know everything is setup correctly as setting a static IP on my desktop (on the LAN) works perfectly.

To confuse things further I went to Kea's logs and did a few searches which seem to indicate that it is seeing a DHCP request from my desktop and trying to issue a DHCP lease to it -- at least per my reading of the logs.  I've attached the results of a search of the desktop MAC (which is not getting an IP via DHCP).

It turns out that while setting up I did unintentionally activate dnsmasq, but that has been stopped and OPNSense has been rebooted so I hope that's now just a red herring.