Messages

Updating that interface on 25 Gbe NIC works now, was just a cable that got loose. 

Hi Patrick, thank you for your answer.
My WAN interface is my internet connection connected to the provider box.
My LAN and WAN have the same network adress (192.168.1.xxx) is it a problem?
After your recommandation i was able ton ping correctly just 1 time my internal DNS.
[img width=50% height=50%]https://ibb.co/JwkwybPW[/img]

As Patrick said, yes this is a huge problem. The router doesn't know which port to send data to if any two ports have ANY overlapping IP address space.  This would explain why you got exactly one ping.

Ok, tried installing to disk and it's working beautifully now.  I'm sure going to HDD over USB helped, but it feels like it was just a bad install.

I do indeed have a PERC card, but still only installed to one disk to avoid config issues (I really just want to get this working at the moment, happy to experiment later).  I did notice that ZFS seems to be the default now, I had a terrible experience with that a few years ago where USB drives did NOT like it (I was only trying to do a Zmirror) and kept getting "corrupted" (they were recoverable by reformatting) but going back to UFS fixed it instantly. I take it ZFS is seen as stable now?  Needless to say this is not a ZFS install at the moment.

The good news is that OPNsense does show me the interfaces (mce0 & mce1) on the card. The bad news is that even though they report to be up in the assignments page (green icon instead of red), the VLANs on top all seem to be down. I think there's a trick I'm missing, but that's a tomorrow me problem!

I've got most of my config moved over, so I'm calling this a win for the night.

TL;DR:
I installed OPNsense on a Dell R620 which should be massive overkill, but it's painfully slow and having problems that normally just don't happen.  Are there known issues with OPNsense on Dells, or anything that needs to be done starting fro ma fresh install?

Note that this is bare metal, no virtualization.

Background:
I'm trying to find a good host for an OPNsense router noting that I have a bajillion VLANs and 25 Gbe internal networking.

Previously I tried a miniforums MS-01 and that was AWFUL.  So I figured for sure a Dell server would be a quick solution that would just work while I tried other more questionable solutions.

Unfortunately it's anything but! I was able to do the installation to a USB (as I normally do) which was slow, but that's a one time thing.  However running was painfully slow:

But for instance if I'm at the command interface and I press '8' to open a shell, it took 21.86 seconds to load a shell and present a prompt.

The process of setting the IP of a network interface was 1 minute 33 seconds -- 37.09 seconds  of which were walking through the prompts of me not changing anything (for those who want to accuse me of being a slow typist).

I also never even GOT to the web interface because I couldn't get it started. One problem I did notice was that when I changed the network interface & set an IP, the route to the IP stayed on the old physical interface -- for instance by default igb0 was WAN & igb1 was LAN but I wanted to change LAN to igb2. When I made that change the route 192.168.1.0/24 stayed on igb1 even though nothing (LAN nor WAN) was on igb1 so naturally the 'interface was down'.  Changing the interface manually seems to have worked enough for pinging to work but not enough to trigger the webgui to start.

I'm retrying now on a HDD instead of USB (sadly these) as I did find a post saying USB drivers can be slow.  But that's a little confusing to me as I thought OPNSense lived in RAM while running.

Any recommendations on how to get this hardware to work or reasons this hardware won't work (if there's an issue, I'll happily take recommendations)? See specs below.

Specs:
Dell R620
Dual Intel(R) Xeon(R) CPU E5-2650L v2 @ 1.70GHz
192 GB DDR3 ECC RAM

A much belated update after much stress & chaos -- but a solution! You were correct Passeri, this is NOT a Kea related issue at all.

There were a few problems, but I believe the two big ones were:

1. VLAN #1 being treated differently by different systems.  Some have it as a default, some have it as a normal VLAN. The equipment I started on treated it normal so I made the mistake of relying on it. Won't do that again!

2. It turns out that the MS-01 uses vPro -- an IPMI that uses a shared network port (unlike iDrac) with the host OS. Further, a subset of vPro (conveniently including the MS-01) does not turn off entirely even when you specifically tell it to turn off, and continues to have certain parts running which happen to include some sort of network management which interferes with certain packets (such as DHCP packets) before the OS has a chance to see them.

So the packets were being sent to the hardware just fine, but the OS never saw them. As soon as I swapped to ports outside of vPro's scope it all worked fine!

Hopefully others will find this information useful, it sure wasn't easy to find or diagnose (until I knew exactly what to look for).

Update: I found out that part of the problem is that when I introduced the new router I also introduced a new switch.  It turns out that part of the problem is that the two switches (old & new) handle VLAN 1 differently, which is leading packets to be tagged to incorrect VLANs on incorrect ports leading to unexpected behavior.  Since VLAN 1 is where my management ports are, I suspect that this is a major factor in the problems I'm seeing on that network.

While I don't know that this explains all the problems I'm running into, I do believe it explains enough that it warrants investigation.  This "investigation" so to speak, is a complete rebuild of my network.  This will obviously take some time, and I will report back when done -- hopefully with success.

I am happy that it seems (at least for now) that the problem is not Kea, or OPNSense at all.

"coatmaker618, this may seem pedantic but your answers are not actually clarifying things."  Nah, that's a pretty important aspect of answering questions, given that I'm not running for office.  Also, I appreciate your breakdown of assumptions.

What I really should've clarified is that I'm not sure I'm understanding what you were asking for exactly.


1. Check your client configuration is (if possible) pointed at your DHCP server
I'm not sure what configurations you want me to check when I set network info to "automatic". Doesn't that, by definition mean there are no configs?

2. In Kea reserve any address so long as it is outside your pool, using the client's MAC address
I appreciate you saying outside your pool instead of above here.  I have several reserved/statically assigned IPs in Kea that work fine on other interfaces.  In fact this is my preferred method of IP assignment.  This includes the desktop, it already has a statically reserved IP in Kea on the LAN interface.

3. Ask the client to renew its lease
I admit I mostly do this by unplugging a cable rather than using release/refresh/renew commands, I assume that is sufficient (overkill) if I wait a few seconds?  if not I can look up the commands, but given how messy things are right now, I do lean towards simple/overkill.

4. Show exactly what was the client configuration before you tried
Again, not sure what client configuration you would like.

5. Show /var/log/kea/latest.log where there is any record of the client's MAC or IP.
At the moment let us assume there is not -- since as you said my logs don't match yours.  What does this mean, especially on the LAN?

An interesting note:
I ran into an issue a few days ago where I lost internet connection -- TL;DR that ended up being Verizon's fault but I was obviously suspicious & troubleshooting since things currently aren't stable.

During the course of troubleshooting I brought in my laptop (Linux) and noted that the laptop also does not get a DHCP lease on the LAN but also works fine with static IP. However laptop does get an IP on other network interfaces which interestingly the windows desktop does not.  I don't know what logs you want from where, but let me know and I'll be happy to share.

This does make me think that there's something wrong with desktop, but I think there's also something wrong with LAN although it may NOT be Kea (I do not know what else it could be, gateway maybe as that's the other piece of info you provide in a static IP?).

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing? Is communication from LAN client to server verified as happening, e.g. with a ping?

What does the log show in that situation? Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.

The situation is somewhat confused for me because I do not have a consistent case of basic client configuration with an associated log. You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing?
Not sure I understand the questions. I thought the gateway came with DHCP assignment from the DHCP server?

Is communication from LAN client to server verified as happening, e.g. with a ping?
Again, not sure what you mean here. Without the client having a valid IP on the subnet how can I ping?
That said, yes if the client is set to DHCP it gets nothing.

What does the log show in that situation?
The OPNSense log or client log?  I posted the OPNSense log here https://forum.opnsense.org/index.php?topic=49321.msg250177#msg250177  only filtered by MAC, so it should have everything with the desktop.  If there's another filter or something else you want me to post, just let me know.

Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.
My pool is 100-199, so everything below 100 is actually reserved.

The situation is somewhat confused for me.
Me too!

You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.
I have every firewall rule log enabled. I've had too many issues in the past where things weren't obvious and it's cause a firewall rule was allowing/blocking it unexpectedly.

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing? Is communication from LAN client to server verified as happening, e.g. with a ping?

What does the log show in that situation? Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.

The situation is somewhat confused for me because I do not have a consistent case of basic client configuration with an associated log. You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.

Just to confirm, if the client knows the gateway and is set to get an IP by DHCP then it gets nothing?
Not sure I understand the questions. I thought the gateway came with DHCP assignment from the DHCP server?

Is communication from LAN client to server verified as happening, e.g. with a ping?
Again, not sure what you mean here. Without the client having a valid IP on the subnet how can I ping?
That said, yes if the client is set to DHCP it gets nothing.

What does the log show in that situation?
The OPNSense log or client log?  I posted the OPNSense log here https://forum.opnsense.org/index.php?topic=49321.msg250177#msg250177  only filtered by MAC, so it should have everything with the desktop.  If there's another filter or something else you want me to post, just let me know.

Have you tried giving the client's MAC a reserved address above the .1-.199 pool? I found that where I wanted a fixed client IP I needed to reserve rather than relying on manual IP configuration of the client.
My pool is 100-199, so everything below 100 is actually reserved.

The situation is somewhat confused for me.
Me too!

You can also enable logs of LAN rules to verify the packets are passed on that interface. I am stopping short of packet tracing although that may be a next step.
I have every firewall rule log enabled. I've had too many issues where things weren't obvious and it's cause a firewall rule was allowing/blocking it unexpectedly.

Update: The rest of the interfaces (everything but LAN) are now working.  I forgot that by default all network traffic on those interfaces is blocked (which is a good default).

So now it's ONLY the LAN interface that's acting up.

Note: I did check the LAN, it has default allow all rules.  So sadly it's not the same problem there too.

There are no requests for, or allocations of, 192.168.1.42 in either log.

So what happens if you set assignment to DHCP rather than manual?

If I set assignments to DHCP (automatic) I don't get an IP on the client.
    "inet 169.254.41.44/16 brd 169.254.255.255 scope global dynamic"

From the server side, I guess nothing because I was jumping back and forth a lot with this.

Was there something more specific you wanted me to check?

Ok, just making sure it wasn't a problem. 

Static assignment config is attached.

I don't think it's a client issue because everything works with the old OPNSense router.  It's just old hardware and has some quirks built up after years of use, a fresh start seemed to be in order.

Ahhh, I can explain. So the 192.168.x.y is a format I'm using.  The x represents the subnet, easy enough. The y is 3 for the router since there's a longterm goal of using this router in a HA/failover setup.  I did setup CARP on each interface to be the .1 address but I turned that off days ago as it adds more complexity to troubleshooting.

But that's why you're seeing a strange number choice. I can turn CARP back on (or reboot yet again) if that would help (eg: if something is looking for .1 -- it shouldn't be a problem since this is the only router so it's always master/main on the CARP interface). But I hope that helps explain the strange IPs you're seeing (.3 for a router instead of .1).

Note that this is the same on 192.168.1.y & 192.168.10.y

Also, is 'implied gateway' just because a.b.c.1 is the started gateway or is it stated somewhere in the log/settings? I didn't see it, but I sure could be looking right at it and missing it.

The total logfile is a bit long (a little over 5k lines, but I did a search for the MAC of my desktop as well as the MAC of a server getting a static assignment successfully via DHCP) so you can see the results of each.  I guess I've been restarting the server a lot while debugging!

Per the command request:
root@OPNsense02:~ # sockstat -ln | egrep -ai 'user|:67'
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
0        kea-dhcp4  73038 15  udp4   192.168.1.3:67        *:*
0        kea-dhcp4  73038 17  udp4   192.168.2.3:67        *:*
0        kea-dhcp4  73038 19  udp4   192.168.3.3:67        *:*
0        kea-dhcp4  73038 21  udp4   192.168.10.3:67       *:*
0        ntpd       32097 23  udp6   fe80::5a47:caff:fe79:6752%igc0:123 *:*

Here's what I can piece together. I've disabled a bunch of entries just for the sake of testing but it's still problematic.