[SOLVED] WAN stops working randomly?

[Dowd]

So I am 90% sure I found the issue and I am currently testing to see if it goes away. I noticed upon boot this time around that it output the WAN address to the IPMI interface despite not having any cable in the IPMI interface, which struck me as really odd. I went into BIOS and under BMC there is a feature? function? called LAN Failover (why is this default I have no idea), but the Mac Address for that matched the rogue one that kept popping up in the logs.

Since I cant toggle it on/off for the failover in BIOS I just disabled BMC completely. If I dont reply back to this thread, then for anyone in the future this was the culprit. Apparently there is a way to turn it off in the firmware as well.

A similar thread on reddit had the same issue from 8 years ago when I was trying to find a fix for it - https://www.reddit.com/r/homelab/comments/58ojk1/psa_if_you_are_using_an_asrock_board_with_ipmi/

This also explains why when I first setup the system it didn't disconnect for long periods of time, because I had the BMC cable plugged in.

[Patrick M. Hausen]

I went into BIOS and under BMC there is a feature? function? called LAN Failover (why is this default I have no idea), but the Mac Address for that matched the rogue one that kept popping up in the logs.

It's the default because it makes sense. You want IPMI for any server - at least I know that I do. And piggybacking the IPMI interface on the first network port is common practice so you have IPMI available with just a single cable.

I don't have any monitor and keyboard anywhere for about one hundred systems. IPMI is the way ...

If you disabled the BMC how do you intend to monitor and control your fan speeds, for example?

And BTW - since at least with the Supermicro boards I run, IPMI is always piggybacked to the first network port - why is that WAN in your installation? The first port is LAN in new OPNsense installs so you changed that? If you keep the standard order of interfaces, IPMI will receive an IP address from your OPNsense DHCP server and will be reachable from LAN - which makes much more sense.

Kind regards,
Patrick

[opnfwb]

Nice work finding the cause and yes, this is what I was talking about in reference to HP servers. They allow their lights out functions (remote management) to move around the physical integrated NICs and sometimes this causes confusion. Seems Asrock has this feature as well.

[cookiemonster]

> If I dont reply back to this thread, then for anyone in the future this was the culprit. Apparently there is a way to turn it off in the firmware as well.

What about coming back to mark it as resolved and confirm it was? And to say thanks to people who stopped to try to give a hand. Would take all of a about 30 secs.

[Dowd]

I went into BIOS and under BMC there is a feature? function? called LAN Failover (why is this default I have no idea), but the Mac Address for that matched the rogue one that kept popping up in the logs.

It's the default because it makes sense. You want IPMI for any server - at least I know that I do. And piggybacking the IPMI interface on the first network port is common practice so you have IPMI available with just a single cable.

I don't have any monitor and keyboard anywhere for about one hundred systems. IPMI is the way ...

If you disabled the BMC how do you intend to monitor and control your fan speeds, for example?

And BTW - since at least with the Supermicro boards I run, IPMI is always piggybacked to the first network port - why is that WAN in your installation? The first port is LAN in new OPNsense installs so you changed that? If you keep the standard order of interfaces, IPMI will receive an IP address from your OPNsense DHCP server and will be reachable from LAN - which makes much more sense.

Kind regards,
Patrick

I dont disagree with you at all about the benefits of IPMI/BMC. My issue is with how terrible this implementation is that by default if BMC is not plugged in, it will hijack a port is not allocated to it for BMC. Lets suppose you run a server that is issued a public IP address or at the edge of your network. By default your BMC can take over your IP address exposing your IPMI to the web assuming that you did not properly create firewall rules (if they are supported). What happens if your BMC loses connectivity from a switch going down, or cable goes bad? It will take over whatever traffic is happening on your main interface, killing whatever is running on it. BMC going down is peanuts compared to say your email server or web server going down because your BMC decided it wants the main port. Its a terrible design feature that needs to be changed so you opt INTO bonding interfaces not opt OUT as the defaults are not secure in the slightest. I prefer the old days where management ports were dedicated and if you wanted to use them, you could turn them on or off, but it would be that specific port.

As for why I disabled it, I only use BMC currently for Firmware and BIOS flashing currently, fan speeds dont really matter. Its a server after all, if fan loudness matters in this case I would just order quieter fans honestly.

> If I dont reply back to this thread, then for anyone in the future this was the culprit. Apparently there is a way to turn it off in the firmware as well.

What about coming back to mark it as resolved and confirm it was? And to say thanks to people who stopped to try to give a hand. Would take all of a about 30 secs.

Its marked as resolved and given rep.