[Solved] Odd occurence I am having with LAN...

[Amodin]

Solved - My last post explained it.

So, this is something I've had a problem with since before OPN, so I don't think it's a problem there, but it's certainly an issue with my LAN that I am tired of hearing about from my wife.

At any rate, whenever I lose internet connectivity, like what just happened (ATT fiber), my LAN seems to also disconnect wireless and access to my NAS.

I have a really simple setup:
172.x.x.x/16 subnet
.1 is the OPN LAN
ATT fiber router is set to passthrough to OPN on second NIC.

APs are Unifi and controlled by Unifi Network software.  All clients, APs, devices:
Gateway address, .1, is the OPN
OPN is also DHCP Server. 
DNS is also set in DHCP for all clients, including static assignments.
DNS are my two internal recursive Pi-hole VMs, .30 and .40 addresses.
All clients use those two DNS servers.
NAS, servers, etc. all set up this same way.

If the internet goes down, the first thing I notice is the APs go offline, even though they are up and functioning.  When internet comes back, so do the APs.  I had this same problem when I was using Sophos UTM and XG with their APs.

Is there something I'm missing that I am just completely blind to?

Any help would be appreciated so my Minister of Finance (wife) will remain happy.

[cookiemonster]

Realtek NICs perhaps ?
What's your hardware for OPN and rest of the setup, physically not logical ?

[meyergru]

What do you mean by "ATT fiber router is set to passthrough to OPN on second NIC"?

How is that second NIC set up?

Can you tell what IP address that ATT fiber router has?

[Amodin]

I am using the Minisforum MS-01 Mini workstation device, using 10Gb SFP+ port to connect to my LAN core switch, and the connection to the ATT router is ETH.  I believe those are the Intel 226 cards in the MS-01 for ethernet, I don't recall offhand what the fiber NICs are, but I don't believe they are Realtek. 

*I had this same issue with a Super Micro server I have, where I was using an Intel NIC for my connections.*

The router is set to passthrough mode, not bridge mode.  I have an internal IP from the ATT router that assigns an address, and it passes through the external IP to the second NIC (not cascaded), which I can see my external IP in OPN on the Interfaces > Overview.  The services for that connection in the ATT router, i.e., NAT, firewall, and packet filtering are all disabled.  I also have all wireless on the ATT router disabled, and also disabled on the OPN device (it has built-in WiFi that I don't utilize).

[meyergru]

So you have either a double NAT setup or you have set up a back route to 172.x.x.x/16 on the ATT router and that second NIC is the WAN of your OpnSense (either NATed or routed as a default route).

I have a similar setup, but just with an ONT, also on a Minisforum.

Whatever the case, it should be separated from your LAN and the latter should continue to work if the WAN fails... if the ATT router was connected to your LAN somehow, it could explain why with both OpnSense and Sophos, your LAN gets ballistic when WAN is disconnected.

The Unifi controller is local?

On an unrelated side note: Did you disable ASPM for the Intel I226s?

[Amodin]

It's basically a DHCP address from the router to OPN.  I'd love to just bypass it completely but was assured I couldn't with the router I have and wouldn't be able to download the certs.  Besides, I don't know how to do all of that, lol.

Regards to the MS-01, no I didn't change that and didn't know about that issue.  My connection to the router is using igc1, so for me it's the 226-LM card. the 10Gb ports are Intel cards, X710s (just looked on shell with pciconf).  But I don't have an intermittent drop either that I've noticed, and I am on here pretty much every day, most of the day.  Wife works from home and is on it all day.

The only goofy thing I've noticed (aside from my original post), is when I lose internet either by router reboot, or a drop from the ISP, when it returns, the passthrough doesn't give the WAN IP to OPN until I go into the Overview tab and reload the connection, then it will grab the IP.

Regards to Unifi, yes it's local and running in a VM on Proxmox.  The setup I had when using Sophos was using their APs on UTM and later XG, utilizing their controller built into the firewall.  My setup now is separated, specifically to avoid what's happening now.  Guess that didn't work so well, haha.

[meyergru]

In that case, it is clear what is happening: Your OpnSense does not know if/when the ATT router has a new address. DHCP is a pull mechanism. OpnSense would try to get a new address only if the physical layer is down for a bit.

Interesting, though, that you cannot use a static IP, because it is unsual for a router to change the LAN address range on its own. Maybe you could set up gateway monitoring in order to automate the reset process.

[Amodin]

But that makes no sense to me why my LAN connectivity would drop off.  I should have a completely functioning LAN if I had no WAN access.  If my internet connection drops, I lose all connection locally - to my NAS, my APs drop off, my clients lose connection, etc.

I do have the router assigning the same IP, I can either do it as a static address, or DHCP, and I've just left it at DHCP and tied the MAC to the IP so it's always the same address.

OPN assigns all of my internal addresses, I am wondering if this is the issue as it's the only constant between both setups with Sophos and OPN.  If I create my DHCP scope to an internal server and off OPN, I wonder if that would change something...

[meyergru]

If it did, that cannot be the fault of OpnSense, as it happened with Sophos before. As I said, I wonder what is strange in your setup, because if LAN and WAN are separated, nothing from your ISP router could make your LAN get ballistic. Maybe you should do a network sniff to see what exactly goes wrong in these situations.

E.g.: Is it DNS or ARP or broadcast storms?

[Amodin]

Yeah, like I said, I don't believe this is an OPN problem, it's just something that's been plaguing me for some time.

I think I'll try a separate DHCP server and see what that does this evening and go from there.

Not seeing any storms at all, my LAN isn't even close to being saturated, and nothing weird, it's running actually really well. 

[Amodin]

I believe I found the issue, and it was Unifi Controller software monitoring the OPN IP address. 

The software wants to monitor an IP address to be able to say that there's connectivity... kind of silly, but I get it.  I had that pointed at my OPN LAN IP, so anytime I'd reboot, the controller believed connectivity was down, so I'd lose ALL connectivity. 

Just switched it over to a server IP that I have on all the time, solved this issue.