Random and non client specific disconnects

[Callahan]

So I've been living with this issue for about 2 weeks but it's really impacting things now. I've done lots of testing and my results are below.

Environment

• 2 Windows 2012 Domain Controllers each running DNS & DHCP.
• Windows/Mac clients.
• Clients are a mix of hard wired Ethernet connected devices as well as wifi devices (the problem affects both).
• DHCP served by a pair of load balanced Windows DHCP servers.
• DNS forwarder on the DNS servers is pointing to a Pihole running on docker and secondary forwarder is the OPNSense firewall.
• Default gateway is my OPNSense box running on an Intel NUC with additional NIC feed from the internal mini pci-e port. All has worked fine for about 8/9 months or more.
• Running the latest stable version of OPNSense

Problem
Random disconnects of clients.

This almost always happens after a client machine has restarted. I can log onto the client machine and it picks up an address from the DHCP server along with the correct DNS servers/default gateway and correct subnet mask. I get no indication that there is any issue with the connection in the sense that Windows will generally tell you if it can't resolve any hosts on the Internet by showing a yellow exclamation mark against your network connection in the toolbar. I then try to browse the Internet and can't resolve any pages.

• Next, I try and ping google.com. No reply. So I assume it's DNS and ping 8.8.8.8. No reply.
• So next you would assume it's a FW rule (it's not as I'm allowing anything on the LAN out to the Internet on any port just while I get this working again).

The only change I have made of any significance in the last month is that I have set up a VPN to IPVanish via OpenVPN set up on OPNSense that I use with an Alias list to route certain hosts over the IPVanish OpenVPN/OPNSense gateway. In order to ensure that OPNSense doesn't just push traffic down the default gateway when the VPN to IPVanish is down (I still don't see how this is the acceptable default option, it does exactly the opposite of what anyone would expect), I enabled:

Firewall/Settings/Advanced
Gateway Monitoring
Skip rules when gateway is down = disable

I also enabled:

System/Settings/General
DNS Server Options
Do not use the local DNS service as a nameserver for this system

Now the weirdest part is that when a client machine fires up and I log on and the above issue happens, the existing clients that have already gone through the random wait time (upwards of 5 mins most of the time), remain online and working. Clients retain their allocated IP addresses from the DHCP server, the gateway (OPNSense) is still pingable, as is the DHCP/DNS servers. However, beyond that, I can't get anywhere on outside of my own network. Oddly, the web GUI of OPNSense during this down time, despite being pingable from the problem hosts, is unaccessable until you wait out the 5/10 mins for the connection to be suddenly restored. I make no changes to the FW to make this happen during this time. If I restart OPNSense during the time the newly booted hosts can't get Internet access, it fixes the issue (until the next machine fires up and needs Internet access).

So from what I've seen so far:

• It can't be firewall rules as the devices all sit on the same subnet
• It can't be general routing issues as its specific to machines that have just powered up (it also happens randomly throught the day).
• It can't be the connection to the Internet (suggesting modem issues) as the rest of the machines have active connections while the problem machines can't get out the Internet.
• This isn't specific to wired or wifi connections as it happens to both.

I am at a loss to figure out why OPNSense is preventing access. I want to point the finger at the modem connection but as I say, multiple devices are working when the problem devices don't.

Appreciate this is a long post but better to get as much detail in the first one than add the extra details over another 12. :-)

If anyone has any idea as to what could be causing this, I'd really appreciate some pointers.

Thanks.

[deekdeeker]

I dunno if this helps but i had a similar problem with having ipv4 and ipv6 wan addresses and for some reason it was causing a problem. I removed the ipv6 gateway bound to the WAN and fixed my issue.

[Callahan]

Thanks for the suggestion but thats the first thing I disabled when setting it up months ago. All gateways are IPv4.