WAN drop-outs

[colordrops]

I've got OPNSense on bare metal, and am seeing intermittent WAN dropouts anywhere from a couple seconds to a minute.  It happens a lot more with high traffic.  At first, I thought it was due to running in KVM/libvirt, so I bought router hardware, and am seeing the same issue.  We called Spectrum, our ISP, and they say everything looks good on their end. 

I've tried every setting I could find through exploration and through other forum posts, e.g. disabling IDS, disabling IPv6, changing firewall optimization, enabling gateway monitoringj, disabling state killing on gateway failure, "override MTU" on the WAN on and off, and experimented with other settings as well.

I still don't know whether this is an ISP problem, an OPNSense problem, or maybe a mixture of both.  It's never been 100% stable, even when using other routers, i.e. the UDM-Pro, but it's far worse now with OPNSense.  The drops get much worse with high traffic.

I've tested with the mtr tool, trying to get as close as possible to the modem.  SSHing into OPNSense, then running

Code: [Select]

mtr <ISP gateway IP>

still shows drop-outs.  Is it possible for OPNSense to be at fault this close to the modem, or is this certainly an ISP issue based on this test?

BTW, when the drop-outs occur, mtr outputs "no route to host".  It seems to be more correlated with maxing out uploads rather than downloads.

[colordrops]

More details have emerged after further investigation

I noticed that ping times would go up significantly (eventually creeping up to 1000ms) when running speedtest.  Reading that this was due to bufferbloat, I setup QoS with CoDel to successful avoid maxing out the upstream, which kept ping times close to 10ms with about 80% utilization of bandwidth.

To my dismay, the drops still occurred with nearly every speedtest.

With QoS on now, the Speedtest actually killed the connection.  The following were observed:

1. Unbound was pinned at > 100% CPU.  Logs showed a bunch of "udp connect failed: Network is unreachable for <several root DNS IPs>"
2. A service "dpinger" (gateway monitor) that was not normally in the service list, was now listed, and shown as down
3. The WAN_DHCP gateway was stuck in "Pending"

Symptoms (2) & (3) persisted after a reboot as well as a power cycle.  This leads me to believe that there may be something going on with the ISP or at least the cable model, but still not sure.

EDIT: (1) also happens after reboot.

Furthermore, from the OPNSense console, I'm able to do DNS lookup on google.com and ping google.com and 8.8.8.8, but no other domains.  More evidence that this might be an ISP thing.

[seang96]

I believe your issue may be related to mine. I found out that the modem that I had from Spectrum is built by Hitron. Hitron modems from Spectrum have a PUMA chipset. These modems are known for having huge spikes in latency and jitter. I bought a Nighthawk with a broadcom chipset, but it also doubled the normal latency. I had Spectrum service come in again with an Ubee or Technicolor modem which resolved my issue entirely.

Hope this helps!

http://badmodems.com/