SOLVED: WAN Latency causing connection loss

[petro49er]

Good morning folks,

I am having an issue I can't figure out.  I was running pfsense for a couple of years but started having WAN issues.  I didn't troubleshoot it too much, but decided to try opnsense.  I had the same WAN latency and connection issues.  I then replaced the hardware, loaded opnsense back on it and am still having the same issues.  I have set up gateway monitoring on my WAN interface and am still getting errors multiple times a day that look like the following: 

2021-08-13T10:32:39   dpinger[13677]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 0 RTT: 18809us RTTd: 22217us Loss: 0%)   
2021-08-13T10:32:39   dpinger[24815]   WAN_DHCP 96.34.137.232: Clear latency 18809us stddev 22217us loss 0%   
2021-08-13T10:31:50   dpinger[62892]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 7254114us RTTd: 8504542us Loss: 5%)   
2021-08-13T10:31:50   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 7254114us stddev 8504542us loss 5%   
2021-08-13T10:31:01   dpinger[24339]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 12962479us RTTd: 16215018us Loss: 31%)   
2021-08-13T10:31:01   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 12962479us stddev 16215018us loss 31%   
2021-08-13T10:30:58   dpinger[96723]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 336654us RTTd: 673228us Loss: 66%)   
2021-08-13T10:30:58   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 336654us stddev 673228us loss 66%   
2021-08-13T10:30:31   dpinger[33094]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 1660366us RTTd: 2855251us Loss: 22%)   
2021-08-13T10:30:31   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 1660366us stddev 2855251us loss 22%   
2021-08-13T10:29:40   dpinger[56890]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 1927744us RTTd: 4000585us Loss: 0%)   
2021-08-13T10:29:40   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 1927744us stddev 4000585us loss 0%   

How would I go about troubleshooting this any further?  Is this an issue with my modem/ISP?  Thank you for any assistance!

[opnfwb]

Yes this looks ISP related. Try setting up a remote gateway instead of just your default WAN gateway and see if there are obvious latency spikes and/or packet loss.

Also make sure that you enable gateway graphing so that you can view spikes/packet loss trends over time.

See the attached screenshot for these two options. Specific a different Monitor IP, such as 8.8.8.8 or 4.2.2.2.

[opnfwb]

Another thing worth checking would be bandwidth usage. If you have a device(s) on the LAN that are heavy bandwidth users, this can cause latency spikes if you aren't using some kind of traffic shaping.

Do the latency alarm timestamps correlate to large WAN bandwidth spikes shown in Reporting/Health/Traffic/WAN section of the firewall GUI.

[petro49er]

Yeah as far as high bandwidth hosts, I don't think it could be that.  I've been monitoring it today and I put 2 Youtube streams on to ensure something was talking.  It hasn't gotten above 20mbps all day I don't believe.  It also seems to go down at odd hours when no one is really utilizing the connection at all.  For the sake of completeness, our line is 400mbps from Spectrum.

Here is another time it went down today:

2021-08-13T13:51:37   dpinger[74101]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 0 RTT: 37814us RTTd: 40228us Loss: 1%)   
2021-08-13T13:51:37   dpinger[24815]   WAN_DHCP 96.34.137.232: Clear latency 37814us stddev 40228us loss 1%   
2021-08-13T13:50:42   dpinger[26704]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 21351229us RTTd: 16578448us Loss: 0%)   
2021-08-13T13:50:42   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 21351229us stddev 16578448us loss 0%   
2021-08-13T13:49:55   dpinger[47103]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 6733170us RTTd: 6224914us Loss: 22%)   
2021-08-13T13:49:55   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 6733170us stddev 6224914us loss 22%   
2021-08-13T13:49:24   dpinger[94857]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 10746488us RTTd: 9258024us Loss: 3%)   
2021-08-13T13:49:24   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 10746488us stddev 9258024us loss 3%   
2021-08-13T13:48:46   dpinger[96159]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 23925829us RTTd: 16391029us Loss: 3%)   
2021-08-13T13:48:46   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 23925829us stddev 16391029us loss 3%   
2021-08-13T13:48:29   dpinger[22140]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 17642us RTTd: 12673us Loss: 75%)   
2021-08-13T13:48:29   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 17642us stddev 12673us loss 75%   
2021-08-13T13:47:57   dpinger[89418]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 6801678us RTTd: 8226958us Loss: 22%)   
2021-08-13T13:47:57   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 6801678us stddev 8226958us loss 22%   
2021-08-13T13:47:34   dpinger[15487]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 5102817us RTTd: 7708468us Loss: 0%)   
2021-08-13T13:47:34   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 5102817us stddev 7708468us loss 0%   
2021-08-13T13:47:23   dpinger[76094]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 5186909us RTTd: 7746197us Loss: 0%)   
2021-08-13T13:47:23   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 5186909us stddev 7746197us loss 0%   
2021-08-13T13:47:11   dpinger[11173]   GATEWAY ALARM: WAN_DHCP (Addr: 96.34.137.232 Alarm: 1 RTT: 9641us RTTd: 3019us Loss: 22%)   
2021-08-13T13:47:11   dpinger[24815]   WAN_DHCP 96.34.137.232: Alarm latency 9641us stddev 3019us loss 22%

I suppose my next call should be to Spectrum?  Should I attempt to get a new modem?

Thanks for your assistance.

[opnfwb]

Only Spectrum can tell you if you'll need a new modem.

I think you need to get more data. Enable the options that I mentioned in the first post, and also double check your bandwidth as shown in the screenshot below. Once you've ensured that the latency spikes aren't coinciding with bandwidth spikes, the next step is to determine what the ISP says.

[petro49er]

Ok I have enabled the settings you recommended.  I had it go out again right around 6:11 - 6:15 this evening.  I will attach screenshots.  Bandwidth utilization looks low.  Let me know what you think.

[errored out]

You can ask your ISP to test your connection. It's a simple test to check out the quality of your connection.  Also ask them what the results are such as attenuation, and signal to noise ratio.    It's possible something on your line is cause this issue especially since you are sharing it with your neighbors. 

Also, you stated you replaced the hardware.  Which exactly, the computer running your FW; could be your modem has failed / not operating correctly.

[petro49er]

The modem was replaced not long ago as the previous one failed completely. That was within the past month. We had these same issues with that other modem as well so I assumed it was the hardware for the firewall at that point. It’s currently running on an Optiex 7020 with an i7 4790/8GB/128GB SSD with an Intel 4 port NIC. The NIC was pulled from the old hardware so I suppose it could be that. I tried reassigning interfaces but have the same results no matter what interfaces I utilize.

[errored out]

It is possible you received a replacement modem which is bad.  Highly unlikely, although not unheard of.

Something could be radiating causing interference against the cable, or it could be the cable itself.    I'm still thinking its the coaxial cable.  You can ask Spectrum to go out and conduct a line test.  If the cables were not properly terminated, that would cause this issue.  They shouldn't charge for the tech to go out and check. 

Have you tried replacing the Ethernet cable?  Unlikely, but it could be the cable itself.   Perhaps some interference if it's not shielded?

[petro49er]

Spectrum seems to think it’s because both modems before were DOCSIS 3.0 modems. Today they exchanged it for a DOCSIS 3.1 modem. I have had a monitor set up on both the internal and external gateways and the only one that ever experienced any latency is the one pointed outside. Fingers crossed that this fixes it.

Thanks for your assistance in this.

[errored out]

Since some time has passed, has the situation improved?  If fix, can you change the subject to resolved?

[petro49er]

Yes it seems to have fixed my issue. Thank you for your help in troubleshooting this issue.