Quality Problems

[stefks]

Hi, I have some quality problems, my Quality chart looks like the attached. Also, my RTT and RTTd values start increasing over time. What solves it is restarting the ATT fiber modem. What is the root cause? How do I diagnose the issue? Thank you!

[stefks]

I chatted to AT&T support agent a few days ago, they said they fixed something on their end, but problem persists. I had to again manually restart my modem today. Might try to call the ISP again. But is there a way to figure out what's going on from OPNsense logs? Could my OPNsense router be misconfigured in some way? Thanks!

I don't believe this is linked to updating OPNsense. I updated on 2024-06-23 (to 24.1.9), and the problems started on 2024-07-02. Next time I updated OPNsense (to 24.1.10) was today.

[axsdenied]

Very much looks like an AT&T problem not an opnsense problem.  I've had similar issues with Cox Cable.

[Seimus]

This graph represents the GW probing, usually by default its the 1st HOP after your OPNsense e.g. 1st ISP device in the path. If you see loss or latency this indicates issue on the cable between OPNsense and ISP device or the ISP device itself.

Regards,
S.

[stefks]

Replaced my ISP device, AT&T sent me a new modem, problem still persists. Delays are starting to grow again. Maybe it's really just the Ethernet cable between my router and modem??? That would be so weird. I'll try replacing that next. Maybe I'll also then try to look at the modem logs, but that will require disabling IP passthrough... ugh. Any other things I should try?

[opnfwb]

What IP is OPNsense selecting for the WAN gateway monitor? Is it directly on the ATT provided gateway, or is this an IP further upstream on their network?

If it's an IP upstream, can you duplicate the higher response times when you manually ping the same IP from a client on your LAN?

This may be a case of ATT starting to de-prioritize repetitive ping traffic over a period of time from the same device on their network.

Two other things I would try:
Go to System/Gateways/Configuration and click the edit button for your WAN DHCP gateway.

In there set your data length to a small number, usually 3 to 5. This will assign a byte value to your ICMP packets and some ISPs will handle them differently after you do this and therefore, you may see more consistent gateway monitoring.

On the same configuration page, you can also try to change the monitor IP to a known stable WAN IP (4.2.2.2, 8.8.8.8, 1.1.1.1, 9.9.9.9, etc.). This will be useful if your ISP is slowly throttling your pings to their own internal gateway over a period of time. Using an external monitor IP will also tell you if the ping replies continue to increase, the issue lies somewhere within the ISP side and is impacting all your traffic, not just the 1st hop pings.

[stefks]

- What IP is OPNsense selecting for the WAN gateway monitor? Is it directly on the ATT provided gateway, or is this an IP further upstream on their network?

It's not selecting anything, I chose this one: 8.8.4.4 (this is google DNS I believe)

- If it's an IP upstream, can you duplicate the higher response times when you manually ping the same IP from a client on your LAN?

yes

- In there set your data length to a small number, usually 3 to 5.
Ok, changed to 4

- change the monitor IP to a known stable WAN IP
Ok, changed to 8.8.8.8


Rebooting router and opnsense, and seeing what will happen now...

[stefks]

Delays and packet losses are accumulating again, even after those changes.

By the way, very interesting pattern:

WAN_DHCP6 - *only* packet loss becomes apparent, not delay(!). With WAN_DHCP both packet loss and delay start to grow after some time. The ipv6 address for monitor IP I was using for WAN_DHCP6 was 2001:4860:4860::8888, also google's public DNS.

I will try one more thing: remove the monitor IPs from the configuration, and let OPNsense figure out a monitor IP by itself. Rebooting modem now...

[opnfwb]

Each ISP is different however, I've found the default gateway WAN_DHCP6 monitoring to be really flaky. In my case it just uses an fe80 address and the latency and packet loss are all over the place, to the point of making the interface appear "down" due to packet loss and causing issues.

For my WAN_DHCP6 I've always had to use a separate, external monitor IP. That stays super consistent and has essentially zero packet loss. In my case, I'm using Quad9 for my WAN_DHCP6 monitor (2620:fe::9).

For my WAN_DHCP monitor I'm just using the default value that OPNsense discovers when it requests an IP. Again in my case, its worked well for years.

I have the byte value for both of my monitors at 3.

[stefks]

Still same problem, with default gateway monitoring. Latency and loss increase exponentially over time. Gonna change the re-try timing to every 10 seconds and reboot gateway now...

[stefks]

Completely disabled monitoring, still same problem persists.

Here is today, before and after restarting the modem.

Will look at the logs in the AT&T modem now. Also will try to selectively disable a few things I'm running on my network. Maybe AT&T doesn't like certain kinds of traffic.

[stefks]

Completely bypassed my bgw320-500, now everything works well. It was a problem with the gateway.