I've been meaning to ask about this for some time to satisfy my curiosity. I upgraded to 25.7 and let it run for 24 hours, to see if some recent changes around ICMP (like this one: https://forum.opnsense.org/index.php?topic=45991.0) would possibly affect this, but it hasn't.
Let's call my local network 'Site 1' and with ISP provider 'A'. The ISP is a large cable internet provider with coaxial to the home, terminated at the DOCSIS modem, and then Ethernet to the OPNsense box. This firewall is using dark UI theme.
Let's call a remote network 'Site 2' and with ISP provider 'B'. The ISP is a large mobile telecom provider with fiber to the home, terminated at an exterior ONT, and then Ethernet to the OPNsense box. This firewall is using light UI theme.
Both sites are using UniFi brand switches for LAN clients, just different models. Everything purchased within the last year. All new Cat6 cabling (pre-made). Router specs are as listed in my sig.
---
Site 1 almost never has a perfectly "smooth" GW health graph, at least as compared to Site 2 which is the opposite- it almost never has a "spikey" one. These are examples of the typically observed graphs (1 minute granularity views), although Site 1 is behaving quite well today. It usually has more loss spikes, often times with clusters of short-duration spikes throughout the day.
Site1.png
Site2.png
Only when I zoom in closer on a section without those loss spikes does Site 1 start to resemble Site 2. I consider this the baseline.
Site1-zoomed.png
It's a similar thing on both of the gateways, DHCP and DHCPv6, so doesn't much matter which one I'm looking at. The curious thing though is that the loss spikes in Site 1 don't always line up between the gateways. Sometimes the v4 gateway will have them at different times than the v6 gateway and vice-versa. So, based on that, I think I can rule out exterior cabling (ISP lines) as the culprit, as I would expect both gateways to experience disruptions at the same times if there were an ISP infrastructure issue. (I could be naive about this.)
The other clue is that the losses are typically very short (1 min. duration) and very uniform (vast majority of the time have the same amplitude of just under 1% loss, but never exceeding 20% AFAIK). The uniformity of it does make me wonder if this is a behavior of the router or of the different spec Intel NICs? Maybe even an issue with the ISP provided modem. I'm not sure how to even begin diagnosing.
What do your graphs look like? Are these spikes typical, for those of you with varied experience? Is it normal for some residential ISPs to do this?
TIA!
P.S. - I don't notice the effect of these loss spikes subjectively, but doesn't mean they aren't problematic for connection quality or stream disruptions. I don't have data to prove that.
Adding '1h' granularity views as well with longer data trends. The smaller spikes are drowned out in this view but the overall difference is notable.
Again, not enough to cause any felt disturbance. More of a curiosity.
Site1-1h-granularity.png
Site2-1h-granularity.png