Intermittent high ping only to gateway (opnsense)

[db]

Hi folks, I know this probably isn't an opnsense issue but thought there may be someone with ideas reading these forums anyway .

I've been tearing my hair out with a difficult to diagnose network issue which manifests as intermittent high ping to the gateway (which is opnsense), only from one host. This will occur for seemingly random amounts of time, maybe only seconds, or maybe 20+ minutes, a few times every day. Packet captures on that host reveal nothing that seems very interesting, other than a lot of tcp retries/spurious retransmits etc which I suspect are symptoms and not cause. The host is unusable from a network standpoint when this happens (no pages load, etc).

What's odd is that when this happens, this same host can ping any other host on the network without issue, including others on the same switch. The opnsense box can also ping any other host just fine, and they can ping it. Pinging from the opnsense box to the affected host is also slow. Packet captures on the opnsense box reveal that it receives pings and replies immediately (~0.1ms), on the affected host the captures show pings and replies being far apart (10-250ms).

The opnsense box is connected to a zyxel switch, the affected host is connected to a mikrotik switch which is connected to that same zyxel switch. Nothing on either switch seems interesting either, no massive error counts or anything like that.

I'm led to believe the issue lies with the host itself and not the network, but I have no idea what... It's connected with an Asus XG-C100C adapter, I found updated drivers for this and installed them but hasn't changed the problem.

Anyone have any ideas? Things I might try, other things to look for?

[db]

Update, disabled ethernet on the host, connected via Wifi instead (so different adapter, driver, etc). Same behavior! I'm very confused. Maybe I was mistaken and this does somehow have something to do with opnsense? Completely odd to me that the high/bad ping is only between the host and opnsense, and nothing else. I seem to be able to fix it by momentarily unplugging the switch and plugging it back in. (On wifi, the host is connected to an AP which is connected to the same switch it was in before).

I've also tried swapping out the switch for another, same issue.

[axsdenied]

Whats the logical network look like? Any VLANS? Any way to start eliminating components in between?  Are you able to plug the host into the switch zyxel switch?

[WN1X]

I think you need to figure out what is common when connected with wifi and ethernet. It sounds like a bad cable somewhere...perhaps the connection between your switches.

[db]

No VLANs, it's a flat network. It's too far to plug the host directly into the zyxel.

Bad cable is something I'm starting to suspect, as the link for this switch and the zyxel is only 5Gbe. Is it possible that a bad cable would cause issues that will only show up between two hosts? What's odd to me is that the bad ping is only between the host and the gateway, but the host can ping other hosts on the network without issue. Also, it will resolve itself eventually, last two captures I have it resolved in ~20M and ~45M.

[ctr]

Have you checked your outbound connections while you observe the issue?
This symptom sounds familiar when running into the problem described here: https://forum.opnsense.org/index.php?topic=31431.0

[db]

Interesting, I haven't looked at that. I did learn that this issue happens on multiple hosts, not just the one, and at the same times. (I just left a ping running every 5s, sure enough, when I notice the issue on one host it's happening to _some_ of the others, but not all...).

Frustrated, I ran another cable along the top of the walls between the switches, thought I might have resolved it as the ping test was good all night but encountered the same issue this morning. Noticed the zyxel had broadcast storm control enabled, though logs don't seem to indicate that was being enforced I turned it off anyway, because I have no idea what else to do.

Seeing as it's affecting multiple hosts now however, I'm back to wondering if there might be something happening with opnsense here. I don't know what but, it's the only point of commonality that I haven't switched out. I can't log into the opnsense box directly from an affected host while this is happening, but I can ssh into a neighboring host which can then ssh into the opnsense box (?), and everything seems fine, 99% idle, no single core pegged or anything like that, tons of memory free. Pinging from opnsense to any non-affected host is fine.

The affected hosts effectively have no internet (as they can't seem to communicate with opnsense). Ping still 'works' between affected hosts and opnsense though it's quite a bit more latent than normal. Ping between other hosts and affected hosts seems normal.

Maybe something quirky happening with the firewall? That would only occur on random occasion and self-resolve? I know I'm grasping at straws here..

[db]

Update, I seem to have discovered what was going on, I had a wifi extender device (TP-Link RE605X) I was using to bridge an ethernet-only device into the wifi network, but it seems to have some crazy behavior of announcing as owning the IPs of basically every other device on the wifi network, including the gateway... for example:

12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.1.1
12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.102.1
12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.0.1
12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.3.1
12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.4.1
12/18/2022 10:17:37 AM    Flip flop    A6-A2-F4-97-F2-4A takes other's IP: 10.0.10.1
etc...

Not sure why it's trying to proxy arp for the entire world, feeling like it's some sort of bug but not sure, I've posted in their forums but no reply yet. I have a RE600X configured in exactly the same way but it doesn't do this. In any event, what would happen is that when I thought I was pinging the gateway, I was actually going through this extender... and so was the rest of the entire network, probably. Then I suppose the gateway would garp again and I'd be good for a while, rinse/repeat.

Suppose this type of this might be a reason why an arpwatch module for opnsense would be nice. Seems like there is one for pfsense, anyone know of an effort to do the same in opnsense?