[SOLVED] Regular packet loss / latency spikes of ~30 seconds

[crayon]

Hi Forum,

I'm trying to debug a curious problem that's been plaguing us for a while now (first time reported to me October 2020). We're seeing regular packet loss / latency spikes for ~30 seconds, which is particularly disrupting during video calls.

Our setup:

PC Engines APU2D4 (co-located) currently running OPNsense 20.7.8 as router/firewall. The system has been in place for 2 years, and we've only seen issues recently.

Interface 1 (if1): WAN
Interface 2 (if2): carries 3 VLANs (our internal network) to our offices in another building
Interface 3 (if3): DMZ

Ifs 1 and 2 are connected to the server room switch, if3 is connected to two servers through our own small switch.

We only observe the issues on if1 and if2, never on if3.

I'm monitoring all three interfaces from OPNsense through monit ping tests to the next available system (upstream router on if1, our own switch on if2, both servers on if3).

Because we only see issues on if1 and if2, e.g. the ones connected to the server room network equipment, I thought the problem could be there. While I'm still monitoring that, I think it can increasingly rule out this possibility.

Things I've (unsuccessfully) tried on the router:
- downgrade to 20.7.4 (the version preceding the first problem report)
- disable flow control (dev.igb.X.fc=0)
- disable TSO (net.inet.tcp.tso=0)

I don't see any suspicious messages in the logs.

My gut feeling is that this may be an issue with the new iflib-ified igb(4), but I'm puzzled that we don't see these issues on if3.

I've though about downgrading to OPNsense 20.1.9 as a test; would that be possible from a 20.7.8-based system, or result in issues with the configuration?

Thanks for reading, if anyone has an idea I'm all ears.

Thanks!

[thowe]

If it is some kind of incompatibility between the firewall and the central switch, interposing a small switch might help for a test. This might solve the problem by "normalizing" the communication. It would be worth a try.

[crayon]

Good suggestion, I'll give it a try, thanks. Updating to 21.1 did not help in any case.

[linuxmail]

hi,

can you tell me, what for a network card you use ?

Because: https://forum.opnsense.org/index.php?topic=21663.0

[crayon]

hi,

can you tell me, what for a network card you use ?

Because: https://forum.opnsense.org/index.php?topic=21663.0

NIC in use: Intel i210AT. The problems I have sound different from the ones you describe.

I have since updated to OPNsense 21.1.2.

The issue on the WAN connection appears to have been fixed by moving to a different switch port. The issues on the LAN port persist though; there have been fixes related to iflib/igb/VLAN in FreeBSD, so I'm trying to figure out whether any of them may help.

[crayon]

Just a not that we eventually figured this problem out. It was due to an overheating fanless switch; after replacing it the issues went away.

[thowe]

Thanks for the update! 

If you wish, you could add [SOLVED] to the title then.