Degraded Speed Ghost

[juicemain]

----

At a glance:

Internet connection speed:   1gig, usual speeds 940 mbps down / 940 mbps up
Temporary slowed speed state:   ~700 mbps down / ~50 mbps up , (all pings/ latency stay generally consistent no degredation)
Triggers: Multiple, the one I found for testing was playing the video game Overwatch while doing an Ookla speed test in a web browser
Temporary fixes:   Rebooting the router usually leads to a normal speed state most of the time, will revert eventually to the degraded speed state, which will eventually revert back to full speed overtime itself oscillating back and forth.

Degraded speed state DOES NOT occur on the ISP provided router solution, Calix U6 router.

When this happens it CAN also cause disconnects from the Overwatch servers during the degraded speed state, but other services remain connected.

----

Hardware:

Systems:
Lenovo ThinkCentre m720q tiny (original system)
Dell OptiPlex 7070 SFF (second system)

NICS:
ULANSeN Dual-Port PCIe card (Intel 82575/82576 chipset)
NICGIGA 4-port i350 card (Amazon white-label)
Genuine Dell I350-T4 (4-port, used in both m720q and 7070)

Network config:
Client PC -> [LAN port] -> Opnsense machine -> [WAN port] -> modem

----

Timeline of troubleshooting steps taken:

(on Lenovo m720q tiny)
Multiple fresh installs of latest Opnsense
Disabled hardware offloading
Set tuneables
Observed pause frames and link flaps
Replaced original Ulansen 82575/82576 card with NICGIGA i350 4-port card
Tried multiple third-party PCIe risers
Isolated WAN/LAN port assignments on the multi-port cards
Removed internal wi-fi card
Scoured every possible BIOS setting in regards to PCIE slots and power management
Implemented FQ-CoDel traffic shaping
Isolated all network components down to PC -> m720q OPNSENSE router
Sourced genuine Dell I350-T4 with proper m720q low profile bracket
Checked ethernet cables

(on Dell Optiplex 7070 SFF)
Migrated to Dell Optiplex 7070 SFF with native PCIe slot
Retried all tunables
Installed and tested pfsense
Tested double NAT (7070 behind ISP router)

----

Synopsis:

This ghost persists across different hardware, OS, and configurations.  I believe that I have isolated the issue either down to the PCIE bus / NIC itself, the Intel chipset igb drivers, and/or Freebsd itself.  I am an amateur, and this is for a home network setup.  If I'm missing anything glaring, please excuse.  This has been my last resort to come to the forums for this issue.  I'm sick of talking to an LLM.  I'm sure this is usually a place for enterprise issues, but thank you in advance.  You're my only hope lol.  Any and all suggestions appreciated.  Has anyone else ever had any issues like this when using a desktop PC as an opnsense machine?  Is it recommended that I move on to dedicated hardware? 

----

Next steps:

1. Ordered a realtek chipset nic to tryout different drivers
2. WIll move to Protectli hardware next with soldered nic instead of pcie nic.

[nero355]

Ordered a realtek chipset nic to tryout different drivers

That's a waste of time for sure!

Any Intel NIC should perform a lot better!

I am guessing something gets flooded somehow (A game so I guess UDP packets/traffic ?!) but I don't know what exactly...
Have you changed any settings or applied any tuneables that are non-default and influence the traffic ?
And how does the rest of your network look like ?

[juicemain]

Ordered a realtek chipset nic to tryout different drivers

That's a waste of time for sure!

Any Intel NIC should perform a lot better!

I am guessing something gets flooded somehow (A game so I guess UDP packets/traffic ?!) but I don't know what exactly...
Have you changed any settings or applied any tuneables that are non-default and influence the traffic ?
And how does the rest of your network look like ?

Thank you.  Unfortunately, no I haven't changed any default settings.  This occurs on multiple fresh installations of both opnsense and pfsense.  In regards to my network, unfortunately the issue occurs when the client pc is connected directly to the LAN port on the opnsense machine, and with the WAN port directly connected to the modem.  And the issue occurs on multiple client systems.

[nero355]

This occurs on multiple fresh installations of both opnsense and pfsense.  In regards to my network, unfortunately the issue occurs when the client pc is connected directly to the LAN port on the opnsense machine, and with the WAN port directly connected to the modem.

I think the only thing left to check is the BIOS/UEFI of the two Mini PCs and make sure the system does not do some kind of "Dynamic PCIe Lanes Speed Switching" during operation that might have this strange outcome ?!

In case you are using a PCIe Riser that's not original/the same brand as the Mini PC you might want to look into that too!

[juicemain]

This occurs on multiple fresh installations of both opnsense and pfsense.  In regards to my network, unfortunately the issue occurs when the client pc is connected directly to the LAN port on the opnsense machine, and with the WAN port directly connected to the modem.

I think the only thing left to check is the BIOS/UEFI of the two Mini PCs and make sure the system does not do some kind of "Dynamic PCIe Lanes Speed Switching" during operation that might have this strange outcome ?!

In case you are using a PCIe Riser that's not original/the same brand as the Mini PC you might want to look into that too!

Thank you again for your suggestions.  Unfortunately, I have checked all that I know in regards to bios power / PCIE settings on these devices.  Their options there are pretty limited.  In regards to the PCIE riser on the Lenovo m720q, I replaced it three times and eventually made the move to the Dell Optiplex 7070 SFF which doesn't require a riser thankfully.  I believe that should rule that out unless I am mistaken.  I am still new to all of this.

What is strange to me though, is that from my research, both of these systems were recommended for this sort of thing.  Yet, I can't really find much about an issue like this.

[pfry]

[...]This occurs on multiple fresh installations of both opnsense and pfsense.[...]

Any unusual log messages? From any of the OPNsense logs.

[...]"Dynamic PCIe Lanes Speed Switching" during operation[...]

Easy enough to check while in operation (e.g. "pciconf -lcv [device]", where device is the network driver, e.g. "em0"). But...?

[Seimus]

Triggers:  Multiple, the one I found for testing was playing the video game Overwatch while doing an Ookla speed test in a web browser

What is your PPS (packet per second) when you see the degradation?
Cause this potentially indicates that your HW can not cope with amount of packets sent thru the system.

Regards,
S.

[juicemain]

Thank you all.  Okay, I will check Opnsense logs, as well as various commands in regards to PCIe lane status before and during the speed state, and PPS and report back with hopefully some relevant data.

One thing to note is that during the testing that I did do with the magical LLM guiding me through, we did observe pause frames and link flaps.  A quick summary of that:


----

Observed Pause Frames & Link Flaps

Pause Frames (xoff_recvd):

Non-zero pause frame counters observed on igb ports during degraded state (e.g., dev.igb.1.mac_stats.xoff_recvd: 3 or similar values).
Indicates flow control is being triggered, suggesting transmit-side queue pressure or bufferbloat-like behavior on the NIC.

Link Flaps:

Repeated igbX: link state changed to DOWN followed immediately by UP events in dmesg during high-load periods (especially when Overwatch is running).
These flaps coincide with the gradual speed decay (~940 → ~700 Mbps down / ~70 Mbps up).
The link eventually recovers after some time or a reboot.

Additional Context:

No obvious errors in netstat -I counters.
Behavior is consistent across multiple motherboards (Lenovo m720q and Dell OptiPlex 7070), OSes (OPNsense and pfSense), and Intel igb cards (including genuine Dell I350-T4).
Works fine when using the ISP-provided Calix U6 router directly.

This pattern strongly suggests a compatibility issue between Intel igb NICs and the ISP link under sustained high-pps UDP traffic.

[Seimus]

RX Pause frames is part of the flow control suite,
are an indication that the other side (device connected on this port) is not capable to keep up.
If this happens device that receives the Pause Frames, if supported and enable will throttle down effectively decreasing pps.
Flow control is a suite that throttles the connection in order to not overwhelm the receiver or sender and prevent possible packet drops.

It is advised to disable Flow control between network devices, such as FW, router, Switches etc.

Link Flaps
So if you see flapping during high load, be it throughput or pps this is not good, could be caused by ASPM.

Try to disable it.

Regards,
S.

[juicemain]

Thank you.  Unfortunately I did disable flow control, ASPM, and tried other tunables during the first round of troubleshooting.  Nothing has helped so far.  Hopefully, I will have some new data to report momentarily.

What I tried before:

For flow control:
dev.igb.0.fc=0
dev.igb.1.fc=0
dev.igb.2.fc=0
dev.igb.3.fc=0

For link flaps:
dev.igb.*.iflib.rx_budget=-1 (and tx_budget equivalent)
hw.igb.num_queues=0 (auto queue configuration)
Various per-port versions (e.g., dev.igb.0.iflib.rx_budget=-1)

Have you ever had these issues like this on a desktop machine running a pcie nic before? 

[pfry]

[...]Okay, I will check Opnsense logs, as well as various commands in regards to PCIe lane status before and during the speed state[...]

Before is good. "During" should show exactly what "before" showed. I would expect a full device reset to change link parameters.

The pause frames are odd. Do you have the sender MAC? Is it the expected value (provider side of the link)?

The link flaps are also odd. That would seem to indicate a device/driver issue, but multiple issues with multiple loci seems unlikely. Likely useless test: place a switch between the OPNsense device and upstream (L2 only).

[Seimus]

The Pause frames received are just 3 if this is pasted from the latest visible issue

dev.igb.1.mac_stats.xoff_recvd: 3

In case the peer cant keep up it would be a constant flood of Pauses.

Regards,
S.