10%-30% packet loss on uploading of data only

[Sisko]

Hello,

I am using a custom appliance (Qotom) /w Intel NICs (2.5G) using an Intel Denverton chipset - Bare Metal install. Never had issues with upload or download packets getting dropped until recently. Just noticed as I started to get worse scores on the bufferbloat test over at Waveform.

Now when I do this:

ping google.com -f -l 1472

I get back 4 responses and usually 1-2 of those 4 don't make it back. The two that make it back are what they should be, about less than 20 ms.

When I play Marvel Rivals or Fortnite and use both of their network diag tools, I can see that only the upload is being affected. My download is rock solid on both. I see levels as low as 1-3%, yet up to 50% a few times.

If I use a software VPN and encapsulate all of the traffic on the PC running those games, then I get almost no packet loss on my uploads.

My first upgrade to v26 was a disaster. Worked for less than 24 hrs then woke up to a failing to boot router. Rebuilt it using the old config, upgraded to v26 and migrated to the new rules.

I'm using this appliance bridged /w x5 2.5G ports, one being WAN plus 4 SPF+ ports of which x2 I use for a 10G/SR Fiber connection back to my Mini PC running Proxmox for a homelab. The router is an Intel 8C/8T CPU, so it has plenty of resources on a 32GB Optane SSD /w 32GB of DDR4 RAM. I ordered 5x 16GB sticks of Optane SSD sticks and this appliance has the ability to RAID x2 of them, so I am planning on a RAID 1 rebuild soon.

I also recently replaced my ISP's cable modem /w a Netgear CM2000 which meets and exceeds the specs of the ISP's modem.

I appreciate any help pointing me in the right direction of what to check for.

TIA

EDIT:

I was tired posting this and tried one more thing before going to bed, I added CoDel to the upload queue (and not just the shaper). Rebooted and it seems to be working now. I played some matches of Fortnite today and didn't see any upload packets drop, whereas yesterday I saw up to 50% get tossed. Calling this a win for now. Saving config and rebuilding this tonight or tomorrow to take advantage of a RAID setup vs the one SSD in this appliance now.

[connervt]

I also recently replaced my ISP's cable modem /w a Netgear CM2000 which meets and exceeds the specs of the ISP's modem.

I would start here.  Plug your modem directly into a known good computer and test.

When troubleshooting, I have always been from the school of "What was the last thing changed?"

[nero355]

I ordered 5x 16GB sticks of Optane SSD sticks and this appliance has the ability to RAID x2 of them, so I am planning on a RAID 1 rebuild soon.

You are probably better off with a Software RAID1/OpenZFS Mirror in order to keep the T.R.I.M. commands working correctly !!

Most Intel SSD's have very decent Garbage Collection performance, but still : Avoid Hardware RAID or any kind of "Chipset RAID" coming from the Motherboard or it's BIOS/UEFI ;)

[OPNenthu]

Good advice to isolate the new modem first.  You also upgraded OPNsense so take them one variable at a time.

Side note: I want to try this myself but the ping command you gave doesn't work in OPNsense.  Both the Linux and FreeBSD versions say they need superuser privilege for '-l 1472' but FreeBSD throws a usage error in any case.  Weird.

ping google.com -f -l 1472

[pfry]

[...]I want to try this myself but the ping command you gave doesn't work in OPNsense.[...]

But it does work in Windows:

Code Select Expand

Microsoft Windows [Version 10.0.19045.4529]
(c) Microsoft Corporation. All rights reserved.

C:\Users\User>ping google.com -f -l 1472

Pinging google.com [142.251.116.102] with 1472 bytes of data:
Reply from 142.251.116.102: bytes=1472 time=19ms TTL=104
Reply from 142.251.116.102: bytes=1472 time=14ms TTL=104
Reply from 142.251.116.102: bytes=1472 time=19ms TTL=104
Reply from 142.251.116.102: bytes=1472 time=19ms TTL=104

Ping statistics for 142.251.116.102:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 14ms, Maximum = 19ms, Average = 17ms

C:\Users\User>


[,,,]I also recently replaced my ISP's cable modem /w a Netgear CM2000 which meets and exceeds the specs of the ISP's modem.[...]

What cable service? More asymmetric ones are going away, but many are still around. Just a data point.

[Patrick M. Hausen]

Windows:

-f - don't fragment
-l - size

FreeBSD:

-D - don't fragement
-s - size

It helps to read the documentation:

Code Select Expand

man ping

HTH,
Patrick

[OPNenthu]

That makes more sense.

I'm not on Windows so didn't realize that -f and -l meant something different, but I should have questioned why I was trying to flood (-f) ping a specific number of preload packets (-l)

;-)

Linux:

-M do - don't fragment
-s - size

I'm seeing times of 30-40ms on my end, so note to self:  maybe time to ditch cable for good.  Not seeing the packet loss though (OPNsense 26.1.2_5).