Messages

Whew.  Well my fellow internetians, I believe the problem was solved with the ONT replacement the tech did today.  Not even going to mention the flack I got just to get it done.  But preliminary testing is phenomenal (fingers crossed).  I'm connected via the Opnsense router machine now into the new ONT, and the performance is night and day so far, with NO decayed speed state.  The cause had to have been either a failing ONT, or that model isn't performant to modern network demands.  What a journey this has been.

I am going to give it a few more days, before I call it for myself.  But I will go ahead and mark this as solved.  Thank you all so much, and cheers.

Now to return all the shit I bought when troubleshooting back to Amazon... -_-

Sigh.  So the ISP, after denying that the ONT included logs, sent me this in our conversation:



Full resolution: https://i.postimg.cc/pxVcqhYW/unnamed.png

I'm assuming these are the ONT logs right?  Does anyone know what kind of system this is?  From what I can see, it is showing nonzero Downstream and Upstream Bip error counts.  This isn't normal correct?  Does this confirm errors at the ONT?  Does this point to any particular issue?  Any and all feedback much appreciated.  Fighting an uphill battle here.

I know this isn't the place for this conversation at this point, but I greatly appreciate the help in troubleshooting assistance that you all have offered. 

Thank you, I will give that a go.  But they are literally giving me pushback trying to say that it could still be something "local."  I even just recorded a video capturing an instance of one machine displaying the behavior when connected directly to the ONT, and another instance in the same video of two machines connected directly to the ISP router displaying the same behavior.  That has to completely rule out anything in the home right?

Okay, sorry to post again.  But the ISP just told me that the ONT does NOT have logs.  AI LLM says otherwise.  I am unable to find that information for myself.  Does anyone know for sure whether or not the Calix GigaPoint 803G Model No: 100-04255 11 should have logs accessible by the ISP?  Thanks.

Sent a detailed email to my ISP explaining the situation and telling them that I observed the behavior when directly connected to the ONT, and they literally responded with an email only saying "it's a poor idea to run directly connected to an ONT."  -_-

Hi guys, back with what will probably be the final update here.  So I did what I should have done from the start and tested with the computer directly connected to the ONT.  I observed the slowdown here as well.  So it's definitely a problem with the ONT, or upstream.  Thanks for all the help.  I guess I can mark this one as solved since we've ruled out Opnsense entirely.  I will post another update when or if the problem gets resolved.  Take care.

If anyone has had a similar issue before, or any advice with dealing with this situation moving forward feel free to share.  Cheers.

Well I have two more frustrating updates:

1. I observed the degraded speed state on the ISP provided router while playing Overwatch.  It is much rarer on the ISP router, and it definitely resolves itself much quicker.  It has never caused disconnects unlike 3rd party routers, but I did observe the degraded speed just now.

2. I tried the Protectli Vault FW2B, and wow.  Not good performance.  I couldn't even get it to hit full gig speeds.  It only performed at around half, ~500m down / ~500m up.  Even with that though, I definitely still noticed the decayed state while testing as it decayed to ~500m down / ~50m up.

Whew.  I feel I've exhausted all of my options, save for contacting the ISP directly.  I did call the ISP customer support line, just to ask if we could check the logs from router and modem.  The representative told me I could check those logs by logging into the Calix U6 through the ip address browser portal.  Unfortunately, that's not the case.  These routers are locked down.  Can't shell into them, and the browser UI options are extremely limited.

If anyone has any clue as to what this could be at this point, anything would be much appreciated.  I have no idea what to check next, and I highly doubt the ISP is going to be willing to help much.  Thanks again for all the help already.

[3. [the most important] I decided to retry the double NAT solution, this time observing the speed during the degraded state on a second computer that was connected to the initial primary NAT, the ISP provided router (not in bridge mode), to see what speed it would show. I was expecting it to show a normal speed even when the opnsense machine was showing degraded speed if this were a problem local to the opnsense machine. To my surprise, BOTH client pc's, one connected to the opnsense machine AND the one connected to the ISP provided router showed the decayed speed state. Does this show that the problem then is being generated by the opnsense machine (since it never happens with just the ISP provided router alone) and is affecting my modem handshake / connection to my ISP?]

Well, a few interesting updates with one I believe is very important:

1. On the parameter suggestions, opnsense logs, pcie lane status, and PPS everything was nominal both before and during the decayed speed state.  No unexpected messages, pcie lane status checked out the whole time, etc.  It was even harder this time to catch the link flaps for some reason.  I can post the command line results if needed, but none of it adds new information to the situation.  If there are any other metrics or parameters I should check, please advise.

2. The opnsense machine didn't even recognize the realtek nic that I tried to use with it, so with that combined with how abysmal everyone says the realtek drivers perform, I decided not to pursue that any further.  Unless it's recommended that I do for testing sake.  Just gonna return it.

3. [the most important] I decided to retry the double NAT solution, this time observing the speed during the degraded state on a second computer that was connected to the initial primary NAT, the ISP provided router (not in bridge mode), to see what speed it would show.  I was expecting it to show a normal speed even when the opnsense machine was showing degraded speed if this were a problem local to the opnsense machine.  To my surprise, BOTH client pc's, one connected to the opnsense machine AND the one connected to the ISP provided router showed the decayed speed state.  Does this show that the problem then is being generated by the opnsense machine (since it never happens with just the ISP provided router alone) and is affecting my modem handshake / connection to my ISP? 

To be clear, the setup here was as follows:

client PC #1 -> [Opnsense LAN port] -> Opnsense machine -> [Opensense WAN port] -> ISP provided router (client PC #2 connected via LAN port) -> ISP provided modem

May be important to note here that this behavior persists across opnsense and pfsense as well.

4. I have ordered a Protectli device just to hopefully rule out the PCIE bus / PCIE NIC cards once and for all.

Hopefully, any of this helps add new information.  I feel like the next step is to reach out to my ISP, but I have no idea where to even begin the conversation there, or what to speculate could be the problem.  They've been receptive enough so far with helping out, but I have a feeling they might put this request on the back burner.  They've been pretty shy about providing support for 3rd party router solutions.  Thanks again all.

Thank you.  Unfortunately I did disable flow control, ASPM, and tried other tunables during the first round of troubleshooting.  Nothing has helped so far.  Hopefully, I will have some new data to report momentarily.

What I tried before:

For flow control:
dev.igb.0.fc=0
dev.igb.1.fc=0
dev.igb.2.fc=0
dev.igb.3.fc=0

For link flaps:
dev.igb.*.iflib.rx_budget=-1 (and tx_budget equivalent)
hw.igb.num_queues=0 (auto queue configuration)
Various per-port versions (e.g., dev.igb.0.iflib.rx_budget=-1)

Have you ever had these issues like this on a desktop machine running a pcie nic before? 

[Observed Pause Frames & Link Flaps]

Thank you all.  Okay, I will check Opnsense logs, as well as various commands in regards to PCIe lane status before and during the speed state, and PPS and report back with hopefully some relevant data.

One thing to note is that during the testing that I did do with the magical LLM guiding me through, we did observe pause frames and link flaps.  A quick summary of that:


----

Observed Pause Frames & Link Flaps

Pause Frames (xoff_recvd):

Non-zero pause frame counters observed on igb ports during degraded state (e.g., dev.igb.1.mac_stats.xoff_recvd: 3 or similar values).
Indicates flow control is being triggered, suggesting transmit-side queue pressure or bufferbloat-like behavior on the NIC.

Link Flaps:

Repeated igbX: link state changed to DOWN followed immediately by UP events in dmesg during high-load periods (especially when Overwatch is running).
These flaps coincide with the gradual speed decay (~940 → ~700 Mbps down / ~70 Mbps up).
The link eventually recovers after some time or a reboot.

Additional Context:

No obvious errors in netstat -I counters.
Behavior is consistent across multiple motherboards (Lenovo m720q and Dell OptiPlex 7070), OSes (OPNsense and pfSense), and Intel igb cards (including genuine Dell I350-T4).
Works fine when using the ISP-provided Calix U6 router directly.

This pattern strongly suggests a compatibility issue between Intel igb NICs and the ISP link under sustained high-pps UDP traffic.

This occurs on multiple fresh installations of both opnsense and pfsense.  In regards to my network, unfortunately the issue occurs when the client pc is connected directly to the LAN port on the opnsense machine, and with the WAN port directly connected to the modem.

I think the only thing left to check is the BIOS/UEFI of the two Mini PCs and make sure the system does not do some kind of "Dynamic PCIe Lanes Speed Switching" during operation that might have this strange outcome ?!

In case you are using a PCIe Riser that's not original/the same brand as the Mini PC you might want to look into that too!

Thank you again for your suggestions.  Unfortunately, I have checked all that I know in regards to bios power / PCIE settings on these devices.  Their options there are pretty limited.  In regards to the PCIE riser on the Lenovo m720q, I replaced it three times and eventually made the move to the Dell Optiplex 7070 SFF which doesn't require a riser thankfully.  I believe that should rule that out unless I am mistaken.  I am still new to all of this.

What is strange to me though, is that from my research, both of these systems were recommended for this sort of thing.  Yet, I can't really find much about an issue like this.

Ordered a realtek chipset nic to tryout different drivers

That's a waste of time for sure!

Any Intel NIC should perform a lot better!

I am guessing something gets flooded somehow (A game so I guess UDP packets/traffic ?!) but I don't know what exactly...
Have you changed any settings or applied any tuneables that are non-default and influence the traffic ?
And how does the rest of your network look like ?

Thank you.  Unfortunately, no I haven't changed any default settings.  This occurs on multiple fresh installations of both opnsense and pfsense.  In regards to my network, unfortunately the issue occurs when the client pc is connected directly to the LAN port on the opnsense machine, and with the WAN port directly connected to the modem.  And the issue occurs on multiple client systems.

[TL;DR: ONT failure]

TL;DR: ONT failure

----

At a glance:

Internet connection speed:   1gig, usual speeds 940 mbps down / 940 mbps up
Temporary slowed speed state:   ~700 mbps down / ~50 mbps up , (all pings/ latency stay generally consistent no degredation)
Triggers: Multiple, the one I found for testing was playing the video game Overwatch while doing an Ookla speed test in a web browser
Temporary fixes:   Rebooting the router usually leads to a normal speed state most of the time, will revert eventually to the degraded speed state, which will eventually revert back to full speed overtime itself oscillating back and forth.

Degraded speed state DOES NOT occur on the ISP provided router solution, Calix U6 router.
UPDATE #2: It does occur on the ISP provided router.  It handles it much more gracefully than the third party routers though.  The third party routers will sometimes disconnect from Overwatch services, where the ISP router hasn't yet.

When this happens (on the 3rd party routers) it CAN also cause disconnects from the Overwatch servers during the degraded speed state, but other services remain connected as does the internet connection itself.

----

Hardware:

Systems:
Lenovo ThinkCentre m720q tiny (original system)
Dell OptiPlex 7070 SFF (second system)

NICS:
ULANSeN Dual-Port PCIe card (Intel 82575/82576 chipset)
NICGIGA 4-port i350 card (Amazon white-label)
Genuine Dell I350-T4 (4-port, used in both m720q and 7070)

Network config:
Client PC -> [LAN port] -> Opnsense machine -> [WAN port] -> modem

----

Timeline of troubleshooting steps taken:

(on Lenovo m720q tiny)
Multiple fresh installs of latest Opnsense
Disabled hardware offloading
Set tuneables
Observed pause frames and link flaps
Replaced original Ulansen 82575/82576 card with NICGIGA i350 4-port card
Tried multiple third-party PCIe risers
Isolated WAN/LAN port assignments on the multi-port cards
Removed internal wi-fi card
Scoured every possible BIOS setting in regards to PCIE slots and power management
Implemented FQ-CoDel traffic shaping
Isolated all network components down to PC -> m720q OPNSENSE router
Sourced genuine Dell I350-T4 with proper m720q low profile bracket
Checked ethernet cables

(on Dell Optiplex 7070 SFF)
Migrated to Dell Optiplex 7070 SFF with native PCIe slot
Retried all tunables
Installed and tested pfsense
Tested double NAT (7070 behind ISP router)

----

Synopsis:

This ghost persists across different hardware, OS, and configurations.  I believe that I have isolated the issue either down to the PCIE bus / NIC itself, the Intel chipset igb drivers, and/or Freebsd itself.  I am an amateur, and this is for a home network setup.  If I'm missing anything glaring, please excuse.  This has been my last resort to come to the forums for this issue.  I'm sick of talking to an LLM.  I'm sure this is usually a place for enterprise issues, but thank you in advance.  You're my only hope lol.  Any and all suggestions appreciated.  Has anyone else ever had any issues like this when using a desktop PC as an opnsense machine?  Is it recommended that I move on to dedicated hardware? 

----

Next steps:

1. Ordered a realtek chipset nic to tryout different drivers
2. WIll move to Protectli hardware next with soldered nic instead of pcie nic.
3. Contact ISP directly

----

UPDATE #1:

Re-performed the double NAT test.  This time, checking the speed at both routers.  To my surprise, the decayed speed is observed on BOTH routers, the ISP provided router in front connected to the modem, and the Opnsense machine connected to the ISP provided router.  Does this indicate that the problem is generated by the Opnsense machine, and then affects my modem handshake / ISP connection?  Could this be a problem on their end?

To be clear, the setup here was as follows:

client PC #1 -> [Opnsense LAN port] -> Opnsense machine -> [Opensense WAN port] -> ISP provided router (client PC #2 connected via LAN port) -> ISP provided modem

----

UPDATE #3:

I observed the degraded speed state with the PC directly connected to the ISP ONT.  Since this rules out Opnsense and the other hardware entirely, I will mark this solved.  Sorry to take up eveyrone's time, and thank you for the help.

----

FINAL UPDATE:

It would appear that an ONT upgrade has solved the problem.  It had to be either a failing ONT or a model that is out of spec.  Either way, may I finally be able to lay this to rest, and Godspeed to the next poor soul who has to walk this path.  /salute

----

CONCLUSION:

For anyone in the future: aside from logical troubleshooting and process of elimination as well as all of the help that you guys offered here, two of the things that pointed the strongest to ONT failure were

1. testing with the pc directly connected to the ONT and observing the degraded speed state and
2. power cycling the ONT to resolve the speed state when it occurred, which turned out to be a guaranteed and immediate fix every time (compared to router reboot or command prompt dhcp lease renewal)

I shouldn't have stopped at the router level when isolating the issue initially.  The thought process was that since there were so many more points of failure on an Opnsense machine, that I had to have made a mistake in my hardware or software configuration there (To be fair, the ISP router did handle the ONT component failure more gracefully and made observation of the issue when the ISP router was in use more difficult, which in turn led me more toward the idea that it was user error on my part in my Opnsense machine configuration).  Had I done these things in the beginning, that would have saved me a lot of time and trouble especially since these two tests demanded much less time and energy than going down the fractal of troubleshooting my potentially flawed Opnsense configuration. 

This was such a brutal process, because every single granular change required me to test a video game for thirty minutes to an hour just to trigger the decayed speed state as I was going down a self induced wild goose chase for weeks, JUST to be able to prove to my ISP that I had ruled out any component in the home save for the ONT or anything upstream, JUST for them to then give me pushback to even replace the damn component.  Thank you all so much and cheers.