So my WAN down and up started to tank sometime in december and all I can think is an update caused it. Today I had some time and tweaked the MTU a little with middling to no change:
ersion Date
26.1.5 (installed) 2026-03-24
26.1.4 2026-03-11
26.1.3 2026-03-04
26.1.2 2026-02-12
26.1.1 2026-02-04
26.1 2026-01-28
25.7.11 2026-01-15
25.7.10 2025-12-18
25.7.9 2025-12-04
25.7.8 2025-11-26
25.7.7 2025-11-06
25.7.6 2025-10-22
25.7.5 2025-10-08
25.7.4 2025-09-30
25.7.3 2025-09-09
Timestamp (GMT) Server id Server name Download Upload Latency
2026-03-26T16:40:47 69076 Ripplefiber, Detroit, MI 154.27 119.56 34.63
2026-03-26T16:39:46 53778 123NET, Southfield, MI 321.78 162.60 23.07
2026-03-26T16:39:26 53778 123NET, Southfield, MI 427.84 150.26 18.03
2026-03-26T16:30:47 53778 123NET, Southfield, MI 303.88 158.49 23.59
2026-03-26T16:30:18 69076 Ripplefiber, Detroit, MI 175.65 118.15 35.37
2026-03-26T15:55:38 69076 Ripplefiber, Detroit, MI 282.70 66.56 46.06
2026-03-26T15:54:56 53778 123NET, Southfield, MI 309.22 197.95 22.45
2026-03-26T15:18:56 53778 123NET, Southfield, MI 298.12 178.34 31.09
2026-03-26T15:12:58 15342 Jefferson County Cable, Toronto, OH 532.03 87.85 81.72
2026-03-26T15:09:05 69076 Ripplefiber, Detroit, MI 337.01 80.93 33.71
2026-03-26T15:08:41 69076 Ripplefiber, Detroit, MI 269.19 96.78 36.84
2026-03-26T14:56:52 69076 Ripplefiber, Detroit, MI 147.98 103.90 31.88
2026-03-26T14:56:14 69076 Ripplefiber, Detroit, MI 258.65 82.40 33.43
2026-03-26T14:48:14 69076 Ripplefiber, Detroit, MI 132.60 117.18 39.13
2026-03-26T14:12:39 73050 Omni Fiber, Detroit, MI 135.47 69.68 62.72
2026-03-26T13:59:46 73050 Omni Fiber, Detroit, MI 156.37 80.17 50.09
2026-03-26T13:56:54 69076 Ripplefiber, Detroit, MI 207.24 98.28 45.51
2026-03-26T13:53:49 25491 Cable Co-op, Oberlin, OH 43.46 5.73 1035.91
2026-03-25T11:43:25 16973 Spectrum, Livonia, MI 193.45 98.21 46.64
2026-03-25T11:42:35 74288 EZEE Fiber, Detroit, MI 144.72 96.05 61.52
2026-03-25T11:42:05 5709 Merit Network, Inc, Ann Arbor, MI 288.97 93.82 70.66
2026-03-25T11:41:17 71374 Wyandotte Municipal Services, Wyandotte, MI 310.37 120.34 49.01
2025-12-28T14:45:18 6124 Merit Network, Inc, Southfield, MI 306.94 94.94 64.64
2025-12-13T14:27:32 1778 Comcast, Detroit, MI 517.04 191.64 43.55
2025-12-13T14:26:55 64415 Surf Internet, Southfield, MI 144.70 100.25 55.62
2025-07-12T20:47:45 1778 Comcast, Detroit, MI 1846.12 260.60 13.40
2025-06-28T09:38:00 53778 123NET, Southfield, MI 1472.07 346.66 14.02
2025-06-28T09:36:58 1778 Comcast, Detroit, MI 2310.24 351.93 13.62
2025-06-20T14:30:56 1778 Comcast, Detroit, MI 2304.69 349.62 13.60
2025-06-06T16:46:48 69076 Ripplefiber, Detroit, MI 2144.83 317.85 31.49
2025-06-06T16:45:58 6124 Merit Network, Inc, Southfield, MI 2204.34 347.66 26.89
2025-06-06T16:45:21 1778 Comcast, Detroit, MI 2279.34 352.00 13.46
2025-05-28T14:58:20 1778 Comcast, Detroit, MI 2311.01 339.32 13.22
2025-04-20T20:51:52 53778 123NET, Southfield, MI 1252.25 314.12 9.47
2025-04-20T20:47:59 1778 Comcast, Detroit, MI 2017.36 351.83 14
The difference is significant, but not on an obvious (bandwidth) boundary. A rather asymmetric service... DOCSIS? Comcast? I'd expect a (more) symmetric offering these days.
Not much to go on. Start by characterizing performance through the firewall (that is not clear here), then (as necessary) without it, watching for errors and/or packet loss.
Did you actually try to keep the server ID fixed? Or try to use one of the fast Comcast servers now? Obviously, those were never used in 2026 for whatever reasons, thus the numbers are not comparable as such.
This is what I have for history. In December I got really busy with work and it's only now started to cool off and I can start to look at some issues.
Yes this is on a comcast DOCSIS line.
The system is an Intel Atom C3758R with intel i226V . Opensense is running as a VM in proxmox currently by itself.
The throughput has drastically changed. I tested directly on the modem with a laptop and I'm able to get around 800Mbps D / 300 Mbps U using fast.com or speedtest.net . Yes this is not the speed I use to get but the upload is really what I rely on. The throughput of opensense has taken a dive and I'm not even sure where to begin.
I'm trying to test the LAN side with iperf and I'm not sure what I'm doing wrong here but I can't connect to the client on on opensense, firewall rule is in place to allow any lan source to talk to the firewall on any port
Do you have any IDS/IPS or other filtering installed? Possible one of those changed.
Do you have ZFS and snapshots before and after the December updates? You could roll back to a previous and see if the speeds go back up and work forward from that point.
Glad you did a direct test on your modem, Spectrum was horrible when we had it, more down and than when we finally cancelled. Dealing with T-Mobile Home internet now (5g) and it's OK enough.
Quote from: nullspace on March 27, 2026, 12:54:18 AM[...]The throughput has drastically changed. I tested directly on the modem with a laptop and I'm able to get around 800Mbps D / 300 Mbps U using fast.com (https://fast.com/) or speedtest.net (https://speedtest.net/) . Yes this is not the speed I use to get but the upload is really what I rely on. The throughput of opensense has taken a dive and I'm not even sure where to begin.
Ugh. That doesn't narrow it down much, unfortunately. Could even be something like a modem firmware push. Or multiple things, just to be even more irritating. I'd try to get a line test (upstream of the modem, assuming you haven't already). In addition to any other troubleshooting. Oh, and this might be barking up the wrong tree, but I've had to deal with a few silent, erroneous service downgrades in the past. You might want to verify your service (as much as possible, from an administrative standpoint). (I worked for my provider, so I had internal visibility and contacts, and it was still a pain in the ass.)
Light work day and spent it going down a rabbit hole
got iperf working, I tested the LAN side that has a 10GB SFP+ X553 DAC into my switch. Looks like I'm getting full bandwidth there. So started to really look hard at the I226-V which connects directly to the modem. After looking the interface:
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN (wan) options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG> ether 20:7c:14:f4:3c:51 inet 73.18.207.231 netmask 0xfffffe00 broadcast 255.255.255.255 inet6 fe80::227c:14ff:fef4:3c51%igc1 prefixlen 64 scopeid 0x5 inet6 2001:558:6007:b6:7117:e650:28b1:f71b prefixlen 128 pltime 202602 vltime 202602 media: Ethernet 2500Base-T (2500Base-T <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Confirming my:
-IDS is off
-Shaper Rules: None
-No spike on a single cpu when under load ( top -aSH)
-checked counters for errors or anything obvious netstat -i-looked for any kill states pfctl -F state- looked through sysctl -a | grep dev.igc.1.mac_stats for anything that might stick out
Looked through dmesg:
dmesg | grep -i igc
for resets, watchdog events, link renegotiations, DMA/ring issues
started to think I needed to look at the threads on the forum about i-226V
I was running NVM version 2.14 ... so I'm updating to 2.32 tonight. successfully updated a couple of the spare NICs already. but not seeing a speed improvement when move my WAN port over to an updated one... though I might be getting them mixed up between the proxmox and opensense. I have failover WAN port that I also use and maintenance por.
Quote from: nullspace on March 28, 2026, 02:09:43 AMLight work day and spent it going down a rabbit hole
got iperf working, I tested the LAN side that has a 10GB SFP+ X553 DAC into my switch. Looks like I'm getting full bandwidth there. So started to really look hard at the I226-V which connects directly to the modem. After looking the interface:
igc1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN (wan) options=4800028<VLAN_MTU,JUMBO_MTU,HWSTATS,MEXTPG> ether 20:7c:14:f4:3c:51 inet 73.18.207.231 netmask 0xfffffe00 broadcast 255.255.255.255 inet6 fe80::227c:14ff:fef4:3c51%igc1 prefixlen 64 scopeid 0x5 inet6 2001:558:6007:b6:7117:e650:28b1:f71b prefixlen 128 pltime 202602 vltime 202602 media: Ethernet 2500Base-T (2500Base-T <full-duplex>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Confirming my:
-IDS is off
-Shaper Rules: None
-No spike on a single cpu when under load ( top -aSH)
-checked counters for errors or anything obvious netstat -i-looked for any kill states pfctl -F state- looked through sysctl -a | grep dev.igc.1.mac_stats for anything that might stick out
Looked through dmesg:
dmesg | grep -i igc
for resets, watchdog events, link renegotiations, DMA/ring issues
started to think I needed to look at the threads on the forum about i-226V
I was running NVM version 2.14 ... so I'm updating to 2.32 tonight. successfully updated a couple of the spare NICs already. but not seeing a speed improvement when move my WAN port over to an updated one... though I might be getting them mixed up between the proxmox and opensense. I have failover WAN port that I also use and maintenance por.
For the fun of it, have you tried speed testing directly from the proxmox?
If there were no specific sysctl settings that may have got wipes during updates, then things are most likely still all the same, which then leans an issue on the modem side.
But, iperf will be the goto tool to verify.
I will also note, with some recent work not related to OPNsense, PPS and Throughput are two very different things, so testing net load from those two angles is a better view of things.
Recommended Sysctl Offloading Settings for i226 NIC on FreeBSD
To optimize the performance of the i226 NIC on FreeBSD, you can adjust several sysctl settings. Below are the recommended configurations:
Key Sysctl Settings
Setting Value Description
net.isr.dispatch "deferred" Improves performance by allowing deferred processing of interrupts.
hw.ix.flow_control "0" Disables flow control to enhance throughput.
hw.ix.max_interrupt_rate Increase to 20000 Raises the maximum interrupt rate for better performance.
so after updating the firmware I didn't see any difference in the speed. So I got a SFP+ 2.5/5/10 module and just made the change over to the that as the WAN as I had a couple spare SFP+ X553 . That also didn't change the speed. So I've run out of ideas. I'll see if I can get into the modems admin interface and see if I can see anything.
So this is Proxmox... Can it handle the hardware offloading, or is this turned off? If you have it off, can you turn it on. Or the other way around depending on the circumstances.
Quote from: Greg_E on March 30, 2026, 08:26:41 PMSo this is Proxmox... Can it handle the hardware offloading, or is this turned off? If you have it off, can you turn it on. Or the other way around depending on the circumstances.
The NICs are given to opensense with PCI passthrough raw.
Sigh. I wish people would actually say if they use Proxmox underneath anything before asking seemingly unrelated questions...
Didn't I write something to that extent? Ah, yes, here, point 16 (https://forum.opnsense.org/index.php?topic=42985.0).
While we are at this, also take a look at points 10, 22 and 27.
Also: How exactly did you set up your OpnSense under PVE? Virtio or passthru NICs? Did you use multiqueue on the PVE NICs in the VM definition?
You now answered that - in case you use Realtek physical NICs, you inherit all the problems in point 6 of the READ ME FIRST article.
Maybe it is time to also look at: https://forum.opnsense.org/index.php?topic=44159.0, with a caveat that the "hardware checksumming" on virtio interfaces may be fixed already and can be left enabled. If you disabled it before, maybe that explains why the VM became slower.
Quote from: meyergru on March 30, 2026, 09:25:15 PMVirtio or passthru NICs?
They wrote passthrough just above :-)
I know. I edited in parallel and now augmented my post.
post #3 I state that I'm using proxmox also I noted the NIC hardware I was using which is all intel, i266V and X553. No realtek. It has 8GBs of RAM. All suggested tunables were configured on this more than a year ago. This is system has run very well for around 2+ years and I kept it up to date weekly.
Please note I'm now fully on X553 NICs for WAN and LAN. They are shared to the VM via passthrough on proxmox (host). It is still slow for incoming bandwidth... upload looks to be back to normal. ( ~300 Mbps Down ( this should be 2000 Mbps) , ~280 Mbps up). I changed nothing on the device all last year except the firmware and only some where in the 4th quarter of 2025 did it drop off a cliff.
Quote from: nullspace on March 31, 2026, 02:13:36 AMThis is system has run very well for around 2+ years and I kept it up to date weekly.
What's the status on the cooling of the system :
- Dust free ?
- Cooling paste not too old ?
Just a thought since I don't see anything about the type of Server Hardware you are using :)