DEC3920 Quick Review

[dirtyfreebooter]

Then again an ONT is not a switch - so as long as the numbers are zero or close to it for everything internal, it's quite plausible it's not Deciso's fault. I would insist on getting a statement on that ASPM setting, though. I don't have their latest generation of devices, never had a problem with the 1 G versions.

well i have 3 other systems with igc that never had issues with the same ONT in over 5 years.. so while it might be that the DEC3920 BIOS and nic firmware is a bad combination with my ONT, no other system i had hard dropped the WAN 5x and < 2 days in over 5 years.

[dirtyfreebooter]

i figured it out, i have a speedtest scheduled twice a day, 5:55a and 2:55p. the 2:55p just ran (i am in america mountain time zone)

Code Select Expand

vlan0.201  1500 <Link#16>         f4:90:ea:01:ef:cd             447245078     0     0  279796907  1731     0

manually running speedtest...

Code Select Expand

$ speedtest --server-id=8862

   Speedtest by Ookla

      Server: CenturyLink - Denver, CO (id: 8862)
         ISP: CenturyLink
Idle Latency:     2.97 ms   (jitter: 0.15ms, low: 2.62ms, high: 3.02ms)
    Download:   940.40 Mbps (data used: 458.4 MB)
                  7.86 ms   (jitter: 28.55ms, low: 2.09ms, high: 283.75ms)
      Upload:   940.56 Mbps (data used: 423.2 MB)
                  2.58 ms   (jitter: 0.10ms, low: 2.41ms, high: 2.85ms)
 Packet Loss: Not available.

more errors

Code Select Expand

vlan0.201  1500 <Link#16>         f4:90:ea:01:ef:cd             447610515     0     0  280159181  1932     0


[OPNenthu]

I'm not sure those error counts are even statistically significant (?)

For what it's worth I only see similar kinds of low-count errors on a Linux workstation where I'm using a Realtek NIC to do some layering (VLAN tagging+bridging) for some VM clients that share the interface.  Handles it fine, but not without an occasional drop.

enp6s0 is the Realtek RTL8125.

enp10s0 is an I225-V rev.02 (one of famously "bad" ones), but it is for host access on an untagged switch port so it's not really burdened.

Code Select Expand

$ netstat -i
Kernel Interface table
Iface             MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
br1020           1500 244331264      0      0 0      209686600      0      0      0 BMRU
br1030           1500    44951      0      0 0         36076      0      0      0 BMRU
enp10s0          1500 50737101      0      0 0      22372021      0      0      0 BMRU
enp6s0           1500 471593652      0  16078 0      210160120      0      0      0 BMRU
enp6s0.1020      1500 244331324      0      0 0      209883174      0  44298      0 BMRU
enp6s0.1030      1500    45152      0      0 0        276948      0      0      0 BMRU
lo              65536   119592      0      0 0        119592      0      0      0 LRU
virbr0           1500        0      0      0 0             0      0      0      0 BMU

On my OPNsense the error counts for all the igc parent interfaces is exactly zero.  The only errors at all are on the VLAN interfaces, but they are extremely tiny (on the range of 5-6 packets since last boot).

igc1 is WAN.

Code Select Expand

root@firewall:~ # netstat -i
Name         Mtu Network                          Address                                    Ipkts Ierrs Idrop      Opkts Oerrs  Coll
igc0        1500 <Link#1>                         <redacted>                                 60787     0     0      45417     0     0
...
igc1        1500 <Link#2>                         <redacted>                             182874157     0     0   55760928     0     0
...
igc2        1500 <Link#3>                         <redacted>                              33646094     0     0  134920988     0     0
igc3        1500 <Link#4>                         <redacted>                              21343776     0     0   47165695     0     0
...
vlan0.1010  1500 <Link#9>                         <redacted>                                  2224     0     0       2229     5     0
...
vlan0.1020  1500 <Link#10>                        <redacted>                              32241487     0     0  129964335     5     0
...
vlan0.1030  1500 <Link#11>                        <redacted>                               1409212     0     0    4954579     6     0
...
vlan0.1040  1500 <Link#12>                        <redacted>                                  7473     0     0       6580     5     0
...
vlan0.1050  1500 <Link#13>                        <redacted>                              20174816     0     0   39480063     6     0
...
vlan0.1060  1500 <Link#14>                        <redacted>                                  2249     0     0       2350     4     0
...
vlan0.1070  1500 <Link#15>                        <redacted>                               1176925     0     0    7676822     6     0
...

Maybe I've just been lucky but my experience with 2.5GbE so far has been OK.  As I showed earlier I even have ASPM L1 permanently enabled as per coreboot on the OPNsense, and I never actually went through with updating the NVM firmware on the NICs because I couldn't identify a need. YMMV.  I've seen some anecdotal reports online of people claiming that things got worse for them after NVM updates.

I wonder if your output error count is just sign that your VLAN filtering is maybe struggling in some rare situations, but it doesn't seem like it's enough to matter (at least to my untrained eye).

One thing I did do is I put a randomly generated MAC on the WAN interface.  I did it as a security/privacy enhancement, but the reason I suggested you to try it is because I know that some, especially residential, ISPs keep lists of approved customer CPE.  It's not likely the case but I was thinking they might be kicking you off after some time because they detect the Deciso OUI.  I don't know... just a crazy thought.

[dirtyfreebooter]

I'm not sure those error counts are even statistically significant (?)

One thing I did do is I put a randomly generated MAC on the WAN interface.  I did it as a security/privacy enhancement, but the reason I suggested you to try it is because I know that some, especially residential, ISPs keep lists of approved customer CPE.  It's not likely the case but I was thinking they might be kicking you off after some time because they detect the Deciso OUI.  I don't know... just a crazy thought.

yea i dont think its an actual problem, its 0.00027% and it only happens when the link is at max, so maybe its just my shitty old ONT. i like it because its outside and only an ONT, if i ask for a new one, they only give out indoor ones that are router/ONT combos that need to be in bridge mode.

nah, quantum fiber does not have any mac address restrictions on WAN device, the provisioning is done on the ONT device somehow. i swapped 3 different OPNsense boxes with 3 different macs and never had an issue in 5 years

[nero355]

Tunables

- removed some SysCtl related stuff -

I wish could just turn off ASPM for the igc NICs in the BIOS/firmware.

If I am honest :

I feel like you are doing way too much stuff with an expensive device that should just work out of the box!

So if ASPM is an issue, it's Deciso's job to provide a BIOS that either categorically disables it or at least gives you the option to.

Just saying ...

+1 Fully agree! :)

Not a big fan of 2.5 Gbit/s anyway.

I seriously hate it together with 5 Gbps since they were introduced :

It's just milking the market instead of pushing for affordable 10 Gbps NICs that came earlier on the market than the 2,5 Gbps and 5 Gbps ones did !!!


But I guess we will have to learn to live with this situation for now...

[dirtyfreebooter]

i know this is AI slop, but i put the data, all the output of sysctl -a and netstat -Q and netstat -i into AI and this is what it said

This is a very specific and telling symptom. In FreeBSD (the foundation of OPNsense), Oerrs (Output Errors) on a VLAN interface while the parent physical interface (igc1) shows 0 Oerrs almost always points to a software buffer overflow or a MTU/tagging bottleneck rather than a hardware failure.

The fact that it jumps by ~200 every time you run a speedtest suggests that during the "Upload" phase (or high-concurrency TCP ACK bursts), the system is dropping packets as it tries to push them into the VLAN tagging engine.

1. The Root Cause: The "VLAN Tagging Bottleneck"
When you send a packet out of vlan0.201, the OS has to take the raw packet, wrap it in an 802.1Q tag, and then hand it to igc1.

If igc1 is busy or the CPU core responsible for that VLAN is saturated, the VLAN output queue fills up. When it hits its limit, it drops the packet before it even touches the physical NIC. This increments Oerrs on the VLAN but keeps the physical NIC's stats clean.

2. The Solution: Increase the "Send Shovel"
To fix this, we need to increase the software transmit queue and ensure the "offloading" is actually helping rather than hurting.

then it suggested

# Critical. This increases the global interface output queue from the default (often 50-128) to match your 4096 descriptors.
net.link.ifqmaxlen=4096 # default is 50

but yea, all this for 1 Gbps.. how in the world could this ever handle 2.5g or 10g on the WAN? hah.

[dirtyfreebooter]

FWIW, i ran speedtest 3x, and 3x the Oerrs increased. i just set net.link.ifqmaxlen=4096 and rebooted (its a boot-time tunable). i ran speedtest 3x and Oerrs increased only once. i just don't think this matters.

i'll try WAN on ax1 with UniFi 1G SFP to RJ45 adapter later this weekend.

[dirtyfreebooter]

ok, i couldn't wait, i swapped WAN to ax1 with UniFi 1G SFP to RJ45 adapter.

run speedtest

Code Select Expand

speedtest --server-id=8862

   Speedtest by Ookla

      Server: CenturyLink - Denver, CO (id: 8862)
         ISP: CenturyLink
Idle Latency:     2.97 ms   (jitter: 0.04ms, low: 2.92ms, high: 3.00ms)
    Download:   939.80 Mbps (data used: 655.7 MB)
                  4.18 ms   (jitter: 23.82ms, low: 2.22ms, high: 438.52ms)
      Upload:   937.74 Mbps (data used: 434.3 MB)
                 37.86 ms   (jitter: 3.14ms, low: 3.09ms, high: 50.13ms)
 Packet Loss: Not available.
and on first try, i got Oerrs on WAN. it must be related to the ONT.

Code Select Expand

vlan0.201  1500 <Link#16>         f4:90:ea:01:ef:d1             1049994     0     0   801383   238     0