Periodic NIC issues (?) with Protectli Vault, Intel i226-V

[BrandyWine]

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Has the 226's NVM been updated?

Before installing new kernel,
I would try (from ssh or in a start script)
ifconfig igc0 -rxcsum -txcsum -rxcsum6 -txcsum6
ifconfig igc1 -rxcsum -txcsum -rxcsum6 -txcsum6
should be no need to down/up the device

Then monitor.

Offloading puts the function into hardware.

Disabled or not, iflib sits between kernel and driver. So the functional test logic may be:
1) Disabling offload runs ok, then the issue is likely igc/nic problem.
2) Disabling offload & issue comes back, this points to something else, maybe iflib.c

[fornax]

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Code Select Expand

dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 2544406
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 3606676
dev.igc.0.mac_stats.tx_frames_512_1023: 14052
dev.igc.0.mac_stats.tx_frames_256_511: 17180
dev.igc.0.mac_stats.tx_frames_128_255: 26586
dev.igc.0.mac_stats.tx_frames_65_127: 105743
dev.igc.0.mac_stats.tx_frames_64: 5542
dev.igc.0.mac_stats.mcast_pkts_txd: 2184
dev.igc.0.mac_stats.bcast_pkts_txd: 83
dev.igc.0.mac_stats.good_pkts_txd: 3775779
dev.igc.0.mac_stats.total_pkts_txd: 3775779
dev.igc.0.mac_stats.good_octets_txd: 5499410065
dev.igc.0.mac_stats.good_octets_recvd: 179427410
dev.igc.0.mac_stats.rx_frames_1024_1522: 28848
dev.igc.0.mac_stats.rx_frames_512_1023: 11786
dev.igc.0.mac_stats.rx_frames_256_511: 10339
dev.igc.0.mac_stats.rx_frames_128_255: 22011
dev.igc.0.mac_stats.rx_frames_65_127: 1596460
dev.igc.0.mac_stats.rx_frames_64: 26780
dev.igc.0.mac_stats.mcast_pkts_recvd: 3350
dev.igc.0.mac_stats.bcast_pkts_recvd: 44496
dev.igc.0.mac_stats.good_pkts_recvd: 1696224
dev.igc.0.mac_stats.total_pkts_recvd: 1705605
dev.igc.0.mac_stats.mgmt_pkts_txd: 0
dev.igc.0.mac_stats.mgmt_pkts_drop: 0
dev.igc.0.mac_stats.mgmt_pkts_recvd: 0
dev.igc.0.mac_stats.unsupported_fc_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.recv_length_errors: 0
dev.igc.0.mac_stats.missed_packets: 197
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 612
dev.igc.0.queue_rx_3.rxd_head: 613
dev.igc.0.queue_rx_3.interrupt_rate: 20832
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 368
dev.igc.0.queue_rx_2.rxd_head: 369
dev.igc.0.queue_rx_2.interrupt_rate: 20832
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 830
dev.igc.0.queue_rx_1.rxd_head: 830
dev.igc.0.queue_rx_1.interrupt_rate: 4032
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 605
dev.igc.0.queue_rx_0.rxd_head: 606
dev.igc.0.queue_rx_0.interrupt_rate: 20832
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 173
dev.igc.0.queue_tx_3.txd_head: 173
dev.igc.0.queue_tx_3.interrupt_rate: 20832
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 1000
dev.igc.0.queue_tx_2.txd_head: 1000
dev.igc.0.queue_tx_2.interrupt_rate: 20832
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 46
dev.igc.0.queue_tx_1.txd_head: 46
dev.igc.0.queue_tx_1.interrupt_rate: 4032
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 735
dev.igc.0.queue_tx_0.txd_head: 735
dev.igc.0.queue_tx_0.interrupt_rate: 20832
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1075578433
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 3
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.tso_tcp_flags_mask_last_segment: 3967
dev.igc.0.tso_tcp_flags_mask_middle_segment: 3958
dev.igc.0.tso_tcp_flags_mask_first_segment: 4086
dev.igc.0.dmac: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
        CTRL     401c0641
        STATUS   00380683
        CTRL_EXIT        100000c0

Interrupt Registers
        ICR      00000000

RX Registers
        RCTL     04408002
        RDLEN    00004000
        RDH      0000025e
        RDT      0000025d
        RXDCTL   02040808
        RDBAL    75d88000
        RDBAH    00000000

TX Registers
        TCTL     a503f0fa
        TDBAL    750f8000
        TDBAH    00000000
        TDLEN    00004000
        TDH      000002df
        TDT      000002df
        TXDCTL   0201011f
        TDFH     00000000
        TDFT     00000000
        TDFHS    00000000
        TDFPC    00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303
dev.igc.0.enable_aim: 1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 613
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 612
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 369
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 368
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 831
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 830
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 606
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 605
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 256058
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 256060
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0060 pidx_tail: 0060 cidx: 0060 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 512129
dev.igc.0.iflib.txq3.txq_processed: 512169
dev.igc.0.iflib.txq3.txq_in_use: 44
dev.igc.0.iflib.txq3.txq_cidx_processed: 169
dev.igc.0.iflib.txq3.txq_cidx: 129
dev.igc.0.iflib.txq3.txq_pidx: 173
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 195065
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 195065
dev.igc.0.iflib.txq2.ring_state: pidx_head: 0505 pidx_tail: 0505 cidx: 0505 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 390078
dev.igc.0.iflib.txq2.txq_processed: 390118
dev.igc.0.iflib.txq2.txq_in_use: 42
dev.igc.0.iflib.txq2.txq_cidx_processed: 998
dev.igc.0.iflib.txq2.txq_cidx: 958
dev.igc.0.iflib.txq2.txq_pidx: 1000
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 3023378
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 3023383
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0535 pidx_tail: 0535 cidx: 0535 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 6046724
dev.igc.0.iflib.txq1.txq_processed: 6046764
dev.igc.0.iflib.txq1.txq_in_use: 42
dev.igc.0.iflib.txq1.txq_cidx_processed: 44
dev.igc.0.iflib.txq1.txq_cidx: 4
dev.igc.0.iflib.txq1.txq_pidx: 46
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 301485
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 301488
dev.igc.0.iflib.txq0.ring_state: pidx_head: 0432 pidx_tail: 0432 cidx: 0432 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 602805
dev.igc.0.iflib.txq0.txq_processed: 602845
dev.igc.0.iflib.txq0.txq_in_use: 42
dev.igc.0.iflib.txq0.txq_cidx_processed: 733
dev.igc.0.iflib.txq0.txq_cidx: 693
dev.igc.0.iflib.txq0.txq_pidx: 735
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 3
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.allocated_msix_vectors: 5
dev.igc.0.iflib.use_extra_msix_vectors: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%iommu: rid=0x100
dev.igc.0.%parent: pci1
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V


[tuto2]

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Ok, getting closer..

Can you try:

Code Select Expand

# opnsense-update -zkr 26.1.11-igc_aspm

and reboot?

Ref: https://github.com/freebsd/freebsd-src/pull/2318

[tuto2]

Hold up...

Code Select Expand

dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303


This is before any of the fixes Intel pushed for related stuff.. The baseline should be 2.25.

[Greg_E]

Uggg... I just received a brand new VP4650 with 6x i226 ports to replace my old recycled OPNsense server. I'm also on Business so fixes might be a while.

I guess I wait and maybe burn it in on my lab?

I'll be watching this. And before I bought this the sales person spoke to Protectli, and I spoke to Protectli about using this with OPNsense, the only thing they said was that I probably want to run Coreboot, but can switch back and forth at any time.

Considering the difficulty of people in the USA for getting Deciso based hardware, Protectli is about the best choice. And they were on government contract which helped me buy it for work. Seems like there might be a spot for more of a partnership between Protectli and Deciso/OPNsense.

[edit] I sent an email to the sales contact I made a few months ago, linked this thread. Maybe we can get Protecli on this to help out. The OPNsense hardware wasn't really an option for me here in the USA, which is why I ended up with a Protectli device.

[BrandyWine]

You're troubleshooting while on old problematic NVM. Why?
Your 226's are five (5) revs behind. Start at step #1, update NVM.

Also a note for readers, in this thread the NVM is the 2MB version. 2MB versions have more features in it, makes me wonder if this issue is specific to the 2MB version?

v2.25
Bug Fix:
• HSD 13010560068: Link Flaps with Energy Efficient Ethernet Enabled

There's also a 2MB v2.27 and a v2.32 available.

Oddly from the billy curtis notes (fixes section), v2.22 states the link flap issue was fixed, but in v2.25 the same issue is fixed again.

Also to note, ASPM and EE are not the same. Best recommendation is to disable ASPM and then tune EE C states so that nothing can ever go 100% sleep.

EEE for igc can be done via sysctl
sysctl hw.igc.eee_setting=1
sysctl dev.igc.0.eee_control=1
sysctl dev.igc.1.eee_control=1

And I not sure why, I think this was mentioned by another user elsewhere, the logic is backwards in the actual c code. "1" means to disable (which should be default), and "0" means enable. I mentioned 802.3az in the other thread some time ago.

When EEE is enabled, the igc NIC uses the IEEE 802.3az "Energy Efficient Ethernet" behavior: during periods of low/no transmit activity it can reduce link power by entering Low Power Idle (LPI) (often coordinated with the switch/peer), then wake up quickly when traffic resumes. The goal is to cut power use during low utilization while keeping normal connectivity.

[franco]

@Greg_E the kernel mentioned above is also on the business mirror to try.


Cheers,
Franco

[fornax]

Ok, thanks for the additional comments. As mentioned in the original post, I haven't updated the NVM because frankly it's a bit intimidating. As far as I can tell Protectli only provides an update for one model (that isn't mine), so I'm going third-party for something that can brick the interface. Sounds like that may be what I need to do anyway though. I'll try a plea to Protectli while I prepare.

[OPNenthu]

There are two fixes offered in this thread, one in the FreeBSD kernel (in development) and one in the OPNsense custom kernel (-iflib variant, requiring newer NVM).  Are they redundant or targeting different issues?

BTW, Protectli have a statement about why they can't distribute i226 NVM updates at the bottom of this document: https://kb.protectli.com/kb/how-to-update-intel-nic-nvm-firmware-on-protectli-vaults/.  They link to the thread here with the BillyCurtis images, for those needing to do it.