Periodic NIC issues (?) with Protectli Vault, Intel i226-V

[BrandyWine]

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Has the 226's NVM been updated?

Before installing new kernel,
I would try (from ssh or in a start script)
ifconfig igc0 -rxcsum -txcsum -rxcsum6 -txcsum6
ifconfig igc1 -rxcsum -txcsum -rxcsum6 -txcsum6
should be no need to down/up the device

Then monitor.

Offloading puts the function into hardware.

Disabled or not, iflib sits between kernel and driver. So the functional test logic may be:
1) Disabling offload runs ok, then the issue is likely igc/nic problem.
2) Disabling offload & issue comes back, this points to something else, maybe iflib.c

[fornax]

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Code Select Expand

dev.igc.0.interrupts.rx_desc_min_thresh: 0
dev.igc.0.interrupts.asserts: 2544406
dev.igc.0.mac_stats.tso_txd: 0
dev.igc.0.mac_stats.tx_frames_1024_1522: 3606676
dev.igc.0.mac_stats.tx_frames_512_1023: 14052
dev.igc.0.mac_stats.tx_frames_256_511: 17180
dev.igc.0.mac_stats.tx_frames_128_255: 26586
dev.igc.0.mac_stats.tx_frames_65_127: 105743
dev.igc.0.mac_stats.tx_frames_64: 5542
dev.igc.0.mac_stats.mcast_pkts_txd: 2184
dev.igc.0.mac_stats.bcast_pkts_txd: 83
dev.igc.0.mac_stats.good_pkts_txd: 3775779
dev.igc.0.mac_stats.total_pkts_txd: 3775779
dev.igc.0.mac_stats.good_octets_txd: 5499410065
dev.igc.0.mac_stats.good_octets_recvd: 179427410
dev.igc.0.mac_stats.rx_frames_1024_1522: 28848
dev.igc.0.mac_stats.rx_frames_512_1023: 11786
dev.igc.0.mac_stats.rx_frames_256_511: 10339
dev.igc.0.mac_stats.rx_frames_128_255: 22011
dev.igc.0.mac_stats.rx_frames_65_127: 1596460
dev.igc.0.mac_stats.rx_frames_64: 26780
dev.igc.0.mac_stats.mcast_pkts_recvd: 3350
dev.igc.0.mac_stats.bcast_pkts_recvd: 44496
dev.igc.0.mac_stats.good_pkts_recvd: 1696224
dev.igc.0.mac_stats.total_pkts_recvd: 1705605
dev.igc.0.mac_stats.mgmt_pkts_txd: 0
dev.igc.0.mac_stats.mgmt_pkts_drop: 0
dev.igc.0.mac_stats.mgmt_pkts_recvd: 0
dev.igc.0.mac_stats.unsupported_fc_recvd: 0
dev.igc.0.mac_stats.xoff_txd: 0
dev.igc.0.mac_stats.xoff_recvd: 0
dev.igc.0.mac_stats.xon_txd: 0
dev.igc.0.mac_stats.xon_recvd: 0
dev.igc.0.mac_stats.alignment_errs: 0
dev.igc.0.mac_stats.crc_errs: 0
dev.igc.0.mac_stats.recv_errs: 0
dev.igc.0.mac_stats.recv_jabber: 0
dev.igc.0.mac_stats.recv_oversize: 0
dev.igc.0.mac_stats.recv_fragmented: 0
dev.igc.0.mac_stats.recv_undersize: 0
dev.igc.0.mac_stats.recv_no_buff: 0
dev.igc.0.mac_stats.recv_length_errors: 0
dev.igc.0.mac_stats.missed_packets: 197
dev.igc.0.mac_stats.defer_count: 0
dev.igc.0.mac_stats.sequence_errors: 0
dev.igc.0.mac_stats.symbol_errors: 0
dev.igc.0.mac_stats.collision_count: 0
dev.igc.0.mac_stats.late_coll: 0
dev.igc.0.mac_stats.multiple_coll: 0
dev.igc.0.mac_stats.single_coll: 0
dev.igc.0.mac_stats.excess_coll: 0
dev.igc.0.queue_rx_3.rx_irq: 0
dev.igc.0.queue_rx_3.rxd_tail: 612
dev.igc.0.queue_rx_3.rxd_head: 613
dev.igc.0.queue_rx_3.interrupt_rate: 20832
dev.igc.0.queue_rx_2.rx_irq: 0
dev.igc.0.queue_rx_2.rxd_tail: 368
dev.igc.0.queue_rx_2.rxd_head: 369
dev.igc.0.queue_rx_2.interrupt_rate: 20832
dev.igc.0.queue_rx_1.rx_irq: 0
dev.igc.0.queue_rx_1.rxd_tail: 830
dev.igc.0.queue_rx_1.rxd_head: 830
dev.igc.0.queue_rx_1.interrupt_rate: 4032
dev.igc.0.queue_rx_0.rx_irq: 0
dev.igc.0.queue_rx_0.rxd_tail: 605
dev.igc.0.queue_rx_0.rxd_head: 606
dev.igc.0.queue_rx_0.interrupt_rate: 20832
dev.igc.0.queue_tx_3.tx_irq: 0
dev.igc.0.queue_tx_3.txd_tail: 173
dev.igc.0.queue_tx_3.txd_head: 173
dev.igc.0.queue_tx_3.interrupt_rate: 20832
dev.igc.0.queue_tx_2.tx_irq: 0
dev.igc.0.queue_tx_2.txd_tail: 1000
dev.igc.0.queue_tx_2.txd_head: 1000
dev.igc.0.queue_tx_2.interrupt_rate: 20832
dev.igc.0.queue_tx_1.tx_irq: 0
dev.igc.0.queue_tx_1.txd_tail: 46
dev.igc.0.queue_tx_1.txd_head: 46
dev.igc.0.queue_tx_1.interrupt_rate: 4032
dev.igc.0.queue_tx_0.tx_irq: 0
dev.igc.0.queue_tx_0.txd_tail: 735
dev.igc.0.queue_tx_0.txd_head: 735
dev.igc.0.queue_tx_0.interrupt_rate: 20832
dev.igc.0.fc_low_water: 32752
dev.igc.0.fc_high_water: 32768
dev.igc.0.rx_control: 71335938
dev.igc.0.device_control: 1075578433
dev.igc.0.watchdog_timeouts: 0
dev.igc.0.rx_overruns: 0
dev.igc.0.link_irq: 3
dev.igc.0.dropped: 0
dev.igc.0.eee_control: 1
dev.igc.0.tso_tcp_flags_mask_last_segment: 3967
dev.igc.0.tso_tcp_flags_mask_middle_segment: 3958
dev.igc.0.tso_tcp_flags_mask_first_segment: 4086
dev.igc.0.dmac: 0
dev.igc.0.rs_dump: 0
dev.igc.0.reg_dump: General Registers
        CTRL     401c0641
        STATUS   00380683
        CTRL_EXIT        100000c0

Interrupt Registers
        ICR      00000000

RX Registers
        RCTL     04408002
        RDLEN    00004000
        RDH      0000025e
        RDT      0000025d
        RXDCTL   02040808
        RDBAL    75d88000
        RDBAH    00000000

TX Registers
        TCTL     a503f0fa
        TDBAL    750f8000
        TDBAH    00000000
        TDLEN    00004000
        TDH      000002df
        TDT      000002df
        TXDCTL   0201011f
        TDFH     00000000
        TDFT     00000000
        TDFHS    00000000
        TDFPC    00000000


dev.igc.0.fc: 3
dev.igc.0.debug: -1
dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303
dev.igc.0.enable_aim: 1
dev.igc.0.nvm: -1
dev.igc.0.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq3.rxq_fl0.cidx: 613
dev.igc.0.iflib.rxq3.rxq_fl0.pidx: 612
dev.igc.0.iflib.rxq3.cpu: 3
dev.igc.0.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq2.rxq_fl0.cidx: 369
dev.igc.0.iflib.rxq2.rxq_fl0.pidx: 368
dev.igc.0.iflib.rxq2.cpu: 2
dev.igc.0.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq1.rxq_fl0.cidx: 831
dev.igc.0.iflib.rxq1.rxq_fl0.pidx: 830
dev.igc.0.iflib.rxq1.cpu: 1
dev.igc.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.0.iflib.rxq0.rxq_fl0.cidx: 606
dev.igc.0.iflib.rxq0.rxq_fl0.pidx: 605
dev.igc.0.iflib.rxq0.cpu: 0
dev.igc.0.iflib.txq3.r_abdications: 0
dev.igc.0.iflib.txq3.r_restarts: 0
dev.igc.0.iflib.txq3.r_stalls: 0
dev.igc.0.iflib.txq3.r_starts: 256058
dev.igc.0.iflib.txq3.r_drops: 0
dev.igc.0.iflib.txq3.r_enqueues: 256060
dev.igc.0.iflib.txq3.ring_state: pidx_head: 0060 pidx_tail: 0060 cidx: 0060 state: IDLE
dev.igc.0.iflib.txq3.txq_cleaned: 512129
dev.igc.0.iflib.txq3.txq_processed: 512169
dev.igc.0.iflib.txq3.txq_in_use: 44
dev.igc.0.iflib.txq3.txq_cidx_processed: 169
dev.igc.0.iflib.txq3.txq_cidx: 129
dev.igc.0.iflib.txq3.txq_pidx: 173
dev.igc.0.iflib.txq3.no_tx_dma_setup: 0
dev.igc.0.iflib.txq3.txd_encap_efbig: 0
dev.igc.0.iflib.txq3.tx_map_failed: 0
dev.igc.0.iflib.txq3.no_desc_avail: 0
dev.igc.0.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq3.m_pullups: 0
dev.igc.0.iflib.txq3.mbuf_defrag: 0
dev.igc.0.iflib.txq3.cpu: 3
dev.igc.0.iflib.txq2.r_abdications: 0
dev.igc.0.iflib.txq2.r_restarts: 0
dev.igc.0.iflib.txq2.r_stalls: 0
dev.igc.0.iflib.txq2.r_starts: 195065
dev.igc.0.iflib.txq2.r_drops: 0
dev.igc.0.iflib.txq2.r_enqueues: 195065
dev.igc.0.iflib.txq2.ring_state: pidx_head: 0505 pidx_tail: 0505 cidx: 0505 state: IDLE
dev.igc.0.iflib.txq2.txq_cleaned: 390078
dev.igc.0.iflib.txq2.txq_processed: 390118
dev.igc.0.iflib.txq2.txq_in_use: 42
dev.igc.0.iflib.txq2.txq_cidx_processed: 998
dev.igc.0.iflib.txq2.txq_cidx: 958
dev.igc.0.iflib.txq2.txq_pidx: 1000
dev.igc.0.iflib.txq2.no_tx_dma_setup: 0
dev.igc.0.iflib.txq2.txd_encap_efbig: 0
dev.igc.0.iflib.txq2.tx_map_failed: 0
dev.igc.0.iflib.txq2.no_desc_avail: 0
dev.igc.0.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq2.m_pullups: 0
dev.igc.0.iflib.txq2.mbuf_defrag: 0
dev.igc.0.iflib.txq2.cpu: 2
dev.igc.0.iflib.txq1.r_abdications: 0
dev.igc.0.iflib.txq1.r_restarts: 0
dev.igc.0.iflib.txq1.r_stalls: 0
dev.igc.0.iflib.txq1.r_starts: 3023378
dev.igc.0.iflib.txq1.r_drops: 0
dev.igc.0.iflib.txq1.r_enqueues: 3023383
dev.igc.0.iflib.txq1.ring_state: pidx_head: 0535 pidx_tail: 0535 cidx: 0535 state: IDLE
dev.igc.0.iflib.txq1.txq_cleaned: 6046724
dev.igc.0.iflib.txq1.txq_processed: 6046764
dev.igc.0.iflib.txq1.txq_in_use: 42
dev.igc.0.iflib.txq1.txq_cidx_processed: 44
dev.igc.0.iflib.txq1.txq_cidx: 4
dev.igc.0.iflib.txq1.txq_pidx: 46
dev.igc.0.iflib.txq1.no_tx_dma_setup: 0
dev.igc.0.iflib.txq1.txd_encap_efbig: 0
dev.igc.0.iflib.txq1.tx_map_failed: 0
dev.igc.0.iflib.txq1.no_desc_avail: 0
dev.igc.0.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq1.m_pullups: 0
dev.igc.0.iflib.txq1.mbuf_defrag: 0
dev.igc.0.iflib.txq1.cpu: 1
dev.igc.0.iflib.txq0.r_abdications: 0
dev.igc.0.iflib.txq0.r_restarts: 0
dev.igc.0.iflib.txq0.r_stalls: 0
dev.igc.0.iflib.txq0.r_starts: 301485
dev.igc.0.iflib.txq0.r_drops: 0
dev.igc.0.iflib.txq0.r_enqueues: 301488
dev.igc.0.iflib.txq0.ring_state: pidx_head: 0432 pidx_tail: 0432 cidx: 0432 state: IDLE
dev.igc.0.iflib.txq0.txq_cleaned: 602805
dev.igc.0.iflib.txq0.txq_processed: 602845
dev.igc.0.iflib.txq0.txq_in_use: 42
dev.igc.0.iflib.txq0.txq_cidx_processed: 733
dev.igc.0.iflib.txq0.txq_cidx: 693
dev.igc.0.iflib.txq0.txq_pidx: 735
dev.igc.0.iflib.txq0.no_tx_dma_setup: 0
dev.igc.0.iflib.txq0.txd_encap_efbig: 0
dev.igc.0.iflib.txq0.tx_map_failed: 0
dev.igc.0.iflib.txq0.no_desc_avail: 0
dev.igc.0.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.0.iflib.txq0.m_pullups: 3
dev.igc.0.iflib.txq0.mbuf_defrag: 0
dev.igc.0.iflib.txq0.cpu: 0
dev.igc.0.iflib.override_nrxds: 0
dev.igc.0.iflib.override_ntxds: 0
dev.igc.0.iflib.allocated_msix_vectors: 5
dev.igc.0.iflib.use_extra_msix_vectors: 0
dev.igc.0.iflib.use_logical_cores: 0
dev.igc.0.iflib.separate_txrx: 0
dev.igc.0.iflib.core_offset: 0
dev.igc.0.iflib.tx_abdicate: 0
dev.igc.0.iflib.rx_budget: 0
dev.igc.0.iflib.disable_msix: 0
dev.igc.0.iflib.override_qs_enable: 0
dev.igc.0.iflib.override_nrxqs: 0
dev.igc.0.iflib.override_ntxqs: 0
dev.igc.0.iflib.driver_version: 1
dev.igc.0.%iommu: rid=0x100
dev.igc.0.%parent: pci1
dev.igc.0.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.0.%location: slot=0 function=0 dbsf=pci0:1:0:0
dev.igc.0.%driver: igc
dev.igc.0.%desc: Intel(R) Ethernet Controller I226-V