Periodic NIC issues (?) with Protectli Vault, Intel i226-V

[franco]

I'd like to reiterate that this is a real world easy to test approach that requires *actual feedback*:

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Ok, getting closer..

Can you try:

Code Select Expand

# opnsense-update -zkr 26.1.11-igc_aspm

and reboot?

Ref: https://github.com/freebsd/freebsd-src/pull/2318

[fornax]

Ok, the issue popped up again this morning, verifying that the NVM update didn't in itself fix anything. I just installed the igc_aspm kernel, so we'll see how that goes now.

Code Select Expand

root@scutum:~ # uname -a
FreeBSD scutum.nightsky.internal 14.3-RELEASE-p16 FreeBSD 14.3-RELEASE-p16 igc_aspm-n272154-6f771ce68454 SMP amd64


[BrandyWine]

Ok, the issue popped up again this morning, verifying that the NVM update didn't in itself fix anything. I just installed the igc_aspm kernel, so we'll see how that goes now.

Code Select Expand

root@scutum:~ # uname -a
FreeBSD scutum.nightsky.internal 14.3-RELEASE-p16 FreeBSD 14.3-RELEASE-p16 igc_aspm-n272154-6f771ce68454 SMP amd64


In post #30 you mention you installed a kernel, which kernel was that one? Between #30 and #31 you installed two kernels ??

[fornax]

In post #30 you mention you installed a kernel, which kernel was that one? Between #30 and #31 you installed two kernels ??

From post #16 until this morning I was on the -iflib kernel mentioned in post #2 (post #30 quoted post #16). Prior to that I was on the stock kernel.

[meyergru]

As @Franco recommended, you should absolutely try "opnsense-update -zkr 26.1.11-igc_aspm", because that should fix the hangup condition.

BTW: This is the original bug report.

[fornax]

As @Franco recommended, you should absolutely try "opnsense-update -zkr 26.1.11-igc_aspm", because that should fix the hangup condition.

Yep, as mentioned, that's what I'm on now.

[Lucid1010]

https://github.com/freebsd/freebsd-src/pull/2318

Is this patch included in OPNsense v26.7?

[tuto2]

https://github.com/freebsd/freebsd-src/pull/2318

Is this patch included in OPNsense v26.7?

It will be, though feedback is the important bit here.

[Greg_E]

I got a response from Protectli:

#1 my VP4650 shouldn't have this problem.

#2 they offered another document regarding ASPM for the older devices:
https://kb.protectli.com/wp-content/uploads/sites/9/2026/02/TSB-2025-001_VP2440-ASPM-Network-Interface-Performance-Issue-v2_0.pdf

Hopefully they watch this thread to see the outcome and distill it back into another KB document.

[fornax]

#2 they offered another document regarding ASPM for the older devices:
https://kb.protectli.com/wp-content/uploads/sites/9/2026/02/TSB-2025-001_VP2440-ASPM-Network-Interface-Performance-Issue-v2_0.pdf

Sounds like their coreboot fix is doing the same/similar thing as the kernel patch? If so, I wonder why they didn't extend that to their other models with i226-Vs.

[OPNenthu]

Update:

I had installed the 'igc_aspm' kernel on a system with all i226-V NICs having NVM firmware 2.13 (an old version).  The patch appears safe on this older firmware and hasn't caused any problems that I've seen so far.  The earlier advice to start from at least a 2.25 baseline is probably sound, but at least there's no observed impact here.

I was not having link flaps or speed degradation in any case.  I was hoping to clear up the frequent dpinger drops that I see which cause a lot of unecessary failovers for gatways groups (such as VPNs), but unfortunately the patch didn't help that.  I'm guessing my issue is not localized and could just be the ISP infrastructure or upstream routers.

Might work up the courage to upgrade the NVM but I really don't have any of the typical problem indications that are worth the risk AFAICT.

WAN link data for reference...

Code Select Expand

root@firewall:~ # netstat -I igc1
Name    Mtu Network                 Address                                   Ipkts Ierrs Idrop     Opkts Oerrs  Coll
igc1   1500 <Link#2>                xx:xx:xx:xx:xx:xx                     100279348     0     0  38220783     0     0
igc1      - fe80::%igc1/64          fe80::xx:xxxx:xxxx:xxxx%igc1               2242     -     -      2728     -     -
igc1      - 69.xxx.xxx.xxx/22       xxx.xxx.xx.comcast.net                 42381670     -     -  13944744     -     -
igc1      - 10.0.0.0/30             10.0.0.2                                      0     -     -         0     -     -
igc1      - xxxx:xx:xxxx:1b77::/64  xxxx:xx:xxxx:1b77:xxxx:xxxx:xxxx:xxxx    331055     -     -    394721     -     -


Code Select Expand

root@firewall:~ # sysctl dev.igc.1
dev.igc.1.interrupts.rx_desc_min_thresh: 0
dev.igc.1.interrupts.asserts: 107822263
dev.igc.1.mac_stats.tso_txd: 0
dev.igc.1.mac_stats.tx_frames_1024_1522: 599302
dev.igc.1.mac_stats.tx_frames_512_1023: 460622
dev.igc.1.mac_stats.tx_frames_256_511: 210904
dev.igc.1.mac_stats.tx_frames_128_255: 13631408
dev.igc.1.mac_stats.tx_frames_65_127: 23107378
dev.igc.1.mac_stats.tx_frames_64: 213312
dev.igc.1.mac_stats.mcast_pkts_txd: 46
dev.igc.1.mac_stats.bcast_pkts_txd: 181
dev.igc.1.mac_stats.good_pkts_txd: 38222926
dev.igc.1.mac_stats.total_pkts_txd: 38222926
dev.igc.1.mac_stats.good_octets_txd: 4995909586
dev.igc.1.mac_stats.good_octets_recvd: 138811724686
dev.igc.1.mac_stats.rx_frames_1024_1522: 96022714
dev.igc.1.mac_stats.rx_frames_512_1023: 472886
dev.igc.1.mac_stats.rx_frames_256_511: 483889
dev.igc.1.mac_stats.rx_frames_128_255: 1734860
dev.igc.1.mac_stats.rx_frames_65_127: 1327632
dev.igc.1.mac_stats.rx_frames_64: 239883
dev.igc.1.mac_stats.mcast_pkts_recvd: 98216
dev.igc.1.mac_stats.bcast_pkts_recvd: 137
dev.igc.1.mac_stats.good_pkts_recvd: 100281864
dev.igc.1.mac_stats.total_pkts_recvd: 100283378
dev.igc.1.mac_stats.mgmt_pkts_txd: 0
dev.igc.1.mac_stats.mgmt_pkts_drop: 0
dev.igc.1.mac_stats.mgmt_pkts_recvd: 0
dev.igc.1.mac_stats.unsupported_fc_recvd: 0
dev.igc.1.mac_stats.xoff_txd: 0
dev.igc.1.mac_stats.xoff_recvd: 0
dev.igc.1.mac_stats.xon_txd: 0
dev.igc.1.mac_stats.xon_recvd: 0
dev.igc.1.mac_stats.alignment_errs: 0
dev.igc.1.mac_stats.crc_errs: 0
dev.igc.1.mac_stats.recv_errs: 0
dev.igc.1.mac_stats.recv_jabber: 0
dev.igc.1.mac_stats.recv_oversize: 0
dev.igc.1.mac_stats.recv_fragmented: 0
dev.igc.1.mac_stats.recv_undersize: 0
dev.igc.1.mac_stats.recv_no_buff: 0
dev.igc.1.mac_stats.recv_length_errors: 0
dev.igc.1.mac_stats.missed_packets: 0
dev.igc.1.mac_stats.defer_count: 0
dev.igc.1.mac_stats.sequence_errors: 0
dev.igc.1.mac_stats.symbol_errors: 0
dev.igc.1.mac_stats.collision_count: 0
dev.igc.1.mac_stats.late_coll: 0
dev.igc.1.mac_stats.multiple_coll: 0
dev.igc.1.mac_stats.single_coll: 0
dev.igc.1.mac_stats.excess_coll: 0
dev.igc.1.queue_rx_3.rx_irq: 0
dev.igc.1.queue_rx_3.rxd_tail: 733
dev.igc.1.queue_rx_3.rxd_head: 734
dev.igc.1.queue_rx_3.interrupt_rate: 20832
dev.igc.1.queue_rx_2.rx_irq: 0
dev.igc.1.queue_rx_2.rxd_tail: 621
dev.igc.1.queue_rx_2.rxd_head: 622
dev.igc.1.queue_rx_2.interrupt_rate: 20832
dev.igc.1.queue_rx_1.rx_irq: 0
dev.igc.1.queue_rx_1.rxd_tail: 145
dev.igc.1.queue_rx_1.rxd_head: 146
dev.igc.1.queue_rx_1.interrupt_rate: 20832
dev.igc.1.queue_rx_0.rx_irq: 0
dev.igc.1.queue_rx_0.rxd_tail: 808
dev.igc.1.queue_rx_0.rxd_head: 809
dev.igc.1.queue_rx_0.interrupt_rate: 20832
dev.igc.1.queue_tx_3.tx_irq: 0
dev.igc.1.queue_tx_3.txd_tail: 329
dev.igc.1.queue_tx_3.txd_head: 329
dev.igc.1.queue_tx_3.interrupt_rate: 20832
dev.igc.1.queue_tx_2.tx_irq: 0
dev.igc.1.queue_tx_2.txd_tail: 471
dev.igc.1.queue_tx_2.txd_head: 471
dev.igc.1.queue_tx_2.interrupt_rate: 20832
dev.igc.1.queue_tx_1.tx_irq: 0
dev.igc.1.queue_tx_1.txd_tail: 980
dev.igc.1.queue_tx_1.txd_head: 980
dev.igc.1.queue_tx_1.interrupt_rate: 20832
dev.igc.1.queue_tx_0.tx_irq: 0
dev.igc.1.queue_tx_0.txd_tail: 240
dev.igc.1.queue_tx_0.txd_head: 240
dev.igc.1.queue_tx_0.interrupt_rate: 20832
dev.igc.1.fc_low_water: 32752
dev.igc.1.fc_high_water: 32768
dev.igc.1.rx_control: 71335938
dev.igc.1.device_control: 404489793
dev.igc.1.watchdog_timeouts: 0
dev.igc.1.rx_overruns: 0
dev.igc.1.link_irq: 18
dev.igc.1.dropped: 0
dev.igc.1.eee_control: 1
dev.igc.1.tso_tcp_flags_mask_last_segment: 3967
dev.igc.1.tso_tcp_flags_mask_middle_segment: 3958
dev.igc.1.tso_tcp_flags_mask_first_segment: 4086
dev.igc.1.dmac: 0
dev.igc.1.rs_dump: 0
dev.igc.1.reg_dump: General Registers
CTRL 181c0641
STATUS 00780683
CTRL_EXIT 100000c0

Interrupt Registers
ICR 00000000

RX Registers
RCTL 04408002
RDLEN 00004000
RDH 00000329
RDT 00000328
RXDCTL 02040808
RDBAL 76e4c000
RDBAH 00000000

TX Registers
TCTL a503f0fa
TDBAL 76dbc000
TDBAH 00000000
TDLEN 00004000
TDH 000000f0
TDT 000000f0
TXDCTL 0201011f
TDFH 00000000
TDFT 00000000
TDFHS 00000000
TDFPC 00000000


dev.igc.1.fc: 3
dev.igc.1.debug: -1
dev.igc.1.fw_version: EEPROM V2.13-0 eTrack 0x80000286
dev.igc.1.enable_aim: 1
dev.igc.1.nvm: -1
dev.igc.1.iflib.rxq3.rxq_fl0.buf_size: 2048
dev.igc.1.iflib.rxq3.rxq_fl0.credits: 1023
dev.igc.1.iflib.rxq3.rxq_fl0.cidx: 734
dev.igc.1.iflib.rxq3.rxq_fl0.pidx: 733
dev.igc.1.iflib.rxq3.cpu: 3
dev.igc.1.iflib.rxq2.rxq_fl0.buf_size: 2048
dev.igc.1.iflib.rxq2.rxq_fl0.credits: 1023
dev.igc.1.iflib.rxq2.rxq_fl0.cidx: 622
dev.igc.1.iflib.rxq2.rxq_fl0.pidx: 621
dev.igc.1.iflib.rxq2.cpu: 2
dev.igc.1.iflib.rxq1.rxq_fl0.buf_size: 2048
dev.igc.1.iflib.rxq1.rxq_fl0.credits: 1023
dev.igc.1.iflib.rxq1.rxq_fl0.cidx: 146
dev.igc.1.iflib.rxq1.rxq_fl0.pidx: 145
dev.igc.1.iflib.rxq1.cpu: 1
dev.igc.1.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.igc.1.iflib.rxq0.rxq_fl0.credits: 1023
dev.igc.1.iflib.rxq0.rxq_fl0.cidx: 809
dev.igc.1.iflib.rxq0.rxq_fl0.pidx: 808
dev.igc.1.iflib.rxq0.cpu: 0
dev.igc.1.iflib.txq3.r_abdications: 0
dev.igc.1.iflib.txq3.r_restarts: 0
dev.igc.1.iflib.txq3.r_stalls: 0
dev.igc.1.iflib.txq3.r_starts: 7796885
dev.igc.1.iflib.txq3.r_drops: 0
dev.igc.1.iflib.txq3.r_enqueues: 7797973
dev.igc.1.iflib.txq3.ring_state: pidx_head: 1237 pidx_tail: 1237 cidx: 1237 state: IDLE
dev.igc.1.iflib.txq3.txq_cleaned: 7985439
dev.igc.1.iflib.txq3.txq_processed: 7985479
dev.igc.1.iflib.txq3.txq_in_use: 42
dev.igc.1.iflib.txq3.txq_cidx_processed: 327
dev.igc.1.iflib.txq3.txq_cidx: 287
dev.igc.1.iflib.txq3.txq_pidx: 329
dev.igc.1.iflib.txq3.no_tx_dma_setup: 0
dev.igc.1.iflib.txq3.txd_encap_efbig: 0
dev.igc.1.iflib.txq3.tx_map_failed: 0
dev.igc.1.iflib.txq3.no_desc_avail: 0
dev.igc.1.iflib.txq3.mbuf_defrag_failed: 0
dev.igc.1.iflib.txq3.m_pullups: 0
dev.igc.1.iflib.txq3.mbuf_defrag: 0
dev.igc.1.iflib.txq3.cpu: 3
dev.igc.1.iflib.txq2.r_abdications: 0
dev.igc.1.iflib.txq2.r_restarts: 0
dev.igc.1.iflib.txq2.r_stalls: 0
dev.igc.1.iflib.txq2.r_starts: 5389171
dev.igc.1.iflib.txq2.r_drops: 0
dev.igc.1.iflib.txq2.r_enqueues: 5389985
dev.igc.1.iflib.txq2.ring_state: pidx_head: 1697 pidx_tail: 1697 cidx: 1697 state: IDLE
dev.igc.1.iflib.txq2.txq_cleaned: 5407150
dev.igc.1.iflib.txq2.txq_processed: 5407190
dev.igc.1.iflib.txq2.txq_in_use: 41
dev.igc.1.iflib.txq2.txq_cidx_processed: 470
dev.igc.1.iflib.txq2.txq_cidx: 430
dev.igc.1.iflib.txq2.txq_pidx: 471
dev.igc.1.iflib.txq2.no_tx_dma_setup: 0
dev.igc.1.iflib.txq2.txd_encap_efbig: 0
dev.igc.1.iflib.txq2.tx_map_failed: 0
dev.igc.1.iflib.txq2.no_desc_avail: 0
dev.igc.1.iflib.txq2.mbuf_defrag_failed: 0
dev.igc.1.iflib.txq2.m_pullups: 0
dev.igc.1.iflib.txq2.mbuf_defrag: 0
dev.igc.1.iflib.txq2.cpu: 2
dev.igc.1.iflib.txq1.r_abdications: 0
dev.igc.1.iflib.txq1.r_restarts: 0
dev.igc.1.iflib.txq1.r_stalls: 0
dev.igc.1.iflib.txq1.r_starts: 18676639
dev.igc.1.iflib.txq1.r_drops: 0
dev.igc.1.iflib.txq1.r_enqueues: 18677616
dev.igc.1.iflib.txq1.ring_state: pidx_head: 1904 pidx_tail: 1904 cidx: 1904 state: IDLE
dev.igc.1.iflib.txq1.txq_cleaned: 33628074
dev.igc.1.iflib.txq1.txq_processed: 33628114
dev.igc.1.iflib.txq1.txq_in_use: 42
dev.igc.1.iflib.txq1.txq_cidx_processed: 978
dev.igc.1.iflib.txq1.txq_cidx: 938
dev.igc.1.iflib.txq1.txq_pidx: 980
dev.igc.1.iflib.txq1.no_tx_dma_setup: 0
dev.igc.1.iflib.txq1.txd_encap_efbig: 0
dev.igc.1.iflib.txq1.tx_map_failed: 0
dev.igc.1.iflib.txq1.no_desc_avail: 0
dev.igc.1.iflib.txq1.mbuf_defrag_failed: 0
dev.igc.1.iflib.txq1.m_pullups: 0
dev.igc.1.iflib.txq1.mbuf_defrag: 0
dev.igc.1.iflib.txq1.cpu: 1
dev.igc.1.iflib.txq0.r_abdications: 0
dev.igc.1.iflib.txq0.r_restarts: 0
dev.igc.1.iflib.txq0.r_stalls: 0
dev.igc.1.iflib.txq0.r_starts: 6354534
dev.igc.1.iflib.txq0.r_drops: 0
dev.igc.1.iflib.txq0.r_enqueues: 6358392
dev.igc.1.iflib.txq0.ring_state: pidx_head: 1400 pidx_tail: 1400 cidx: 1400 state: IDLE
dev.igc.1.iflib.txq0.txq_cleaned: 6376647
dev.igc.1.iflib.txq0.txq_processed: 6376687
dev.igc.1.iflib.txq0.txq_in_use: 41
dev.igc.1.iflib.txq0.txq_cidx_processed: 239
dev.igc.1.iflib.txq0.txq_cidx: 199
dev.igc.1.iflib.txq0.txq_pidx: 240
dev.igc.1.iflib.txq0.no_tx_dma_setup: 0
dev.igc.1.iflib.txq0.txd_encap_efbig: 0
dev.igc.1.iflib.txq0.tx_map_failed: 0
dev.igc.1.iflib.txq0.no_desc_avail: 0
dev.igc.1.iflib.txq0.mbuf_defrag_failed: 0
dev.igc.1.iflib.txq0.m_pullups: 0
dev.igc.1.iflib.txq0.mbuf_defrag: 0
dev.igc.1.iflib.txq0.cpu: 0
dev.igc.1.iflib.override_nrxds: 0
dev.igc.1.iflib.override_ntxds: 0
dev.igc.1.iflib.allocated_msix_vectors: 5
dev.igc.1.iflib.use_extra_msix_vectors: 0
dev.igc.1.iflib.use_logical_cores: 0
dev.igc.1.iflib.separate_txrx: 0
dev.igc.1.iflib.core_offset: 0
dev.igc.1.iflib.tx_abdicate: 0
dev.igc.1.iflib.rx_budget: 0
dev.igc.1.iflib.disable_msix: 0
dev.igc.1.iflib.override_qs_enable: 0
dev.igc.1.iflib.override_nrxqs: 0
dev.igc.1.iflib.override_ntxqs: 0
dev.igc.1.iflib.driver_version: 1
dev.igc.1.%iommu: rid=0x200
dev.igc.1.%parent: pci2
dev.igc.1.%pnpinfo: vendor=0x8086 device=0x125c subvendor=0x8086 subdevice=0x0000 class=0x020000
dev.igc.1.%location: slot=0 function=0 dbsf=pci0:2:0:0
dev.igc.1.%driver: igc
dev.igc.1.%desc: Intel(R) Ethernet Controller I226-V