Periodic NIC issues (?) with Protectli Vault, Intel i226-V

[BrandyWine]

EEE is disabled. It's not an ASPM or EEE issue.

NVM 2.32 fixes a lane swapping issue, which can present issues like link working ok now and then fails later.

NVM 2.32 - Fixes an issue in MDI lane swap handling that caused the P (positive) and N (negative) signal polarities to be swapped incorrectly.

The buggy lane swapping can occur for numerous reasons.

Hard to follow which user has what NVM, but the base config needs to be i226 with latest NVM.

[tuto2]

What's the status of 'igc' related changes in the kernel shipped with 26.7?  Do we still need to patch post upgrade?

The kernel patches have not been shipped. Neither patch confirmed or solved anything, there's also no custom kernel for these patches on the 26.7 ABI.

[fornax]

Ok, turning off flow control and swapping the LAN-side cable didn't change anything. As we're 5 pages in and have multiple setups being talked about, I'm going to sort of "reboot" the OP:

Symptoms are one or more of the following:

• Games (particularly GTA Online, but I suspect I also had an issue with the online features of Death Stranding at one point) unable to contact servers.
• Sites (most or all) not loading on first attempt, but 10-20 seconds later retrying is successful.
• DHCP stops working. Something I've found since I started this thread is that it seems to happen on some VLANs but not all at once, and the VLANs that do/don't work aren't the same in every incident.
• There are NO link flaps noted in dmesg/syslog other than the ones I force to fix the issue.

The only thing that ties these issues together is that they're all fixed by bouncing the firewall's LAN interface (ifconfig igc0 down && ifconfig igc0 up).

Current state:
OPNSense was last updated 7/12 (version 26.1.11_6). Kernel is the -igc_aspm kernel first mentioned in post #17. i226-V NVM version on both interfaces is 2.32. LAN link is statically set to 1000Mbps full duplex, FC off on both ends. The only tunables set are dev.igc.0.fc=0 and dev.igc.1.fc=0.

Things I've tried so far in this thread:

• Various tunables mentioned in the OP.
• The -iflib kernel mentioned in post #2 and the -igc_aspm kernel.
• Swapping the LAN-side cable.
• Updating the NVM from stock 2.17 to current 2.32.

Currently the system is experiencing the issue in that I'm getting the usual error preventing me from playing GTA Online and DHCP is not working on one VLAN. I'm going to leave it in this state for a while to troubleshoot since most everything else is still working. If anyone has any other suggestions, I'm all ears. Thanks for the guidance so far.

Oh, and I've had continuous pings running from the firewall both to the LAN and WAN, and both are still happily plugging away with 100% return.

[fornax]

Well, to complicate matters it looks like there's a separate issue with dnsmasq. Although in the past I've definitely run into VLANs losing DHCP which was solved with an interface bounce, this particular incident appears to be something else. It was not solved by bouncing the interface. (dnsmasq is throwing "no address range available for DHCP request via vlan01" despite vlan01 being in the interfaces list and having a range assigned.)

Still open to further suggestions, but I'm now leaning toward starting from scratch with a reinstall of OPNSense.

[tuto2]

A kernel patch related to your issue has been developed, assuming you're on 26.1.11_6, you can install it with:

# opnsense-update -zkr 26.1.11-igc

and reboot.

Otherwise if you're on 26.7:

# opnsense-update -zkr 26.7-igc

Relevant events should be logged in System -> Log Files -> General. Can you share these here?

[nero355]

I'm going to sort of "reboot" the OP:

Symptoms are one or more of the following:

• Games (particularly GTA Online, but I suspect I also had an issue with the online features of Death Stranding at one point) unable to contact servers.

Currently the system is experiencing the issue in that I'm getting the usual error preventing me from playing GTA Online

May I suggest not involving this game into these issues ?

My connection has always been working just fine across multiple Routers and that stupid P2P game still finds a way to spit out similar issues to yours for some dumb reason... :(

And now that they have released the new DLC a couple of days ago I bet the issues are even worse than before !!

[Lucid1010]

# opnsense-update -zkr 26.7-igc

Will the official patch be included in the next update?

[franco]

Not until we know what's going on.

[BrandyWine]

Symptoms are one or more of the following:

• Games (particularly GTA Online, but I suspect I also had an issue with the online features of Death Stranding at one point) unable to contact servers.
• Sites (most or all) not loading on first attempt, but 10-20 seconds later retrying is successful.
• DHCP stops working. Something I've found since I started this thread is that it seems to happen on some VLANs but not all at once, and the VLANs that do/don't work aren't the same in every incident.
• There are NO link flaps noted in dmesg/syslog other than the ones I force to fix the issue.

[/list]

The only thing that ties these issues together is that they're all fixed by bouncing the firewall's LAN interface (ifconfig igc0 down && ifconfig igc0 up).

Well, that's poor analysis as-is.
No indicators if your issue is WAN or LAN side.
DHCP stops working, so just for the LAN side?
VLANs? How do you mean? Two i226's and VLAN's on LAN side, so you run .1q or your VLAN's are L3 ?

If there's no indicator that the i226 link goes down, then perhaps it is working normally.

Can you SSH into the FW using a non i226 iface? I suggest you do that and start monitoring stuff. Run a ping to the FW LAN IP address, then wait for the no-traffic issue to arise, what does the ping show?

Start a 2nd ssh session, then
dmesg -w | grep -E 'igc'

see if any kernel messages arrive when your issue happens.

[fornax]

A kernel patch related to your issue has been developed, assuming you're on 26.1.11_6, you can install it with:

# opnsense-update -zkr 26.1.11-igc

Sure, I'll install that now. What's the change(s) in this one?

May I suggest not involving this game into these issues ?

Yeah, I know the game is buggy as hell. If it was the only issue I was having I wouldn't be here. But it's a useful canary; as I usually check into the game fairly early, it's often my first indication that something's going wrong. And the fact that bouncing the LAN interface on the firewall makes it immediately start working again (and the fact that I never had these issues before switching to the new firewall) leads me to believe this particular issue isn't the game's fault.

No indicators if your issue is WAN or LAN side.

I haven't dug too much into the WAN side because only bouncing the LAN interface has any effect. I've tried bouncing WAN first on a few occasions and it never changes anything.

DHCP stops working, so just for the LAN side?

Yes, the firewall's DHCP client works just fine, it's the LAN-side clients on one or more VLANs that stop working.

VLANs? How do you mean? Two i226's and VLAN's on LAN side, so you run .1q or your VLAN's are L3 ?

Yes, 802.1q VLANs on the LAN side only.

Can you SSH into the FW using a non i226 iface? I suggest you do that and start monitoring stuff. Run a ping to the FW LAN IP address, then wait for the no-traffic issue to arise, what does the ping show?

Start a 2nd ssh session, then
dmesg -w | grep -E 'igc'

see if any kernel messages arrive when your issue happens.

The firewall has just the two i226-Vs. I could set it up to run in the background like the previous one, though. FreeBSD's dmesg doesn't appear to have a follow option, but neither dmesg nor the syslog have never shown any indication that they see anything wrong anyway.