Messages

Well, drivers don't log that much data in general. Another test kernel has been made available which includes disabling ASPM, a cleanup of some suspicious spots and more verbose logging (but not disruptive). If anyone is on version 26.1.x and wants to test:

# opnsense-update -zkr 26.1.11-igc2

and reboot.

Again, the timestamp of a failure and the relevant "System -> Log Files -> General" snippet of the same time is most relevant here.

Edit: kernel for 26.7 is up as well:

# opnsense-update -zkr 26.7-igc2

A kernel patch related to your issue has been developed, assuming you're on 26.1.11_6, you can install it with:

# opnsense-update -zkr 26.1.11-igc

and reboot.

Otherwise if you're on 26.7:

# opnsense-update -zkr 26.7-igc

Relevant events should be logged in System -> Log Files -> General. Can you share these here?

What's the status of 'igc' related changes in the kernel shipped with 26.7?  Do we still need to patch post upgrade?

The kernel patches have not been shipped. Neither patch confirmed or solved anything, there's also no custom kernel for these patches on the 26.7 ABI.

Unfortunately the -igc_aspm kernel didn't resolve my issue. It happened again this morning, and as usual bouncing the LAN interface with ifconfig down/up resolved it. There's nothing interesting in dmesg and the only recent entries in the syslog relate to a DHCP renewal for the WAN interface (which is not something I've seen in other instances).

One more thing to try before I'm going to suggest a more complicated troubleshooting setup to help classify the issue:

Disable flow control on all igc ports through System -> Settings -> Tunables. Tunable: "dev.igc.X.fc", Value: "0" (where X is the port). For 4 ports I'd expect 4 entries.

Intel seems to have "fixed" the hang problem by detecting it in the Linux driver and issuing a reset - FreeBSD has only now picked up that change in the OS driver (that was what my bug report was about).

"TX hangs" have been around forever, which is why watchdog timeouts have been baked into iflib from the get-go (but broke somewhere along the way 5 years ago, the logic wasn't missing), the same goes for the Linux driver - the reset logic has been in there since the first commit. Nothing special about it. However, it being broken definitely hinted at other platforms only reporting "micro freezes" instead of full freezes.

https://github.com/freebsd/freebsd-src/pull/2318

Is this patch included in OPNsense v26.7?

It will be, though feedback is the important bit here.

I think the following patch should help:

# opnsense-patch https://github.com/opnsense/core/commit/0e1e8cfa163bb27b01e33cc0d34fde2eac512e4e

Seems to have been an ordering issue in rendered content vs resize trigger.

Hold up...

Code Select Expand

dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303


This is before any of the fixes Intel pushed for related stuff.. The baseline should be 2.25.

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Ok, getting closer..

Can you try:

Code Select Expand

# opnsense-update -zkr 26.1.11-igc_aspm

and reboot?

Ref: https://github.com/freebsd/freebsd-src/pull/2318

I just purchased DEC3940 and it shows weird values about processor: AMD Ryzen Embedded V3C18 8-Core Processor (4 cores, 8 threads)

V3C18 should have 8-cores and that's what Deciso also says in their tech specs about this model. Does anyone have similar model and what it says about CPU?

Where did you see (4 cores, 8 threads)?

# sysctl hw.model
hw.model: AMD Ryzen Embedded V3C18 8-Core Processor

# sysctl kern.smp.cpus
kern.smp.cpus: 8

# sysctl kern.smp.cores
kern.smp.cores: 8

If it's this issue, you should see "watchdog timeout" messages appearing in your dmesg/system log. After this, an automatic reset should recover connectivity. If this happens, can you share these logs?

Yeah, a lot of the stuff I've read indicates I should see interfaces flapping in the logs, but I haven't actually seen any evidence of that. There's no incidence of "watchdog" or "timeout" anywhere in dmesg or the month or so of system logs that have built up, and the only link state changes I see appear to correspond to when I uncheck/check the box and apply. So it's possible this is something else entirely.

I wasn't referring to instances of "watchdog" or "timeout" in your historic logs, these messages should pop up as a result of installing the custom kernel. If this is the case, I'd like to know if you still have to intervene manually.

For what it's worth, no report I've read so far has mentioned any link flapping, this seems to happen entirely without link state changes, but it does seem to happen with autonegotiation speed degradation, hence my questions to establish a baseline (which is why the sysctl output is important as well).

Cheers,
Stephan

From the docs:

Since log lines are stored on the system without an exact match to the rule in question, we do need to translate the sequence in the file back to the rule definition stored in the system. Due to this fact, the information is less accurate historically if the firewall was reconfigured. (labels may be incorrect when looking at older data)

https://docs.opnsense.org/manual/logging_firewall.html#live-view

And I forgot to ask, since you mention that toggling offloading fixes it,

does

Code Select Expand

# ifconfig igcX down && ifconfig igcX up
also fix it?

Cheers,
Stephan

Hi there,

While having a look at this issue, I noticed a potential bug in the iflib code making an automatic reset in case of a TX hang impossible, a custom kernel has been published which resolves this (though likely not the final patch version). Would you mind installing this kernel to see if this changes anything about the issue?

Code Select Expand

# opnsense-update -zkr 26.1.10-iflib
The commit in question is https://github.com/opnsense/src/commit/8dd26e6351d72a53fab5d47a16d053d5f8648353.

If it's this issue, you should see "watchdog timeout" messages appearing in your dmesg/system log. After this, an automatic reset should recover connectivity. If this happens, can you share these logs?

Your description of the issue sounds similar to others, however, there are still a lot of gaps to fill. Most notably, do you always need manual intervention to fix the issue? or does it recover on its own? Is it always the same igc interface? What is the auto-negotiated link state at the time of failure (# ifconfig igcX)?  If there's no auto-negotiation, what link speed did you set it to?

Also, and perhaps most importantly, can you share a snapshot of

Code Select Expand

# sysctl dev.igc.X (where X is the affected interface) after the failure?

Lastly, please do these tests with all default tunables. As far as I know, dev.igc.0.eee_control=0 will *enable* EEE.

Cheers,
Stephan

I am curious to see what would be so busy then given that the first debug level shouldn't spew so much data out that it fully utilizes a CPU.

Same original boot problem when linked on ax1 after a reinstallation?