Messages

I tested this on all the Deciso appliance variants running the E810 with 10G-SR and 25G-SR fiber optics. All link and communicate just fine with the DDP package loaded. The link may take a few cycles to come up, but this shouldn't be a problem.

Reading all of the reports in here, as well as https://forum.opnsense.org/index.php?topic=52405 I'm seeing quite some variablity in reported firmware version and indeed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296989 also reports versions >= 4.60. The only report in the forum post concerning a DEC4280 was on a non-standard firmware version (DEC devices run NVM 4.30), which is not something that's supported.

You have to make your browser window smaller like it is on mobile devices to reproduce this scrolling issue. I had it also in Safari on my Mac but only when the browser window is small enought. I've experienced it at least for the new firewall rules list.

This was indeed the trigger, thanks for reporting.

It should be fixed with

# opnsense-patch https://github.com/opnsense/core/commit/6e3cda346c858d0b070b77439bdd322d6a0db7c1

Assuming the other patch Franco mentioned wasn't reverted.

Make sure the proposals are actually the same for IKE (the connection or phase 1). Seems like it fails there instead of your phase 2, which use a separate set of proposals.

Well, drivers don't log that much data in general. Another test kernel has been made available which includes disabling ASPM, a cleanup of some suspicious spots and more verbose logging (but not disruptive). If anyone is on version 26.1.x and wants to test:

# opnsense-update -zkr 26.1.11-igc2

and reboot.

Again, the timestamp of a failure and the relevant "System -> Log Files -> General" snippet of the same time is most relevant here.

Edit: kernel for 26.7 is up as well:

# opnsense-update -zkr 26.7-igc2

A kernel patch related to your issue has been developed, assuming you're on 26.1.11_6, you can install it with:

# opnsense-update -zkr 26.1.11-igc

and reboot.

Otherwise if you're on 26.7:

# opnsense-update -zkr 26.7-igc

Relevant events should be logged in System -> Log Files -> General. Can you share these here?

What's the status of 'igc' related changes in the kernel shipped with 26.7?  Do we still need to patch post upgrade?

The kernel patches have not been shipped. Neither patch confirmed or solved anything, there's also no custom kernel for these patches on the 26.7 ABI.

Unfortunately the -igc_aspm kernel didn't resolve my issue. It happened again this morning, and as usual bouncing the LAN interface with ifconfig down/up resolved it. There's nothing interesting in dmesg and the only recent entries in the syslog relate to a DHCP renewal for the WAN interface (which is not something I've seen in other instances).

One more thing to try before I'm going to suggest a more complicated troubleshooting setup to help classify the issue:

Disable flow control on all igc ports through System -> Settings -> Tunables. Tunable: "dev.igc.X.fc", Value: "0" (where X is the port). For 4 ports I'd expect 4 entries.

Intel seems to have "fixed" the hang problem by detecting it in the Linux driver and issuing a reset - FreeBSD has only now picked up that change in the OS driver (that was what my bug report was about).

"TX hangs" have been around forever, which is why watchdog timeouts have been baked into iflib from the get-go (but broke somewhere along the way 5 years ago, the logic wasn't missing), the same goes for the Linux driver - the reset logic has been in there since the first commit. Nothing special about it. However, it being broken definitely hinted at other platforms only reporting "micro freezes" instead of full freezes.

https://github.com/freebsd/freebsd-src/pull/2318

Is this patch included in OPNsense v26.7?

It will be, though feedback is the important bit here.

I think the following patch should help:

# opnsense-patch https://github.com/opnsense/core/commit/0e1e8cfa163bb27b01e33cc0d34fde2eac512e4e

Seems to have been an ordering issue in rendered content vs resize trigger.

Hold up...

Code Select Expand

dev.igc.0.fw_version: EEPROM V2.17-0 eTrack 0x80000303


This is before any of the fixes Intel pushed for related stuff.. The baseline should be 2.25.

Ok, reverted all tunables to defaults this morning and installed this kernel. Didn't have to wait long. The issue just popped up again. Again it was a down/up on the LAN interface (igc0) that resolved it. I do not see any references to watchdog timeouts, or indeed anything other that my work this morning and the reset I just did in dmesg or syslog. Here's the sysctl output on the interface while the issue was happening:

Ok, getting closer..

Can you try:

Code Select Expand

# opnsense-update -zkr 26.1.11-igc_aspm

and reboot?

Ref: https://github.com/freebsd/freebsd-src/pull/2318

I just purchased DEC3940 and it shows weird values about processor: AMD Ryzen Embedded V3C18 8-Core Processor (4 cores, 8 threads)

V3C18 should have 8-cores and that's what Deciso also says in their tech specs about this model. Does anyone have similar model and what it says about CPU?

Where did you see (4 cores, 8 threads)?

# sysctl hw.model
hw.model: AMD Ryzen Embedded V3C18 8-Core Processor

# sysctl kern.smp.cpus
kern.smp.cpus: 8

# sysctl kern.smp.cores
kern.smp.cores: 8

If it's this issue, you should see "watchdog timeout" messages appearing in your dmesg/system log. After this, an automatic reset should recover connectivity. If this happens, can you share these logs?

Yeah, a lot of the stuff I've read indicates I should see interfaces flapping in the logs, but I haven't actually seen any evidence of that. There's no incidence of "watchdog" or "timeout" anywhere in dmesg or the month or so of system logs that have built up, and the only link state changes I see appear to correspond to when I uncheck/check the box and apply. So it's possible this is something else entirely.

I wasn't referring to instances of "watchdog" or "timeout" in your historic logs, these messages should pop up as a result of installing the custom kernel. If this is the case, I'd like to know if you still have to intervene manually.

For what it's worth, no report I've read so far has mentioned any link flapping, this seems to happen entirely without link state changes, but it does seem to happen with autonegotiation speed degradation, hence my questions to establish a baseline (which is why the sysctl output is important as well).

Cheers,
Stephan

From the docs:

Since log lines are stored on the system without an exact match to the rule in question, we do need to translate the sequence in the file back to the rule definition stored in the system. Due to this fact, the information is less accurate historically if the firewall was reconfigured. (labels may be incorrect when looking at older data)

https://docs.opnsense.org/manual/logging_firewall.html#live-view