Messages

I second that. Different map file types should be mentioned in the tutorial.

I even use type "reg", because sometimes, I use HAproxy as a first-level reverse proxy for CG-NAT backends that are only reachable via IPv4.
I want to use that for *.domain.xyz (DOMAIN_dyndns), while keeping another local backend (DOMAIN_backend) for www.domain.xyz and domain.xyz.

This does not even work with map type "beg" and ".domain.xyz", because the first dot will be ignored. You have to use a regex map with a setup like:

(?i)^www\.domain\.xyz$            DOMAIN_backend
(?i)^.+\.domain\.xyz$             DOMAIN_dyndns
(?i)^domain\.xyz$                 DOMAIN_backend

Note the order, which is relevant for "reg" maps.

[cannot]

Would you admit that it cannot be the OS drivers when the card is not detected in the BIOS before the OS even boots (which was the case)?

The cards definitely sometimes were not detected on the PCIe bus, which I also told in the Computerbase thread. Also, I linked several other reports about those instabilities here: https://www.computerbase.de/forum/threads/erfahrungsbericht-realtek-rtl8126-rtl8127-instabilitaeten-auf-x570-pcie-4-0.2265596/post-31314530

Also, I noticed corrupted data, which destroyed my Windows installation a few times and even became noticeable as data errors on a restore, so I definitely can say there was a hardware problem. Had it been on the ethernet wire, it would have got detected and corrected, but there is no such correction on the PCIe bus.

Thus, I know for a fact that there were signaling problems between my X570 mainboard and two different specimens of modern Realtek adapters (one 8126 and one 8127). There are reports of other people experiencing similar results, which is not to say that it cannot be the mainboard that was at fault. Yet, these problems were not present with any other brand NIC that I tried.

Potentially, there are EMI problem with higher PCIe rates, but whatever the case, the drivers played no role in this.

That is why I answered: the title suggests a general problem, while it seems very specific (maybe you consider changing the title).

If you want to investigate, you can look at the command line that is called underneath (with Proxmox, that is possible). It will be derived from the options you choose and may show what causes it. I had to explicitely use the "args" line above to even enable it.

I do not use Hyper-V extensions because they are strictly needed only for Windows guests and when I tried, I even noticed a slightly higher CPU load with Hyper-V enabled for OpnSense.

Perhaps noteworthy: This does not happen on all KVM-based hypervisors and not under all circumstances, so this does not look like a generic problem.

E.g., with Proxmox, you normally do not get Hyper-V enlightenments unless you specifically ask to do so via "args: -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" in the VM configuration.

When that feature is absent, "sysctl kern.timecounter.hardware" will give you "kvmclock", which does not have this problem at all.
Even when I specifically requested the Hyper-V feature, it gave "Hyper-V-TSC", but:

Code Select Expand

# sysctl kern.eventtimer.timer
kern.eventtimer.timer: LAPIC
# sysctl kern.eventtimer.et.LAPIC
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 500015996
kern.eventtimer.et.LAPIC.flags: 7

Which seems correct and I saw no slowdown, either.

I did not get HPET detected with 26.7, so this seems to be specific to the supplied KVM flags or what the underlying hypervisor uses as defaults.

When you read the thread you will find that it is an incompatibility of Realtek 8126 / 8127 devices with X570 chipsets that cannot be healed by drivers, since the device is not even detected. It is a hardware incompatibility.

It seems like with the new kernel, there are now race conditions with the early microcode loading under certain conditions. For me, it was the presence of vidconsole (but I did not check if that problem would prevail if the microcode update was disabled - I still have it running).

I think that the recent kernel changes of memory layout and such is responsible for that, although at times, it even surfaced earlier than with 26.7: https://github.com/opnsense/ports/issues/230. Whatever the specific condition is, it seems that the early loading is always part of the problem, so disabling that should fix it.

Alas, since many mainboard manufacturers do not supply microcode updates in the BIOS and it seems there is no way of doing the microcode update even earlier in FreeBSD, there is currently no fix for this situation other than to disable os-cpu-microcode-intel.

P.S.: Doing the update later via cpuctl is a tough decision, because:

1. There is code in the kernel that detects certain features at boot, which might change because of the update and cause problems.
2. Race conditions may occur when the update is being applied while other code executes, which is why the preferred way was early updates in the first place.

Yes.

[before]

There is an even bigger problem behind that with 26.7.:

I upgraded and after reboot, saw this on my stalled boot console:

You cannot view this attachment.

The "Witty Woodpecker" part is just because that is only updated after a sucessful reboot. But you can see the error message "console vidconsole is unavailable". This now keeps the system from properly booting - it did not do that ever before.

My console settings were "vt" = enabled, primary console = efi, secondary console = VGA console. I did this in case I ever had to boot the disk on a legacy system, while my current one is UEFI. With 26.7, this setting kept the system from even booting. What is worse is that you cannot even use boot environments - at least it did not work for me.

I had to switch the secondary console to serial or none in order to be up and running again. IDK if the vidconsole driver would work on its own. For safety reasons, I switched to EFI console on another UEFI system before doing the upgrade.

This is noteworthy for people who update from remote: Change the console setting before the update!

In rare cases it might also be caused by the counterpart because of slight incompatibilities (or settings, like different flow control or speed/duplex).

That really looks like link flapping to me which may indicate a flaky connection to your modem or ONT.

Queries for what names? Also, there are some settings on "Services: Unbound DNS: Advanced" that should NOT be applied, like "Strict QNAME Minimization" - there are warnings in their help texts.

I only assumed that the reset code was added later to Linux because it was missing in the FreeBSD driver, which otherwise looked like a port. I really did not do any software archeology. :-)

If that problem is handled in iflib in FreeBSD, it is not really needed in the hardware-specific driver.

If Intel knew that problem existed from the get-go, which would be implied if the hangup fix was there from the first commit, they should have better fixed it in hardware, though.

When I had the problem on my Minisforum MS-01, it did not resolve by a NIC firmware upgrade. Not even disabling global ASPM helped (it did on some china boxes I have).

The problem went away after a BIOS update from Minisforum. So I figure that there are conditions where micro-sleeps trigger a hangup. ASPM potentially allows for those sleep states. It might be the case that Intel also tried to fix it in NIC firmware, however, there are several chip revisions out for I226V and some may have a hardware idiosyncrasy that causes this which cannot be healed via NIC firmware.

Intel seems to have "fixed" the hang problem by detecting it in the Linux driver and issuing a reset - FreeBSD has only now picked up that change in the OS driver (that was what my bug report was about).

So:

1. There are chip revisions (e.g., the latest ones have a smaller structure width and consume less power). You can see that via " pciconf -lv | grep -B3 igc", look at the "rev" value (mine is 0x04 on my current china box, I have others with 0x03). I think that early versions had a hardware problem that cause hangups. I did not investigate up to what revision these problems exist. I also do not remember which revision was on my Minisforum MS-01.

2. On some boxes, disabling ASPM globally via BIOS settings or tuneable does fix the problem, so you should try this.

3. On some boxes, you can get BIOS upgrades (e.g. Protectli, Minisforum).

4. My experience is that if the problems persist, updating the NIC firmare does not help.

5. In the future, we will have a FreeBSD driver with Intel's workaround that handles the remaining problems, the fixed kernel can be tested right now.

Not happening here, neither from a connected client nor from OpnSense itself. So this must be specific to some setups.

Do you have altered MTUs, like with PPPoE or do you use MSS clamping?

Since you deinstalled Zenarmor, did you first enable some kind of hardware offloading, like LRO, to improve performance and forgot to disable it again?

If the default deny rule catches traffic, it is because nothing else allowed the traffic. For TCP, that could be out-of-state traffic that is not caught by an existing rule, which is most often the cause somebody wonders why that rule does not apply.

However, this is UDP traffic, which is stateless, so I figure you have no rule to allow that traffic. For convenience, there is a default "allow any" rule created for the first LAN and for that (V)LAN only. Thus, if you do not allow DNS traffic to your firewall on any other (V)LAN, it will be caught by the default deny rule and thus be blocked.

You may not have noticed yet, because many browsers do not even use DNS on port 53 any more (doing DoT or DoH instead), such that DNS worked despite port 53 being blocked. Maybe you have a rule allowing internet access for your second VLAN, such that 8.8.8.8 can be reached.

P.S.: You created the "noise" all by yourself, because you enabled logging for the default deny rule. The whole purpose of the default deny rule is that you can log what it does - otherwise, you would not need it. While enabling logging on that can be helpful to diagnose something going wrong, blocking packets is the whole purpose of a firewall. Normally, you don't watch it doing its job.