Messages

Naja, man kann bei Deciso ins Regal greifen, um eine spezialisierte Appliance zu bekommen, die eben 7.9 GBps Threat Protection Durchsatz hat bei gleichzeitig niedrigem Stromverbrauch (DEC3920) oder irgendwas im Server-Bereich als 19"-Rack-Lösung (dann aber Performance) oder selbst bauen auf Basis aktueller High-End-Prozessoren. Die meisten Desktop-Plattformen könnten dann eben kein ECC-RAM, alternativ brauchst Du Server-CPUs und Mainboards, die fast ebenso teuer sind wie fertige 19"-Server von Dell.

Was auch in Frage käme, wäre so etwas: https://minisforumpc.eu/de/products/ms-a2-mini-pc, da reichen eventuell auch die kleineren CPUs, da die Single-Thread-Leistung entscheidend ist.

Ich zweifele aber ohnehin daran, dass das wesentlich günstiger als mit einer DEC3920 geht. Selbst die kann z.B. bei Wireguard nur ca. 4 Gbps. Es gibt noch fettere Lösungen, ich habe aber damit keine Erfahrungen - bei mir reichen 10 Gbps geroutet im Intranet und 2 Gbps im WAN. Außerdem nutze ich keine DPI-Lösungen.

Je nach Größe des Regelsatzes wirst Du da um Enterprise Hardware kaum herumkommen, wenn Du den Speed wirklich ausnutzen willst. Suricata braucht hohe Single-Core Performance, mit Mehrkern-Server-Prozessoren mit niedrigen Frequenzen kommst Du da nicht weit. Das wird laut und energiefressend.

Der N355 bietet keinen sehr hohen Grundtakt, wird das bei größeren Regelsätzen einbrechen.

See: https://forum.opnsense.org/index.php?topic=52441.0

See: https://forum.opnsense.org/index.php?topic=52441.0

Bei DPI zählt CPU-Performance. Am Rande bemerkt, hängt das davon ab, was Du dafür einsetzen willst. Soweit mir bekannt, läuft Zenarmor in der Gratisversion nur single-threaded, was die Performance per se einschränkt - ich glaube kaum, dass irgendeine CPU das mit 10 Gpbs schafft.

Für reines Firewalling wäre ein N355 ausreichend, das packt sogar ein N1x0.

Auch hier gilt: You gotta pay to play!

Im Grunde ist es doch ganz einfach:

Welcher DNS-Server ist auf Deinem VPN-Client konfiguriert (Du hast übrigens nicht die DHCP-Einstellungen gezeigt)? Wenn es Deine OpnSense und nicht der Mullvad-VPN-Server ist, müsstest Du eine Regel auf der OpnSense haben, die für die VPN-Clients den DNS auf Mullvad umbiegt. Das ist ziemlich komplex, also würde ich entweder per DHCP diesen Clients den Mullvad DNS mitteilen oder diese Clients statisch konfigurieren.

@franco I rather appreciate that upstream change - what are we looking for in the WG context, specifically?

The patch was about the issue that the peer was not initialized during startup. The reresolve script only changes parameters in the Wireguard connection, it does not restart from scratch.

A Wireguard connection can have a peer (like with outbound s2s or a c2s client), but it does not have to (like with inbound s2s or a c2s server), so the "allowedips" attribute can become empty during WG startup. It definitely will be when the DNS cannot be resolved on startup and the old script did not handle that case yet.

Essentially, the old script handled the case of a changed DNS entry (by reresolving the peer), but it could not create the full connection if the peer was not resolvable in the start phase due to timing conditions (it set the peer, but allowedips was still empty).

As I said, I had two cards from different vendors. When you look at the RTL8126/7 pinouts, you will find that the chip is directly connected to the PCIe lanes, giving the PCB manufacturer little to vary, except for routing the connections. Of course there may be problems with power filtering and such.

BTW: I do not have those cards any more, as the ASUS NIC - which is based on another chip - works nicely, as discussed.

A driver problem that is not cleared up during the next boot would explain the initialisation failures, but not the data corruption - the latter strongly suggests a signaling problem. I am using Occam's Razor here because a signaling problem explains both symptoms. I did not investigate any further because I lack the exquipment and skills to verify the finding and it simply was not worth the effort for me.

Das bedeutet, dass die DNS-Anfragen von Deinem Client nicht den Weg über die VPN-Verbindung nehmen, sondern vermutlich über den DNS-Server auf Deiner OpnSense. Du schreibst nicht, was das für ein Client ist (OS, Browser, welche DNS-Einstellungen nutzt der?).

Offenbar hast Du das VPN auf der OpnSense und nicht auf dem Client eingerichtet.

Wenn Du beispielsweise nur bestimmte Clients oder VLANs über Dein Mullvad-VPN-Gateway routest, hast Du eventuell die Routen für lokale Netze trotzdem ohne VPN konfiguriert, weil Du z.B. per SSH, RDP oder sonstiges diese VPN-Clients aus Deinem LAN kontroliieren willst und sie direkt erreichen musst. Das erlaubt natürlich die Nutzung der OpnSense als DNS-Server. Du musst auf den VPN-Clients "irgendwie" dafür sorgen, dass sie die Mullvad-DNS-Server nutzen, z.B. per DHCP, lokaler Konfiguration oder per DNS-Umleitung. Das wirkt allerdings eventuell nur begrenzt, falls z.B. im Browser DoT oder DoH eingestellt ist (obwohl diese Adressen eher nicht in einem lokalen Bereich liegen und daher wohl per VPN geroutet werden).

Ein Fehler könnte auch darin liegen, dass Du Regeln verwendest, die jeden DNS-Request per NAT auf Deine OpnSense umleiten - dann könnte sonstwas auf dem VPN-CLient eingestellt sein, jeder Request wird dann auf die OpnSense geleitet, was Du bei VPN-Clients nicht willst.

I second that. Different map file types should be mentioned in the tutorial.

I even use type "reg", because sometimes, I use HAproxy as a first-level reverse proxy for CG-NAT backends that are only reachable via IPv4.
I want to use that for *.domain.xyz (DOMAIN_dyndns), while keeping another local backend (DOMAIN_backend) for www.domain.xyz and domain.xyz.

This does not even work with map type "beg" and ".domain.xyz", because the first dot will be ignored. You have to use a regex map with a setup like:

(?i)^www\.domain\.xyz$            DOMAIN_backend
(?i)^.+\.domain\.xyz$             DOMAIN_dyndns
(?i)^domain\.xyz$                 DOMAIN_backend

Note the order, which is relevant for "reg" maps.

[cannot]

Would you admit that it cannot be the OS drivers when the card is not detected in the BIOS before the OS even boots (which was the case)?

The cards definitely sometimes were not detected on the PCIe bus, which I also told in the Computerbase thread. Also, I linked several other reports about those instabilities here: https://www.computerbase.de/forum/threads/erfahrungsbericht-realtek-rtl8126-rtl8127-instabilitaeten-auf-x570-pcie-4-0.2265596/post-31314530

Also, I noticed corrupted data, which destroyed my Windows installation a few times and even became noticeable as data errors on a restore, so I definitely can say there was a hardware problem. Had it been on the ethernet wire, it would have got detected and corrected, but there is no such correction on the PCIe bus.

Thus, I know for a fact that there were signaling problems between my X570 mainboard and two different specimens of modern Realtek adapters (one 8126 and one 8127). There are reports of other people experiencing similar results, which is not to say that it cannot be the mainboard that was at fault. Yet, these problems were not present with any other brand NIC that I tried.

Potentially, there are EMI problem with higher PCIe rates, but whatever the case, the drivers played no role in this.

That is why I answered: the title suggests a general problem, while it seems very specific (maybe you consider changing the title).

If you want to investigate, you can look at the command line that is called underneath (with Proxmox, that is possible). It will be derived from the options you choose and may show what causes it. I had to explicitely use the "args" line above to even enable it.

I do not use Hyper-V extensions because they are strictly needed only for Windows guests and when I tried, I even noticed a slightly higher CPU load with Hyper-V enabled for OpnSense.

Perhaps noteworthy: This does not happen on all KVM-based hypervisors and not under all circumstances, so this does not look like a generic problem.

E.g., with Proxmox, you normally do not get Hyper-V enlightenments unless you specifically ask to do so via "args: -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time" in the VM configuration.

When that feature is absent, "sysctl kern.timecounter.hardware" will give you "kvmclock", which does not have this problem at all.
Even when I specifically requested the Hyper-V feature, it gave "Hyper-V-TSC", but:

Code Select Expand

# sysctl kern.eventtimer.timer
kern.eventtimer.timer: LAPIC
# sysctl kern.eventtimer.et.LAPIC
kern.eventtimer.et.LAPIC.quality: 600
kern.eventtimer.et.LAPIC.frequency: 500015996
kern.eventtimer.et.LAPIC.flags: 7

Which seems correct and I saw no slowdown, either.

I did not get HPET detected with 26.7, so this seems to be specific to the supplied KVM flags or what the underlying hypervisor uses as defaults.

When you read the thread you will find that it is an incompatibility of Realtek 8126 / 8127 devices with X570 chipsets that cannot be healed by drivers, since the device is not even detected. It is a hardware incompatibility.

It seems like with the new kernel, there are now race conditions with the early microcode loading under certain conditions. For me, it was the presence of vidconsole (but I did not check if that problem would prevail if the microcode update was disabled - I still have it running).

I think that the recent kernel changes of memory layout and such is responsible for that, although at times, it even surfaced earlier than with 26.7: https://github.com/opnsense/ports/issues/230. Whatever the specific condition is, it seems that the early loading is always part of the problem, so disabling that should fix it.

Alas, since many mainboard manufacturers do not supply microcode updates in the BIOS and it seems there is no way of doing the microcode update even earlier in FreeBSD, there is currently no fix for this situation other than to disable os-cpu-microcode-intel.

P.S.: Doing the update later via cpuctl is a tough decision, because:

1. There is code in the kernel that detects certain features at boot, which might change because of the update and cause problems.
2. Race conditions may occur when the update is being applied while other code executes, which is why the preferred way was early updates in the first place.