Messages

The thing is, such N1x0 units were like 300€ complete with 16 GByte RAM and 256 GByte SSD a year ago. Now they are more like 600€ (both assuming good brands, not el cheapo no-name SSDs that fail after one year of heavy ZFS use).

Thanks. This does not look like the LAPIC calibration problem from the thread I linked.

Your LAPIC frequency of about 500 MHz looks sane and is almost exactly what was reported for the working Proxmox case there. The broken case had a frequency off by roughly three orders of magnitude and around 65k timer interrupts/sec on every vCPU.

The `vmstat -i` numbers are interesting, but note that the displayed rate is averaged since boot, so it may hide what happens during one of the short stalls.

Since you are using `kvmclock`, I think one simple A/B test would still be worthwhile:

Code Select Expand

sysctl kern.timecounter.hardware=ACPI-fast

Leave everything else unchanged and see whether the stalls still occur. With several events per hour it should not take too long to get a useful result.

You can switch back with:

Code Select Expand

sysctl kern.timecounter.hardware=kvmclock

I would not conclude from the DTrace samples yet that the TCP retransmission timers are the cause. They may also be a consequence of the actual stall: if packet processing stops briefly, retransmission timers expire and `softclock_thread` subsequently has a lot of work to do.

The snapshot-related KVM clock problem I mentioned earlier also looks less likely in your case, since you have many events which clearly do not coincide with snapshots or backups.

That was only a boot issue and does not explain these short outages. I would try disabling multiqueue on the NICs of the OPNsense VM.

There have also been reports of a current issue that may be related:
https://forum.opnsense.org/index.php?topic=52420.0

Can you give the output of:

Code Select Expand

sysctl kern.timecounter.hardware
sysctl kern.timecounter.choice
sysctl kern.timecounter.fast_gettime

sysctl kern.eventtimer.timer
sysctl kern.eventtimer.choice
sysctl kern.eventtimer.periodic

sysctl kern.eventtimer.et.LAPIC.frequency
sysctl kern.eventtimer.et.LAPIC.quality

Also, do the outages coincide with VM snapshots or backups?

Just applied and rebooted that - came up fine just as usual. PPPoE over VLAN setup.

Maybe shop around on the used market for something nice with 4 x Intel and something like the Intel N100 SoC for a good price ??
Or maybe something totally different that turns out to be a steal somehow ^_^

Alas, the "cheap" hardware options have left the building a while ago :-(

I think the main problem with the diagram is that it mixes different abstraction layers and then shows them as if they were consecutive hops in the packet path.

For example:

• The gateway and the VLAN interface are not really separate hops. From the client's point of view, the gateway normally is an IP address on the OPNsense VLAN interface.
• A WireGuard interface, instance and peer are not three consecutive network elements either. The instance provides the WireGuard interface, while the peer is configuration belonging to that instance.
• The firewall is not one single box which the packet passes only once. Filtering happens at specific interfaces/directions and state tracking is involved.
• With WireGuard there are also two packet layers: the inner IP packet and the outer encrypted UDP packet.

Very simplified, outbound traffic would look more like:

Code Select Expand

Client
-> VLAN
-> OPNsense VLAN interface
-> firewall / routing decision
-> WireGuard interface
-> WireGuard processing / peer selection / encryption
-> WAN interface
-> ISP / Internet
-> remote WireGuard peer

And incoming traffic:

Code Select Expand

remote WireGuard peer
-> Internet / ISP
-> WAN
-> firewall (encrypted UDP packet)
-> WireGuard processing / decryption
-> WireGuard interface
-> firewall / routing (inner IP packet)
-> VLAN interface
-> Client

So I would probably either draw a packet-flow diagram, or a configuration-object diagram showing the relationships between VLANs, interfaces, WireGuard instances and peers.

Mixing both concepts into one left-to-right chain is what makes the current diagram somewhat misleading.

Dann hast Du aber sowieso ein Problem in Deiner Topologie, weil Dein PC offenbar gar nicht über das LAN, wie Du schriebst, sondern über das WAN reinkommt. Das hört sich nämlich danach an, dass der PC im "Transfernetz" zwischen der Fritzbox und der OpnSense sitzt - eventuell solltest Du zu dem Thema mal dies lesen:

https://forum.opnsense.org/index.php?topic=39556

Any specific reason not to leave handling of the physical NIC to PVE via virtio, as described here?

That would also take the igc/iflib driver path inside OPNsense out of the equation.

Also virtio an einer Bridge, was Treiberprobleme weitgehend ausschließt. Wie ich ja schon schrieb: dass "pfctl -d" es zum Fliegen bringt, legt nahe, dass es ein Firewallproblem ist.

Dann würde ich in den Einstellungen das Logging für die Default-Block-Regeln einschalten und mir bei eingeschalteter Firewall im Live Log ansehen, welche Pakete geblockt werden. Dann musst Du feststellen, wieso. Offenbar ziehen dann ja Regeln nicht, die die per Default geblockten Pakete durchlassen sollten. Diese Regeln musst Du finden und schauen, weshalb sie nicht (mehr) greifen.

Da Du mit eingeschalteter Firewall keinen Zugriff mehr auf die OpnSense haben dürftest, musst Du das Vorgehen wahrscheinlich anpassen:

Zuerst ganz vorne eine Allow-All-Regel für Deine Workstation anlegen, von der aus Du die Analyse durchführst. Und dann die Zugriffe von einem anderen PC aus versuchen und ins Live Log sehen.

[welche?]

Ohne Details der Topologie und ggf. eingesetzter Hardware ist dazu keine belastbare Aussage möglich. AKA: Zu wenig Information!

Wenn ich raten sollte, könnte es beispielsweise folgendes sein:

Du hast die OPNsense nicht wie empfohlen mit virtio, sondern mit Pass-Trough (1) und außerdem obendrein mit nicht empfohlener Realtek-Hardware unter Proxmox am laufen (2). Mit neueren OPNsense-Versionen gibt es bekannte Probleme mit den Realtek-Treibern, gerade wenn die früher notfalls empfohlenen Hersteller-Treiber (os-realtek-re) eingesetzt werden (3).

Außerdem: https://forum.opnsense.org/index.php?topic=42985.0, Punkt 16.

Andererseits spricht die Tatsache, dass pfctl -d, also das Abschalten der Firewall, das Problem löst, eher dafür, dass Du beispielsweise bei einem größeren Sprung von der alten Version (welche?) die neuen Firewall-Regeln subtile Änderungen zur Folge haben. Dann führt nichts daran vorbei, Deine Regeln zu zeigen oder noch besser, das Logging einzuschalten und dann festzustellen, welche Regel jetzt blockiert bzw. welche Pass-Regel jetzt nicht mehr trifft.


(1) Siehe https://forum.opnsense.org/index.php?topic=44159.0
(2) Siehe https://forum.opnsense.org/index.php?topic=42985.0, Punkt 6
(3) Siehe z.B.: https://forum.opnsense.org/index.php?topic=53004


 

I suppose you could ask @Monviech (Cedrik) about that - AFAIK he writes most of the documentation.

It is approximately every 24h, with the expression "if (time.time() - fstat.st_mtime) < (86400 - 90)" refering to the timestamp of the last update (see geoip.py).

But why does that bother you, when you can even update the alias on the spot via Firewall: Aliases -> Actions -> Update GeoIP?

Nur kurz gelesen, klingt nach Standardproblemen:

1. Ist das Kabel-Modem wirklich ein Modem oder ein Router mit 192.168.0.x als Transfernetz dahinter? Hint: Reply-To und "Wann ist ein WAN ein WAN"?
2. Erfolgt der Zugriff auf den Port von hinter der OpnSense aus und auf die externe IP via DNS? Ist das die vermeintliche oder die richtige WAN-IP? Ist NAT Reflection an?

Je nach Antwort auf diese Fragen ist Bobs Frage eventuell sehr naheliegend. Ich empfehle ggf: https://forum.opnsense.org/index.php?topic=42985.0

You could try to hide the "physical" link-state changes from OPNsense by assigning LAN to a bridge interface, with hn1 as a bridge member and an additional virtual interface that stays permanently up.

A bridge containing only hn1 probably won't help, as the bridge itself will go down when its last active member goes down.

A LAGG might be another option, although the same caveat probably applies if it only has a single active member.

You may need to create the "dummy" interface on Hyper-V, though, as I do not see any valid interface type on OpnSense (modulo epair, which you cannot create with the web UI).

The point is that in the route section of the web UI, you can define a route to an existing gateway or to a predefined "Null4" or "Null6" gateway.

The way the route is created seems to be bound by using these specific entries and they use the B (blackhole) flag, you can see this results in "USB" flags for the route.

Since you cannot set the "Reject" flag for a self-defined gateway either, I do not see any way to to this via the web UI. So your are left with either a feature request on Github or by creating a specific pf entry by other means than the web UI.