Messages

...because the documentation knows better than you if your ISP needs a VLAN and if your modem supplies it in that case?

There are two cases:

1. your ISP does not need a VLAN - then your use the NIC device without one.
2. your ISP needs a VLAN - then you can do one of two things:
   a. if your modem supports it, you can set the VLAN in the modem and your NIC device without VLAN.
   b. if your modem does not support it or you do not want to lose modem access via the untagged VLAN, you can use the NIC device with VLAN.


I prefer 2b if needed for the reason given.

You only need that for active FTP, which I assumed why the question was asked. These days, most FTP servers can do passive FTP.

[Important limitation:]

The official documentation currently does not describe the complete setup of os-ftp-proxy and the old how-to explains the concept, but it is about 10 years old and the screenshots are no longer available.

The current documentation only mentions the basic principle.

The important missing step is that merely enabling the FTP proxy service is not enough. For a transparent forward proxy you also need a NAT port-forward rule for the FTP control connection, for example:

Code Select Expand

Interface:        LAN
Protocol:         TCP
Source:           LAN net
Destination:      any
Destination port: 21

Redirect target:  127.0.0.1
Redirect port:    8021

With the proxy enabled on 127.0.0.1:8021, but without that redirect, active FTP failed here exactly as expected:

Code Select Expand

> EPRT |1|192.168.10.97|22827|
< 425 Connections to other hosts not allowed.

After adding the redirect rule, the same active FTP test worked immediately.

The proxy handles PORT/EPRT, substitutes an externally reachable endpoint and dynamically creates the required PF rules for the incoming FTP data connection. Therefore no permanent WAN rule for the FTP data ports is required.

This can easily be tested against the public Rebex FTP test server:

Code Select Expand

curl.exe -v --ftp-port - --user demo:password ftp://test.rebex.net/pub/example/readme.txt

Important limitation: this only works for plain, unencrypted FTP.

It cannot work for FTPS/TLS, because once the FTP control connection is encrypted, ftp-proxy can no longer inspect or rewrite commands such as PORT, EPRT, PASV or EPSV, nor derive the required dynamic firewall rules from them.

The underlying ftp-proxy behaviour is documented here.


So I think the current OPNsense documentation should explicitly include

• the required NAT redirect to [tt]127.0.0.1:8021[/tt],
• an explanation that ftp-proxy dynamically opens and redirects the active-mode data connection, and
• the limitation that this cannot work with encrypted FTP control connections (FTPS/TLS).

The thing is, such N1x0 units were like 300€ complete with 16 GByte RAM and 256 GByte SSD a year ago. Now they are more like 600€ (both assuming good brands, not el cheapo no-name SSDs that fail after one year of heavy ZFS use).

Thanks. This does not look like the LAPIC calibration problem from the thread I linked.

Your LAPIC frequency of about 500 MHz looks sane and is almost exactly what was reported for the working Proxmox case there. The broken case had a frequency off by roughly three orders of magnitude and around 65k timer interrupts/sec on every vCPU.

The `vmstat -i` numbers are interesting, but note that the displayed rate is averaged since boot, so it may hide what happens during one of the short stalls.

Since you are using `kvmclock`, I think one simple A/B test would still be worthwhile:

Code Select Expand

sysctl kern.timecounter.hardware=ACPI-fast

Leave everything else unchanged and see whether the stalls still occur. With several events per hour it should not take too long to get a useful result.

You can switch back with:

Code Select Expand

sysctl kern.timecounter.hardware=kvmclock

I would not conclude from the DTrace samples yet that the TCP retransmission timers are the cause. They may also be a consequence of the actual stall: if packet processing stops briefly, retransmission timers expire and `softclock_thread` subsequently has a lot of work to do.

The snapshot-related KVM clock problem I mentioned earlier also looks less likely in your case, since you have many events which clearly do not coincide with snapshots or backups.

That was only a boot issue and does not explain these short outages. I would try disabling multiqueue on the NICs of the OPNsense VM.

There have also been reports of a current issue that may be related:
https://forum.opnsense.org/index.php?topic=52420.0

Can you give the output of:

Code Select Expand

sysctl kern.timecounter.hardware
sysctl kern.timecounter.choice
sysctl kern.timecounter.fast_gettime

sysctl kern.eventtimer.timer
sysctl kern.eventtimer.choice
sysctl kern.eventtimer.periodic

sysctl kern.eventtimer.et.LAPIC.frequency
sysctl kern.eventtimer.et.LAPIC.quality

Also, do the outages coincide with VM snapshots or backups?

Just applied and rebooted that - came up fine just as usual. PPPoE over VLAN setup.

Maybe shop around on the used market for something nice with 4 x Intel and something like the Intel N100 SoC for a good price ??
Or maybe something totally different that turns out to be a steal somehow ^_^

Alas, the "cheap" hardware options have left the building a while ago :-(

I think the main problem with the diagram is that it mixes different abstraction layers and then shows them as if they were consecutive hops in the packet path.

For example:

• The gateway and the VLAN interface are not really separate hops. From the client's point of view, the gateway normally is an IP address on the OPNsense VLAN interface.
• A WireGuard interface, instance and peer are not three consecutive network elements either. The instance provides the WireGuard interface, while the peer is configuration belonging to that instance.
• The firewall is not one single box which the packet passes only once. Filtering happens at specific interfaces/directions and state tracking is involved.
• With WireGuard there are also two packet layers: the inner IP packet and the outer encrypted UDP packet.

Very simplified, outbound traffic would look more like:

Code Select Expand

Client
-> VLAN
-> OPNsense VLAN interface
-> firewall / routing decision
-> WireGuard interface
-> WireGuard processing / peer selection / encryption
-> WAN interface
-> ISP / Internet
-> remote WireGuard peer

And incoming traffic:

Code Select Expand

remote WireGuard peer
-> Internet / ISP
-> WAN
-> firewall (encrypted UDP packet)
-> WireGuard processing / decryption
-> WireGuard interface
-> firewall / routing (inner IP packet)
-> VLAN interface
-> Client

So I would probably either draw a packet-flow diagram, or a configuration-object diagram showing the relationships between VLANs, interfaces, WireGuard instances and peers.

Mixing both concepts into one left-to-right chain is what makes the current diagram somewhat misleading.

Dann hast Du aber sowieso ein Problem in Deiner Topologie, weil Dein PC offenbar gar nicht über das LAN, wie Du schriebst, sondern über das WAN reinkommt. Das hört sich nämlich danach an, dass der PC im "Transfernetz" zwischen der Fritzbox und der OpnSense sitzt - eventuell solltest Du zu dem Thema mal dies lesen:

https://forum.opnsense.org/index.php?topic=39556

Any specific reason not to leave handling of the physical NIC to PVE via virtio, as described here?

That would also take the igc/iflib driver path inside OPNsense out of the equation.

Also virtio an einer Bridge, was Treiberprobleme weitgehend ausschließt. Wie ich ja schon schrieb: dass "pfctl -d" es zum Fliegen bringt, legt nahe, dass es ein Firewallproblem ist.

Dann würde ich in den Einstellungen das Logging für die Default-Block-Regeln einschalten und mir bei eingeschalteter Firewall im Live Log ansehen, welche Pakete geblockt werden. Dann musst Du feststellen, wieso. Offenbar ziehen dann ja Regeln nicht, die die per Default geblockten Pakete durchlassen sollten. Diese Regeln musst Du finden und schauen, weshalb sie nicht (mehr) greifen.

Da Du mit eingeschalteter Firewall keinen Zugriff mehr auf die OpnSense haben dürftest, musst Du das Vorgehen wahrscheinlich anpassen:

Zuerst ganz vorne eine Allow-All-Regel für Deine Workstation anlegen, von der aus Du die Analyse durchführst. Und dann die Zugriffe von einem anderen PC aus versuchen und ins Live Log sehen.

[welche?]

Ohne Details der Topologie und ggf. eingesetzter Hardware ist dazu keine belastbare Aussage möglich. AKA: Zu wenig Information!

Wenn ich raten sollte, könnte es beispielsweise folgendes sein:

Du hast die OPNsense nicht wie empfohlen mit virtio, sondern mit Pass-Trough (1) und außerdem obendrein mit nicht empfohlener Realtek-Hardware unter Proxmox am laufen (2). Mit neueren OPNsense-Versionen gibt es bekannte Probleme mit den Realtek-Treibern, gerade wenn die früher notfalls empfohlenen Hersteller-Treiber (os-realtek-re) eingesetzt werden (3).

Außerdem: https://forum.opnsense.org/index.php?topic=42985.0, Punkt 16.

Andererseits spricht die Tatsache, dass pfctl -d, also das Abschalten der Firewall, das Problem löst, eher dafür, dass Du beispielsweise bei einem größeren Sprung von der alten Version (welche?) die neuen Firewall-Regeln subtile Änderungen zur Folge haben. Dann führt nichts daran vorbei, Deine Regeln zu zeigen oder noch besser, das Logging einzuschalten und dann festzustellen, welche Regel jetzt blockiert bzw. welche Pass-Regel jetzt nicht mehr trifft.


(1) Siehe https://forum.opnsense.org/index.php?topic=44159.0
(2) Siehe https://forum.opnsense.org/index.php?topic=42985.0, Punkt 6
(3) Siehe z.B.: https://forum.opnsense.org/index.php?topic=53004


 

I suppose you could ask @Monviech (Cedrik) about that - AFAIK he writes most of the documentation.

It is approximately every 24h, with the expression "if (time.time() - fstat.st_mtime) < (86400 - 90)" refering to the timestamp of the last update (see geoip.py).

But why does that bother you, when you can even update the alias on the spot via Firewall: Aliases -> Actions -> Update GeoIP?