Messages

Aha. Found it:

https://github.com/flaviuvlaicu/opnsense-topo-map

I had a look at the code. The plugin does not actually discover the network topology. It merely collects clients from ARP/Kea; the relationships between clients, switches and APs are then defined manually by the user via drag & drop and stored in topology.json.

There is no LLDP, SNMP, switch MAC-table or controller data being queried. Therefore, the plugin cannot determine which switch/port a client is actually connected through, nor can it detect when a wireless client roams from one AP to another.

So essentially, it is a graphical network diagram editor with an automatically populated client list, not an automatic topology discovery tool.

That said, I would be rather cautious about installing software from arbitrary GitHub repositories on an OPNsense firewall.

So do I.

OpnSense does not know anything about this other than on which interface a client is connected to, so what you want is not feasible using it.

There is network equipment available that can show you things like that, however. Take a look at Unifi switches and APs. My personal opinion is that they are quite good at making those, less good at making routers (I use OpnSense for that).

You do not need a dream box or any of their routers, though. The network controller can be run on x64 hardware or as a VM (the software is called Unifi OS server and is free).

That is a collision of interest of the purest kind.

You can block DoT/DoH, but obviously, you cannot intercept and modify that traffic, because it is encrypted and protected via certificates. Normally, such an approach would be used in conjunction with a traffic interception of normal, unencrypted DNS traffic to be able to block certain DNS domains. If anyone wants to use your network, he must fall back to standard DNS in such a scenario, i.e. abide by your imposed rules.

If you want to give a device internet access that does not play by those rules, you can, but then you cannot block DoT/DoH, as you have seen.
In that case, I would put that device into a separate VLAN (probably also a separate WLAN), such that it can do whatever it chooses, but has no possibility to access any of your own LAN ressources. Of course, you can allow specific services, like accessing a printer.

Or to put it short: You cannot have the cake and eat it, too.

BTW: If there was a canary domain, you would be able to see a DNS request (and, obviously, it would be via normal DNS).

Since Chromium deliberately chose not to implement a Firefox-style canary domain, there is no DNS-side mechanism to force such a fallback; and if the Chromebook is managed by the school, the DoH-only setting is most likely policy-locked anyway.

...because the documentation knows better than you if your ISP needs a VLAN and if your modem supplies it in that case?

There are two cases:

1. your ISP does not need a VLAN - then your use the NIC device without one.
2. your ISP needs a VLAN - then you can do one of two things:
   a. if your modem supports it, you can set the VLAN in the modem and your NIC device without VLAN.
   b. if your modem does not support it or you do not want to lose modem access via the untagged VLAN, you can use the NIC device with VLAN.


I prefer 2b if needed for the reason given.

You only need that for active FTP, which I assumed why the question was asked. These days, most FTP servers can do passive FTP.

[Important limitation:]

The official documentation currently does not describe the complete setup of os-ftp-proxy and the old how-to explains the concept, but it is about 10 years old and the screenshots are no longer available.

The current documentation only mentions the basic principle.

The important missing step is that merely enabling the FTP proxy service is not enough. For a transparent forward proxy you also need a NAT port-forward rule for the FTP control connection, for example:

Code Select Expand

Interface:        LAN
Protocol:         TCP
Source:           LAN net
Destination:      any
Destination port: 21

Redirect target:  127.0.0.1
Redirect port:    8021

With the proxy enabled on 127.0.0.1:8021, but without that redirect, active FTP failed here exactly as expected:

Code Select Expand

> EPRT |1|192.168.10.97|22827|
< 425 Connections to other hosts not allowed.

After adding the redirect rule, the same active FTP test worked immediately.

The proxy handles PORT/EPRT, substitutes an externally reachable endpoint and dynamically creates the required PF rules for the incoming FTP data connection. Therefore no permanent WAN rule for the FTP data ports is required.

This can easily be tested against the public Rebex FTP test server:

Code Select Expand

curl.exe -v --ftp-port - --user demo:password ftp://test.rebex.net/pub/example/readme.txt

Important limitation: this only works for plain, unencrypted FTP.

It cannot work for FTPS/TLS, because once the FTP control connection is encrypted, ftp-proxy can no longer inspect or rewrite commands such as PORT, EPRT, PASV or EPSV, nor derive the required dynamic firewall rules from them.

The underlying ftp-proxy behaviour is documented here.


So I think the current OPNsense documentation should explicitly include

• the required NAT redirect to 127.0.0.1:8021,
• an explanation that ftp-proxy dynamically opens and redirects the active-mode data connection, and
• the limitation that this cannot work with encrypted FTP control connections (FTPS/TLS).

The thing is, such N1x0 units were like 300€ complete with 16 GByte RAM and 256 GByte SSD a year ago. Now they are more like 600€ (both assuming good brands, not el cheapo no-name SSDs that fail after one year of heavy ZFS use).

Thanks. This does not look like the LAPIC calibration problem from the thread I linked.

Your LAPIC frequency of about 500 MHz looks sane and is almost exactly what was reported for the working Proxmox case there. The broken case had a frequency off by roughly three orders of magnitude and around 65k timer interrupts/sec on every vCPU.

The `vmstat -i` numbers are interesting, but note that the displayed rate is averaged since boot, so it may hide what happens during one of the short stalls.

Since you are using `kvmclock`, I think one simple A/B test would still be worthwhile:

Code Select Expand

sysctl kern.timecounter.hardware=ACPI-fast

Leave everything else unchanged and see whether the stalls still occur. With several events per hour it should not take too long to get a useful result.

You can switch back with:

Code Select Expand

sysctl kern.timecounter.hardware=kvmclock

I would not conclude from the DTrace samples yet that the TCP retransmission timers are the cause. They may also be a consequence of the actual stall: if packet processing stops briefly, retransmission timers expire and `softclock_thread` subsequently has a lot of work to do.

The snapshot-related KVM clock problem I mentioned earlier also looks less likely in your case, since you have many events which clearly do not coincide with snapshots or backups.

That was only a boot issue and does not explain these short outages. I would try disabling multiqueue on the NICs of the OPNsense VM.

There have also been reports of a current issue that may be related:
https://forum.opnsense.org/index.php?topic=52420.0

Can you give the output of:

Code Select Expand

sysctl kern.timecounter.hardware
sysctl kern.timecounter.choice
sysctl kern.timecounter.fast_gettime

sysctl kern.eventtimer.timer
sysctl kern.eventtimer.choice
sysctl kern.eventtimer.periodic

sysctl kern.eventtimer.et.LAPIC.frequency
sysctl kern.eventtimer.et.LAPIC.quality

Also, do the outages coincide with VM snapshots or backups?

Just applied and rebooted that - came up fine just as usual. PPPoE over VLAN setup.

Maybe shop around on the used market for something nice with 4 x Intel and something like the Intel N100 SoC for a good price ??
Or maybe something totally different that turns out to be a steal somehow ^_^

Alas, the "cheap" hardware options have left the building a while ago :-(

I think the main problem with the diagram is that it mixes different abstraction layers and then shows them as if they were consecutive hops in the packet path.

For example:

• The gateway and the VLAN interface are not really separate hops. From the client's point of view, the gateway normally is an IP address on the OPNsense VLAN interface.
• A WireGuard interface, instance and peer are not three consecutive network elements either. The instance provides the WireGuard interface, while the peer is configuration belonging to that instance.
• The firewall is not one single box which the packet passes only once. Filtering happens at specific interfaces/directions and state tracking is involved.
• With WireGuard there are also two packet layers: the inner IP packet and the outer encrypted UDP packet.

Very simplified, outbound traffic would look more like:

Code Select Expand

Client
-> VLAN
-> OPNsense VLAN interface
-> firewall / routing decision
-> WireGuard interface
-> WireGuard processing / peer selection / encryption
-> WAN interface
-> ISP / Internet
-> remote WireGuard peer

And incoming traffic:

Code Select Expand

remote WireGuard peer
-> Internet / ISP
-> WAN
-> firewall (encrypted UDP packet)
-> WireGuard processing / decryption
-> WireGuard interface
-> firewall / routing (inner IP packet)
-> VLAN interface
-> Client

So I would probably either draw a packet-flow diagram, or a configuration-object diagram showing the relationships between VLANs, interfaces, WireGuard instances and peers.

Mixing both concepts into one left-to-right chain is what makes the current diagram somewhat misleading.

Dann hast Du aber sowieso ein Problem in Deiner Topologie, weil Dein PC offenbar gar nicht über das LAN, wie Du schriebst, sondern über das WAN reinkommt. Das hört sich nämlich danach an, dass der PC im "Transfernetz" zwischen der Fritzbox und der OpnSense sitzt - eventuell solltest Du zu dem Thema mal dies lesen:

https://forum.opnsense.org/index.php?topic=39556

Any specific reason not to leave handling of the physical NIC to PVE via virtio, as described here?

That would also take the igc/iflib driver path inside OPNsense out of the equation.