Messages

Some IPv6 clients act in an unexpected way w/r to DHCPv6 and its options. For example, Android devices cannot use DHCPv6 at all but use router advertisements (RA) instead. Some can use the RDNSS option.

That being said, I use RA in "unmanaged" mode for many reasons, but mainly because that is guaranteed to work, but I do not use IPv6 DNS servers - those are not strictly needed if your clients can also do IPv4, because the IPv4 DNS server will also serve IPv6 adresses. This is all described here.

I would rather instruct OpnSense itself to make use of your PiHole as upstream server and not instruct clients to use that directly.

Alas, I cannot give much info about how to do it with DNSmasq, because I use Kea and Unbound. All I know is that DNSmasq has restrictions on its builtin RA mechanism, however, you can disable that and use RADVD instead.

1. The reason why iperf probably shows a higher performance is that it is able to use multiple TCP streams, which you may have selected by using -Px (see https://forum.opnsense.org/index.php?topic=42985.0, point 10).

2. wget, curl rsync and scp use a signle TCP stream, where problems induced that can be induced by double NAT and the Proxmox virtualisation layer come into play. The fact that the rsync starts out fast and then drops speaks for buffer overruns.

I would try to further reduce the MTU size to 1360 on both sides first, regardless of if the pings work with your current settings.

Did you use virtio networking on the OpnSense VM with the settings depicted here, including RSS in OpnSense and on the VM NIC settings? In this specific case, multiqueue on the VM's NIC might be harmful, as it can cause out-of-order packets.

If you passthru the NIC, the VM might not get all interrupts in due time.

If you have to option, try to use a bare metal OpnSense in site B to isolate virtualisation issues.

Wireguard uses UDP and does not benefit from TCP buffer algorithms, so you might also try to use traffic shaping to ensure that buffers are not overrun.

The physical interface is limited to 1 Gbps, I doubt that the driver itself does anything w/r to timing. However, virtio on the vistualisation border should work just fine. IDK how SR-IOV comes into play here, because you do not / should not pass thru the NICs.

Just to make this clear: There are two basic ways you can do this:

1. By passing thru the physical NIC hardware to the VM guest. This absolutely needs a working FreeBSD for the hardware. I do not recommend it. This is where technologies like SR-IOV comes into play.

2. By attaching a virtual network card to the PVE host bridge. In the VM, you can then select which emulated hardware you want to present to the guest, so either E1000 or virtio or whatever hardware your guest supports. For OpnSense, I recommend this way and also, using the virtio drivers.
Virtio would show up as virtioX as network device names under OpnSense.

You MTU looks O.K., so this should work fine. Maybe the physical NIC has some optimisations to coalesce interrupts. This is often the case for high-speed NICs in order to handle traffic more efficiently. It may well be that this interferes and makes low-volume traffic look "choppy".

I would try to use another NIC type to rule out a hardware/driver problem on the PVE side of things, especially, because there are many problem reports for those adapters over on the Proxmox forum: https://forum.proxmox.com/search/19853344/?q=I40e+nic

I suppose there are no FreeBSD drivers for i40e, so you can only use virtio networking. The virtio drivers are pretty good in that there is no additional overhead to emulate any "real" hardware, so there should be no slowness, provided you do everything that is mentioned in the guide.

What I do not get: Even if virtio emulation does not work as expected and E1000 does, this is but a matter of performance, not functionality per se. So what is the problem with using E1000? AFAIK, there is no specific limit with either virtio nor E1000, modulo that any emulation layer through virtualisation limits performance with speeds way higher than 1 Gbps.

Without any specific details, it is hard to say more. Partial loading and general network slowness can be caused by many things that could also be occur if Proxmox was not involved, like wrong MTU settings (did you try to use jumbo packets?). By using OpnSense under Proxmox, you certainly make the setup more complex, that's for sure.

Probably this helps: https://forum.opnsense.org/index.php?topic=44159.0

[unless]

Actually I think @meyergru forgot we're already down to 199 days for public certificates and come next March the value will be 100 days, and 45 days in another year.

What I wanted to stress is the fact that you cannot use your own long-lived certificates unless you use a trick that OpnSense has not got under its sleeve (maybe that would be a good feature request): namely, you cannot set the start date of an issued certificate to "-startdate 20190630120000Z", which I always do with my own CA. This is because "old" certificates can last arbitrarily long. I tend to issue them for at least 10 years, which is way longer than 825, 397, 199, 100 or even 47 days - and 10 years definitely does not work when the "Not Before" date is not manipulated. I changed my CA script to use that "Not Before" date and never looked back because that eliminates the need to ever think about this again ("i.e. "have your cake and eat it").

On the other hand, it simply does not matter how long ACME certificates can last, just because OpnSense can (and will) also reissue them at the respective appropriate intervals, even when the duration changes in the future.

I updated the guide to make this even more obvious.

Did you check if the firewall state tables gets exhausted? If so, increase Firewall ➔ Settings ➔ Advanced : Max States

[VPN: IPsec: Connections]

Hi everyone,


I usually prefer Wireguard for its simplicity, but I found that some ISPs block it using Deep Packet Inspection (even for the purpose of fighting copyright violations). IPsec, being the more "enterprise" VPN protocol, is less often blocked, so it is handy to have a fallback.

While setting up an IKEv2 EAP-MSCHAPv2 Roadwarrior connection using the modern VPN: IPsec: Connections module according to the official OPNsense Roadwarrior (swanctl) Documentation, you might run into situations where the connection seems established on the firewall (swanctl --list-sas), but 0 packets / 0 bytes are being transmitted.

To save you hours of structural troubleshooting on the FreeBSD kernel or routing layers, here is a definitive list of bullet points on what actually causes issues with modern iOS/macOS clients—and what you can safely ignore.

⚠️ The Real Problems (What you must avoid / fix)

• Avoid Manual Profile Configuration (The DNS Trap)
  
  • The Issue: Typing the VPN credentials directly into the native iOS VPN settings menu.
    
  • The Impact: iOS manually configured profiles strictly ignore the DNS Configuration Payload sent by the server. This happens regardless of whether you use Split Tunneling or Full Tunneling (0.0.0.0/0). Internal DNS resolution will be completely broken.
    
  • The Fix: You must deploy the configuration via a .mobileconfig file to explicitly inject the DNS structure into the Apple network stack.
    You can create such profiles with the free Imazing app.
    
• Avoid "long" Server Certificate Lifetimes
  
  • The Issue: Creating a server certificate with a really long validity period, like more than a year.
    
  • The Impact: The Apple crypto subsystem silently and rigidly rejects any server certificate with too long lifetimes exceeding.
    
  • Note: While this rule can technically be bypassed by backdating the "Not Before" timestamp years into the past via OpenSSL CLI commands, the OPNsense GUI does not allow backdating (it always forces creation date to time()). Therefore, stick to ACME/Let's Encrypt (90 days) or issue short-lived certificates.
    Note: This will not be any problem when using ACME-type certificates. Also, the System : Trust : Certificate dialogue will preset the certificate lifetime with 397 days for this very reason.
    
• Avoid Missing Server Authentication EKU
  
  • The Issue: Generating the server certificate as a generic or unconstrained type.
    
  • The Impact: The certificate must explicitly contain the Extended Key Usage (EKU) attribute TLS Web Server Authentication. Without it, iOS drops the connection instantly during the IKE_AUTH phase. This is also done automatically in System : Trust : Certificate if you select "Server Authentication".
    
• Avoid Non-RSA certificates
  
  • The Issue: Generating the server certificate as an EC certificate.
    
  • The Impact: The certificate must be an RSA certificate to be recognized. This is also done automatically in System : Trust : Certificate per default.
    

ℹ️ The Cosmetic Illusion (Do not judge the connection by this)

• The Missing VPN Icon
  
  • The Reality: In current iOS versions, the VPN icon in the status bar standardly disappears after a successful IKEv2 handshake. Do not assume a vanished icon means a routing failure, a bad configuration, or a silent disconnect. The tunnel remains fully active (ESTABLISHED / INSTALLED in swanctl), even when the icon is missing. The status display is not tied to a failing DNS check.
    

🚫 Mythbusting (What is NOT the problem)

If your tunnel is up but registers 0 packets on active SAs, do not waste your time troubleshooting the following theoretical network pitfalls, as iOS handles them perfectly fine:

• Overlapping / Supernet Routing: iOS handles broad local traffic selectors like 192.168.0.0/16 flawlessly alongside CGNAT or carrier cellular networks. However, use a disjoint network for your IP pools.
  
• MTU sizes: IP fragmentation is handled correctly, but AI agents will often misinterpret log messages as to try to make the
  
• Multiple LocalNets / Comma-separated Lists: The native Apple client parses multiple distinct local networks in the traffic selector correctly.
  
• Strict SAN / Wildcard Validation: Wildcard certificates (*.domain.tld) or strict Subject Alternative Name (SAN) match anomalies are not the cause of 0-byte transmission stalls.
  
• IPv6 problems: There are none. AI bots may also misinterpret mixed IP adressing for the connection itself and the tunnel IPs, but that is not a problem.
  

Summary for a working setup:
Follow the official documentation, make sure your certificate is short-lived with the correct Server-EKU, ignore the missing status bar icon, and deploy the client configuration exclusively via a tailored .mobileconfig profile to get proper DNS access.

The first rule is an "in" rule for your mediaserver interface, yet it applies only to source adresses in the LAN network, so it probably never applies. More often than not, you will specify either the interface network or even "any" as source address. Remember, the source adresses will probably be from the interface network range - but that is implicitely given by the fact that the interface they arrive on is specified anyway.

The second rule blocks anything from the mediaserver interface to anywhere. If there is no preceeding rule, it will block any traffic passing the firewall.

Essentially, these rules would allow only level 2 traffic on the mediaserver network that does not pass the firewall. Also, order is usually important (well, not if the rules do not work out, such as these).

You should familiarize yourself with the basic concepts of OpnSense firewalling, especially with how rules are applied (packets going "in" on an interface), rule precedence and network coverage. If you want to block access to "the internet" (which is destination "any"), you may still need rules preceeding the block rule that in turn allow your other VLANs (like allow to "RFC1918").

If you want to analyse what really happens, just imagine a packet with source and destination adresses and ports and apply the set rules in order.

[You]

So you are saying that when I measure iPerf3 speeds between OPNsense and a Client that the speed will always be lower than between two any other type of Clients on the same subnet ?!

Yes, of course, just because when OpnSense is itself involved, iperf3 runs on top of the actual routing. Well, "on the same subnet" is incorrect, of course, because OpnSense would not be involved at all if the traffic were between two clients on layer 2.

Ofcourse you will always have to measure with as much threads as possible and sometimes even raise the window size and stuff like that... :)

You may know, but the OP did not state how exactly the measurements were taken, which is why I pointed it out again. RSS will not do any good with just one connection.

OK, but this clearly shows there is something wrong with the OPNsense Mini PC itself :

iperf3:
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

These two should always be equal IMO :)

No:

1. There is a big difference between OpnSense routing sessions between different partners and OpnSense being the endpoint (the latter one is slower).

2. Since this is iperf, I also like to point out this article, point 10. I totally depends on how many TCP sessions you use.

Pulling these together, I see ~1.87 Gbps with iperf -P1 vs. 3.56 Gbps with iperf -P4 vs. 6 Gbps when OpnSense routes only.

Peaks at about 40%.

iperf3:
1.48 Gbits/sec from the router to a public server.
1.28 Gbits/sec from Win11 to the same public server.
1.60 Gbits/sec from Win11 to OPNsense iperf3 server.
2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

I forgot to mention this :

I did update my i226 firmwares to v2.32 tho i dont think that mattered.

So have a look at this topic : https://forum.opnsense.org/index.php?topic=48695.0

Maybe that's the issue here...

I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.

NICE! :)

So far I only knew that a N100 can satisfy the 2,5 Gbps NICs out there as long as you don't use any IDS/IPS stuff.

W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.

He got good speeds to his DIY NAS :

2.37 Gbits/sec from Win11 to Unraid (N95) iperf3 server.

So I am guessing that's not the issue...

There are big timing differences between internet and local connections and also, the endpoints can behave differently, thus there can be any number of problems w/r to timing and/or buffering. The latter depends on which TCP algorithms are in use, such that flow control and buffering or interrupt coalescing can very well play a role.

If your connection was still on PPPoE, it would blame that, but it is not.

I have tested the pure routing speed of N100-based OpnSense systems between VLANs to ~6 Gbps, so that should work.

W/R to the Windows 11 system, I have seen network drivers that needed tuning to exceed 1.5 Gbps on some network adapters, like disabling interrupt moderation, enlarge buffer sizes or changing flow of control settings.

a. If your concern is to tighten security, you can use the client MAC to enforce rules.

b. If you aim to cause clients to use only the DHCPv6-assigned address for outbound access, you want to disable the "autonomous address-configuration flag" from RFC4862.

With Kea and RADVD, you must use "Managed" mode for the interface if you want DHCPv6, see this table. "Assisted" mode would allow for DHCPv6, but still allows the client to use privacy extensions. As I do not use DNSmasq, I cannot tell how to do it when it sends RAs by itself, but I guess it is documented.

c. You can also configure individual clients to disable privacy extensions completely.

That being said, I never tried nor verified it, because I actually want clients to use privacy extensions. If I want to control or limit a client, I do that regardless of the used IPv6 via its MAC (method a.). The only thing I can imagine is if you want to look into logs and identify the actual client involved.

[inkl.]

Sooooo. Ich komme nochmal auf das Thema zurück, weil ich jetzt die ultimative Lösung für das Problem "Unifi mit OpnSense" gefunden habe:

Technisch ist es offenbar so, dass Unifi "intern" das VLAN 1 nutzt, wenn Ports einfach als Access Ports aufgeschaltet werden. Dabei ist eben VLAN 1 das "Native VLAN" und VLANs stehen auf "Block All". Das "Native VLAN" wird einfach egress vom Tag 1 befreit und ingress wird das Tag 1 hinzugefügt. Dadurch ist das Standard-VLAN 1 an Access Ports verfügbar und sonst nichts. Will man ein anderes VLAN an diesem Access Port nutzen, nimmt man eben das als "Native VLAN".

You cannot view this attachment.

Der Normalfall für Trunk Ports (also z.B. andere Unifi Switche oder APs) ist, dass "Allow All" für VLANs genutzt wird, und weiterhin das VLAN 1 das "Native VLAN" ist - obwohl man das im Netzwerk gar nicht manuell auswählen kann: man lässt das Feld dann leer:



und nur in der Übersicht der Netzwerke steht unter VLAN ID dann die 1:

You cannot view this attachment.

Das Port-Profil dazu sieht etwa so aus:

You cannot view this attachment.

Somit wird das Native VLAN an solchen Ports untagged angeboten, zusätzlich werden aber alle VLANs entsprechend verteilt - bis auf VLAN 1, denn das Tag dafür wird egress entfernt und ingress hinzugefügt.

Hierdurch ergibt sich das beschriebene "Problem": Unifi rechnet damit, dass das Management VLAN untagged an allen Ports zur Verfügung steht, was das einfache Anlernen neuer Unifi-Devices ermöglicht. Weitere VLANs werden dann getaggt.

OpnSense hingegen hat bekanntlich in bestimmten Bereichen Probleme mit dem Mischen von tagged und untagged VLANs auf dem selben physischen Port.

Zwar kann man auch bei Unifi das Management VLAN auf ein getaggtes VLAN legen, dazu müssen die Geräte aber erstmal worden angelernt sein - ein typischer Catch22.

Die Lösung ist eigentlich ganz einfach, ähnlich wie Patrick es beschrieb: Man kann nämlich auch ein Profil für Trunk Ports anlegen, bei dem das "Native VLAN" = "none" ist. Dann findet einfach kein egress/ingress-Tagging mehr statt - aber das Endgerät kann selbst jedes VLAN nutzen - inkl. VLAN 1. Wenn man dann das Management-Interface in OpnSense als VLAN 1 anlegt und das physische Parent-Interface nicht konfiguriert, funktioniert alles auf einmal: Unifi behandelt die Ports, wie es will und OpnSense taggt alle VLANs.

You cannot view this attachment.

Ich beschreibe jetzt absichtlich NICHT Schritt-für-Schritt, wie man den Übergang schafft, ohne sich dabei wegzuschießen, aber als Hinweis:

1. Man sollte sich zuerst das VLAN 1 in OpnSense anlegen und sicherstellen, dass man mindestens Zugriff auf die Konsole hat (oder Layer-2-Zugriff auf ein anderes Netzwerkinterface, als das untagged), um dort agieren zu können.

2. Man sollte ebenfalls dafür sorgen, dass man Layer-2-Zugriff auf die Unifi Console hat, der nicht von der OpnSense abhängig ist. Zuerst sollte man sich ein neues Trunk Profil mit "Default VLAN" = "none" und "Allow All" für den OpnSense-Port bauen, auf das man dann umschalten kann.

3. Vorsichtig, wenn man die Assigments der Interfaces ändert - der DHCP-Server bekommt die Änderung nicht mit, den muss man selbst neu starten (oder OpnSense rebooten), anderenfalls verlieren die Unifi-Devices nach Ablauf des Leases die IP-Adressen und sind nicht mehr erreichbar.

Das ist notwendig, weil man nacheinander beide Ports (am Switch und an der OpnSense) umschalten muss - wenn etwas schiefgeht, kann man sonst nicht mehr zurück (been there, done that).