Messages

There is a cron job called "Restart Wireguard on stale connections" - try enabling that.

Did you use the Realtek vendor or the FreeBSD native driver on OpnSense?

O.K., I never had the need to look at those logs. I do adblocking using browser plugins.

If you normally use OpnSense for all things network-centric, you may be better off to have everything pertaining to logging and things there. Also, if you have DNS problems because of excessive blocking, you can switch centrally on OpnSense only this way, because otherwise you would have to wait for your clients to pick up the alternative DNS server IP.

Who wants to look at a DNS query log and for what purpose? And even if you do, why not look at OpnSense's DNS logs, if you care about who asks for what?

[Only use if you know what you are doing.]

Du hast vermutlich "Strict QNAME Minimisation" angehakt. Einfach mal den Hilfetext lesen:
   
"Send minimum amount of information to upstream servers to enhance privacy. Do not fall-back to sending full QNAME to potentially broken nameservers. A lot of domains will not be resolvable when this option in enabled. Only use if you know what you are doing."
   

Some IPv6 clients act in an unexpected way w/r to DHCPv6 and its options. For example, Android devices cannot use DHCPv6 at all but use router advertisements (RA) instead. Some can use the RDNSS option.

That being said, I use RA in "unmanaged" mode for many reasons, but mainly because that is guaranteed to work, but I do not use IPv6 DNS servers - those are not strictly needed if your clients can also do IPv4, because the IPv4 DNS server will also serve IPv6 adresses. This is all described here.

I would rather instruct OpnSense itself to make use of your PiHole as upstream server and not instruct clients to use that directly.

Alas, I cannot give much info about how to do it with DNSmasq, because I use Kea and Unbound. All I know is that DNSmasq has restrictions on its builtin RA mechanism, however, you can disable that and use RADVD instead.

1. The reason why iperf probably shows a higher performance is that it is able to use multiple TCP streams, which you may have selected by using -Px (see https://forum.opnsense.org/index.php?topic=42985.0, point 10).

2. wget, curl rsync and scp use a signle TCP stream, where problems induced that can be induced by double NAT and the Proxmox virtualisation layer come into play. The fact that the rsync starts out fast and then drops speaks for buffer overruns.

I would try to further reduce the MTU size to 1360 on both sides first, regardless of if the pings work with your current settings.

Did you use virtio networking on the OpnSense VM with the settings depicted here, including RSS in OpnSense and on the VM NIC settings? In this specific case, multiqueue on the VM's NIC might be harmful, as it can cause out-of-order packets.

If you passthru the NIC, the VM might not get all interrupts in due time.

If you have to option, try to use a bare metal OpnSense in site B to isolate virtualisation issues.

Wireguard uses UDP and does not benefit from TCP buffer algorithms, so you might also try to use traffic shaping to ensure that buffers are not overrun.

The physical interface is limited to 1 Gbps, I doubt that the driver itself does anything w/r to timing. However, virtio on the vistualisation border should work just fine. IDK how SR-IOV comes into play here, because you do not / should not pass thru the NICs.

Just to make this clear: There are two basic ways you can do this:

1. By passing thru the physical NIC hardware to the VM guest. This absolutely needs a working FreeBSD for the hardware. I do not recommend it. This is where technologies like SR-IOV comes into play.

2. By attaching a virtual network card to the PVE host bridge. In the VM, you can then select which emulated hardware you want to present to the guest, so either E1000 or virtio or whatever hardware your guest supports. For OpnSense, I recommend this way and also, using the virtio drivers.
Virtio would show up as virtioX as network device names under OpnSense.

You MTU looks O.K., so this should work fine. Maybe the physical NIC has some optimisations to coalesce interrupts. This is often the case for high-speed NICs in order to handle traffic more efficiently. It may well be that this interferes and makes low-volume traffic look "choppy".

I would try to use another NIC type to rule out a hardware/driver problem on the PVE side of things, especially, because there are many problem reports for those adapters over on the Proxmox forum: https://forum.proxmox.com/search/19853344/?q=I40e+nic

I suppose there are no FreeBSD drivers for i40e, so you can only use virtio networking. The virtio drivers are pretty good in that there is no additional overhead to emulate any "real" hardware, so there should be no slowness, provided you do everything that is mentioned in the guide.

What I do not get: Even if virtio emulation does not work as expected and E1000 does, this is but a matter of performance, not functionality per se. So what is the problem with using E1000? AFAIK, there is no specific limit with either virtio nor E1000, modulo that any emulation layer through virtualisation limits performance with speeds way higher than 1 Gbps.

Without any specific details, it is hard to say more. Partial loading and general network slowness can be caused by many things that could also be occur if Proxmox was not involved, like wrong MTU settings (did you try to use jumbo packets?). By using OpnSense under Proxmox, you certainly make the setup more complex, that's for sure.

Probably this helps: https://forum.opnsense.org/index.php?topic=44159.0

[unless]

Actually I think @meyergru forgot we're already down to 199 days for public certificates and come next March the value will be 100 days, and 45 days in another year.

What I wanted to stress is the fact that you cannot use your own long-lived certificates unless you use a trick that OpnSense has not got under its sleeve (maybe that would be a good feature request): namely, you cannot set the start date of an issued certificate to "-startdate 20190630120000Z", which I always do with my own CA. This is because "old" certificates can last arbitrarily long. I tend to issue them for at least 10 years, which is way longer than 825, 397, 199, 100 or even 47 days - and 10 years definitely does not work when the "Not Before" date is not manipulated. I changed my CA script to use that "Not Before" date and never looked back because that eliminates the need to ever think about this again ("i.e. "have your cake and eat it").

On the other hand, it simply does not matter how long ACME certificates can last, just because OpnSense can (and will) also reissue them at the respective appropriate intervals, even when the duration changes in the future.

I updated the guide to make this even more obvious.

Did you check if the firewall state tables gets exhausted? If so, increase Firewall ➔ Settings ➔ Advanced : Max States

[VPN: IPsec: Connections]

Hi everyone,


I usually prefer Wireguard for its simplicity, but I found that some ISPs block it using Deep Packet Inspection (even for the purpose of fighting copyright violations). IPsec, being the more "enterprise" VPN protocol, is less often blocked, so it is handy to have a fallback.

While setting up an IKEv2 EAP-MSCHAPv2 Roadwarrior connection using the modern VPN: IPsec: Connections module according to the official OPNsense Roadwarrior (swanctl) Documentation, you might run into situations where the connection seems established on the firewall (swanctl --list-sas), but 0 packets / 0 bytes are being transmitted.

To save you hours of structural troubleshooting on the FreeBSD kernel or routing layers, here is a definitive list of bullet points on what actually causes issues with modern iOS/macOS clients—and what you can safely ignore.

⚠️ The Real Problems (What you must avoid / fix)

• Avoid Manual Profile Configuration (The DNS Trap)
  
  • The Issue: Typing the VPN credentials directly into the native iOS VPN settings menu.
    
  • The Impact: iOS manually configured profiles strictly ignore the DNS Configuration Payload sent by the server. This happens regardless of whether you use Split Tunneling or Full Tunneling (0.0.0.0/0). Internal DNS resolution will be completely broken.
    
  • The Fix: You must deploy the configuration via a .mobileconfig file to explicitly inject the DNS structure into the Apple network stack.
    You can create such profiles with the free Imazing app.
    
• Avoid "long" Server Certificate Lifetimes
  
  • The Issue: Creating a server certificate with a really long validity period, like more than a year.
    
  • The Impact: The Apple crypto subsystem silently and rigidly rejects any server certificate with too long lifetimes exceeding.
    
  • Note: While this rule can technically be bypassed by backdating the "Not Before" timestamp years into the past via OpenSSL CLI commands, the OPNsense GUI does not allow backdating (it always forces creation date to time()). Therefore, stick to ACME/Let's Encrypt (90 days) or issue short-lived certificates.
    Note: This will not be any problem when using ACME-type certificates. Also, the System : Trust : Certificate dialogue will preset the certificate lifetime with 397 days for this very reason.
    
• Avoid Missing Server Authentication EKU
  
  • The Issue: Generating the server certificate as a generic or unconstrained type.
    
  • The Impact: The certificate must explicitly contain the Extended Key Usage (EKU) attribute TLS Web Server Authentication. Without it, iOS drops the connection instantly during the IKE_AUTH phase. This is also done automatically in System : Trust : Certificate if you select "Server Authentication".
    
• Avoid Non-RSA certificates
  
  • The Issue: Generating the server certificate as an EC certificate.
    
  • The Impact: The certificate must be an RSA certificate to be recognized. This is also done automatically in System : Trust : Certificate per default.
    

ℹ️ The Cosmetic Illusion (Do not judge the connection by this)

• The Missing VPN Icon
  
  • The Reality: In current iOS versions, the VPN icon in the status bar standardly disappears after a successful IKEv2 handshake. Do not assume a vanished icon means a routing failure, a bad configuration, or a silent disconnect. The tunnel remains fully active (ESTABLISHED / INSTALLED in swanctl), even when the icon is missing. The status display is not tied to a failing DNS check.
    

🚫 Mythbusting (What is NOT the problem)

If your tunnel is up but registers 0 packets on active SAs, do not waste your time troubleshooting the following theoretical network pitfalls, as iOS handles them perfectly fine:

• Overlapping / Supernet Routing: iOS handles broad local traffic selectors like 192.168.0.0/16 flawlessly alongside CGNAT or carrier cellular networks. However, use a disjoint network for your IP pools.
  
• MTU sizes: IP fragmentation is handled correctly, but AI agents will often misinterpret log messages as to try to make the
  
• Multiple LocalNets / Comma-separated Lists: The native Apple client parses multiple distinct local networks in the traffic selector correctly.
  
• Strict SAN / Wildcard Validation: Wildcard certificates (*.domain.tld) or strict Subject Alternative Name (SAN) match anomalies are not the cause of 0-byte transmission stalls.
  
• IPv6 problems: There are none. AI bots may also misinterpret mixed IP adressing for the connection itself and the tunnel IPs, but that is not a problem.
  

Summary for a working setup:
Follow the official documentation, make sure your certificate is short-lived with the correct Server-EKU, ignore the missing status bar icon, and deploy the client configuration exclusively via a tailored .mobileconfig profile to get proper DNS access.

The first rule is an "in" rule for your mediaserver interface, yet it applies only to source adresses in the LAN network, so it probably never applies. More often than not, you will specify either the interface network or even "any" as source address. Remember, the source adresses will probably be from the interface network range - but that is implicitely given by the fact that the interface they arrive on is specified anyway.

The second rule blocks anything from the mediaserver interface to anywhere. If there is no preceeding rule, it will block any traffic passing the firewall.

Essentially, these rules would allow only level 2 traffic on the mediaserver network that does not pass the firewall. Also, order is usually important (well, not if the rules do not work out, such as these).

You should familiarize yourself with the basic concepts of OpnSense firewalling, especially with how rules are applied (packets going "in" on an interface), rule precedence and network coverage. If you want to block access to "the internet" (which is destination "any"), you may still need rules preceeding the block rule that in turn allow your other VLANs (like allow to "RFC1918").

If you want to analyse what really happens, just imagine a packet with source and destination adresses and ports and apply the set rules in order.

[You]

So you are saying that when I measure iPerf3 speeds between OPNsense and a Client that the speed will always be lower than between two any other type of Clients on the same subnet ?!

Yes, of course, just because when OpnSense is itself involved, iperf3 runs on top of the actual routing. Well, "on the same subnet" is incorrect, of course, because OpnSense would not be involved at all if the traffic were between two clients on layer 2.

Ofcourse you will always have to measure with as much threads as possible and sometimes even raise the window size and stuff like that... :)

You may know, but the OP did not state how exactly the measurements were taken, which is why I pointed it out again. RSS will not do any good with just one connection.