Messages

I know of LibeQoS (Dave Täht's last project), but not much about how to employ it on a small non-ISP scale. Especially not in conjunction with OPNsense.

Care to elaborate?

It seems to just be another appliance that you would deploy between your OPNsense WAN port and your ISP equipment, as a transparent bridge

https://libreqos.readthedocs.io/en/latest/docs/v2.0/design.html

Wireguard running as expected on our 25.1 deployment (just private Wireguard servers).

May I suggest to check firewall configuration on the Wireguard interface?

Hope this helps

I don't have any firewall config on that interface really - there are no rules (I don't want to accept any incoming traffic even if the provider theoretically allows it, which I don't think Nord do anyway), the interface is enabled but with very little config (the local IP is set in my Wireguard config and the gateway set via System/Gateways). But I removed everything and double-checked everything anyway.

Wireguard running fine, I get near line speed at 800 GBit/s via iperf3, just with one thread.

And NordVPN runs at 1 GBit/s, so whatever the problem is, seems not to be Wireguard.

I have found that either peering or some kind of blocking at target sites sometimes bites me with Wireguard. Try changing the outlet server.

This appears to have "solved" the issue. I picked a random server in Albania (which is their first country alphabetically, no other reason) and the performance came right back up to where it should be (around 0.4 seconds for the same HTTPS test I did above). So yeah, in this case looks like maybe some Cloudflare oddities that affected both NordVPN and ProtonVPN on their EU exit nodes.

At some point I might come back and troubleshoot it further, but it looks like it's working fine for now and it was just coincidence that I noticed this right after upgrading OPNsense to 25.1.

Has anyone noticed any issues with Wireguard performing badly since the 25.1 upgrade? I've done minimal troubleshooting with this so far besides some basic tinkering but I've noticed that even basic TCP/HTTPS connections with Wireguard are taking impossibly long times and more often than not failing.

My basic setup is that I generally have two NordVPN tunnels to different locations - one uses Wireguard, one uses OpenVPN (because otherwise the remote gateway IPs conflict; Wireguard and OpenVPN appear to use different overlay networks so I've had success with combining the protocols). The two gateways are in a gateway group with Wireguard as Tier 1, OpenVPN as Tier 2. Tonight, I noticed that I was having issues loading websites in a browser so I pared everything back a little to the most basic site I could think of (https://ipv4.icanhazip.com). If I do this from a Linux container on my LAN which matches a firewall forcing traffic through this gateway group then the connection runs for three minutes without success:

Code Select Expand

root@5087ef50935a:/# time curl https://ipv4.icanhazip.com
^C

real    2m52.575s
user    0m0.009s
sys     0m0.010s

If I invert the priorities on the gateway group, then this works almost immediately

Code Select Expand

root@5087ef50935a:/# time curl https://ipv4.icanhazip.com
87.120.102.196

real    0m0.229s
user    0m0.019s
sys     0m0.015s

In order to isolate NordVPN issues specifically, I set up another Wireguard connection to ProtonVPN

Code Select Expand

root@5087ef50935a:/# time curl https://ipv4.icanhazip.com
^C

real    2m0.533s
user    0m0.005s
sys     0m0.016s

So in summary:

Wireguard/NordVPN: times out, has to be cancelled after a couple of minutes
Wireguard/ProtonVPN: times out, has to be cancelled after a couple of minutes
OpenVPN/NordVPN: works immediately

The gateway checks on both of the Wireguard VPNs is reporting that the gateways are up, so ICMP traffic appears not to be affected (NordVPN is showing a low level of packet loss but only 1%, which I wouldn't expect would cause this).

I find it hard to believe that two separate VPN providers would somehow blocklist a very safe site like this, while NordVPN still allows it via OpenVPN, which makes me think that maybe there's a performance regression in Wireguard somewhere since that's "the common factor" with the failures.

I intend to troubleshoot more tomorrow, all I've seen so far is that tcpdump'ing the Wireguard interface shows SYN packets leaving, but no SYN/ACK coming back (which would normally suggest a provider issue, but this is on two different providers), and I saw a GitHub issue about WAN MTUs being wrong so I applied the recommended patch and that didn't help. But if anyone else can confirm either that they're seeing similar issues, or that they're still using Wireguard successfully then I'd love that.

[edit]

I have similar symptoms occasionally.

If you can, log in via SSH and run `pftop`. I have something running on my LAN (it's a Docker container but I still need to spend the time to narrow down which one) which seems to hold connections open, but only sometimes.

Earlier this evening, after the 24.7.10_1 update, I couldn't SSH into the box anymore but I happened to have a serial cable connected. I was able to run `pftop` and see that there were 15000+ states open. Shortly after that, the box kernal panicked, spewed a load of debug output via serial and then rebooted itself. After the reboot, I stopped the Docker daemon on my home NAS and the number of states is currently hovering around 2000 states. I'm going to start the Docker daemon again and see if the problem comes back - if it does then I need to figure out which container it is because all I can see on OPNsense is the source IP which just comes back as the NAS because of how Docker networking works by default.

In your case, it sounds like maybe you also have too many states open, so your box gets to the point that for some reason it can't accept new connections or the DDoS protection (syncookies) is coming into play, or something like that.

edit - I started the Docker daemon. Within a few seconds, `pftop` was showing 7500 states, so the amount tripled.

Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else?

They would be in /var/unbound/etc/dot.conf

Hmm, ok the link he quoted mentioned the unbound.conf.  My dot.conf file other than a single forwarding zone is empty.

The documentation for `unbound.conf` just shows every available option - Unbound is one of the (sensible) apps which allows for options to be spread across multiple configuration files, for example some provided by a package manager (eligible for overwriting) and some manually (which should not be overwritten). Or separated out by purpose/feature.

So `/var/unbound/etc/dot.conf` will contain a rendered config file with the configuration entries from the `unbound.conf` man page, which are relevant for DNS-over-TLS (or `dot`).

Apparently it's a feature they coined to be for "Windows" and default to off?

tls-win-cert: yes

instead of tls-cert-bundle... can anyone confirm?


Thanks,
Franco

https://nlnetlabs.nl/documentation/unbound/unbound.conf/

I don't see either of these entries in my unbound.conf file.  Should I check somewhere else?

They would be in /var/unbound/etc/dot.conf

Patch works for me.

I have an OPNsense 22.7.2 box where I was trying to debug some traffic.

I have a Floating rule with the settings:

   
• Interface: Guest, VPN
• Direction: In
• Protocol: IPv4+6 TCP
• Source: any
• Destination: ALIAS (The alias contains 2 RFC1918 IPv4 addresses)
• Port: MS DS (445)
• Description: server CIFS

On the Floating Rules page, if I push "Inspect" then the UI updates to show me a new "States" column, which shows 1 session. This is as I expect (there is currently one client connected from the VPN interface, so this lines up).

If I click the "1" text, then I get taken to the Firewall/Diagnostics/States screen where there are several matches, one of which is the one I expect, and several of which don't match the criteria. For example there are matches with the destination port of 80, there are IPv6 matches even though the alias only contains IPv4 addresses, etc. I've attached a couple of screenshots, some redacting has been done but there's enough to show that these states should not match the rule they claim to be matching

the GUI reboot works if and especially when it wants: after giving it, think about it, go back to the "dashboard" and, with ease, start again, obviously whatever you do you will need to log in.

Out of interest, do you use the memory filesystem, and have features like Insights enabled with backups to disk?

On a previous release, I used to see that when I had this combination of features, during the reboot OPNsense would try to back up the memory-backed mountpoint to disk. This took so long that the "check for when the box has rebooted" spinner thing would start firing, see that the UI was up, and assume the reboot was finished even though it wasn't. Then when the backup script actually finished, the reboot would happen.

I did some tinkering a while back and opened https://github.com/opnsense/core/issues/5278 about changing the compression algorithm; by default it uses gzip which gives good compression but is slow. I changed my appliance to use zstd level-1 which was slightly larger but compressed in around 15% of the time

Is there anything to be aware of if someone applied this test base/kernel, with 22.1.9 being released? i.e. should you avoid 22.1.9 altogether? Should you upgrade to 22.1.9, then reapply this preview? Should you do something else?

I'm looking to use the shaping functionality to make for smoother audio calls (like everyone else, I'm working at home...)

The UI for the shaper is "enough", but it's less feature rich than I might like (ports specified manually, can't use aliases, etc). So one thought I had is simply to create myself some low/med/high priority queues and then configure rules for certain DSCP values which could then be applied by firewall rules. However, to know whether that would work properly depends on how the packets flow through the firewall, which could be (for example):

Code Select Expand

ingress --> filter --> shaper --> egress
(this could work because the filter could apply a DSCP class to the packet, the shaper could see it and handle it in the correct queue)

Code Select Expand

ingress --> shaper --> filter --> egress
(this could not work because by the time the filter saw the packet to mark it, the shaping decision would have already been made)

So in a system with both packet filtering and shaping enabled, does anyone know what's the order?