Messages

Hi! We have the problem with the certificate, also discussed here and maybe somewhere else. I have correct SSL setup with Captive Portal and it worked fine before some update (cant tell specifically which and when).

The main problem cause is this site:
https://connectivitycheck.gstatic.com/generate_204
or https://msftconnecttest.com
or some other device online checking urls.

It is not allowed, so it redirects back to hotspot login like this, for example:
https://hotspot.domain:8000/index.html?redirurl=connectivitycheck.gstatic.com/generate_204

...and this causes problem with the certificate as it tries to use local certificate for the gstatic.com site. Devices which doesn't use this generate_204 link, work fine.  Anyway - before the updates this behavior was correct. It's possible to connect, but there are certificate errors everywhere.

Edit: I'm starting to wonder if this is the issue, because using Firefox always likes the certificate and Chrome does like it only when already logged in. Perhaps this is something certificate itself related

EditV2: In the end I had now to include FULL CA certificate and Chrome based browsers started accepting it now. Guess its some update in Chrome that changed something.

This can be closed.

Greetings.
I am having this problem since installing OPNsense on 2013 and I haven't fixed it yet (partly also because the sites have to be online and there are not many opportunities to shut them down). There is Wireguard site to site tunnel configuration executed based on official documentation step by step. Overall it works, but there is speed problem on one site, but only on one direction (site1 upload). It looks like a standard MSS/MTU or clamping issue for Wireguard, but that has been checked numerous times and no difference has been seen. Also the tunables have been checked and modified with no effect. Also all HW offloading is turned off on both OPNsense VMs in its settings.

The main configuration is as follows:
SITE1 PROXMOX host -> OPNsense guest (and some more VMs using LAN vmbr(0)) vmbr1 -> WAN bridge / Wireguard roadwarrior setup with a pair of clients, that can also access SITE2 trough SITE1
SITE2 PROXMOX host -> OPNsense guest (and some more VMs using LAN vmbr(0)) vmbr1 -> WAN bridge

Running OPNsense 25.7.3_7-amd64 on both hosts, always updated in the same time. Both guests are connected with site to site link. Link itself works and latency is low. I'm gonna add some speed tests. Internet link speed is 250/250 on both sides. Running with multiple connections on iperf doesnt help. Copying files using SMB results in same speed.

Code Select Expand

root@site2:~# iperf3 -c 10.44.1.2
Connecting to host 10.44.1.2, port 5201
[  5] local 10.33.0.2 port 40808 connected to 10.44.1.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  18.7 MBytes   157 Mbits/sec  1002   71.3 KBytes       
[  5]   1.00-2.00   sec  8.69 MBytes  72.9 Mbits/sec   47   61.0 KBytes       
[  5]   2.00-3.00   sec  3.48 MBytes  29.1 Mbits/sec   81   20.8 KBytes       
[  5]   3.00-4.00   sec  10.4 MBytes  87.5 Mbits/sec   70   48.0 KBytes       
[  5]   4.00-5.00   sec  7.82 MBytes  65.6 Mbits/sec   69   64.8 KBytes       
[  5]   5.00-6.00   sec  9.56 MBytes  80.2 Mbits/sec   61   51.9 KBytes       
[  5]   6.00-7.00   sec  8.69 MBytes  72.9 Mbits/sec   52   31.1 KBytes       
[  5]   7.00-8.00   sec  7.82 MBytes  65.6 Mbits/sec   37   70.0 KBytes       
[  5]   8.00-9.00   sec  8.69 MBytes  72.9 Mbits/sec   50   71.3 KBytes       
[  5]   9.00-10.00  sec  9.56 MBytes  80.2 Mbits/sec   69   68.7 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  93.4 MBytes  78.3 Mbits/sec  1538             sender
[  5]   0.00-10.01  sec  91.0 MBytes  76.3 Mbits/sec                  receiver

___
root@site1:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.33.0.2, port 40800
[  5] local 10.44.1.2 port 5201 connected to 10.33.0.2 port 40808
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  16.6 MBytes   140 Mbits/sec                 
[  5]   1.00-2.00   sec  8.79 MBytes  73.7 Mbits/sec                 
[  5]   2.00-3.00   sec  3.64 MBytes  30.6 Mbits/sec                 
[  5]   3.00-4.00   sec  9.99 MBytes  83.8 Mbits/sec                 
[  5]   4.00-5.00   sec  7.64 MBytes  64.1 Mbits/sec                 
[  5]   5.00-6.00   sec  9.60 MBytes  80.5 Mbits/sec                 
[  5]   6.00-7.00   sec  8.59 MBytes  72.1 Mbits/sec                 
[  5]   7.00-8.00   sec  8.57 MBytes  71.9 Mbits/sec                 
[  5]   8.00-9.00   sec  8.37 MBytes  70.2 Mbits/sec                 
[  5]   9.00-10.00  sec  9.06 MBytes  76.0 Mbits/sec                 
[  5]  10.00-10.01  sec  53.2 KBytes  80.1 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  91.0 MBytes  76.3 Mbits/sec                  receiver
And then the other way:

Code Select Expand

root@site1:~# iperf3 -c 10.33.0.2
Connecting to host 10.33.0.2, port 5201
[  5] local 10.44.1.2 port 42628 connected to 10.33.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  32.4 MBytes   272 Mbits/sec  100    908 KBytes       
[  5]   1.00-2.00   sec  23.8 MBytes   199 Mbits/sec  422    175 KBytes       
[  5]   2.00-3.00   sec  26.2 MBytes   220 Mbits/sec   54    122 KBytes       
[  5]   3.00-4.00   sec  28.8 MBytes   241 Mbits/sec   49   81.7 KBytes       
[  5]   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec    0    208 KBytes       
[  5]   5.00-6.00   sec  26.2 MBytes   220 Mbits/sec   54    170 KBytes       
[  5]   6.00-7.00   sec  18.8 MBytes   157 Mbits/sec   69   66.1 KBytes       
[  5]   7.00-8.00   sec  25.0 MBytes   210 Mbits/sec    0    197 KBytes       
[  5]   8.00-9.00   sec  25.0 MBytes   210 Mbits/sec   58    170 KBytes       
[  5]   9.00-10.00  sec  25.0 MBytes   210 Mbits/sec   62    113 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   257 MBytes   216 Mbits/sec  868             sender
[  5]   0.00-10.01  sec   255 MBytes   214 Mbits/sec                  receiver

root@site2:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.44.1.2, port 42614
[  5] local 10.33.0.2 port 5201 connected to 10.44.1.2 port 42628
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  28.8 MBytes   242 Mbits/sec                 
[  5]   1.00-2.00   sec  24.3 MBytes   204 Mbits/sec                 
[  5]   2.00-3.00   sec  26.0 MBytes   218 Mbits/sec                 
[  5]   3.00-4.00   sec  29.4 MBytes   246 Mbits/sec                 
[  5]   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec                 
[  5]   5.00-6.00   sec  25.4 MBytes   213 Mbits/sec                 
[  5]   6.00-7.00   sec  19.3 MBytes   162 Mbits/sec                 
[  5]   7.00-8.00   sec  24.7 MBytes   207 Mbits/sec                 
[  5]   8.00-9.00   sec  24.6 MBytes   206 Mbits/sec                 
[  5]   9.00-10.00  sec  26.1 MBytes   219 Mbits/sec                 
[  5]  10.00-10.01  sec   154 KBytes   218 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   255 MBytes   214 Mbits/sec                  receiver
On SITE1 there is also roadwarrior configuration, which also lets to access SITE2 local network. That all also works fine, just the speed problem. We can also see that the retry count is high when its throttling. When using roadwarrior on SITE1, we can see the exact same behavior (for example when trying speedtest.net test on a remote computer using SITE1 gateway). When utilizing internet (no WG) on both sites locally, I can get full 250/250 speed. That would seem that the problem is most likely only on SITE1 side and when using Wireguard.

The LAN segment on both sides works perfectly utilizing 10G link.

Then there is the fact, that WAN is configured using Proxmox virtual bridge (vmbr1) instead of using the real WAN interface as slave (vmbr0 localnet works fine with full passtrough). That is so on both servers. I have tried disabling every hardware offload function (tso,gso etc...) with ethtool -K on Proxmox hosts. Some entries (when checking with ethtool -k) show off [fixed] or off [forced on] even after trying to turn off. I have also read through this forum multiple times with no real solution. I have disabled 'Reply-To' in firewall settings and maybe some other tweaks that I forgot while writing this topic.

I am looking for an opportunity to enable iommu and pass the real wan interface trough Proxmox to the OPNsense VM's. Then attach it directly as WAN interface, but haven't tried it yet. I tried to fix is using the current setup, but it should be theoretically possible to work using current setup. Maybe someone can give some more ideas, or I should just try passing the PCI network adapter directly...

K

I have faced the same error with the latest version OPNsense 25.7.3_7-amd64 - two dashes instead of one. I had to edit them manually, haven't tested the result, but there are no errors and should probably be fine.

EDIT: it turned out that sqlite database was malformed and dashes were set correctly already. I deleted the database and auth started working, now just there is the SSL problem.

Have you tried using more than one stream (iperf -P8)?

Yeah, same results. Currently - no matter what I tweak, the speed is still somewhere capped and test results come out identical. Now all systems are up to date, the problem persists.

I haven't checked any firewall rules, next thing I will probably try disabling rules one by one.
Maybe I should try super low MTU/MSS, I haven't gone below 1200.

If nothing fixes the problem, I will probably have to migrate to OpenVPN or IPSec. I Hope I will find some solution :D

Hello guys. I have tried almost everything in this topic, including microcode updates, playing with MTU and some recommended tunables... and I still get very bad speeds in one direction of tunnel.
Here's iperf3 in straight and reverse directions:

https://i.imgur.com/aUkYz9d.png

One direction is OK, max speed. Reverse - there seems to be a peak at 130-150Mbps and then the speed drops to steady 60-80Mbps. Cpu usage is low. The CPUs are pretty new Xeons, plenty of power.
When using iperf3 directly (no Wireguard), then the speeds are okay.

The configuration:
* 2 sites, 2 servers
* 250/250 optic uplink, latency 3-5ms
* Proxmox with OpnSense VM
* Virtio virtualized interfaces for both LAN and WAN (I havent tried direct pass-though yet), Proxmox bridge networking (vmnet0, vmnet1). Underlying interfaces are 1Gbps

I am using Netflow and Insight, haven't tried disabling those.

How to properly troubleshoot this? I will still try to find the culprit, the speeds are terribly slow.