Issues with OPNsense WiFi Speed Compared to Untangle – Need Assistance

[CyberSenseSteve]

Hi All

I've been a long-time user of Untangle and had a wonderful experience with it. Recently, I decided to transition to OPNsense, given its features and modern appeal. While I'm genuinely excited about OPNsense, I've stumbled upon some issues that I'm finding hard to diagnose, specifically concerning WiFi speeds. I'm hoping the wealth of expertise here can help me troubleshoot this.

My Setup:
I'm running OPNsense on a VM inside Proxmox. Here are the VM specs:

   
• Cores: 6 (Host mode)
• CPU: Host (Ryzen 5900X)
• Memory: 8192
• Machine: q35
• NIC: Intel i350 T2V2 (Passed through to the VM)
• OPNSense 23.7.1

Performance Metrics:
Using a wired connection, the network performs admirably:

Local iperf Test:
Average: ~944 Mbits/sec

"iperf3 -c 10.0.77.21
Connecting to host 10.0.77.21, port 5201
[  5] local 10.0.77.88 port 58522 connected to 10.0.77.21 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   958 Mbits/sec    0    489 KBytes
[  5]   1.00-2.00   sec   113 MBytes   951 Mbits/sec    0    592 KBytes
[  5]   2.00-3.00   sec   112 MBytes   937 Mbits/sec    0    650 KBytes
[  5]   3.00-4.00   sec   112 MBytes   944 Mbits/sec    0    680 KBytes
[  5]   4.00-5.00   sec   112 MBytes   944 Mbits/sec    0    680 KBytes
[  5]   5.00-6.00   sec   112 MBytes   944 Mbits/sec    0    680 KBytes
[  5]   6.00-7.00   sec   112 MBytes   944 Mbits/sec    0    963 KBytes
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0   1.09 MBytes
[  5]   8.00-9.00   sec   112 MBytes   944 Mbits/sec    0   1.09 MBytes
[  5]   9.00-10.00  sec   112 MBytes   944 Mbits/sec    0   1.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec    0             sender
[  5]   0.00-10.05  sec  1.10 GBytes   937 Mbits/sec                  receiver"

Speedtest Results:
Download: 731.63 Mbit/s
Upload: 192.63 Mbit/s

"Retrieving speedtest.net configuration...
Testing from XXXXXXXX ...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXXXXX [71.61 km]: 2.085 ms
Testing download speed................................................................................
Download: 731.63 Mbit/s
Testing upload speed...
Upload: 192.63 Mbit/s"

However, when switching to a WiFi device, the performance drops:

WiFi iperf to Same Server:
Average: ~462 Mbits/sec

"Connecting to host 10.0.77.21, port 5201
[  5] local 10.0.77.30 port 63020 connected to 10.0.77.21 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  53.6 MBytes   449 Mbits/sec
[  5]   1.00-2.00   sec  56.6 MBytes   474 Mbits/sec
[  5]   2.00-3.00   sec  55.8 MBytes   468 Mbits/sec
[  5]   3.00-4.00   sec  56.0 MBytes   469 Mbits/sec
[  5]   4.00-5.00   sec  52.4 MBytes   439 Mbits/sec
[  5]   5.00-6.00   sec  56.6 MBytes   474 Mbits/sec
[  5]   6.00-7.00   sec  56.5 MBytes   474 Mbits/sec
[  5]   7.00-8.00   sec  54.6 MBytes   458 Mbits/sec
[  5]   8.00-9.00   sec  53.5 MBytes   448 Mbits/sec
[  5]   9.00-10.00  sec  54.7 MBytes   459 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   550 MBytes   462 Mbits/sec                  sender
[  5]   0.00-10.01  sec   549 MBytes   460 Mbits/sec                  receiver"


WiFi Speedtest Results:
Download: 217.18 Mbit/s
Upload: 161.76 Mbit/s

"Retrieving speedtest.net configuration...
Testing from XXXXXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXXXXX [71.61 km]: 5.964 ms
Testing download speed................................................................................
Download: 217.18 Mbit/s
Testing upload speed...
Upload: 161.76 Mbit/s"

The problem

On my wired connection, I consistently achieve speeds nearing 1Gig, accounting for expected variations and overhead. However, when I conduct similar tests on my WiFi connection (Unifi), there's a significant drop in performance. Interestingly, when I use Untangle as my firewall, WiFi speeds regularly reach upwards of 400Mbps — almost double of what I record with OPNsense. Yet, wired connection results are comparably consistent between both firewalls. It's noteworthy that this performance discrepancy seems exclusive to WiFi devices. When I revert to Untangle from OPNsense, WiFi speeds instantly return to their peak of over 400Mbps, aligning with my throughput tests. I've included a screenshot detailing my WiFi experience on Unifi. The transition point where I switch to Untangle is evident due to the noticeable boost in WiFi performance, compared to when I use OPNsense. While there's a 10% dip in the user experience score on Unifi, the actual speed sees a more significant reduction of 50%.



Troubleshooting Steps Taken:

   
• I've tried system tuning, based on some suggestions and guides. These didn't help.
• Checked and ensured that the NIC and other hardware are compatible and efficiently passed through to the VM.
• Monitored system logs for any evident issues like CPU load etc. Nothing stands out and my CPU load is rarely ever more than 5%.
• I have rebooted my entire network after each change of firewall.

Current Roadblock:

To be completely honest, I'm stumped about the WiFi performance discrepancy between Untangle and OPNsense. I'm not sure where to look next or how to start addressing this problem. I have no idea why the Wifi performance is so bad compared to Untangle when the wired connection seems fine.

I'd be immensely grateful if anyone could provide guidance, insights, or share similar experiences and resolutions. I've included as much detail as I can, hoping it assists in diagnosing the problem.

Thank you in advance for your time and help.

[meyergru]

From the looks of it I would say the WiFi connection itself is the culprit, but I bet that nothing has changed there with the switch from Untangle to OpnSense.

Thus, there must be something that could be different between Untangle and OpnSense that manifests itself more in a WiFi than in a wired connection. What comes into my mind would be fragmentation: If your outside line uses PPPoE or VLANs, it could be that you need to set the MTU of your WAN to less than 1500 Bytes. Larger LAN packets must be fragmented to use this and usually you do that via MSS clamping. However, if the LAN has a larger MTU, them packets need to be fragmented, causing double the number of packets and turnarounds, which might be more of a problem with the latency in wireless connections.

I would try to reduce the LAN MTU to the same size as the WAN MTU.

[CyberSenseSteve]

Hi meyergru,

Thank you for taking the time to provide a suggestion. I really appreciate you sharing your insights.

Per your guidance, I tried reducing the LAN MTU to match the WAN MTU of 1500. I manually set it to 1500, 1480, and 1460 to test if it made any difference. Unfortunately, the changes did not have a material or persistent impact on the inconsistent WiFi speeds I'm seeing.

To provide some more information - my internet connection is DHCP, not PPPoE. I monitored my CPU load during the tests and it remained low, around 5-10%, even on wired speed tests. So I don't think CPU is a bottleneck here.



I ran multiple speedtests over a period of time and the WiFi speeds fluctuated wildly, from 238Mbps up to 510Mbps down. This aligns with the inconsistent performance I'm trying to troubleshoot. Here are some sample results:

"speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXXX [71.69 km]: 6.435 ms
Testing download speed................................................................................
Download: 506.96 Mbit/s
Testing upload speed...
Upload: 136.55 Mbit/s"

and then a few moments later:


"speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXXX [71.61 km]: 7.133 ms
Testing download speed................................................................................
Download: 238.72 Mbit/s
Testing upload speed...
Upload: 166.77 Mbit/s"

While the speeds did spike higher at times, inconsistency and wide fluctuation remain an issue. On Untangle, speeds over WiFi were consistently 400Mbps+.

I appreciate you taking the time to suggest a solution.


While the MTU adjustment did not resolve the problem in my case, I'm thankful you proposed it.


Please let me know if any other potential causes or troubleshooting steps come to mind. I'm happy to keep testing theories to try and isolate the root cause as its quite frustrating.

[meyergru]

If it is not the fragmentation, I am at a loss as to what could be different for wired vs. wireless connections apart from the obvious difference: Are you really sure that the wireless speed is unlimited if you measure from wired to wireless LAN without the OpnSense between them?

The observed inconsistencies point to a reception problem.

There could also be interference from the OpnSense hardware that makes your WiFi slower. As an example, I have some USB receptables which, when I plug in a USB3 thumb drive, my bluetooth mouse stops working unless I bring it very near to the receiver.

[lilsense]

couple of questions:

Are we to presume that the Opnsense and untangle are both on the same proxmox, just shutting down one and starting up the other?

What services are running on both untangle/OPNsense? Zenarmor/Suricatta??? also, how are the WIFI LAN set up on both?

[newsense]

Don't change the MTU, there's no need.


What VHT values have you set on 2.4hz and 5Ghz ?

Did you confirm the trunks negotiate at 1Gbps Full Duplex ? I not try to set it manually.


Also, I'd be interested in a speedtest from a device that's in the same Lan/VLAN as the WiFi device - not from the FW

[newsense]

I believe this is the culprit and may help solve the issue, assuming the VHT and port speed negotiation are not an issue.

https://it-notes.dragas.net/2023/08/14/boosting-network-performance-in-freebsds-vnet-jails/

[CyberSenseSteve]

If it is not the fragmentation, I am at a loss as to what could be different for wired vs. wireless connections apart from the obvious difference: Are you really sure that the wireless speed is unlimited if you measure from wired to wireless LAN without the OpnSense between them?

The observed inconsistencies point to a reception problem.

There could also be interference from the OpnSense hardware that makes your WiFi slower. As an example, I have some USB receptables which, when I plug in a USB3 thumb drive, my bluetooth mouse stops working unless I bring it very near to the receiver.

Hi meyeygru

Given my setup, I don't think the OPNsense hardware itself is causing interference. The server is located in a garage far from the access points, which are wired into a switch. The APs remain constant whether I have Untangle or OPNsense running.

The rack has two computers, a NAS, and the Proxmox server with the firewall VMs. No USB devices are connected during testing. There is an AP in the garage but that also does not change between OSs. The server hardware and environment are identical between Untangle and OPNsense.

I have considered wireless reception as a potential factor. However, since the physical setup is constant, I would expect the reception quality to be the same regardless of which firewall OS is running. I originally considered traffic shaping, queue etc, but none are present.

couple of questions:

Are we to presume that the Opnsense and untangle are both on the same proxmox, just shutting down one and starting up the other?

What services are running on both untangle/OPNsense? Zenarmor/Suricatta??? also, how are the WIFI LAN set up on both?

Hi lilsense,

To answer your questions:

Yes, Opnsense and Untangle are both installed on the same Proxmox host. I switch between them by setting one or the other to boot, and rebooting the server. The hardware is identical between the two - same NIC (Intel i350), specs, and network topology. For services, Untangle has intrusion prevention enabled, while Opnsense currently does not. Otherwise it is just VLANs and static DHCP leases on both.

The WiFi LAN setup is the same - APs are wired into a switch which connects to the firewall VM. Each AP negotiates at 1000Mbps.

So essentially the core infrastructure remains constant between the two configurations. The only change is swapping Untangle for Opnsense as the firewall OS.

Don't change the MTU, there's no need.


What VHT values have you set on 2.4hz and 5Ghz ?

Did you confirm the trunks negotiate at 1Gbps Full Duplex ? I not try to set it manually.


Also, I'd be interested in a speedtest from a device that's in the same Lan/VLAN as the WiFi device - not from the FW

Hi newsense,

Thank you for the tip on leaving the MTU as is. I have confirmed the Unifi APs are set to 160Mhz on 5Ghz bands and 20Mhz on the 2.4Ghz band.

To provide full context on the testing environment:

The client device is a MacBook Pro connected over WiFi
The MacBook is on the default LAN network throughout testing
It connects to the same SSID for all speed tests
Only the firewall OS changes between Untangle and OPNsense

So in summary:

Proxmox server running firewall VM (Untangle or OPNsense)
UniFi APs wired to firewall VM via switch
MacBook Pro client on WiFi
MacBook stays on default LAN, connecting to same SSID
With that consistent setup, here are the speed test results when comparing the two firewalls:

Untangle

Connection details to AP:



Openspeedtest on same network (LXC on Proxmox)



iPerf test from client:



Speedtest from client via CLI:

speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXX [71.61 km]: 6.957 ms
Testing download speed................................................................................
Download: 511.84 Mbit/s
Testing upload speed...
Upload: 185.81 Mbit/s

Speedtest via URL:



OPNSense results

Connection details:



Openspeedtest from client:



iPerf from client:



Speedtest via cli:


First run:
speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXX [71.61 km]: 5.975 ms
Testing download speed................................................................................
Download: 313.87 Mbit/s
Testing upload speed...
Upload: 165.56 Mbit/s


Second run:
speedtest-cli
Retrieving speedtest.net configuration...
Testing from XXXXXX...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by XXXXXXX [71.61 km]: 4.885 ms
Testing download speed................................................................................
Download: 111.88 Mbit/s
Testing upload speed...
Upload: 47.13 Mbit/s

Speedtest via the web:



I have tried to provide a detailed overview of my testing environment and comparative results between Untangle and OPNsense. As you can see from the speed tests, Untangle consistently provides faster and more reliable WiFi speeds in my setup. With OPNsense, the speeds are much lower and fluctuate widely.

I'm still unsure what could be causing this performance difference, since all other factors remain constant. Thank you again to everyone who has provided suggestions and assistance. Your expertise is greatly appreciated.

[newsense]

I believe this is the culprit and may help solve the issue, assuming the VHT and port speed negotiation are not an issue.

https://it-notes.dragas.net/2023/08/14/boosting-network-performance-in-freebsds-vnet-jails/

Thanks for all the details, maybe you missed my post when replying?

[CyberSenseSteve]

I believe this is the culprit and may help solve the issue, assuming the VHT and port speed negotiation are not an issue.

https://it-notes.dragas.net/2023/08/14/boosting-network-performance-in-freebsds-vnet-jails/

Hi newsense,

Thank you for the informative article and suggested optimizations. I really appreciate you taking the time to share that resource.

I went ahead and implemented the tunable changes outlined in the guide:

Disabled LRO and checksum offloading on the igb interfaces
Disabled bridge packet filtering
I rebooted after applying those tweaks. Unfortunately, testing showed no noticeable improvement to the inconsistent wireless speeds in OPNsense. Here is my "/boot/loader.conf" file:

"root@OPNsense:~ # vi /boot/loader.conf

##############################################################
# This file was auto-generated using the rc.loader facility. #
# In order to deploy a custom change to this installation,   #
# please use /boot/loader.conf.local as it is not rewritten, #
# or better yet use System: Settings: Tunables from the GUI. #
##############################################################

loader_brand="opnsense"
loader_logo="hourglass"
loader_menu_title=""

autoboot_delay="3"

# Vital modules that are not in FreeBSD's GENERIC
# configuration will be loaded on boot, which makes
# races with individual module's settings impossible.
carp_load="YES"
if_bridge_load="YES"
if_enc_load="YES"
if_gif_load="YES"
if_gre_load="YES"
if_lagg_load="YES"
if_tap_load="YES"
if_tun_load="YES"
if_vlan_load="YES"
pf_load="YES"
pflog_load="YES"
pfsync_load="YES"

# dynamically generated console settings follow
#comconsole_speed
#boot_multicons
#boot_serial
#kern.vty
console="vidconsole"

# dynamically generated tunables settings follow
hw.ibrs_disable="1"
hw.igb.0.csum_disable="1"
hw.igb.0.lro_disable="1"
hw.igb.1.csum_disable="1"
hw.igb.1.lro_disable="1"
hw.igb.rx_process_limit="-1"
hw.igb.tx_process_limit="-1"

Changes were similarly made in "/etc/sysctl.conf" as discussed in the article.

Wired speeds remain fast and reliable, so it does seem isolated to WiFi performance.

Here is a speedtest:

I believe this is the culprit and may help solve the issue, assuming the VHT and port speed negotiation are not an issue.

https://it-notes.dragas.net/2023/08/14/boosting-network-performance-in-freebsds-vnet-jails/

Thanks for all the details, maybe you missed my post when replying?

Hahaha... Sorry about that. Was busy finalising the post and wanted to test before replying. Hopefully the above helps.

[newsense]

How about setting VHT to 80 ?

[CyberSenseSteve]

How about setting VHT to 80 ?

Hi newsense,

Per your suggestion, I tried setting the UniFi APs to use VHT 80Mhz instead of 160Mhz. Unfortunately I'm still seeing the same inconsistent speeds in OPNsense compared to Untangle.

Here's where things get really puzzling:

Using Wifiman speed tests, I'm now seeing speeds up to 485Mbps on OPNsense. I am not sure how Wifiman tests, but the image at the bottom appears to indicate an external server, so its not internal Wifi speeds.



But immediately after in the same location, Speedtest CLI shows only half that at ~240Mbps.



Fast.com is giving me ~370Mbps.

I'm really at a loss as to why the different speed tests would vary so wildly on the same network. The environment and equipment is unchanged between tests. I don't have visibility into how OPNsense handles traffic behind the scenes, but it seems something is affecting performance in a way I can't determine. The drastic fluctuations remain very perplexing.

[CJ]

Are you moving your wifi device or anything else in between tests?  I'm able to pull 900mbps through OPNSense using my U6LR.

Even though I have 10G for my LAN, I don't bother changing the MTU's from the default.  Also, my AP is on it's own 1G NIC and I run physical OPNSense.

[CyberSenseSteve]

Are you moving your wifi device or anything else in between tests?  I'm able to pull 900mbps through OPNSense using my U6LR.

Even though I have 10G for my LAN, I don't bother changing the MTU's from the default.  Also, my AP is on it's own 1G NIC and I run physical OPNSense.

Hi CJ

Nope... Not at all. My devices are ceiling mounted throughout my house. Speed auto negotiates at 1Gig so its not cable problems.