I've recently created an OPNSense box. Specs:
Dell OptiPlex 7050
CPU: i5-7500
SSD: Lexar 512 GB 3000 MB/s
RAM: 8 GB (2x4) DDR4 2133
NICs: 2x Realtek RTL8125B (I know Realtek isn't the best, especially with OPNSense, but they seem to work for basically everything).
So, the issue I'm having is strange. The reason I posted in the Hardware and Performance subforum is because I'm wondering if my Realtek cards have something to do with it? No idea why they would, but I know they are buggy with OPNSense (and like all other appliance firewalls. Lol)
Basically, everything in my home network works fine, aside from YouTube and the Google Play Store (and possibly other Google apps - I haven't checked yet).
Any Apple products we have, download from their App Store perfectly fine. My desktop, which is hardwired, downloads from Steam at 100+ MB/s. However, playing YouTube videos and downloading from the Google Play Store are a huge struggle (wired, and wireless). YouTube videos constantly buffer, and switch back and forth between low and higher resolution, and downloading from the Play Store when my phone is connected to my WiFi is in the range of like 50 Kb/s.
Now before you ask if my WiFi is having issues, it's not. And again, before you ask if maybe my ISP is having issues... they're not. I did not have these problems at all until I switched to the OPNSense box as my main router/firewall. My phone runs a 300 Mb/s speed test, and my home internet is 1 Gb/s. And I have this problem on both my wired, and wireless devices. JUST YouTube and Google Play Store are slow.
Any ideas?
Just wanting to bump this, as it's actively problematic, and would be nice to try to diagnose with folks who are more familiar with OPNSense than me!
Hi, first post :)
This is just for comparison to yours as I do not have the issue you describe.
CPU: i5-6400
SSD: Integral 256 GB Nvme
RAM: 4GB DDR4 2400
NICs: 2x Intel PRO 1000
Your machine is superior to mine in both CPU and RAM but I have an Intel Pro dual gigabit NIC. Google, Youtube etc work perfectly and I get 940mbits on my gigabit connection. A long shot but it may be your card unless someone brighter than I (i.e, everyone :D ) can shed some light.
See, and I was wondering if it being my Realtek cards was a possibility, but why would it be JUST Google things? Super weird. I planned on trying out a different card for testing purposes, but would ideally likely to have it confirmed before I go buy a new card.
Maybe HTTP/3 and the QUIC protocol could be the culprit why only google is affected. Youtube heavily relies on QUIC these days. It's many small UDP packets.
https://en.wikipedia.org/wiki/HTTP/3
Maybe you could only allow TCP 443 to the internet, and not UDP 443, and see if that makes a performance difference for you.
I just set up my firewall to block traffic from my LAN to anything IPv4/IPv6 UDP port 443 or port 80. Still have the issue. :/ But let me know if that's not how I should do it / if there are any other steps needed.
Decided to post a reply saying that the issue, though still not technically solved, is solved on my end because I bought new hardware.
I'm assuming the issue is something to do with the combination of the Dell machine, the Realtek cards, and the version of FreeBSD / OPNSense, as the new machine I have is using Intel I225-V interfaces, and everything works perfectly now. YouTube and Google Play Store work nicely, and my download speeds are still 1 GB everywhere else in my network.
Realtek NICs can be hit and miss sometimes...
But did you maybe try install the Realtek NIC plugin on OPNsense?
Could help.
Regards,
S.
Yeah, I've noticed that! ;D
Yeah, I installed the Realtek plugin and it still gave me issues. But I think I should be good to go now!
Sad to hear Realtek plugin didnt help in your case,
The biggest pain, when buying a new HW for OPN at least for me is to pick such that has good NICs. Sometime even Intel isn't without fault (some of their i225 revisions were just bad). By default I avoid Realtek for BSD related networking and OPN in order to have one less worry.
Happy to hear you could get Intel NICs and all is good now.
Regards,
S.
I also have issues with Google Play Store and YouTube, but my symptom is a bit different but still very annoying.
- YT: Some of the videos won't load at all while most of the videos work fine. I'm using NewPipe, so I can see in its network error dump that the video's random CDN domain is resolved to the IP (not a DNS issue), but still, it won't load at all.
- Play Store: The store generally works fine for browsing, installing apps is also fine but some of the apps won't update or just partially. The symptom is that the app update process is stuck in pending forever, or loads to some % value (fully random between 1 and 99%), and then the update can't finish ever. Just a few minutes ago I could update ChatGPT from OpenAI but Firefox Focus and Google Calendar can't update.
All my 3 different Android devices produce the same in different subnets even, 2 phones and 1 tablet.
First, I thought it's a Unifi AP issue as both YT/PS would work on LTE (4G or 5G), just not on wifi.
But then I was on Wireguard VPN to my home network from a phone, and YT/PS would also produce the same issue. Wireguard is hosted by OPNsense, and it has nothing to do with the Unifi AP. This is why I landed on that it's an OPNsense issue.
As I remember back, the symptoms started to appear around mid December after an OPNsense upgrade but I can't remember the from-to versions. Since then, I updated to the latest version even, but the problem didn't go away. No hardware changes happened in the meantime, and everything was fine before.
Now the only way to update some apps from Play Store is to go LTE and use precious mobile data, and also skip some videos on Wifi that are unwatchable. :(
I have Intel NICs BTW. OPNsense is virtualized in Proxmox with more than enough resources.
For the last week I have been doing a lot of research about this topic since I experienced the same issue.
My Setup is a HP Thinclient 730 with a Realtek 4 port 2.5gb card (RTL8125).
What I did was the following:
- I disabled all functionalities to make sure my firewall was clean
- Monitored what what traffic was blocked to see if there was any patern
- Tried all the tips and advises from other posts (non if it worked)
- Allow listed all Google Services in the firewall by creating an Alias for the following list
https://www.gstatic.com/ipranges/goog.txt
But the strange thing was I didn't see any traffic of google being dropped, while still Youtube was performing bad and the playstore took ages to download a simple app. Since somebody mentioned that his fix was replacing the Realtek card for an Intel I went to see if Realtek maybe had new FreeBSD drivers, but the plugin package provided by OPNsense has the latest version.
So I was all out of options and ordered an Intel I225-V this week, it arrived today and guess what all my problems are over. Even have a feeling everything runs much more stable now.
In case you are interested I ordered this card:
https://www.amazon.nl/dp/B0C6FGQD9V
And returned this card:
https://www.amazon.nl/dp/B08VNWKLWP
Oh, wow, that's strange, indeed.
These are my NICs:
1G WAN from my mobo: https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U#Specifications
Device-1: Intel I210 Gigabit Network vendor: ASRock driver: igb v: kernel pcie: speed: 2.5 GT/s
lanes: 1 port: e000 bus-ID: 26:00.0 chip-ID: 8086:1533 class-ID: 0200
IF: enp38s0 state: up speed: 1000 Mbps duplex: full mac: ****
10G LAN from Intel X520-DA2 (2x SFP+): https://www.intel.com/content/dam/doc/product-brief/ethernet-x520-server-adapters-brief.pdf
Device-4: Intel Ethernet 10G 2P X520 Adapter driver: ixgbe v: kernel pcie: speed: 5 GT/s
lanes: 8 port: f000 bus-ID: 2d:00.1 chip-ID: 8086:154d class-ID: 0200
IF: enp45s0f1 state: up speed: 10000 Mbps duplex: full mac: ****
Both added to 1-1 Linux bridges in Proxmox, and OPNsense has 1-1 Virtio interfaces to these bridges.
I also tried to change them from Virtio to Intel E1000 in Proxmox but OPNsense didn't recognize them afterwards, so I needed to revert the settings back and restore OPNsense from backup as the interface settings got permanently damaged in the VM somehow, it couldn't match the virtual interfaces to its settings anymore.
The LAN shouldn't affect anything IMHO as the LTE Wireguard connection was only going through the WAN NIC. No packets should go out of the LAN NIC to the switch and then to the wifi AP in this case.
The mobo also has 1 x Realtek RTL8211E for dedicated IPMI but I'm not sure if that could be used for anything else, and I've read that that IF stays up during shutdown but the i210 NICs don't, so I wouldn't experiment remapping it as WAN.
My main question in all this is what happened in mid Dec that started producing this "selective packet loss" or something. No HW changes were made, just the updated OPNsense.
Now I tried to shuffle around the NICs, since I have 2x1G i210 ports, 2x10G X520-DA2 ports and a few SFP+ copper dongles that can handle 1/2.5/5/10G fine:
- WAN on 1G different port than my original setup, LAN on 10G as original - issue persists
- WAN and LAN on 1G - issue persists
- WAN and LAN on 10G - issue persists
I don't see any change across ports on these NICs, I think this is not a HW problem for me :( And everything worked fine until mid-Dec and works fine even now, except most Play Store app updates and a few Youtube videos :o This is very annoying ::)
I did another test with a 4G LTE USB modem I have as a backup WAN (ZTE MF79U), and well, all Android Play Store updates work ::) The whole USB device is passed through from Proxmox to OPNsense, and it's mapped to an interface/gateway as an Ethernet device. So this means, when I have WAN on a 3rd device other than my original 2 Intel NICs, the symptoms are gone. Interesting. However, this doesn't explain at all what the problem really is, how it started at some point and how could I eliminate it.
I bought an Intel X710-DA2 to replace the X520-DA2 as final resolution after tuning everything possible.
And voila, the issue is gone! 8)
All Android devices are now able to update all the previously stucked apps, and the YT videos previously unplayable became playable again.
WOW, a massive headache now gone!
Why X710? It uses the i40e driver and it's PCIe Gen3, while the X520 used ixgbe and it's PCIe Gen2, and I wanted something that is different in both hardware (the X710 is a lot newer) and in driver as well. I don't know what the original issue was but I wanted to solve it once and for good. Maybe this was an issue with the HTTP/3 UDP-based QUIC protocol on the old hardware with an unlucky combination of host and guest kernels? We'll never know, but it's now solved at last.
If anybody else faces the issue, here are the details of the cards for reference:
X520-DA2
lspci
2d:00.0 Ethernet controller: Intel Corporation Ethernet 10G 2P X520 Adapter (rev 01)
Subsystem: Intel Corporation 10GbE 2P X520 Adapter
Kernel driver in use: ixgbe
Kernel modules: ixgbe
lshw
description: Ethernet interface
product: Ethernet 10G 2P X520 Adapter
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:2d:00.1
logical name: enp45s0f1
version: 01
size: 10Gbit/s
capacity: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre 10000bt-fd
configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=6.5.11-8-pve duplex=full
firmware=0x8000042f latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
resources: irq:204 memory:fbd80000-fbdfffff ioport:f000(size=32) memory:fbf00000-fbf03fff
memory:fbd00000-fbd7ffff memory:c0200000-c02fffff memory:c0300000-c03fffff
X710-DA2
lspci
2d:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
Subsystem: Intel Corporation Ethernet 10G 2P X710 Adapter
Kernel driver in use: i40e
Kernel modules: i40e
lshw
description: Ethernet interface
product: Ethernet Controller X710 for 10GbE SFP+
vendor: Intel Corporation
physical id: 0.1
bus info: pci@0000:2d:00.1
logical name: enp45s0f1
version: 02
size: 10Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre autonegotiation
configuration: autonegotiation=off broadcast=yes driver=i40e driverversion=6.5.11-8-pve duplex=full
firmware=6.80 0x80003d72 18.8.9 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
resources: irq:76 memory:f8000000-f8ffffff memory:fa000000-fa007fff memory:fcd00000-fcd7ffff
I am extremely disheartened to learn that I will have to spend even more money on hardware, after already barely keeping my whole 2.5gb LAN upgrade under budget by buying Realtek NICs, which work just fine in all my other machines (which are all Linux or Windows) . However, I guess I can finally stop banging my head against the wall trying to solve my OPNsense config, so that's good.
We do have intel cards, and xeon cpus and most time QUIC/UDP443 not passing the gateway.
sometimes we get some handshake packets back, and then we have big problem. browsers or apps do not switch back to tcp, so everything locks up. if theres just absolute no response loading websites takes 2-3 seconds longer, but as soon as the client does receive some packets back its locked to QUIC.
New Hardware is not a Solution, since we do have kind of high end hardware.
I registered just to respond to the comment above saying hardware isn't the solution: yes, you're right. It's software. But the drivers between the two cards do differ.
I once had an issue like this with an Intel X550-T2 on pfSense 2.7.0, using the ixgbe driver. I had a constant CPU usage or about 10% and some general slowdown and I couldn't understand why. I went back to my previous i225-LM and everything went back to normal. I tried yet another X550-T2 and had the same issue. I updated the X550-T2's firmware and same issue. I ultimately went back to the two onboard Intel Gigabit NICs my computer's motherboard has.
So, just chiming in, it's not their imagination. The ixgbe drivers are weird.
###################
EDIT: my original report on the pfSense subreddit
DHCPv6 server "high" CPU usage on Intel X550-T2, but not on Intel i225-LM
I'm running pfSense 2.7.0 CE and I noticed that, after replacing my 2 port QNAP i225-LM with an X550-T2, not only the CPU usage is generally slightly higher, but the DHCPv6 server process seems to constantly use 3%+ CPU, whereas is used almost 0% with the i225-LM. Disabling DHCPv6 server gets rid of that, but general CPU usage is still a little higher.
All settings are the same. TCP offloading is ON, all other offloading settings are disabled.
Just wondering if anyone here knows of anything (be it in the ixgbe / igb drivers or whatnot) that could cause that?
I also tried updating the NVM on the X550-T2 to version 3.60 (the card came with version 1.00 installed) and, other than losing ASPM for some reason, nothing changed.
EDIT: I switched back to the i225-LM and everything is back to normal. I switched to a second X550-T2 and it displayed the same behavior again.