YouTube and Google Play Store poor performance on OPNSense

[GentlemanJimStacey]

I've recently created an OPNSense box. Specs:

Dell OptiPlex 7050
CPU: i5-7500
SSD: Lexar 512 GB 3000 MB/s
RAM: 8 GB (2x4) DDR4 2133
NICs: 2x Realtek RTL8125B (I know Realtek isn't the best, especially with OPNSense, but they seem to work for basically everything).

So, the issue I'm having is strange. The reason I posted in the Hardware and Performance subforum is because I'm wondering if my Realtek cards have something to do with it? No idea why they would, but I know they are buggy with OPNSense (and like all other appliance firewalls. Lol)

Basically, everything in my home network works fine, aside from YouTube and the Google Play Store (and possibly other Google apps - I haven't checked yet).
Any Apple products we have, download from their App Store perfectly fine. My desktop, which is hardwired, downloads from Steam at 100+ MB/s. However, playing YouTube videos and downloading from the Google Play Store are a huge struggle (wired, and wireless). YouTube videos constantly buffer, and switch back and forth between low and higher resolution, and downloading from the Play Store when my phone is connected to my WiFi is in the range of like 50 Kb/s.

Now before you ask if my WiFi is having issues, it's not. And again, before you ask if maybe my ISP is having issues... they're not. I did not have these problems at all until I switched to the OPNSense box as my main router/firewall. My phone runs a 300 Mb/s speed test, and my home internet is 1 Gb/s. And I have this problem on both my wired, and wireless devices. JUST YouTube and Google Play Store are slow.

Any ideas?

[GentlemanJimStacey]

Just wanting to bump this, as it's actively problematic, and would be nice to try to diagnose with folks who are more familiar with OPNSense than me!

[Cheule]

Hi, first post :)

This is just for comparison to yours as I do not have the issue you describe.

CPU: i5-6400
SSD: Integral 256 GB Nvme
RAM: 4GB DDR4 2400
NICs: 2x Intel PRO 1000

Your machine is superior to mine in both CPU and RAM but I have an Intel Pro dual gigabit NIC. Google, Youtube etc work perfectly and I get 940mbits on my gigabit connection. A long shot but it may be your card unless someone brighter than I (i.e, everyone :D ) can shed some light.

[GentlemanJimStacey]

See, and I was wondering if it being my Realtek cards was a possibility, but why would it be JUST Google things? Super weird. I planned on trying out a different card for testing purposes, but would ideally likely to have it confirmed before I go buy a new card.

[Monviech (Cedrik)]

Maybe HTTP/3 and the QUIC protocol could be the culprit why only google is affected. Youtube heavily relies on QUIC these days. It's many small UDP packets.

https://en.wikipedia.org/wiki/HTTP/3

Maybe you could only allow TCP 443 to the internet, and not UDP 443, and see if that makes a performance difference for you.

[GentlemanJimStacey]

I just set up my firewall to block traffic from my LAN to anything IPv4/IPv6 UDP port 443 or port 80. Still have the issue. :/ But let me know if that's not how I should do it / if there are any other steps needed. 

[GentlemanJimStacey]

Decided to post a reply saying that the issue, though still not technically solved, is solved on my end because I bought new hardware.

I'm assuming the issue is something to do with the combination of the Dell machine, the Realtek cards, and the version of FreeBSD / OPNSense, as the new machine I have is using Intel I225-V interfaces, and everything works perfectly now. YouTube and Google Play Store work nicely, and my download speeds are still 1 GB everywhere else in my network.

[Seimus]

Realtek NICs can be hit and miss sometimes...

But did you maybe try install the Realtek NIC plugin on OPNsense?
Could help.

Regards,
S.

[GentlemanJimStacey]

Yeah, I've noticed that!  ;D

Yeah, I installed the Realtek plugin and it still gave me issues. But I think I should be good to go now!

[Seimus]

Sad to hear Realtek plugin didnt help in your case,

The biggest pain, when buying a new HW for OPN at least for me is to pick such that has good NICs. Sometime even Intel isn't without fault (some of their i225 revisions were just bad). By default I avoid Realtek for BSD related networking and OPN in order to have one less worry.

Happy to hear you could get Intel NICs and all is good now.

Regards,
S.

[Crate2729]

I also have issues with Google Play Store and YouTube, but my symptom is a bit different but still very annoying.

- YT: Some of the videos won't load at all while most of the videos work fine. I'm using NewPipe, so I can see in its network error dump that the video's random CDN domain is resolved to the IP (not a DNS issue), but still, it won't load at all.

- Play Store: The store generally works fine for browsing, installing apps is also fine but some of the apps won't update or just partially. The symptom is that the app update process is stuck in pending forever, or loads to some % value (fully random between 1 and 99%), and then the update can't finish ever. Just a few minutes ago I could update ChatGPT from OpenAI but Firefox Focus and Google Calendar can't update.

All my 3 different Android devices produce the same in different subnets even, 2 phones and 1 tablet.
First, I thought it's a Unifi AP issue as both YT/PS would work on LTE (4G or 5G), just not on wifi.
But then I was on Wireguard VPN to my home network from a phone, and YT/PS would also produce the same issue. Wireguard is hosted by OPNsense, and it has nothing to do with the Unifi AP. This is why I landed on that it's an OPNsense issue.

As I remember back, the symptoms started to appear around mid December after an OPNsense upgrade but I can't remember the from-to versions. Since then, I updated to the latest version even, but the problem didn't go away. No hardware changes happened in the meantime, and everything was fine before.

Now the only way to update some apps from Play Store is to go LTE and use precious mobile data, and also skip some videos on Wifi that are unwatchable. :(

I have Intel NICs BTW. OPNsense is virtualized in Proxmox with more than enough resources.

[pietossie]

For the last week I have been doing a lot of research about this topic since I experienced the same issue.
My Setup is a HP Thinclient 730 with a Realtek 4 port 2.5gb card (RTL8125).

What I did was the following:
- I disabled all functionalities to make sure my firewall was clean
- Monitored what what traffic was blocked to see if there was any patern
- Tried all the tips and advises from other posts (non if it worked)
- Allow listed all Google Services in the firewall by creating an Alias for the following list
https://www.gstatic.com/ipranges/goog.txt

But the strange thing was I didn't see any traffic of google being dropped, while still Youtube was performing bad and the playstore took ages to download a simple app. Since somebody mentioned that his fix was replacing the Realtek card for an Intel I went to see if Realtek maybe had new FreeBSD drivers, but the plugin package provided by OPNsense has the latest version.

So I was all out of options and ordered an Intel I225-V this week, it arrived today and guess what all my problems are over. Even have a feeling everything runs much more stable now.

In case you are interested I ordered this card:
https://www.amazon.nl/dp/B0C6FGQD9V

And returned this card:
https://www.amazon.nl/dp/B08VNWKLWP

[Crate2729]

Oh, wow, that's strange, indeed.

These are my NICs:

1G WAN from my mobo: https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U#Specifications

Code Select Expand

  Device-1: Intel I210 Gigabit Network vendor: ASRock driver: igb v: kernel pcie: speed: 2.5 GT/s
    lanes: 1 port: e000 bus-ID: 26:00.0 chip-ID: 8086:1533 class-ID: 0200
  IF: enp38s0 state: up speed: 1000 Mbps duplex: full mac: ****

10G LAN from Intel X520-DA2 (2x SFP+): https://www.intel.com/content/dam/doc/product-brief/ethernet-x520-server-adapters-brief.pdf

Code Select Expand

  Device-4: Intel Ethernet 10G 2P X520 Adapter driver: ixgbe v: kernel pcie: speed: 5 GT/s
    lanes: 8 port: f000 bus-ID: 2d:00.1 chip-ID: 8086:154d class-ID: 0200
  IF: enp45s0f1 state: up speed: 10000 Mbps duplex: full mac: ****

Both added to 1-1 Linux bridges in Proxmox, and OPNsense has 1-1 Virtio interfaces to these bridges.
I also tried to change them from Virtio to Intel E1000 in Proxmox but OPNsense didn't recognize them afterwards, so I needed to revert the settings back and restore OPNsense from backup as the interface settings got permanently damaged in the VM somehow, it couldn't match the virtual interfaces to its settings anymore.

The LAN shouldn't affect anything IMHO as the LTE Wireguard connection was only going through the WAN NIC. No packets should go out of the LAN NIC to the switch and then to the wifi AP in this case.

The mobo also has 1 x Realtek RTL8211E for dedicated IPMI but I'm not sure if that could be used for anything else, and I've read that that IF stays up during shutdown but the i210 NICs don't, so I wouldn't experiment remapping it as WAN.

My main question in all this is what happened in mid Dec that started producing this "selective packet loss" or something. No HW changes were made, just the updated OPNsense.

[Crate2729]

Now I tried to shuffle around the NICs, since I have 2x1G i210 ports, 2x10G X520-DA2 ports and a few SFP+ copper dongles that can handle 1/2.5/5/10G fine:
- WAN on 1G different port than my original setup, LAN on 10G as original - issue persists
- WAN and LAN on 1G - issue persists
- WAN and LAN on 10G - issue persists

I don't see any change across ports on these NICs, I think this is not a HW problem for me :( And everything worked fine until mid-Dec and works fine even now, except most Play Store app updates and a few Youtube videos :o This is very annoying ::)

[Crate2729]

I did another test with a 4G LTE USB modem I have as a backup WAN (ZTE MF79U), and well, all Android Play Store updates work  ::)  The whole USB device is passed through from Proxmox to OPNsense, and it's mapped to an interface/gateway as an Ethernet device. So this means, when I have WAN on a 3rd device other than my original 2 Intel NICs, the symptoms are gone. Interesting. However, this doesn't explain at all what the problem really is, how it started at some point and how could I eliminate it.