Good evening all.
First time posting something on the internet, so please bare with me. I might have a short but interesting story, if you're interested. I would however greatly appreciate any inputs because I'm completely stumped at this point.
Here is the TL;DR:
Moved into a new apartment with working and tested OPNsense setup. Since the move I can't watch youtube, Netflix or any other streaming service on the internet. Google services are also heavily bandwidth restricted (like app updates and downloads). Independent of the client device, VLAN or medium. Using an old Router and circumventing OPNsense, everything works again. Please see the list below for things, I have already tried.
Here is my hardware:
- Odroid H3+ for OPNsense (Pentium Silver N6005, 8Gb RAM, ZFS Mirror, with 2x 2.5 GBit onboard + 4x 2.5 Gbit NIC (all realtek)) (labelled as Iroh in diagram)
- Mikrotik CSS610-8P-2S+IN Switch
- Ubiquiti UniFi AP FlexHD
My main DNS is quad-nine 9.9.9.9 with DNS over TLS and secondary is cloudflares 1.1.1.1. I have a 500/100mbit line.
Okay, here the detailed Version:
I moved into a new apartment a few weeks ago, so I took the opportunity to build a network from scratch. I bought an Odroid H3+ with the netcard (realtek, yuk I know. Didn't know any better.), a mikrotik PoE switch and a Ubiquiti AP. I set up OPNsense using homenetworkguy.com's guide, fitted for my own needs, of course.
It was an interesting journey and I learned some things.
After I had it setup and working, I was using the new network for a few weeks, to do a test-run, so to speak.
At that point, the box was running behind another router. I configured it, so that OPNsense was in the DMZ, so all incoming traffic was forwarded to the new box. This way, I could still use my old setup with things like nextcloud. The TV is connected to the IoT network, which is sending DNS requests to pihole (Samsung TV with loads of phone-home functions and ads I don't want). The TV also has access to Jellyfin in the Server-net and is my main testing device with this problem.
Had no issues whatsoever. In fact, I was quite pleased with my results 8) . After all, it was a lot of work.
When it was time to move into my first apartment, I disconnected only the necessary cabling and stuck basically the whole network into a moving-box. I labelled the ports I did have to disconnect, so I don't have to reconfigure the network ports on the switch. When I unpacked it, I set the WAN on OPNsense to DHCP and connected it directly to the Modem of the new provider.
Nothing else was changed.
Now I suddenly can't watch Youtube anymore.
It will load the video selection, but when I click on a video, its a 50/50 that it will load and when it loads another 50/50 that it will keep playing. It seems so random.
> enter trial and error mode.
Here is what I have tested so far:
- Connected my laptop directly to the modem -> no problems
- Hooked up an old tplink wifi router directly to the modem -> no problems
- Connected the laptop to the LAN instead of my own client-VLAN -> problem persisted
- Tried from different VLANS -> still a problem
- Disabled unbound and checked "Allow DNS server list to be overridden by DHCP/PPP on WAN" -> no good
- Clean install of OPNsense with minimal config -> still, YouTube won't behave. -> import settings again
- Go to ubuntu.com and download an iso -> abolutely no problem. 200 mbit/s easily.
- Tried streaming from Jellyfin over different VLANs -> buttery smooth.
- Enabled stats for nerds on youtube -> noticed the network speed randomly drop to 300kb -> definitely inaccurate. It's probably 0 with random small bursts of traffic coming through, according to the graph.
- SSH into OPNsense and monitor CPU with top -aSCHIP while testing -> not going over 25%
- Enabled Hardware CRC, TSO, LRO and VLAN Hardware Filtering -> no change
- Called the provider and asked, if there are any necessary config steps for the firewall. -> They answered: straight up DHCP, no VLANS, nothing special
- Tried to assign parent interface of the LAGG, but since the problem persisted after a fresh install with minimal config, I'm guessing, that's not it either.
Of course I constantly upgraded to the newest OPNsense versions, hoping it would magically resolve itself. No luck on that end either.
I'm now starting to run out of things to test and still haven't gotten any closer to a solution. If someone has an idea, I'm all ears. Because having to switch the WAN cable between the tplink router and OPNsense is really not the solution, and I'd love to stay on OPNsense, because so far, its been a blast.
Please accept my (cut down) Network-diagram as an offering. I may post the full one, once I regain access to that.
Looking forward to a discussion and thank you for your time :)