streaming issues and slow bandwidth on selected services after moving apartments

[grassowl]

Good evening all.

First time posting something on the internet, so please bare with me. I might have a short but interesting story, if you're interested. I would however greatly appreciate any inputs because I'm completely stumped at this point.

Here is the TL;DR:
Moved into a new apartment with working and tested OPNsense setup. Since the move I can't watch youtube, Netflix or any other streaming service on the internet. Google services are also heavily bandwidth restricted (like app updates and downloads). Independent of the client device, VLAN or medium. Using an old Router and circumventing OPNsense, everything works again. Please see the list below for things, I have already tried.

Here is my hardware:

• Odroid H3+ for OPNsense (Pentium Silver N6005, 8Gb RAM, ZFS Mirror, with 2x 2.5 GBit onboard + 4x 2.5 Gbit NIC (all realtek)) (labelled as Iroh in diagram)
• Mikrotik CSS610-8P-2S+IN Switch
• Ubiquiti UniFi AP FlexHD

My main DNS is quad-nine 9.9.9.9 with DNS over TLS and secondary is cloudflares 1.1.1.1. I have a 500/100mbit line.



Okay, here the detailed Version:
I moved into a new apartment a few weeks ago, so I took the opportunity to build a network from scratch. I bought an Odroid H3+ with the netcard (realtek, yuk I know. Didn't know any better.), a mikrotik PoE switch and a Ubiquiti AP. I set up OPNsense using homenetworkguy.com's guide, fitted for my own needs, of course.

It was an interesting journey and I learned some things.
After I had it setup and working, I was using the new network for a few weeks, to do a test-run, so to speak.
At that point, the box was running behind another router. I configured it, so that OPNsense was in the DMZ, so all incoming traffic was forwarded to the new box. This way, I could still use my old setup with things like nextcloud. The TV is connected to the IoT network, which is sending DNS requests to pihole (Samsung TV with loads of phone-home functions and ads I don't want). The TV also has access to Jellyfin in the Server-net and is my main testing device with this problem.

Had no issues whatsoever. In fact, I was quite pleased with my results . After all, it was a lot of work.

When it was time to move into my first apartment, I disconnected only the necessary cabling and stuck basically the whole network into a moving-box. I labelled the ports I did have to disconnect, so I don't have to reconfigure the network ports on the switch. When I unpacked it, I set the WAN on OPNsense to DHCP and connected it directly to the Modem of the new provider.
Nothing else was changed.

Now I suddenly can't watch Youtube anymore.
It will load the video selection, but when I click on a video, its a 50/50 that it will load and when it loads another 50/50 that it will keep playing. It seems so random.

> enter trial and error mode.
Here is what I have tested so far:

• Connected my laptop directly to the modem -> no problems
• Hooked up an old tplink wifi router directly to the modem -> no problems
• Connected the laptop to the LAN instead of my own client-VLAN -> problem persisted
• Tried from different VLANS -> still a problem
• Disabled unbound and checked "Allow DNS server list to be overridden by DHCP/PPP on WAN" -> no good
• Clean install of OPNsense with minimal config -> still, YouTube won't behave. -> import settings again
• Go to ubuntu.com and download an iso -> abolutely no problem. 200 mbit/s easily.
• Tried streaming from Jellyfin over different VLANs -> buttery smooth.
• Enabled stats for nerds on youtube -> noticed the network speed randomly drop to 300kb -> definitely inaccurate. It's probably 0 with random small bursts of traffic coming through, according to the graph.
• SSH into OPNsense and monitor CPU with top -aSCHIP while testing -> not going over 25%
• Enabled Hardware CRC, TSO, LRO and VLAN Hardware Filtering -> no change
• Called the provider and asked, if there are any necessary config steps for the firewall. -> They answered: straight up DHCP, no VLANS, nothing special
• Tried to assign parent interface of the LAGG, but since the problem persisted after a fresh install with minimal config, I'm guessing, that's not it either.

Of course I constantly upgraded to the newest OPNsense versions, hoping it would magically resolve itself. No luck on that end either.

I'm now starting to run out of things to test and still haven't gotten any closer to a solution. If someone has an idea, I'm all ears. Because having to switch the WAN cable between the tplink router and OPNsense is really not the solution, and I'd love to stay on OPNsense, because so far, its been a blast.

Please accept my (cut down) Network-diagram as an offering. I may post the full one, once I regain access to that.

Looking forward to a discussion and thank you for your time 

[iMx]

> Enabled Hardware CRC, TSO, LRO and VLAN Hardware Filtering -> no change

These should all be disabled, TSO/LRO especially on a router (enabling breaks things).

CRC can potentially be enabled, but only if you've got good NICs, Intel basically.

I'd probably be looking at/ruling out MTU as being the cause of the problem, to start with.

Followed by checking the port negotiation speeds/duplex and/or port errors, on the full path.

[grassowl]

Good evening iMx

I am very sorry for the late reply and thank you for your suggestion.
I had written a response the day you replied and thought I had posted it. I guess I'm just dumb.

I re-disabled CRC, TSO, LRO, and VLAN hardware-filtering, sadly without much success. Still good to know, thank you.

Regarding your MTU suggestion. I tested this by reducing the MTU on the WAN-interface by 16 bytes and verified the settings with ifconfig. Sadly, this didn't have an effect.

Furthermore, I've taken the switch, Firewall and AP back to my parents' today, to test if the problem persists. Without unplugging anything I hooked it up to the old router and didn't have any problems loading YouTube videos. Under one second from clicking to playback.
I'm now trying to get my modem replaced again, but this time, I'll try to get a router instead. Not happy about that, but maybe I can get it to work before I head to the providers shop. Fingers crossed.

I'll try to read up on testing port negotiation speed and port errors. I'm not even sure the switch has a CLI, but we'll see, when we get there. I'm not convinced, that this is the cause since the problem didn't persist after taking it back to the old router, but I'll give it a shot anyway.

Thanks again for your suggestions and sorry for not replying. This time I'll hit 'post'!

[grassowl]

Hello again

I looked at the stats on the switch, to see if there are any errors. Looks clean (see screenshot). The same with the Interface overview on OPNsense. Link speeds also look good at 2.5GBit full duplex.

I have now switched from a modem to a wifi router, that was provided by the ISP. Sadly, the problem still persists AND it can't be used in bridge mode. Kinda suspected that, still bummed out though
(Plus my homelab is no longer accessible, and the router interface is weird. But that's just me moaning)

I also ran the connectivity and health audits in System > Firmware > Status. Besides a repository update error, there was nothing noteworthy. pkg.opnsense.org was updating and the health-check didn't throw any errors. (no updates either).

I was also combing through some logs when I noticed the connection dropping. Sadly couldn't find anything.
There weren't really any entries within the past 2 hours at all...

Again, the connection-drops seem to happen at random and don't last for the same length of time. I'm at a loss...

Thanks again, cheers!