Intra VLAN performance issues when WAN has issues

[cyrus104]

I'll start by saying that my internet has high latency of around 150ms or more and that the ISP will randomly have slow downs and high packet loss. I have been able to verify this shouldn't be my opnsense setup by using the router provided by the ISP to test with.

My OPNsense hardware is running on an AMD Epyc 3000, 32gb ram, nvme drive, 1GBe interface to ISP ONT, and a 10GBe network to my 10GBe switch. I know it's overkill but the small machine was meant for something else that I never got around to doing.

My OPNsense configuration, has 3-4 VLANs for management, users, guests, iot like things and the OPNsense does the routing.

Normally this setup works pretty well without issues but when my ISP drops packets or it's connection, all of my intraVLAN traffic suffers. I was using Plex from one VLAN to another (waiting on another interface card for Plex server to avoid this) and it wouldn't play because of how much buffer it had to do. I did a couple of tests like manually down the WAN interface and unplug the WAN interface, both of these "fixed" the internal network issues and Plex streamed across the router without an issue. I have the same issue with SMB shares dropping connections if I am trying to transfer a file.

I wanted to see if anyone had some thoughts or ideas to help me troubleshoot.

Thanks

[cyrus104]

Any thoughts on this?

[Xelas]

I'm sorry I have nothing solid to contribute, but it sounds like an issue with how the gateways between VLANs are set up. Personally, I never wrapped my head around the gateway config in opnsence, so I'm hoping seeing responses to this thread can shed some light on how the gateways work.

[cyrus104]

Yeah, I'm a little confused by this as well.

I'm having a really difficult time right now and might look at doing a double router solution to avoid the internal VLANs from being disrupted. I got a few tips from the pfsense forums but none of them were the exact problem I'm having.

I have a very poor WAN connection and it keeps dropping / super high packet drop or latency spikes and during the time the WAN is coming back up I loose all intraVLAN traffic. The main 2 rules I have are for the VLANs to talk, I used aliases with the internal address of each VLAN, gateway is * (I think is default). The second rule is an inverse of anything in the above alias which routes it out to the WAN gateway.

[errored out]

I am also running into this issue.  However, I am using OpenVPN and the connections on this FW are clients, not Servers. 

Looks like there is are existing threads on this, but last post was from 2020. 
https://forum.opnsense.org/index.php?topic=20045.0 

Additional information, but no responses.
https://forum.opnsense.org/index.php?topic=21533
https://forum.opnsense.org/index.php?topic=15818.0