unable to maintain stable connection. (SEMI-SOLVED)

[tanto.g]

when disabling multi wan (3 of the 4 wan ports) all works perfectly. Will troubleshoot multiwan settings.



multi wan, (x4) gateway is GWGRP all tier 1, same weight.
whenever i connect out, i get interrupts. created FW rules to allow all/all with no limits (except netbios) to eliminate rules causing this.
disabled flowcontrol on all nics via sysctl

testing it by going to netflix to see if I can maintain a stable connection and am unable to stream, experiencing same with voip testing (MS teams)

netstat below shows that when i load nflx traffic goes up, however it then drops, and streaming never really starts.
Any ideas, where to look ?
sticky and shared forwarding are disabled, no IDS, unbound is working,
most of the time the error i get for http/s  is ERR_CONNECTION_TIMED_OUT, however the page will load a few seconds later with no issues.


root@fw:~ # netstat -ihw 1
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
        31     0     0       9.6K         36     0       9.3K     0
        14     0     0       2.7K         13     0       2.6K     0
        18     0     0       3.8K         18     0       3.8K     0
        33     0     0       8.0K         40     0        19K     0
        16     0     0       3.3K         18     0       3.4K     0
        29     0     0       8.3K         36     0        12K     0
        11     0     0       3.7K          9     0       3.6K     0
       117     0     0        41K        128     0        19K     0
        42     0     0        14K         43     0        13K     0
        56     0     0        20K         64     0        33K     0
        91     0     0        34K        101     0        17K     0
       187     0     0       130K        182     0       130K     0
       125     0     0        41K        130     0        32K     0
        73     0     0        27K         77     0        22K     0
       445     0     0       273K        466     0       231K     0
      5.7k     0     0       5.4M       5.7k     0       5.4M     0
      6.5k     0     0       6.6M       6.5k     0       6.6M     0
      4.8k     0     0       4.9M       4.8k     0       4.9M     0
      4.6k     0     0       4.6M       4.6k     0       4.6M     0
        21     0     0       4.9K         22     0       4.9K     0
        28     0     0        17K         30     0        16K     0
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
        31     0     0       6.8K         36     0        18K     0
        85     0     0        48K         85     0        48K     0
        34     0     0       6.2K         35     0       6.2K     0

[tanto.g]

OPNsense 20.7.4-amd64 with multi wan, GW is all wan ports in a group with tier 1 and equal weights for LB purposes. All wan are served by DHCP
internal is LAN, static assigned to nic, with dhcp servers behind it for clients.
Unbound for DNS, and resolution works as expected.

When connecting via http/s sporadically the page will no show up with the error of  ERR_CONNECTION_TIMED_OUT. If we wait a bit then the page will load. some time later when refreshing said page, sporadically the same error comes back, and will go away a few second later.

To replicate, MS teams/discord will sporadically state that it is not connected, and will reconnect a short while later. The best way I found to force this issue, is by attempting to stream anything. because streaming wont allow me to habe interrupted connections. Youtube, Netflix, amazon, etc... all can not stream. When loading up initially I see a spike in network traffic, which dies down after a few seconds and never comes back, and streaming never commences.