MultiWan and VLAN Firewall Rules

[wiesel2482]

Hi,
I've got Version 20.7.4 and have a problem with configuring the Firewall Rules for my WAN Gateway Group. On Friday I''ve bought a 5G/LTE Router (ZTE MC801A) with a provider contract to use it as a Load Balancing opportunity for my normal Internet home connection (WISP). Cause we have two teenagers who use the Internet connection externsivley with Home Schooling and a Dad who need it for Home Office 
Ok enough of explaining!
I have a setup with VLAN's and two ISP's. I've tried to bring my normal Clearnet VLAN to work with my Gateway Load Balancing Group, but everytime I switch my Firewall Rule for Outgoing Wan traffic to use the GW Group everything stop working like the MGMT Webui of OpnSense (can't reach anymore) and the Internet connection. Don't have any glue what I'am doing wrong. Tried tons of tutorials and how to's from OpnSense Official down to reddit and something else....
If I deactivate the Clearnet to any rule (you can see it in the Firewall Rule Screenshot) then everything stop working. With this any rule the GW Group is also not working cause this rule will allow the whole traffic out without any policy routing from what I understand and this is what I don't want....Yes I'am a noob 
Maybe somebody can help a stressed dad to get the family peace back to normal....
I add screenshots of my Firewall Rules and other configs.
Thank you
Regards
Wiesel

[wiesel2482]

Maybe I have the same problem -> https://forum.opnsense.org/index.php?topic=17116.0
Don't know but I have sticky connection activated...should I deactivate it ? But then it would not work for some Websites and Services or I'am wrong?

[wiesel2482]

I've now deactivated the sticky connection check and now internet works without the clearnet to any rule... But opnsense mgmt webui doesn't work at all.... What is wrong?

[wiesel2482]

Ok some Update: I reinstalled OpnSense and made all configurations fresh. Made everything in Firewall Rules for my VLAN as mentioned in the opnsense docs. Made a default VLAN to any rule with the Load Balance GW Group. Also made the DNS rule. Also made a VLAN to all Local Subnets with default GW above the rule with the Load Balance GW Group. The Problem with losing connection to the Opnsense Webui  and after some time also the internet connection is gone persists.....

I really appreciate some help. Don't have any glue what's going on.

Regards
 

[wiesel2482]

For more information I send you some screenshots. Maybe somebody see any problem
PART 1

[wiesel2482]

For more information I send you some screenshots. Maybe somebody see any problem
PART 2

[Gauss23]

Ok, first of all, please provide a graphical network plan with IP addresses and VLAN definitions.

It's very hard to understand why a rule that should do something towards WAN will lock you out completely from the WebGui of OPNsense. Never happened to me.

[wiesel2482]

Thank you! I've already found out what the issue was! If you use sticky connections and shared forwarding then you experience that the internet connection drops if you activate the policy based routing rule in the firewall with the load balancer group. Found an old Github bug report about this and tried it... Et vous la it works if you deactivate shared forwarding and leave sticky connection enabled. So maybe this is a bug from past that never been corrected... I don't know!? The problem is now that I don't know if traffic shaping is working without this shared forwarding setting. Anyone knows?
Regards

[mimugmail]

Does the  internet drop because nothgin works or do you just see dropped packets but internet in general is working?

[wiesel2482]

If I activate both options in Multiwan Settings in Advanced Firewall Settings than Internet gone for my clients in the VLAN. If I look into the Firewall Live he drops every package that goes to both Wan connections alternatley with the default deny rule. Sometimes the connection works and my clients show some websites as long as the connection doesn't switch from one wan to the other wan -> Screenshot

When I deactivate Sticky Connection it starts working normal and Opnsense using both Wan connections. Also if I deactivate Shared Forwarding and leave sticky connection activated.

[mimugmail]

OK, I reread my old thread collection in #38, it seems this is really a limitation. Was long ago and not in my mind. I'll have a talk to Franco, but no promise, shared forwarding is kind of a hack

[wiesel2482]

ok thank you very much. Maybe there is a possibility..... The problem is that I want to use Traffic Shaping with both Wan connections. And this only possible with Shared Forwarding from what I understand.

[mimugmail]

And there are many sites mit working when using SF with  sticky disabled?

Last time I had a customer with loadsharing 2 x 1gbit svdsl and with Speedtest we reached 1,9 gbit

[wiesel2482]

Yes if disable "Sticky Connections" and leave SF enabled Wan Group is working as expected. It's switching between the two Wan connections. I have a WISP connection with 30/10 Mbps and a 4G with a maximum of 100/40 Mbps. 

[mimugmail]

And do you experience any problems with this setting? If everything is ok, maybe just the documentation needs an update