OPNsense Forum

Archive => 20.7 Legacy Series => Topic started by: wiesel2482 on November 08, 2020, 12:01:03 pm

Title: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 08, 2020, 12:01:03 pm
Hi,
I've got Version 20.7.4 and have a problem with configuring the Firewall Rules for my WAN Gateway Group. On Friday I''ve bought a 5G/LTE Router (ZTE MC801A) with a provider contract to use it as a Load Balancing opportunity for my normal Internet home connection (WISP). Cause we have two teenagers who use the Internet connection externsivley with Home Schooling and a Dad who need it for Home Office  ;D
Ok enough of explaining!
I have a setup with VLAN's and two ISP's. I've tried to bring my normal Clearnet VLAN to work with my Gateway Load Balancing Group, but everytime I switch my Firewall Rule for Outgoing Wan traffic to use the GW Group everything stop working like the MGMT Webui of OpnSense (can't reach anymore) and the Internet connection. Don't have any glue what I'am doing wrong. Tried tons of tutorials and how to's from OpnSense Official down to reddit and something else....
If I deactivate the Clearnet to any rule (you can see it in the Firewall Rule Screenshot) then everything stop working. With this any rule the GW Group is also not working cause this rule will allow the whole traffic out without any policy routing from what I understand and this is what I don't want....Yes I'am a noob  ::)
Maybe somebody can help a stressed dad to get the family peace back to normal.... :'(
I add screenshots of my Firewall Rules and other configs.
Thank you
Regards
Wiesel
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 08, 2020, 12:41:00 pm
Maybe I have the same problem -> https://forum.opnsense.org/index.php?topic=17116.0
Don't know but I have sticky connection activated...should I deactivate it ? But then it would not work for some Websites and Services or I'am wrong?
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 08, 2020, 01:33:43 pm
I've now deactivated the sticky connection check and now internet works without the clearnet to any rule... But opnsense mgmt webui doesn't work at all.... What is wrong?
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 10, 2020, 09:59:51 am
Ok some Update: I reinstalled OpnSense and made all configurations fresh. Made everything in Firewall Rules for my VLAN as mentioned in the opnsense docs. Made a default VLAN to any rule with the Load Balance GW Group. Also made the DNS rule. Also made a VLAN to all Local Subnets with default GW above the rule with the Load Balance GW Group. The Problem with losing connection to the Opnsense Webui  and after some time also the internet connection is gone persists.....

I really appreciate some help. Don't have any glue what's going on.

Regards
 
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 10, 2020, 03:44:26 pm
For more information I send you some screenshots. Maybe somebody see any problem
PART 1
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 10, 2020, 03:45:43 pm
For more information I send you some screenshots. Maybe somebody see any problem
PART 2
Title: Re: MultiWan and VLAN Firewall Rules
Post by: Gauss23 on November 10, 2020, 09:07:12 pm
Ok, first of all, please provide a graphical network plan with IP addresses and VLAN definitions.

It's very hard to understand why a rule that should do something towards WAN will lock you out completely from the WebGui of OPNsense. Never happened to me.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 11, 2020, 05:14:49 pm
Thank you! I've already found out what the issue was! If you use sticky connections and shared forwarding then you experience that the internet connection drops if you activate the policy based routing rule in the firewall with the load balancer group. Found an old Github bug report about this and tried it... Et vous la it works if you deactivate shared forwarding and leave sticky connection enabled. So maybe this is a bug from past that never been corrected... I don't know!? The problem is now that I don't know if traffic shaping is working without this shared forwarding setting. Anyone knows?
Regards
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 17, 2020, 06:36:50 pm
Does the  internet drop because nothgin works or do you just see dropped packets but internet in general is working?
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 17, 2020, 08:23:22 pm
If I activate both options in Multiwan Settings in Advanced Firewall Settings than Internet gone for my clients in the VLAN. If I look into the Firewall Live he drops every package that goes to both Wan connections alternatley with the default deny rule. Sometimes the connection works and my clients show some websites as long as the connection doesn't switch from one wan to the other wan -> Screenshot

When I deactivate Sticky Connection it starts working normal and Opnsense using both Wan connections. Also if I deactivate Shared Forwarding and leave sticky connection activated.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 17, 2020, 08:32:38 pm
OK, I reread my old thread collection in #38, it seems this is really a limitation. Was long ago and not in my mind. I'll have a talk to Franco, but no promise, shared forwarding is kind of a hack
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 17, 2020, 08:39:52 pm
ok thank you very much. Maybe there is a possibility..... ::) The problem is that I want to use Traffic Shaping with both Wan connections. And this only possible with Shared Forwarding from what I understand.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 17, 2020, 09:44:15 pm
And there are many sites mit working when using SF with  sticky disabled?

Last time I had a customer with loadsharing 2 x 1gbit svdsl and with Speedtest we reached 1,9 gbit
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 18, 2020, 09:01:43 am
Yes if disable "Sticky Connections" and leave SF enabled Wan Group is working as expected. It's switching between the two Wan connections. I have a WISP connection with 30/10 Mbps and a 4G with a maximum of 100/40 Mbps. 
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 18, 2020, 03:57:01 pm
And do you experience any problems with this setting? If everything is ok, maybe just the documentation needs an update :)
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 18, 2020, 06:45:22 pm
When I disable "Sticky" and enable "SF" then most of the time the traffic shaping works, but sometimes the bufferbloat returns in the "dslreport" speedtest. I'am using shaping prior to do some FQ-CODEL on my lines cause as you know I'am using 2 Wireless WAN connections and this helps a lot against bufferbloat -> Thx to Opnsense. But I can't even tell, if there is a problem with timeout sessions, if I don't have activate "sticky". So I don't know exactly if it works as expected. For me personally it is a little bit odd, that I can't activate both options without breaking my Wan connections. As you know I'am really happy that such a expert as you are, tries to help me with this.  ;)
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 18, 2020, 08:21:08 pm
Loadsharing uses the firewall software pf, while shaping uses software ipfw. To mix both a kernel Hack is required. Maybe there is alimitation for using both at a time.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on November 19, 2020, 03:47:49 pm
Thank you! I've already found out what the issue was! If you use sticky connections and shared forwarding then you experience that the internet connection drops if you activate the policy based routing rule in the firewall with the load balancer group. Found an old Github bug report about this and tried it... Et vous la it works if you deactivate shared forwarding and leave sticky connection enabled. So maybe this is a bug from past that never been corrected... I don't know!? The problem is now that I don't know if traffic shaping is working without this shared forwarding setting. Anyone knows?
Regards


hmmm seems to be related issue. in my case if both shared at sticky enabled, and multiwan policy selected connectionn will work intermittent... you would need to refresh most of the time which is anoying, connection is notmal if using one gateway or backup mode.
never tried disabling sticky... as it may destroy legit site that require uniue IP. ... well now i migth need to try....
or maybe disable shaping? then again I also have captive portal turned on...
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 19, 2020, 03:51:12 pm
Thank you! I've already found out what the issue was! If you use sticky connections and shared forwarding then you experience that the internet connection drops if you activate the policy based routing rule in the firewall with the load balancer group. Found an old Github bug report about this and tried it... Et vous la it works if you deactivate shared forwarding and leave sticky connection enabled. So maybe this is a bug from past that never been corrected... I don't know!? The problem is now that I don't know if traffic shaping is working without this shared forwarding setting. Anyone knows?
Regards

hmmm seems to be related issue. in my case if both shared at sticky enabled, and multiwan policy selected connectionn will work intermittent... you would need to refresh most of the time which is anoying, connection is notmal if using one gateway or backup mode.
never tried disabling sticky... as it may destroy legit site that require uniue IP. ... well now i migth need to try....
or maybe disable shaping? then again I also have captive portal turned on...

If you have a specfiic site which required one IP you could turn of sharing for this one destination IP with setting a gateway instead of group in firewall rules
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on November 19, 2020, 03:58:06 pm
that would be a problem to list specific sites that would require such (banking sites perhaps) but i migth try that.

is this issue specific to opnsense?
or cant sticky connection may be route servers/site to same gateway until no more connection uses the site. it may not multiwan correctly but should not be a problem.

but in may case it is intermittent, which means it is working at some point or level... it just goes down or a few second then back... i just dont know what happens or why
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 19, 2020, 04:20:32 pm
It's a limitation of OPNsense when combining shaping/captive portal and loadsharing (with sticky).
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 19, 2020, 04:45:22 pm
that would be a problem to list specific sites that would require such (banking sites perhaps) but i migth try that.

is this issue specific to opnsense?
or cant sticky connection may be route servers/site to same gateway until no more connection uses the site. it may not multiwan correctly but should not be a problem.

but in may case it is intermittent, which means it is working at some point or level... it just goes down or a few second then back... i just dont know what happens or why

For me it's the same. Now I tested both checked (sticky and sf) and it works for some seconds before it's gone. Then it comes back and it goes again......
It everything goes into default deny rule and sometimes it passes the traffic from the wan.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on November 20, 2020, 06:43:47 am
I have now tried to disable sticky connection (shared connection enabled)

1 computer I am using has a multiwan policy
so far I dont have isse with disconnection or
invalid IP (ip switch to second wan), tried using ip sesitive site
but no errors yet

but this is for 1 PC only... yet
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 20, 2020, 07:31:44 am
Sounds good. Maybe we just can remove the notice in the docs since everything seems to work ok
Title: Re: MultiWan and VLAN Firewall Rules
Post by: wiesel2482 on November 20, 2020, 08:34:00 pm
For me also everything works without sticky connection....for all of my Clients. Maybe sticky this is now integrated if you're using Multiwan?
Regards
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 20, 2020, 10:46:23 pm
No, I had a similar provlem with sip and multiple hosts, didnt work
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on November 21, 2020, 07:19:28 pm
there is none focus on multiwan feature for opensense?
or any update/upgrade/improvement path?
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 21, 2020, 11:56:23 pm
Maybe there's an improvement after 21.1, but no promise
Title: Re: MultiWan and VLAN Firewall Rules
Post by: jzt308 on November 24, 2020, 12:03:46 pm
Hi,

I have the same issue (see my post: https://forum.opnsense.org/index.php?topic=20132.msg93151#msg93151). I now confirmed the issue is with enabling sticky sessions.
After a couple of  sessions (the amount seems to differ) using any connection to WAN/Internet is impossible. (until  you reset the source tracking table).
For now I disabled the sticky session tracking and all is working fine. I'm using some fw rules for sites that don't like the effect but this is not ideal.
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on November 27, 2020, 12:24:23 am
should this be enabled/checked?

just noticed it, it is checked by default
Title: Re: MultiWan and VLAN Firewall Rules
Post by: mimugmail on November 27, 2020, 08:55:15 am
Hi,

I have the same issue (see my post: https://forum.opnsense.org/index.php?topic=20132.msg93151#msg93151). I now confirmed the issue is with enabling sticky sessions.
After a couple of  sessions (the amount seems to differ) using any connection to WAN/Internet is impossible. (until  you reset the source tracking table).
For now I disabled the sticky session tracking and all is working fine. I'm using some fw rules for sites that don't like the effect but this is not ideal.

Which sites are they?
Title: Re: MultiWan and VLAN Firewall Rules
Post by: tong2x on December 09, 2020, 03:48:20 pm
i have shaper and captiveportal enable in my vlan.

been using with sticky disabled, overall it is better.
i have minor issue with youtube in mobile phone which i currently cant tell if it is really due to multiwan.

playing the vid in mobile will present and error but refrshing or moving the page down the video will reset(error will be gone). then pressing play will play the video in full.