Issue with Multi-Wan and Sticky Connections?

[db]

I was running into an odd issue with a balancing group in my multi wan setup, and disabling sticky connections seemed to fix it. I'm not sure if this is some sort of bug with sticky conns that I've happened to stumble across, or if it's just something unique about my set up (though I haven't changed too much from install).

I basically followed the recipe for setting up multi-wan. I have 3 WAN interfaces, WAN, OPT1 and OPT2. All seem to work fine. I created two gateway groups, one as failover (WAN - Tier 1, OPT1 - Tier2, OPT2 - Tier3). I created a rule on my LAN interface to drive all 'LAN net' traffic to the failover group, and this seemed to work fine, all traffic worked as expected, passing through WAN.

I also created a balancing group, with two GWs as T1 and one as T2 (WAN - Tier 1, OPT1 - Tier1, OPT2 - Tier2). When I switched the rule on my LAN interface to use this balancing group, odd things happened. What I would see is the firewall blocking traffic on the LAN interface, which would have the src IP of the public WAN interface (that is, these packets have src of ISP-provided public IP for the WAN interface). That is, I'd get firewall logs like this:

Code: [Select]

lan Mar 22 13:21:19 <WAN1 ip>:37526 205.196.6.165:443 tcp Default deny rule
lan Mar 22 13:21:19 10.0.1.11:62687 205.196.6.165:443 tcp Default to Download Balancing GWG
lan Mar 22 13:21:19 <WAN1 ip>:40375 205.196.6.142:443 tcp Default deny rule
lan Mar 22 13:21:19 10.0.1.11:62686 205.196.6.142:443 tcp Default to Download Balancing GWG
lan Mar 22 13:21:18 <WAN1 ip>:18871 205.196.6.165:443 tcp Default deny rule
lan Mar 22 13:21:18 10.0.1.11:62685 205.196.6.165:443 tcp Default to Download Balancing GWG

Odd, no? Trying to curl anything would result in 1/2 of the attempts hanging.

Disabling sticky connections seems to make everything work fine, as expected. Curling swaps between the balanced WANs, and so far I haven't seen any adverse effects. But sticky seems like a good idea in general... Any thoughts on what I might follow up with, to see if there really is some oddity with my set up, or if this is just something broken with opnsense?

[tong2x]

some thing like that

there is a thread here where in disableng sticky connection would actually make multiwan connection FAR stable than using sticky...

I think it was somethign to do with how it was implimented, based on the previous thread sticky was sort of a workaround... then again it made it worst...

It's a limitation of OPNsense when combining shaping/captive portal and loadsharing (with sticky).

by: Mimugmail

EDIT:
it was the "shared connection" that is making issues

https://forum.opnsense.org/index.php?topic=19977.msg93076#msg93076
https://forum.opnsense.org/index.php?topic=17116.msg93965#msg93965

[tong2x]

change log 21.7.2: shared forwarding edge cases
On the other hand, the work for FreeBSD 13 migration in 22.1 is ongoing as well to be able to test this rather sooner than later. In this iteration we will take the time to look at shared forwarding edge cases and have already upstreamed a number of patches that have been accumulated over the last couple of years to keep our code base light and tidy.

will this now put focus on the "sticky" connection and shared forward connection?

[mimugmail]

Sounds like progress 

[franco]

Honestly I don't know. It means the edge cases in the networking stack code are being patched proper, but that doesn't mean flip-flop balancing will suddenly work because of it. At least I don't see why it should.


Cheers,
Franco