Multi-WAN + public IP pool setup: some connection drops/timeout

[CDuv]

I have multi-WAN + pool public IP (round robin) OPNsense v18.1.7 setup where users are randomly experiencing blank web pages / timeout issues. On their side it seems the website takes ages to respond (when it does). It not all websites and not always.

It was running fine on 17.7 but when I upgraded to 18.1.1 I stumbled on a alias + Outbound NAT bug: outbound rules could not be loaded and got the error "There were error(s) loading the rules: no IP address found for PUBLICIPS_WAN_A".
So I disabled my round robin rule the time I understand the situation and a fix is created.
Version 18.1.7_1 fixed it: outbound NAT rules loads successfully.
But now, I have this blank web pages / timeout issue.

Here is my setup: 2 WANs, outgoing Internet traffic is loadbalanced between the 2, one of the WAN have a pool of public IP addresses I use with round robin.

• I have 2 WAN connections: WAN_A and WAN_B.
• I have a gateway group (GW_LB) containing both WAN_A and WAN_B at tier 1 (for loadbalancing).
• I have a firewall rule on LAN interface that defines GW_LB as the gateway for LAN clients.
• I have an alias (PUBLICIPS_WAN_A) containing the 9 public IP addresses my WAN_A's ISP gave me.
• Firewall: NAT: Outbound is set to "Hybrid outbound NAT rule generation (automatically generated rules are applied after manual rules)"
• I have an outbound NAT rule on interface WAN_A that actives the round robin on pool : interface=WAN_A, source=LAN, Translation/target=PUBLICIPS_WAN_A, pool_option="Round Robin with Sticky Address").
• In page "Firewall: Settings: Advanced", in section "Multi-WAN", I have "Sticky connections" checked, "Shared forwarding" unchecked, and "Disable force gateway" unchecked

My desired behavior:
Outgoing Internet traffic goes through one of the two Internet connection: WAN_A or WAN_B (if one is down, traffic will go through the other).
Whatever outgoing Internet traffic that ends up going through WAN_A have to use any of the 9 public IP addresses defined in alias PUBLICIPS_WAN_A.

My analysis:
My instinct tells me some outgoing traffic is going out via one IP of the pool (or at least tagged as such by OPNsense) but it's response arrives on one other IP...

Do you find the detailed configuration correct/adequate ?
Do you have any tips on how I could debug the (random) event of outgoing traffic that gets lost?

[CDuv]

Oups, I posted that post on the wrong Forum, it should go to "18.1 Production Series" (https://forum.opnsense.org/index.php?board=26.0), I'll warn the moderator for topic-move.
Sorry for the trouble...
Thanks for the move.

[mimugmail]

Shared Forwarding and Disable force gateway can be checked too.

What I would test is a tcpdum on WAN2 with net from WAN1 to see if there's a match.

[CDuv]

I ran 2 "Packet Capture" tests :
One to capture any "misplaced" traffic incoming on WAN_A:

• Interface: WAN_A
• Promiscuous:
• (checked)
• Address Family: Any
• Protocol: Any
• Host Address: The single IP address of WAN_B
• Port: (empty)
• Packet Length: (empty)
• Count: 50

And another to capture any "misplaced" traffic incoming on WAN_B:

• Interface: WAN_B
• Promiscuous:
• (checked)
• Address Family: Any
• Protocol: Any
• Host Address: "1.2.3.4 or 5.6.7.8/29" (All the 9 IP address of PUBLICIPS_WAN_A)
• Port: (empty)
• Packet Length: (empty)
• Count: 50

Both captures ended up empty.

[CDuv]

Enabling "Shared forwarding" and "Disable force gateway" in "Firewall: Settings: Advanced" made things worse (so I re-disabled them)

I am saying "worse" because I experienced the connection timeout symptom my colleague are complaining about since this morning, which is new because I never experienced it before.

Also, some users had no problem this morning but experienced the problem this afternoon (after I tried some settings).

[CDuv]

I don't understand german but could the issue reported in "NAT rotiert virtual IPs" (https://forum.opnsense.org/index.php?topic=7438.0) be similar to mine? (a Google Translate read seems to say yes...)

[mimugmail]

In this post the guy claims the pool is building nat with round robin and after setting sticky nat as default it's noch changing anymore. Your problems seems a bit different since it's related to Multiwan too.

[Davesworld]

I too use multi-wan with load balancing but enabling sticky caused long dropouts. Without it of course there is the problem of some secure sites not liking the dance between more than one wan ip but most handle it just fine.

As of now without sticky, I can stream movies over Amazon, Hulu, Netflix etc and get aggregated bandwidth, with two 7mbs dsl wans I can stream at 13 or so mbs and watch UHD streams without a hitch so yes, it does more than just balance to one or the other as implied by many depending on the other end. I've had downloads aggregate them as well. Both connections do not go through the same gateway at the CO since one is the Incumbent who owns the actual dsl and wiring and the other is through a reseller who uses their infrastructure. Arp shows two different MAC addresses for the two gateways, I already mentioned both gateways have different IP addresses.

Still, it would be nice if all this worked with sticky secure socket connections precluding the need for some firewall rules, namely to my email server administration gui. The latest update touches on something related but I do not know what the fix really did.