OPNsense - Multi-WAN - established VPN connection still using Tier 2 Gateway

[schtebo]

Hi everyone,
I have successfully made an OPNsense - Multi-WAN configuration. **yeahh** Thank you for great documentation.
The tests were also successful, only with established VPN connections I have a strange behavior.

I have 2 gateways in a gateway group

Tier 1 100Mbps
Tier 2 5Mbps

If I boot the OPNsense and all gateways work as expected, the VPN connections are fast and I feel (Reporting -> Traffic) like I'm going through the Tier 1 gateway.
However, if a failure occurs on Tier 1, tier 2 gateway on the gateway group takes over as expected.
Everything as expected so far.

However, if Tier 1 Gateway is available again, the established VPN connection is still using Tier 2 Gateway.
New connections are established via Tier 1.
Is there a way to "force" all also existing connections to use Tier 1 Gateway as well?

Thank you

[franco]

Hello,

Which version? Details matter.


Cheers,
Franco

[schtebo]

I'm sorry for that. We run on:

Version:
OPNsense 23.7.2-amd64
FreeBSD 13.2-RELEASE-p2

More details:
Trigger level in gateway group is set to "packet loss"
All other values/options are set to default.

It's a Zero Trust Tunnel by Cloudflare:
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/install-and-setup/tunnel-guide/remote/

[franco]

Ok, thanks. 23.7.3 should not change that picture then.

FWIW, if both gateways are online the sessions might stick to secondary just because of stateful firewaling and have no reason to be force-closed. The problem eventually sorts itself.

We could add some sort of "swing back" state killing here optionally but all it will do is disrupt existing and working connections most likely.


Cheers,
Franco

[schtebo]

great news upgraded right now to OPNsense 23.7.3-amd64.
i will check and report if the behaviour is better now.
thank you very much!

I would very much welcome this option, as on my side the Tier 2 gateway is limited (100GB / month) and after that, all connections are slowed down to 64kbit/s.
In my case, a short interruption is much better than reaching the monthly limit.

Thank you very much I really appreciate your work!

[franco]

Sorry, typo. I missed a "not" on that 23.7.3 sentence.

[moroznah]

Same problem on multiple systems.
OPNsense 23.7.10_1-amd64
I think this is because gateway groups are not selectable in OpenVPN settings -> Interface. It is possible to set a gateway group in settings on pfSense, VPN is switching back to main from failover as expected.
I've read multiple forum posts with similar issues, it seems that common solution is to create separate client instances for every WAN and failover between them. Unfortunately this is not possible with my setup.
This could be solved by creating a cronjob that will ping via WAN and restart VPN instance if necessary, however in my book i'd call it an ugly hack.
Is there a reason why OPNsense will not allow setting OpenVPN interface as gateway group?

Regards,
Igor