Hi everyone,
I have successfully made an OPNsense - Multi-WAN configuration. **yeahh** Thank you for great documentation.
The tests were also successful, only with established VPN connections I have a strange behavior.
I have 2 gateways in a gateway group
Tier 1 100Mbps
Tier 2 5Mbps
If I boot the OPNsense and all gateways work as expected, the VPN connections are fast and I feel (Reporting -> Traffic) like I'm going through the Tier 1 gateway.
However, if a failure occurs on Tier 1, tier 2 gateway on the gateway group takes over as expected.
Everything as expected so far.
However, if Tier 1 Gateway is available again, the established VPN connection is still using Tier 2 Gateway.
New connections are established via Tier 1.
Is there a way to "force" all also existing connections to use Tier 1 Gateway as well?
Thank you
Hello,
Which version? Details matter. :)
Cheers,
Franco
I'm sorry for that. We run on:
Version:
OPNsense 23.7.2-amd64
FreeBSD 13.2-RELEASE-p2
More details:
Trigger level in gateway group is set to "packet loss"
All other values/options are set to default.
It's a Zero Trust Tunnel by Cloudflare:
https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/install-and-setup/tunnel-guide/remote/
Ok, thanks. 23.7.3 should not change that picture then.
FWIW, if both gateways are online the sessions might stick to secondary just because of stateful firewaling and have no reason to be force-closed. The problem eventually sorts itself.
We could add some sort of "swing back" state killing here optionally but all it will do is disrupt existing and working connections most likely.
Cheers,
Franco
great news :) upgraded right now to OPNsense 23.7.3-amd64.
i will check and report if the behaviour is better now.
thank you very much!
I would very much welcome this option, as on my side the Tier 2 gateway is limited (100GB / month) and after that, all connections are slowed down to 64kbit/s.
In my case, a short interruption is much better than reaching the monthly limit.
Thank you very much I really appreciate your work!
Sorry, typo. I missed a "not" on that 23.7.3 sentence.
Same problem on multiple systems.
OPNsense 23.7.10_1-amd64
I think this is because gateway groups are not selectable in OpenVPN settings -> Interface. It is possible to set a gateway group in settings on pfSense, VPN is switching back to main from failover as expected.
I've read multiple forum posts with similar issues, it seems that common solution is to create separate client instances for every WAN and failover between them. Unfortunately this is not possible with my setup.
This could be solved by creating a cronjob that will ping via WAN and restart VPN instance if necessary, however in my book i'd call it an ugly hack.
Is there a reason why OPNsense will not allow setting OpenVPN interface as gateway group?
Regards,
Igor
thank you so much :) --> new update (OPNsense 25.1.6) contains feature request.
https://forum.opnsense.org/index.php?topic=47125.msg0;boardseen#new
o system: kill gateways states for failback scenario when a higher priority gateway goes back online
We had this request so often that we decided to find a solution for it.
Theres also docs online now explaining the setup:
https://docs.opnsense.org/manual/how-tos/multiwan.html#failover-and-failback-states
Happy we could help :)
Great news. This is on my list. Thanks
Unfortunately, doesn't work for me for WireGuard. It still sticks to the secondary WAN despite the primary WAN going up again. Only restarting WireGuard forces it to fail back to the primary.
Probably not a firewall issue, but a WireGuard issue?
Not using gateway groups, but default gateway switching.
Did you verify that other states were indeed killed on the failback, e.g., clients with sessions towards the internet?
Essentially I assume Wireguard to follow the default route of the system to initiate the connection.
Though if the other side thinks the socket is still WAN2:51820 and not WAN1:51820 it will probably send the packet there and initiate another handshake.
I think it depends if there are firewall rules that allow a connection to WAN1 and WAN2 on the wireguard ports, or if only outgoing connections are allowed?
I did not test this with wireguard specifically though.
Other states do indeed get killed on failback. I tested this with an SSH session which gets reset when the primary WAN goes up again.
The default route successfully fails back to the primary WAN. WireGuard doesn't seem to be bothered though.
There is no firewall rule which allows incoming connections to the affected wg instance. Also, the secondary WAN is an LTE connection and the ISP blocks all inbound connections anyways.
As far as I know it only kills states that have a gateway attached in pf. Maybe wireguard does not and thus its states are not killed.