Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - Dagothur

Pages1
#1
General Discussion / Re: OPNsense - Multi-WAN - established VPN connection still using Tier 2 Gateway
June 13, 2025, 06:26:16 PM
Hi,
Version: OPNsense 25.1.8_1-amd64
Problem: I think some states get stuck associated to the wrong gateway so state killing fails or kills erroneously.

I've got 2 gateways failing over successfully with the following configured:

Primary has "Failover States" checked.
Secondary has "Failover States" checked and "Failback States" checked.
The gateways are in a group together.
"System -> Settings -> General -> Allow default gateway switching" is checked.
"Firewall -> Settings -> Advanced -> Bind states to interface" is checked.
opt1/vlan0.99 is the Secondary
opt2/vlan0.100 is the Primary

Test case:

I'm running 4 continuous pings to 8.8.8.8, 8.8.4.4, 1.1.1.1, 1.0.0.1.
I disconnect the cable to the Primary and failover triggers and all pings move to Secondary:

Failover:

Code Select
<13>1 2025-06-13T15:52:24+00:00 OPNsense.localdomain opnsense 2465 - [meta sequenceId="1"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T15:52:25+00:00 OPNsense.localdomain opnsense 2465 - [meta sequenceId="2"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2
<13>1 2025-06-13T15:52:25+00:00 OPNsense.localdomain opnsense 2465 - [meta sequenceId="3"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to 85.*.*.*
<13>1 2025-06-13T15:52:25+00:00 OPNsense.localdomain opnsense 6518 - [meta sequenceId="4"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for deferred gateway WAN_SECONDARY_DHCP [cb8015ec-602b-4474-8844-032c38713239]
<13>1 2025-06-13T15:52:25+00:00 OPNsense.localdomain opnsense 6518 - [meta sequenceId="5"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[])
<13>1 2025-06-13T15:52:25+00:00 OPNsense.localdomain opnsense 6518 - [meta sequenceId="6"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 24568 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 24568 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PRIMARY_DHCP
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 24568 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt1
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 24568 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to 10.0.0.1
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 28798 - [meta sequenceId="11"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for unreachable gateway WAN_PRIMARY_DHCP [8b028840-6150-4c90-bf79-1c562f2f0109]
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 28798 - [meta sequenceId="12"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[WAN_PRIMARY_DHCP])
<13>1 2025-06-13T15:52:36+00:00 OPNsense.localdomain opnsense 28798 - [meta sequenceId="13"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[WAN_PRIMARY_DHCP]))
Then, I reconnect the cable but only half the pings move back to the Primary (1.1.1.1 and 8.8.4.4 move back whereas 8.8.8.8 and 1.0.0.1 stick to the Secondary).

Failback:

Code Select
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 21512 - [meta sequenceId="1"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 21512 - [meta sequenceId="2"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 21512 - [meta sequenceId="3"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to 85.*.*.*
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 27262 - [meta sequenceId="4"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for deferred gateway WAN_SECONDARY_DHCP [57ca2e8b-5579-4256-83ed-0c6a641a2226]
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 27262 - [meta sequenceId="5"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[])
<13>1 2025-06-13T15:54:01+00:00 OPNsense.localdomain opnsense 27262 - [meta sequenceId="6"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
<13>1 2025-06-13T15:54:12+00:00 OPNsense.localdomain opnsense 58537 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T15:54:13+00:00 OPNsense.localdomain opnsense 58537 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2
<13>1 2025-06-13T15:54:13+00:00 OPNsense.localdomain opnsense 58537 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to 85.*.*.*
<13>1 2025-06-13T15:54:13+00:00 OPNsense.localdomain opnsense 61983 - [meta sequenceId="10"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for deferred gateway WAN_SECONDARY_DHCP [50a1bf4b-de29-4eae-8336-2be347d4b8d4]
<13>1 2025-06-13T15:54:13+00:00 OPNsense.localdomain opnsense 61983 - [meta sequenceId="11"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[])
<13>1 2025-06-13T15:54:13+00:00 OPNsense.localdomain opnsense 61983 - [meta sequenceId="12"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
Then, I disconnect the Primary again. Failover occurs:

Code Select
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 62144 - [meta sequenceId="1"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 62144 - [meta sequenceId="2"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt2
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 62144 - [meta sequenceId="3"] /usr/local/etc/rc.routing_configure: ROUTING: keeping inet default route to 85.*.*.*
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 64584 - [meta sequenceId="4"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for deferred gateway WAN_SECONDARY_DHCP [64c0f9eb-2187-4d6f-a76d-137ab7d4b98c]
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 64584 - [meta sequenceId="5"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[])
<13>1 2025-06-13T16:05:22+00:00 OPNsense.localdomain opnsense 64584 - [meta sequenceId="6"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[]))
<13>1 2025-06-13T16:05:33+00:00 OPNsense.localdomain opnsense 84039 - [meta sequenceId="7"] /usr/local/etc/rc.routing_configure: ROUTING: entering configure using defaults
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 84039 - [meta sequenceId="8"] /usr/local/etc/rc.routing_configure: ROUTING: ignoring down gateways: WAN_PRIMARY_DHCP
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 84039 - [meta sequenceId="9"] /usr/local/etc/rc.routing_configure: ROUTING: configuring inet default gateway on opt1
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 84039 - [meta sequenceId="10"] /usr/local/etc/rc.routing_configure: ROUTING: setting inet default route to 10.0.0.1
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 88040 - [meta sequenceId="11"] /usr/local/etc/rc.syshook.d/monitor/20-recover: ROUTING: killing states for unreachable gateway WAN_PRIMARY_DHCP [e4b28c89-dc67-4b97-be4e-08b9dabc9b1e]
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 88040 - [meta sequenceId="12"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (1,[WAN_PRIMARY_DHCP])
<13>1 2025-06-13T16:05:34+00:00 OPNsense.localdomain opnsense 88040 - [meta sequenceId="13"] /usr/local/etc/rc.syshook.d/monitor/20-recover: plugins_configure monitor (execute task : dpinger_configure_do(1,[WAN_PRIMARY_DHCP]))

The 2 pings flowing through the Secondary are killed anyway. I think because their state is still associated with the Primary.
All pings now flow through Secondary.

I reconnect Primary and Failback occurs. 3 pings still flow through secondary and 1 ping flows through Primary.
Pages1