[FIXED] IPSec VTI-interface not responding any longer after WAN-IP changed

[netcreator]

Hey folks!

I have a strange problem regarding to site-2-site IPSec-VPN tunnels using VTI-Interfaces...

I use an OPNSense-Firewall in the middle of the site-2-site VPN. So the OPNSense is the VPN-Hub and some ZyXEL Firewalls on remote-sites are the spokes. Iam using VTI-based IPSec-VPN to have the benefit of IP-routing. I dont like the policy-based VPN so much...

each of the site-2-site VPNs have their own tunnel-network/transfer-network with 2x IPs to have an gateway-adress on each site.

Now the problem:

When my WAN-link goes down because of VDSL-resync or because my provider is disconnecting me for some reason, the routing is not working fine. After some time, when the WAN-reconnect has been done, the tunnels will come up again automaticly but the local VTI-interfaces of the OPNSense will not respond any longer when you look from the tunnel-site. So the remote-gateway (ZyXEL firewall) is not longer able to ping the neighbor VTI-interface of the OPNsense and the interface is not able to receive and permit traffic to target-hosts which reside in one of the local-network where the OPNSense is placed. In the other direction the traffic will flow fine. So for example iam on the site where the OPNSense is placed and sitting on an computer in the local network pinging an host which is behind the ZyXEL-firewall on the remote site. the ping will function and i can also connect to the remote-computer via RDP. Now lets assume iam connected to that remote-computer via RDP trying to ping the computer where iam sitting on and that fails as long as i restart the routing-stack or interfaces-stack (services) on the OPNSense. After restarting one of these services the connection will work fine again and the VTI-interface is responding again when you see it from the tunnel-site. Restarting the StrongSwan IPSec and packet filter service on the OPNSens does not have an impact on it and the issue keeps staying present. This case does ONLY happen when the WAN-interface of the OPNSense was down and has received an new public IP-address from my DSL-provider.

Iam sure that there is no problem with missconfigured firewall-rules. i double and tripple checked these and all my rules are fine and will let pass these traffic.

I troubleshooted this problem a long time with tons of different settings in the sections: VPN, Interfaces, Firewall, Gateways and Routes. but i had no success.

So finaly a short summary:

- The VPN-tunnels come up again automaticly after WAN-link was down
- VTI-interface is not longer responding from the tunnel-site when WAN-interface received new public IP (tunnels up)
- I can send traffic in one directon but not in the other (hub-networks to spoke-networks will function, spoke-networks to hub-networks will not function) because VTI-interface can not permit traffic and is not responding
- The VTI-interface is responding when I try to ping it from an local inside network (hub-site)
- The remote-gateway-monitoring is working when the issue is present (the OPNSense can ping the VTI-interface of the ZyXEL-firewall but the ZyXEL-firewall can not ping the VTI-interface of the OPNSense)
- After an reboot of the OPNSense or by restarting the described services, this problem is not existing any longer. (problem will kick-in again when the WAN-link was down and the tunnels reestablished)

Due to the fact that iam in networking since a long time now I can swear that these issue is not resulting by having less knownledge. When I had the ZyXEL-only-setup everything worked great.

That is an really strange issue and i think that its a bug...

thanks to all!

[franco]

Using FQDN?


Cheers,
Franco

[netcreator]

Yepp iam using FQDN via DynDns (Oracle DNS) to target to my OPNSense, which is the Hub for all the ZyXEL-spokes. The update of the DynDns entries to point to the new public IP-address is working fine. The tunnels do reconnect after about 3-4 minutes. But after that the bug of the non-reachable VTI-interface kicks-in. But it seems to be only unreachable from the remote-gateways standpoint. like i described... the VTI-interfaces did ping when i do an ping out of an local network which is residing behind the OPNSense. Short restart of the "routing"-service solves that instandly.

[netcreator]

.

[netcreator]

[UPDATE]

I figured out that the VTI-interface is responding but only out of the same network where its placed. So assumed that I have to search for routing-problems on the OPNSense. And yes... There are routing-problems. When the WAN-interface goes down (and also the tunnels) the OPNSense is removing all active routes which are used together with the IPSec-VPN. After the tunnels have gone up again the routes will not be applied again. Some options/configurations like "Allow defaulf gateway switching",  "gateway-monitoring" or "upstream gateway" do not have an effect on that. No matter what I do or configure... The routes used for the IPSec-tunnels will not come back as long as I have not restarted the "routing"-Service.

[franco]

Sounds like https://github.com/opnsense/core/issues/6354 but FQDN will still eventually cause you issues if the address changes on the other end because IPsec doesn't care.


Cheers,
Franco

[netcreator]

Okay yes maybe. But I can not see any solution there...

You mean that IPSec it self can be the problem even if the tunnels are up again? Because rember: I can communicate in one direction over the IPSec-tunnels, when the problem exists. There is just nothing other than the missing routes to communicate not networks behind the OPNSense firewall. So why is OPNSense for example not doing an restart of the routing-servce itself when this workaround fixes the problem? When I restart the routing-service this is not affecting the connected IPSec-VPNs also, regarding to re-establish or something like that...

[netcreator]

Ahhmmm... You OPNSense-Guys!!! Did you hear my wishes? Since the update I installed today (v. 23.1.3) the routing is working fine after disconnected WAN-If.

In the changelog there is a notice which keept my attention. Its these one:

"ipsec: add a routing hook and execute it for all VTI devices during reconfiguration"

You guys read my thread right?

Thank you very much for fixing this issue!!! 

[franco]

Well I did reference #6354 earlier and since we went looking for 23.1.3 inclusion material this seemed to fit the small scope. It was on the wish list for a while, prompted again by a customer and you recently so a strategy was discussed and implemented.

Thank you for sending feedback on new additions. This is actually very helpful as most of the time things that work/improve are not being talked about.


Cheers,
Franco

[netcreator]

That is very nice Franko! Thank you for doing that so well in a very short time.

I didnt know that youre from the OPNSense team but it seems that you might be. Regarding OPNSense I noticed some nice-to-haves in the past. Next time I remember that things or next time they will come up again i will notice it somewhere and will let know you about my ideas to implement in the future ;-)

Youre doing very well. Keep on! Your product is very good overall.

Benjamin