Frustrated on strange behaviour on tunnel net IPs in IPsec Routed.

[mrzaz]

Hello,

I'm coming from pfsense and is migrating to OPNSense and have stumbled on a strange intermittent issue that I could not find a root cause or solution for.

I know that some but not all of this has been discussed in some threads but even implementing the proposals i do not get it to work solid.

I have a IPSec routed net with phase1 and phase2 setup with a tunnel-net 10.6.110.0/30.
Router1   10.6.110.1/30     (LAN: 192.168.120.221/24)
Router2   10.6.110.2/30     (LAN: 192.168.120.231/24)
- Have enabled "Dynamic gateway policy" and it has created the dynamic Gateways in the gateway tab.
- I have even tried the proposal in the OPNSense manual about creating the gateways manually but that works even worse. :-/  It gives "The following input errors were detected: Cannot add IPv4 Gateway Address because no IPv4 address could be found on the interface." so the handbook config does not work. (Tried to follow it by the book)

- I have also added rule on IPSec+VTI_ifc+LAN with a "Allow Firewall to respond to pings"
  Dir: in, IPv4, ICMP, Any, This Firewall
- I have also tested with or without Firewall / Settings /Advanced/ Disable force gateway enabled.
- I have created a Static route to router2 LAN via VTI gateway. (and same in reverse router)

What happens is that occactionally it is possible to ping the tunnel IP both locally and remote either direct or by specifying the tunnel source IP when done from the router itself. (lets say from router1 ping 10.6.110.2 or even it's own 10.6.110.1) but then later for long times it is not possible to ping at all.  Just gets connection timeout.

If I however tries to ping the LAN IP on the other side of the link it is working 100% successful.
So the link is up and also when i do a manual ping to the LAN ip and capture on link in other side it comes through and replies:

I compared a pfSense routing table with the OPNsense and there i could see that it misses an entry for the remote side:

pfsense: (different site-to-site VTI but same prinnciple. also having dynamic gateway for IPSecVTI)
Destination   Gateway   Flags   Use   Mtu   Netif   Expire
10.6.106.1   link#10   UH   8565   1400   ipsec1000   
10.6.106.2   link#10   UHS   0   16384   lo0

opnsense:
It is all over the place. in router1 neither 106.1 or106.2 is visible in the routing table and in router2 they are but still not possible to to ping remote tunnel-IP.  So weird.

UPDATE:
After a restart of router1 the routes came back to the routing table and now I was able to ping
the other sides tunnel IP.

- Question is why these routes intermittently dissapears from the routing table ? (bug?)
- Still Gateway Monitoring does not work. Still OFFLINE regardless if I set monitor-ip o not but normal ping from commandline or through ping GUI works OK.

UPDATE2:
Now after a short while (5-10min) the 10.6.110.2 entry is again lost from the routing table.
The entry:
ipv4   10.6.110.2   link#7   UH   2446   1400   ipsec1   Router1Router2

It still exists in the router2.

UPDATE3:
I have now reproduced problem and feels like a bug.

1. restart routers. Both routers have the following entries. (reversed order in router2)
Destination        Gateway            Flags     Netif Expire
default            178.132.73.97      UGS      vtnet0
10.6.110.1         link#7             UHS         lo0
10.6.110.2         link#7             UH       ipsec1
2. Go to Gateways and edit the dynamic gateway created from IPsec.
3. Untick the "Disable Gateway Monitoring" and enter the tunnelIP on the other side and press APPLY.
4. Go to Gateways and edit the dynamic gateway created from IPsec again.
5. Tick the "Disable Gateway Monitoring" and remove the tunnelIP so editbox is blank and press APPLY.
6. Now the routing table has lost one entry. (the "10.6.110.2         link#7             UH       ipsec1"
Destination        Gateway            Flags     Netif Expire
default            178.132.73.97      UGS      vtnet0
10.6.110.1         link#7             UHS         lo0

Who do I contact to write this in as a bug-report ?
I tried this on the abslute latest update done 10min ago with same result.


UPDATE:
Have now done a bug report.
https://forum.opnsense.org/index.php?topic=22400.0
https://github.com/opnsense/core/issues/4888


Best regards
Dan Lundqvist
Stockholm, Sweden