Cannot access LAN subnets across gre over ipsec using OSPF

[mutahir]

Hi

The following is the topology of my network

Site A)
FW: OPNsense 20.7.7_1-amd64
WAN:  192.168.100.34/24
LAN: 192.168.60.1/24
IPSEC: 10.101.0.5/31
GRE: 10.100.0.5/31

Machine on Site A Ubuntu
WAN 192.168.60.20/24 connected to Site A FW LAN

Site B)
FW: OPNsense 20.7.7_1-amd64
WAN:  192.168.100.38/24
LAN: 192.168.59.1/24
IPSEC: 10.101.0.4/31
GRE: 10.100.0.4/31

Machine on Site B Freebsd
WAN 192.168.60.20/24 connected to Site B FW LAN

Both firewall WANs connected to a single gateway and can ping each other

I have setup a routed IPSec Tunnel as given in the link
https://docs.opnsense.org/manual/how-tos/ipsec-s2s-route.html

I have also setup a GRE tunnel on the IPSEC tunnel. On both ends OSPF is running using frr.

All the tunnels come up, the routes information is exchanged properly for OSPF. Both FWs routes show the others LAN subnet

Firewall A routes

Code: [Select]

Code Network Administrative Distance Metric Interface Via Time
O 0.0.0.0/0 110 10 gre0 onlink 10.100.0.4 weight
K > * 0.0.0.0/0 0 0 00:26:37 192.168.100.1
K > * 8.8.8.8/32 0 0 00:24:12 192.168.100.1
O 10.100.0.4/31 110 20 gre0 onlink 10.100.0.4 weight
C > * 10.100.0.4/31 0 1 gre0 00:26:37
C > * 10.101.0.4/32 0 1 ipsec1000 00:26:37
O > * 10.101.0.5/32 110 20 gre0 onlink 10.100.0.4 weight
O > * 192.168.59.0/24 110 20 gre0 onlink 10.100.0.4 weight
C > * 192.168.60.0/24 0 1 em2 00:26:37
O 192.168.100.0/24 110 20 gre0 onlink 10.100.0.4 weight
* 192.168.100.0/24 0 1 em1 00:26:37
C > * 192.168.100.0/24 0 1 em0 00:26:37

Firewall B routes

Code: [Select]

Code Network Administrative Distance Metric Interface Via Time
K > * 0.0.0.0/0 0 0 00:31:21 192.168.100.1
K > * 8.8.8.8/32 0 0 00:28:51 192.168.100.1
O 10.100.0.4/31 110 20 gre0 onlink 10.100.0.5 weight
C > * 10.100.0.4/31 0 1 gre0 00:31:21
O > * 10.101.0.4/32 110 20 gre0 onlink 10.100.0.5 weight
C > * 10.101.0.5/32 0 1 ipsec1000 00:31:21
C > * 192.168.59.0/24 0 1 em2 00:31:21
O > * 192.168.60.0/24 110 20 gre0 onlink 10.100.0.5 weight
O 192.168.100.0/24 110 20 gre0 onlink 10.100.0.5 weight
* 192.168.100.0/24 0 1 em1 00:31:21
C > * 192.168.100.0/24 0 1 em0 00:31:21


The problem is that Machine A and Machine B are unable to ping or communicate with each others. Using TCP dumps the icmp packet hop is like this

Code: [Select]

FWA LAN--> 192.168.60.20 > 192.168.59.20: ICMP echo request, id 3994
FWA GRE--> OPNsense.localdomain > 192.168.59.20: ICMP echo request, id 18358, seq 1
FWB GRE--> 10.100.0.5 > 192.168.59.20: ICMP echo request, id 18358, seq 1
FWB LAN--> 10.100.0.5 > 192.168.59.20: ICMP echo request, id 18358, seq 1

FWB LAN--> 192.168.59.20 > 10.100.0.5: ICMP echo reply, id 18358, seq 1
FWB GRE--> 192.168.59.20 > 10.100.0.5: ICMP echo reply, id 18358, seq 1
FWA GRE--> 192.168.59.20 > OPNsense.localdomain: ICMP echo reply, id 18358, seq 1
FWA LAN-->  ***********

However FW A can ping Machine B and likewise FW B can ping Machine A

The attached images shows the NAT outbound rules on both sides.

As an experiment I changed the interface in GRE tunnel from the IPSEC tunnel to the WAN interfaces on both sides and the Machines on the LAN subnets were able to ping each other.

Is there any special NAT rules that I have to add so that both the networks can communicate with each other?. Also why does the communication between Machines A and B work when the GRE tunnel is unencrypted but not with IPSEC?. Has anyone else faced such an issue?