OPNsense Forum

English Forums => Virtual private networks => Topic started by: @Vorona on April 09, 2021, 12:32:18 am

Title: OSPF via GRE/IPSec
Post by: @Vorona on April 09, 2021, 12:32:18 am
Hello!
I try to configure dynamic routing with MikroTik.
Now I have working GRE tunnel between opnsense and routerboard.
After start ospf instanses I have corrects routes on opensense. But on MikroTik I have routes without gateway address and interface.


 /routing ospf> neighbor print
 0 instance=default router-id=172.26.0.1 address=100.100.100.41
   interface=vorona-gate priority=1 dr-address=0.0.0.0
   backup-dr-address=0.0.0.0 state="Full" state-changes=50 ls-retransmits=0
   ls-requests=0 db-summaries=0 adjacency=23m31s

Here we can see, that address 100.100.100.41 (opensense)
But in recieved routes I have:

(http://puu.sh/HwBJI/f78084d14f.PNG)

Why I haven't gateway address here?

Also, I haven't this route in routing table for this reason.

configs:
Tik:
/routing ospf area
set [ find default=yes ] disabled=yes
add area-id=172.16.117.0 name=area1
/routing ospf instance
set [ find default=yes ] router-id=172.16.117.1
/routing ospf interface
add passive=yes
add interface=vorona-gate network-type=point-to-point
/routing ospf network
add area=area1 network=100.100.100.40/30
add area=area1 network=172.16.117.0/24

Opensense:
frr version 7.4
frr defaults datacenter
hostname gate.vorona.su
log syslog
!
interface gre1
 ip ospf area 172.16.117.0
!
interface vtnet1
 ip ospf area 0.0.0.0
!
router ospf
 ospf router-id 172.26.0.1
 passive-interface vtnet1
!
ip prefix-list test seq 10 permit 172.26.0.0/28
!
line vty
!
end

Gre interfaces has addresses from 100.100.100.40/30 network

What I'm doing wrong?
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 09, 2021, 09:05:39 am
Have you asked at the Mikrotik forum in parallel? I think you should.
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 09, 2021, 12:30:39 pm
Yes, here: TAP (https://forum.mikrotik.com/viewtopic.php?f=14&t=174293)
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 09, 2021, 04:15:51 pm
What does a dump of the OSPF link state database look like on both ends? I don't know how to get that - I do OSPF in large networks with Cisco gear, only. But so I know OSPF and the database would be my first point to look.
Or maybe second point - additionally for a very first check: what does the equivalent of "show ip ospf neighbour" show on both sides?
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 09, 2021, 06:20:46 pm
Show neigbors:
At MikroTik:
 > /routing ospf neighbor print
 0 instance=default router-id=172.26.0.1 address=100.100.100.41
   interface=vorona-gate priority=1 dr-address=0.0.0.0
   backup-dr-address=0.0.0.0 state="Full" state-changes=19 ls-retransmits=0
   ls-requests=0 db-summaries=0 adjacency=8h28m11s
At opnsense:
 # show ip ospf neighbor detail
 Neighbor 172.16.117.1, interface address 100.100.100.42
    In the area 172.16.117.0 via interface gre1
    Neighbor priority is 1, State is Full, 5 state changes
    Most recent state change statistics:
      Progressive change 8h29m40s ago
    DR is 0.0.0.0, BDR is 0.0.0.0
    Options 2 *|-|-|-|-|-|E|-
    Dead timer due in 34.152s
    Database Summary List 0
    Link State Request List 0
    Link State Retransmission List 0
    Thread Inactivity Timer on
    Thread Database Description Retransmision off
    Thread Link State Request Retransmission on
    Thread Link State Update Retransmission on

Show links:
MikroTik:
 > /routing ospf lsa print detail
 instance=default area=area1 type=router id=172.16.117.1
   originator=172.16.117.1 sequence-number=0x80000100 age=66 checksum=0x8785
   options="E" body=
     flags=
     links (type, id, data, metric)
         Point-To-Point 172.26.0.1 100.100.100.42 10
         Stub 100.100.100.40 255.255.255.252 10
         Stub 172.16.117.0 255.255.255.0 10

 instance=default area=area1 type=router id=172.26.0.1 originator=172.26.0.1
   sequence-number=0x80000014 age=1481 checksum=0x224A options="E"
   body=
     flags=BORDER
     links (type, id, data, metric)
         Point-To-Point 172.16.117.1 0.0.0.8 10

 instance=default area=area1 type=summary-network id=172.26.0.0
   originator=172.26.0.1 sequence-number=0x80000012 age=1411 checksum=0x6C56
   options="E" body=
     netmask=255.255.255.240
     metric=10


Opnsense:
 # show ip ospf database router

       OSPF Router with ID (172.26.0.1)


                Router Link States (Area 0.0.0.0)

  LS age: 1715
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x3
  Flags: 0x1 : ABR
  LS Type: router-LSA
  Link State ID: 172.26.0.1
  Advertising Router: 172.26.0.1
  LS Seq Number: 80000014
  Checksum: 0x22cb
  Length: 36

   Number of Links: 1

    Link connected to: Stub Network
     (Link ID) Net: 172.26.0.0
     (Link Data) Network Mask: 255.255.255.240
      Number of TOS metrics: 0
       TOS 0 Metric: 10



                Router Link States (Area 172.16.117.0)

  LS age: 211
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x6
  Flags: 0x0
  LS Type: router-LSA
  Link State ID: 172.16.117.1
  Advertising Router: 172.16.117.1
  LS Seq Number: 80000100
  Checksum: 0x8785
  Length: 60

   Number of Links: 3

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 172.26.0.1
     (Link Data) Router Interface address: 100.100.100.42
      Number of TOS metrics: 0
       TOS 0 Metric: 10

    Link connected to: Stub Network
     (Link ID) Net: 100.100.100.40
     (Link Data) Network Mask: 255.255.255.252
      Number of TOS metrics: 0
       TOS 0 Metric: 10

    Link connected to: Stub Network
     (Link ID) Net: 172.16.117.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metric: 10


  LS age: 1625
  Options: 0x2  : *|-|-|-|-|-|E|-
  LS Flags: 0x3
  Flags: 0x1 : ABR
  LS Type: router-LSA
  Link State ID: 172.26.0.1
  Advertising Router: 172.26.0.1
  LS Seq Number: 80000014
  Checksum: 0x224a
  Length: 36

   Number of Links: 1

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 172.16.117.1
     (Link Data) Router Interface address: 0.0.0.8
      Number of TOS metrics: 0
       TOS 0 Metric: 10


Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 09, 2021, 07:17:53 pm
What does default area "area1" on a Mikrotik mean? That should be area 0. Or 0.0.0.0 - one and the same, just a 32 bit value.
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 12, 2021, 11:48:57 am
/routing ospf area
set [ find default=yes ] disabled=yes
add area-id=172.16.117.0 name=area1

area1 - is just local name (description) of area in config. Area, configured at MikroTik is 172.16.117.0, аt opnsense backbone (0.0.0.0). In BB area network 172.26.0.0/28. In area 172.16.117.0 network 172.16.117.0/24 and 100.100.100.40/30 (Gre tunnel's network)
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 12, 2021, 04:31:19 pm
You need to set the area to 0.0.0.0 on the Mikrotik. Two OSPF neighbours must share the same area on the link. Areas can only change when you cross routers, i.e.

Code: [Select]
router 1 ------ router 2 ------ router 3
         area 0          area 1

Router 2 would be called an area border router while routers 1 and 3 are autonomous system border routers.
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 13, 2021, 10:04:19 am
And there is!

One directly attached to opnsense network is in BB area. Directly attached network to Mikrotik and network between opnsense and Mikrotik in other area. I know, how works ospf and I am supporting big network on cisco an juniper hardware too, but this problem is new for me.
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 13, 2021, 10:38:02 am
One directly attached to opnsense network is in BB area. Directly attached network to Mikrotik and network between opnsense and Mikrotik in other area. I know, how works ospf and I am supporting big network on cisco an juniper hardware too, but this problem is new for me.
Sorry, that wasn't obvious to me from your posts. In that case I am running out of ideas as well ...
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 13, 2021, 10:43:45 am
Sadly...  :(

Maybe this is something bug in frr ospfd? I found similar problem, but it was with bgp protocol on old version opnsense.
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on April 13, 2021, 12:43:23 pm
Do you have a change to use routed ipsec where you don't need a GRE tunnel?
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 13, 2021, 01:54:11 pm
You cannot form an OSPF neighbour relation without a dedicated point-to-point link. That is one of the main reasons to use a tunnel interface.

But could you try to disable IPsec just for debugging purposes and try if OSPF works over GRE alone?
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on April 13, 2021, 09:10:59 pm
Isnt this the reason for a route based IPsec? A dedicated interface with p2p address
Title: Re: OSPF via GRE/IPSec
Post by: Patrick M. Hausen on April 13, 2021, 09:19:07 pm
Isnt this the reason for a route based IPsec? A dedicated interface with p2p address
Ah ... ok. Never used these. Traditional Kame IPsec doesn't have that feature. So you end up with GRE or IPIP.
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on April 14, 2021, 09:23:02 am
I currently set up a lab cause of wireguard reports with OSPF, I'll try to test this too
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 16, 2021, 05:25:33 pm
But could you try to disable IPsec just for debugging purposes and try if OSPF works over GRE alone?

I cannot do that. Only one point has real IP address. Otrer point behind NAT. For this reason IPSec works in tunnel mode. Why do you think, that IPSec can break OSPF, wich works in GRE? Tunnel works, firewall on tunnel at both sides are open.

I currently set up a lab cause of wireguard reports with OSPF, I'll try to test this too

You mean RB IPSec or OSPF relations between Opnsense and MikroTik?
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on April 18, 2021, 03:40:05 pm
Ok, I just finished the lab and I'm able to ping from host-A to host-B via FW-A to FW-B which has a IPsec with type transport and inside a GRE tunnel and inside GRE the OSPF protocol.

root@OPNsense:~ # tcpdump -n -i gre0
15:37:40.506482 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Hello, length 48
15:37:40.518025 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.530882 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.537441 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 72
15:37:40.541540 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Database Description, length 72
15:37:40.541671 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Request, length 48
15:37:40.542538 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.542587 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Request, length 48
15:37:40.542741 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 88
15:37:40.543885 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 88
15:37:40.544078 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:40.545609 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:40.725297 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Ack, length 64
15:37:41.359517 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Ack, length 64
15:37:45.574028 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:45.575582 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:45.595667 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Hello, length 48
15:37:45.893086 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Ack, length 44
15:37:46.587098 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Ack, length 44
15:37:46.673133 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44857, length 64
15:37:46.675096 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44857, length 64
15:37:46.872543 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44822, length 64
15:37:46.873544 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44822, length 64
15:37:47.723243 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44858, length 64
15:37:47.723928 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44858, length 64
15:37:47.942779 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44823, length 64
15:37:47.943702 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44823, length 64
15:37:48.793598 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44859, length 64
15:37:48.794412 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44859, length 64
15:37:48.972486 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44824, length 64
15:37:48.973553 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44824, length 64
15:37:49.823147 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44860, length 64
15:37:49.823893 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44860, length 64
15:37:50.042721 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44825, length 64
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on April 21, 2021, 04:47:49 pm
Good!
What hardware do you use with opnsense? What version?

In my gre I see OSPF traffic in both ways too, but route wrong.
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on April 21, 2021, 09:07:47 pm
Just VMs with 21.1.4
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 21, 2021, 11:13:55 pm
And same strange things for me.
On mikrotik side, on LSA tab i found this:

Wrong data (OPNsense)
Code: [Select]
flags=
links (type, id, data, metric)
    Point-To-Point 10.74.0.1 0.0.0.7 10
    Stub 10.77.0.0 255.255.255.0 10

Correct data (Mikrotik)
Code: [Select]
flags=
links (type, id, data, metric)
    Point-To-Point 10.74.0.1 172.16.0.2 10
    Stub 172.16.0.0 255.255.255.252 10
    Stub 10.77.0.0 255.255.255.0 10
Note:
* 10.74.0.1/16 - Router ID of Mikrotik #1
* 10.77.0.1/16 - Router ID of OPNsense
172.16.0.0/30 - GRE tunnel network:
172.16.0.1 - mikrotik ip
172.16.0.2 - opnsense ip

What is "0.0.0.7" ?
Any ideas how to fix it (Make it work?)
Look like OSPF won't announce GRE tunnel network to Mikrotik, so mikrotik don't known how to route 10.77 network (And show it as "empty" gateway and interface)

note2:
In OPNSense, on "Network" side i add two networks: LAN and GRE (to area "0.0.0.0"). On "Interfaces" - LAN and GRE interfaces.
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on July 22, 2021, 05:34:08 am
Screenshots please
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 22, 2021, 09:57:47 am
What exacly you want to see?
OSPF and GRE settings almost same on Mikrotik and OPNsense.

Router #1: 10.74.0.0/16 (Mikrotik, gre ip: 172.16.74.13)
Router #2: 10.77.0.0/16 (OPNsense, gre ip: 172.16.74.14)
GRE between routers: 172.16.74.12/30
Can ping both gre ip's. Can ping and access resource behind both routers if select interface manually or add static route.
But on mikrotik side - ospf won't work. Route added without gateway and interface.

OSPF config on OPNsense:
Code: [Select]
Current configuration:
!
frr version 7.4
frr defaults traditional
hostname gw.dc.sipcolor.ru.sipcolor.ru
log syslog notifications
!
router ospf
 ospf router-id 10.77.0.1
 passive-interface bridge0
 passive-interface vtnet0
 passive-interface vtnet1
 network 10.77.0.0/16 area 0.0.0.0
 network 172.16.74.12/30 area 0.0.0.0
!
line vty
!
end
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on July 22, 2021, 02:58:08 pm
When you capture on GRE .. do you see OSPF packets in and out?
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 22, 2021, 05:29:45 pm
When you capture on GRE .. do you see OSPF packets in and out?
Yep. Hello packets, LSA...
I see neighbors in OSPF on both routers, i see routes. But on OPNsense routes is correct, but on Mikrotik - with empty gateways/interfaces (see screenshot above).

I test OPNsense and Mikrotik (as 2nd router) and found difference:
OPNsense (fw_v21.1.8_1 / FRR_v7.4_6):
Code: [Select]
Type: Stub     ID: 10.77.0.0       Data: 255.255.0.0     Metric: 10
Type: PTP      ID: 10.74.0.1       Data: 0.0.0.8         Metric: 10
Mikrotik (fw_v6.48.3):
Code: [Select]
Type: Stub     ID: 10.77.0.0       Data: 255.255.0.0     Metric: 10
Type: PTP      ID: 10.74.0.1       Data: 172.16.74.14    Metric: 10
Type: Stub     ID: 172.16.74.12    Data: 255.255.255.252 Metric: 11

As you see - mikrotik send an additional "stub" route, but OPNsense - ony strange "0.0.0.8".
On Wiki i found - thats a port number (MIB-II notation) and Mikrotik won't understand this form of record (Or expect a regular route/ip address).

Any ideas how to fix it?
Title: Re: OSPF via GRE/IPSec
Post by: mimugmail on July 22, 2021, 07:20:02 pm
What is 172.16.255? 0.0.0.7 is a wildcard mask.
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 22, 2021, 07:35:49 pm
What is 172.16.255? 0.0.0.7 is a wildcard mask.
172.16.255.74 = 10.74.0.1 (Router ID). Sorry, wrong screenshot (I just try to change router ID)
Title: Re: OSPF via GRE/IPSec
Post by: lilsense on July 22, 2021, 09:38:56 pm
create a loopback and place all your devices in the same subnet for the loopback. place the loopback in the area 0. make sure your loopback is a passive interface.
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 23, 2021, 12:43:46 am
create a loopback and place all your devices in the same subnet for the loopback. place the loopback in the area 0. make sure your loopback is a passive interface.

Ok... I did:
* Interfaces - Other types - Loopback - Add (Description: "172.16.255.77". It is ok? I can't set text here, only ip-like record)
* Interfaces - Assignments - Loopback - Add. Set Enabled, IPv4 = Static, 172.16.255.77/32
* Routing - OSPF - Router ID set to "172.16.255.77", added loopback to passive interfaces
* Routing - OSPF - Networks - Add: 172.16.0.0/16, Area: 0.0.0.0

On mikrotik - add only network to area 0.0.0.0...
Now - i have additional route in ospf routes list on mikrotik... Without gateway/interface...
I think mikrotik just not understand "Wildcard" mask... It is possible to swith it on OPNsense (For compatability)?

On Mikrotik (Now his ID is "172.16.255.74"):
Code: [Select]
flags=
links (type, id, data, metric)
    Stub 10.77.0.0 255.255.0.0 10
    Point-To-Point 172.16.255.74 0.0.0.8 10
    Stub 172.16.255.77 255.255.255.255 0
Title: Re: OSPF via GRE/IPSec
Post by: Fox_exe on July 23, 2021, 11:03:44 am
Hmm.. Seems like its a bug in FRR: https://github.com/FRRouting/frr/issues/3973
And its still not fixed :(
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on November 11, 2021, 02:21:52 pm
Hi!
I see, that this problem can be resolved by changing FRR to bird :)

I tried to upgrade FRR to 7.5.1 from FreeBSD repositories, but interface gre still be unnumbered. I think for that reason I see "empty" routes in ospf at mikrotik side.
Title: Re: OSPF via GRE/IPSec
Post by: @Vorona on August 30, 2023, 09:06:56 am
Hello!
I found working solution!
In last version mikrotik firmware added network type "unnumbered p-t-p". With this type all works fine.