Hello!
I try to configure dynamic routing with MikroTik.
Now I have working GRE tunnel between opnsense and routerboard.
After start ospf instanses I have corrects routes on opensense. But on MikroTik I have routes without gateway address and interface.
/routing ospf> neighbor print
0 instance=default router-id=172.26.0.1 address=100.100.100.41
interface=vorona-gate priority=1 dr-address=0.0.0.0
backup-dr-address=0.0.0.0 state="Full" state-changes=50 ls-retransmits=0
ls-requests=0 db-summaries=0 adjacency=23m31s
Here we can see, that address 100.100.100.41 (opensense)
But in recieved routes I have:
(http://puu.sh/HwBJI/f78084d14f.PNG)
Why I haven't gateway address here?
Also, I haven't this route in routing table for this reason.
configs:
Tik:
/routing ospf area
set [ find default=yes ] disabled=yes
add area-id=172.16.117.0 name=area1
/routing ospf instance
set [ find default=yes ] router-id=172.16.117.1
/routing ospf interface
add passive=yes
add interface=vorona-gate network-type=point-to-point
/routing ospf network
add area=area1 network=100.100.100.40/30
add area=area1 network=172.16.117.0/24
Opensense:
frr version 7.4
frr defaults datacenter
hostname gate.vorona.su
log syslog
!
interface gre1
ip ospf area 172.16.117.0
!
interface vtnet1
ip ospf area 0.0.0.0
!
router ospf
ospf router-id 172.26.0.1
passive-interface vtnet1
!
ip prefix-list test seq 10 permit 172.26.0.0/28
!
line vty
!
end
Gre interfaces has addresses from 100.100.100.40/30 network
What I'm doing wrong?
Have you asked at the Mikrotik forum in parallel? I think you should.
Yes, here: TAP (https://forum.mikrotik.com/viewtopic.php?f=14&t=174293)
What does a dump of the OSPF link state database look like on both ends? I don't know how to get that - I do OSPF in large networks with Cisco gear, only. But so I know OSPF and the database would be my first point to look.
Or maybe second point - additionally for a very first check: what does the equivalent of "show ip ospf neighbour" show on both sides?
Show neigbors:
At MikroTik:
> /routing ospf neighbor print
0 instance=default router-id=172.26.0.1 address=100.100.100.41
interface=vorona-gate priority=1 dr-address=0.0.0.0
backup-dr-address=0.0.0.0 state="Full" state-changes=19 ls-retransmits=0
ls-requests=0 db-summaries=0 adjacency=8h28m11s
At opnsense:
# show ip ospf neighbor detail
Neighbor 172.16.117.1, interface address 100.100.100.42
In the area 172.16.117.0 via interface gre1
Neighbor priority is 1, State is Full, 5 state changes
Most recent state change statistics:
Progressive change 8h29m40s ago
DR is 0.0.0.0, BDR is 0.0.0.0
Options 2 *|-|-|-|-|-|E|-
Dead timer due in 34.152s
Database Summary List 0
Link State Request List 0
Link State Retransmission List 0
Thread Inactivity Timer on
Thread Database Description Retransmision off
Thread Link State Request Retransmission on
Thread Link State Update Retransmission on
Show links:
MikroTik:
> /routing ospf lsa print detail
instance=default area=area1 type=router id=172.16.117.1
originator=172.16.117.1 sequence-number=0x80000100 age=66 checksum=0x8785
options="E" body=
flags=
links (type, id, data, metric)
Point-To-Point 172.26.0.1 100.100.100.42 10
Stub 100.100.100.40 255.255.255.252 10
Stub 172.16.117.0 255.255.255.0 10
instance=default area=area1 type=router id=172.26.0.1 originator=172.26.0.1
sequence-number=0x80000014 age=1481 checksum=0x224A options="E"
body=
flags=BORDER
links (type, id, data, metric)
Point-To-Point 172.16.117.1 0.0.0.8 10
instance=default area=area1 type=summary-network id=172.26.0.0
originator=172.26.0.1 sequence-number=0x80000012 age=1411 checksum=0x6C56
options="E" body=
netmask=255.255.255.240
metric=10
Opnsense:
# show ip ospf database router
OSPF Router with ID (172.26.0.1)
Router Link States (Area 0.0.0.0)
LS age: 1715
Options: 0x2 : *|-|-|-|-|-|E|-
LS Flags: 0x3
Flags: 0x1 : ABR
LS Type: router-LSA
Link State ID: 172.26.0.1
Advertising Router: 172.26.0.1
LS Seq Number: 80000014
Checksum: 0x22cb
Length: 36
Number of Links: 1
Link connected to: Stub Network
(Link ID) Net: 172.26.0.0
(Link Data) Network Mask: 255.255.255.240
Number of TOS metrics: 0
TOS 0 Metric: 10
Router Link States (Area 172.16.117.0)
LS age: 211
Options: 0x2 : *|-|-|-|-|-|E|-
LS Flags: 0x6
Flags: 0x0
LS Type: router-LSA
Link State ID: 172.16.117.1
Advertising Router: 172.16.117.1
LS Seq Number: 80000100
Checksum: 0x8785
Length: 60
Number of Links: 3
Link connected to: another Router (point-to-point)
(Link ID) Neighboring Router ID: 172.26.0.1
(Link Data) Router Interface address: 100.100.100.42
Number of TOS metrics: 0
TOS 0 Metric: 10
Link connected to: Stub Network
(Link ID) Net: 100.100.100.40
(Link Data) Network Mask: 255.255.255.252
Number of TOS metrics: 0
TOS 0 Metric: 10
Link connected to: Stub Network
(Link ID) Net: 172.16.117.0
(Link Data) Network Mask: 255.255.255.0
Number of TOS metrics: 0
TOS 0 Metric: 10
LS age: 1625
Options: 0x2 : *|-|-|-|-|-|E|-
LS Flags: 0x3
Flags: 0x1 : ABR
LS Type: router-LSA
Link State ID: 172.26.0.1
Advertising Router: 172.26.0.1
LS Seq Number: 80000014
Checksum: 0x224a
Length: 36
Number of Links: 1
Link connected to: another Router (point-to-point)
(Link ID) Neighboring Router ID: 172.16.117.1
(Link Data) Router Interface address: 0.0.0.8
Number of TOS metrics: 0
TOS 0 Metric: 10
What does default area "area1" on a Mikrotik mean? That should be area 0. Or 0.0.0.0 - one and the same, just a 32 bit value.
/routing ospf area
set [ find default=yes ] disabled=yes
add area-id=172.16.117.0 name=area1
area1 - is just local name (description) of area in config. Area, configured at MikroTik is 172.16.117.0, аt opnsense backbone (0.0.0.0). In BB area network 172.26.0.0/28. In area 172.16.117.0 network 172.16.117.0/24 and 100.100.100.40/30 (Gre tunnel's network)
You need to set the area to 0.0.0.0 on the Mikrotik. Two OSPF neighbours must share the same area on the link. Areas can only change when you cross routers, i.e.
router 1 ------ router 2 ------ router 3
area 0 area 1
Router 2 would be called an area border router while routers 1 and 3 are autonomous system border routers.
And there is!
One directly attached to opnsense network is in BB area. Directly attached network to Mikrotik and network between opnsense and Mikrotik in other area. I know, how works ospf and I am supporting big network on cisco an juniper hardware too, but this problem is new for me.
Quote from: @Vorona on April 13, 2021, 10:04:19 AM
One directly attached to opnsense network is in BB area. Directly attached network to Mikrotik and network between opnsense and Mikrotik in other area. I know, how works ospf and I am supporting big network on cisco an juniper hardware too, but this problem is new for me.
Sorry, that wasn't obvious to me from your posts. In that case I am running out of ideas as well ...
Sadly... :(
Maybe this is something bug in frr ospfd? I found similar problem, but it was with bgp protocol on old version opnsense.
Do you have a change to use routed ipsec where you don't need a GRE tunnel?
You cannot form an OSPF neighbour relation without a dedicated point-to-point link. That is one of the main reasons to use a tunnel interface.
But could you try to disable IPsec just for debugging purposes and try if OSPF works over GRE alone?
Isnt this the reason for a route based IPsec? A dedicated interface with p2p address
Quote from: mimugmail on April 13, 2021, 09:10:59 PM
Isnt this the reason for a route based IPsec? A dedicated interface with p2p address
Ah ... ok. Never used these. Traditional Kame IPsec doesn't have that feature. So you end up with GRE or IPIP.
I currently set up a lab cause of wireguard reports with OSPF, I'll try to test this too
Quote from: pmhausen on April 13, 2021, 01:54:11 PM
But could you try to disable IPsec just for debugging purposes and try if OSPF works over GRE alone?
I cannot do that. Only one point has real IP address. Otrer point behind NAT. For this reason IPSec works in tunnel mode. Why do you think, that IPSec can break OSPF, wich works in GRE? Tunnel works, firewall on tunnel at both sides are open.
Quote from: mimugmail on April 14, 2021, 09:23:02 AM
I currently set up a lab cause of wireguard reports with OSPF, I'll try to test this too
You mean RB IPSec or OSPF relations between Opnsense and MikroTik?
Ok, I just finished the lab and I'm able to ping from host-A to host-B via FW-A to FW-B which has a IPsec with type transport and inside a GRE tunnel and inside GRE the OSPF protocol.
root@OPNsense:~ # tcpdump -n -i gre0
15:37:40.506482 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Hello, length 48
15:37:40.518025 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.530882 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.537441 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 72
15:37:40.541540 IP 10.253.253.2 > 224.0.0.5: OSPFv2, Database Description, length 72
15:37:40.541671 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Request, length 48
15:37:40.542538 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Database Description, length 32
15:37:40.542587 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Request, length 48
15:37:40.542741 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 88
15:37:40.543885 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 88
15:37:40.544078 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:40.545609 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:40.725297 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Ack, length 64
15:37:41.359517 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Ack, length 64
15:37:45.574028 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:45.575582 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Update, length 64
15:37:45.595667 IP 10.253.253.1 > 224.0.0.5: OSPFv2, Hello, length 48
15:37:45.893086 IP 10.253.253.1 > 224.0.0.5: OSPFv2, LS-Ack, length 44
15:37:46.587098 IP 10.253.253.2 > 224.0.0.5: OSPFv2, LS-Ack, length 44
15:37:46.673133 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44857, length 64
15:37:46.675096 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44857, length 64
15:37:46.872543 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44822, length 64
15:37:46.873544 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44822, length 64
15:37:47.723243 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44858, length 64
15:37:47.723928 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44858, length 64
15:37:47.942779 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44823, length 64
15:37:47.943702 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44823, length 64
15:37:48.793598 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44859, length 64
15:37:48.794412 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44859, length 64
15:37:48.972486 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44824, length 64
15:37:48.973553 IP 192.168.10.2 > 192.168.11.3: ICMP echo reply, id 34831, seq 44824, length 64
15:37:49.823147 IP 192.168.10.2 > 192.168.11.3: ICMP echo request, id 63907, seq 44860, length 64
15:37:49.823893 IP 192.168.11.3 > 192.168.10.2: ICMP echo reply, id 63907, seq 44860, length 64
15:37:50.042721 IP 192.168.11.3 > 192.168.10.2: ICMP echo request, id 34831, seq 44825, length 64
Good!
What hardware do you use with opnsense? What version?
In my gre I see OSPF traffic in both ways too, but route wrong.
Just VMs with 21.1.4
And same strange things for me.
On mikrotik side, on LSA tab i found this:
Wrong data (OPNsense)
flags=
links (type, id, data, metric)
Point-To-Point 10.74.0.1 0.0.0.7 10
Stub 10.77.0.0 255.255.255.0 10
Correct data (Mikrotik)
flags=
links (type, id, data, metric)
Point-To-Point 10.74.0.1 172.16.0.2 10
Stub 172.16.0.0 255.255.255.252 10
Stub 10.77.0.0 255.255.255.0 10
Note:
* 10.74.0.1/16 - Router ID of Mikrotik #1
* 10.77.0.1/16 - Router ID of OPNsense
172.16.0.0/30 - GRE tunnel network:
172.16.0.1 - mikrotik ip
172.16.0.2 - opnsense ip
What is "0.0.0.7" ?
Any ideas how to fix it (Make it work?)
Look like OSPF won't announce GRE tunnel network to Mikrotik, so mikrotik don't known how to route 10.77 network (And show it as "empty" gateway and interface)
note2:
In OPNSense, on "Network" side i add two networks: LAN and GRE (to area "0.0.0.0"). On "Interfaces" - LAN and GRE interfaces.
Screenshots please
What exacly you want to see?
OSPF and GRE settings almost same on Mikrotik and OPNsense.
Router #1: 10.74.0.0/16 (Mikrotik, gre ip: 172.16.74.13)
Router #2: 10.77.0.0/16 (OPNsense, gre ip: 172.16.74.14)
GRE between routers: 172.16.74.12/30
Can ping both gre ip's. Can ping and access resource behind both routers if select interface manually or add static route.
But on mikrotik side - ospf won't work. Route added without gateway and interface.
OSPF config on OPNsense:
Current configuration:
!
frr version 7.4
frr defaults traditional
hostname gw.dc.sipcolor.ru.sipcolor.ru
log syslog notifications
!
router ospf
ospf router-id 10.77.0.1
passive-interface bridge0
passive-interface vtnet0
passive-interface vtnet1
network 10.77.0.0/16 area 0.0.0.0
network 172.16.74.12/30 area 0.0.0.0
!
line vty
!
end
When you capture on GRE .. do you see OSPF packets in and out?
Quote from: mimugmail on July 22, 2021, 02:58:08 PM
When you capture on GRE .. do you see OSPF packets in and out?
Yep. Hello packets, LSA...
I see neighbors in OSPF on both routers, i see routes. But on OPNsense routes is correct, but on Mikrotik - with empty gateways/interfaces (see screenshot above).
I test OPNsense and Mikrotik (as 2nd router) and found difference:
OPNsense (fw_v21.1.8_1 / FRR_v7.4_6):
Type: Stub ID: 10.77.0.0 Data: 255.255.0.0 Metric: 10
Type: PTP ID: 10.74.0.1 Data: 0.0.0.8 Metric: 10
Mikrotik (fw_v6.48.3):
Type: Stub ID: 10.77.0.0 Data: 255.255.0.0 Metric: 10
Type: PTP ID: 10.74.0.1 Data: 172.16.74.14 Metric: 10
Type: Stub ID: 172.16.74.12 Data: 255.255.255.252 Metric: 11
As you see - mikrotik send an additional "stub" route, but OPNsense - ony strange "0.0.0.8".
On Wiki i found - thats a port number (MIB-II notation) and Mikrotik won't understand this form of record (Or expect a regular route/ip address).
Any ideas how to fix it?
What is 172.16.255? 0.0.0.7 is a wildcard mask.
Quote from: mimugmail on July 22, 2021, 07:20:02 PM
What is 172.16.255? 0.0.0.7 is a wildcard mask.
172.16.255.74 = 10.74.0.1 (Router ID). Sorry, wrong screenshot (I just try to change router ID)
create a loopback and place all your devices in the same subnet for the loopback. place the loopback in the area 0. make sure your loopback is a passive interface.
Quote from: lilsense on July 22, 2021, 09:38:56 PM
create a loopback and place all your devices in the same subnet for the loopback. place the loopback in the area 0. make sure your loopback is a passive interface.
Ok... I did:
* Interfaces - Other types - Loopback - Add (Description: "172.16.255.77". It is ok? I can't set text here, only ip-like record)
* Interfaces - Assignments - Loopback - Add. Set Enabled, IPv4 = Static, 172.16.255.77/32
* Routing - OSPF - Router ID set to "172.16.255.77", added loopback to passive interfaces
* Routing - OSPF - Networks - Add: 172.16.0.0/16, Area: 0.0.0.0
On mikrotik - add only network to area 0.0.0.0...
Now - i have additional route in ospf routes list on mikrotik... Without gateway/interface...
I think mikrotik just not understand "Wildcard" mask... It is possible to swith it on OPNsense (For compatability)?
On Mikrotik (Now his ID is "172.16.255.74"):
flags=
links (type, id, data, metric)
Stub 10.77.0.0 255.255.0.0 10
Point-To-Point 172.16.255.74 0.0.0.8 10
Stub 172.16.255.77 255.255.255.255 0
Hmm.. Seems like its a bug in FRR: https://github.com/FRRouting/frr/issues/3973
And its still not fixed :(
Hi!
I see, that this problem can be resolved by changing FRR to bird :)
I tried to upgrade FRR to 7.5.1 from FreeBSD repositories, but interface gre still be unnumbered. I think for that reason I see "empty" routes in ospf at mikrotik side.
Hello!
I found working solution!
In last version mikrotik firmware added network type "unnumbered p-t-p". With this type all works fine.