OSPF weird behaviour

[steilfirn_8000]

Hello everyone,

I stumbled accross a weird routing behaviour on my network.
In general my network is:

Mikrotik CCR2004 as internet & VPN router connected to 2x OPNsense which are connected to a Mikrotik CCR2116 as my network router.
As a failover my CCR2004 is also direct to CCR2116 but with higher costs so any traffic would go through my firewall.

Anytime I modify an OSPF setting on OPNsense and reload the process it gets reconnected but no traffic is going through it - it becomes unreachable.

Both Mikrotik routers show that OSPF is connected and exchanged all information (State = Full).

I connected to an OPNsense VM to see what is happending there and FRRs vtysh also show that it's fully exchanged on I can see all routes.
Only ICMP and traceroute is not working:

Code Select Expand

PING k8s-1.hks.lan (10.0.22.80): 56 data bytes
92 bytes from 172.16.1.2: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
4  5  00 0054 b660   0 0000  01  01 35e7 172.16.1.2  10.0.22.80

92 bytes from 172.16.1.2: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
4  5  00 0054 ef8c   0 0000  01  01 fcba 172.16.1.2  10.0.22.80

92 bytes from 172.16.1.2: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
4  5  00 0054 cbe3   0 0000  01  01 2064 172.16.1.2  10.0.22.80

Traceroute is running between Mikrotik and CCR2116 - I guess until TTL is reached and is then been kicked.


Currently the only solution is to restart CCR2116 to get everything running again.


Is there a bug somewhere?

[steilfirn_8000]

I think I fixed it by setting different costs for interface pointing to CCR2004 and CCR2116

[mimugmail]

Yes, looked like a routing loop which get's fixed with different costs.

[steilfirn_8000]

Unfortunately it seems that it's not related to the costs but seems to be a bug at FRR.
I logged at Github ticket to investigate this behaviour: https://github.com/opnsense/plugins/issues/3846

From my point of view the issue is related to network type point to multipoint as there my error occurs.
Yesterday I changed to regular broadcast and this works without my issues.


In general costs should be fine as my direct connection from CCR2004 to CCR2116 should be not used at all as costs on both interfaces are set to 100 each while OPNsense1 is using 10 and OPNsense2 20.
All routers are in the same area and should be aware of all costs.