Hi!
I have 6 OPNsense firewalls which all share IPv4 routes via OSPF. I am part way through upgrading them all from 23.1.11_1 to 23.7.2. No configuration changes have taken place other than upgrading, however I am aware this upgrade moves to FRR 8. What I am seeing is that OSPF on the upgraded firewalls does not redistribute kernel, or static routes despite having the options set in the configuration.
e.g. from inside vtysh
fw-1# sh run
Building configuration...
Current configuration:
!
frr version 8.5.2
frr defaults traditional
hostname fw-1
log syslog notifications
!
interface openvpn
ip ospf passive
exit
!
interface vmx0
ip ospf passive
exit
!
interface vmx2
ip ospf cost 200
exit
!
interface vmx3
ip ospf passive
exit
!
interface vmx4
ip ospf passive
exit
!
interface vmx5
ip ospf passive
exit
!
router ospf
ospf router-id 7.0.1.1
redistribute kernel
redistribute connected
redistribute static
network 1.2.3.4/16 area 1.1.1.1
network 10.1.1.0/30 area 1.1.1.1
network 10.2.2.0/30 area 1.1.1.1
exit
!
end
My kernel routing table:
root@fw-1:/var/log # netstat -rn
Routing tables
Internet:
Destination Gateway Flags Netif Expire
default x.x.x.x UG1 vmx1
10.0.0.0/16 127.0.0.1 USB lo0
10.1.0.0/24 127.0.0.1 USB lo0
-- snip --
10.2.2.0/24 127.0.0.1 USB lo0
10.2.3.0/24 127.0.0.1 USB lo0
My routing table in FRR - note there are only OSPF and Connected routes in the table. Also note I have had to substitute the IP addresses but they are not relevant to the problem:
fw-1# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
O>* 0.0.0.0/0 [110/1] via 1.1.1.1, vmx1, weight 1, 00:23:35
O>* 1.1.1.1/28 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
O 1.1.1.1/24 [110/10] is directly connected, vmx0, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx0, 00:23:51
O 1.1.1.1/24 [110/10] is directly connected, vmx3, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx3, 00:23:51
O 1.1.1.1/24 [110/10] is directly connected, vmx4, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx4, 00:23:51
O>* 1.1.1.1/28 [110/120] via 1.1.1.1, vmx1, weight 1, 00:23:36
O 1.1.1.1/26 [110/10] is directly connected, ovpns1, weight 1, 00:23:51
C>* 1.1.1.1/26 [0/1] is directly connected, ovpns1, 00:23:51
O 1.1.1.1/30 [110/10] is directly connected, vmx5, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx5, 00:23:51
O>* 1.1.1.1/30 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
O 1.1.1.1/30 [110/10] is directly connected, vmx1, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx1, 00:23:51
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx1, weight 1, 00:23:36
O 1.1.1.1/30 [110/200] is directly connected, vmx2, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx2, 00:23:51
O>* 1.1.1.1/30 [110/400] via1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx1, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/400] via 1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/32 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
Yet here is the routing table from within FRR/vtysh on a firewall which has NOT been upgraded, running the same configuration. As it has not been upgraded it runs FRR 7.5.1
fw-2# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, T - Table,
v - VNC, V - VNC-Direct, A - Babel, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
O>* 0.0.0.0/0 [110/1] via 1.1.1.1, vmx1, weight 1, 00:23:35
O>* 1.1.1.1/28 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
O 1.1.1.1/24 [110/10] is directly connected, vmx0, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx0, 00:23:51
O 1.1.1.1/24 [110/10] is directly connected, vmx3, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx3, 00:23:51
O 1.1.1.1/24 [110/10] is directly connected, vmx4, weight 1, 00:23:51
C>* 1.1.1.1/24 [0/1] is directly connected, vmx4, 00:23:51
O>* 1.1.1.1/28 [110/120] via 1.1.1.1, vmx1, weight 1, 00:23:36
O 1.1.1.1/26 [110/10] is directly connected, ovpns1, weight 1, 00:23:51
C>* 1.1.1.1/26 [0/1] is directly connected, ovpns1, 00:23:51
O 1.1.1.1/30 [110/10] is directly connected, vmx5, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx5, 00:23:51
O>* 1.1.1.1/30 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
O 1.1.1.1/30 [110/10] is directly connected, vmx1, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx1, 00:23:51
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx1, weight 1, 00:23:36
O 1.1.1.1/30 [110/200] is directly connected, vmx2, weight 1, 00:23:51
C>* 1.1.1.1/30 [0/1] is directly connected, vmx2, 00:23:51
O>* 1.1.1.1/30 [110/400] via1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx1, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/210] via 1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/30 [110/400] via 1.1.1.1, vmx2, weight 1, 00:23:36
O>* 1.1.1.1/32 [110/20] via 1.1.1.1, vmx1, weight 1, 00:23:35
K>* 10.0.0.0/16 [0/0] unreachable (blackhole), 01:32:28
K>* 10.1.0.0/24 [0/0] unreachable (blackhole), 01:32:28
K>* 10.2.2.0/24 [0/0] unreachable (blackhole), 01:32:28
K>* 10.2.3.0/24 [0/0] unreachable (blackhole), 01:32:28
Same config file, different version of FRR, different result. Does FRR 8 no longer redistribute blackhole/unreachable routes? I use this to inject routes to policy-based VPN tunnels into the OSPF process.
Check in OSPF - General Tab - Route Redistribution - and enable Kernel Routes in the drop down menu. Seems to be Off by default
Thanks for the suggestion, but this setting is still on post upgrade - it's what puts the redistribute lines into the ospfd.conf file. Configuration hasn't changed, not in opnsense or in frr.
Looks like there's an incomplete issue raised already on Github, might be best to comment there as well or open a new one referencing it
https://github.com/opnsense/plugins/issues/3543 (https://github.com/opnsense/plugins/issues/3543)
As only frr changed from 7 to 8 maybe someone can search the web if there changed anything inside frr or needs some additional configs