Routing Issues on Reboot

[LikelyInfection]

When I reboot my secondary router and then my primary router, the primary router seems to have some issue installing the route? I'm specifically trying to get to a loopback ip on the secondary router: 192.168.131.2.

Code Select Expand

root@router01:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 1.2.3.4
        fib: 0
  interface: vtnet3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
 
router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
  Known via "bgp", distance 200, metric 0, best
  Last update 00:05:52 ago
  * 192.168.255.4, via vtnet0, weight 1

So, it's in FRR's routing table but not the OS routing table? Let's reset the bgp session:

Code Select Expand

router01.example.com# clear ip bgp 192.168.255.4
router01.example.com#

root@router02:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 192.168.131.2
    gateway: 192.168.255.4
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,HOST,DONE,PROTO1>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Any thoughts on why this might be the case?

[Seimus]

There is no FRR routing table, there is only one routing table called GRT (cause OPNsense doesn't support VRFs).

The BGP contains a table/list of prefixes, these prefixes are installed into the routing table under certain conditions:
1. More specific (longest prefix wins)
2. Better AD (lowest AD wins)
3. Protocol specific metric/attributes

So the originator of that route/IP is the Router2 and you advertise it over BGP to Router1?
You say you have problem to install it into the Routing table, meaning it never installs there or it takes time?

Regards,
S.

[LikelyInfection]

So the originator of that route/IP is the Router2 and you advertise it over BGP to Router1?
You say you have problem to install it into the Routing table, meaning it never installs there or it takes time?

Yes, that IP is assigned to a loopback device on router2 and it advertises the /32 over BGP to router1.

If I do my standard upgrade procedure, which is to upgrade router2, reboot router2, upgrade router1, reboot router1, the route never seems to get installed. Last time I waited about 6 weeks before poking it :)

I'm not sure of the specifics, but by saying that it's in the FRR "routing table" I just meant that (as shown above) I could use "show ip route" in vtysh and it would tell me that the best route is what I expect, unlike using "route -n get" from bash.

[Seimus]

You mention here upgrade procedure,

So the issue is appearing when you perform upgrade to the latest OPNsense release or you mean in it happens each time you perform reboot?

One very important note, BGP will advertise a prefix only in case it has a route for it installed in the routing table.

What I mean by that is,
Router2 has to have the route towards that destination (loopback) in his route table in order for it to advertise it over BGP.

When this issues happens, can you check

1. if your Router2 advertises that BGP prefix in the BGP table? (not the route table)

Code Select Expand

show ip bgp neighbors IPneighbors advertised-routes
2. if your Router1 recieves that BGP prefix in the BGP table? (not the routing table)

Code Select Expand

show ip bgp neighbors IPneighbors routes
https://docs.frrouting.org/en/latest/bgp.html#clicmd-show-bgp-afi-safi-neighbor-PEER-routes-advertised-routes-received-routes-A.B.C.D-M-X-X-X-X-M-detail-json


Another question, as this is actually an interface on R2 you try to advertise.
How do you advertise it? (network statement or connected?)
Can you show your BGP config?

Regards,
S.

[LikelyInfection]

I just checked and it happens when I reboot router 1 (which just so happens to be the last thing I do when I upgrade both of them which is why I noticed it in that context).

Code Select Expand

root@router01:~ # w
 7:14PM  up 2 mins, 1 user, load averages: 0.13, 0.17, 0.08
USER       TTY      FROM              LOGIN@  IDLE WHAT
me         pts/0    192.168.128.147   7:10PM     - w
root@router01:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 1.2.3.4
        fib: 0
  interface: vtnet3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
root@router01:~ # vtysh
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 3: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 4: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 5: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 6: SO_RCVBUF set to 8388608 (requested 16777216)

Hello, this is FRRouting (version 10.4.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
  Known via "bgp", distance 200, metric 0, best
  Last update 00:02:30 ago
  * 192.168.255.4, via vtnet0, weight 1

router01.example.com# show ip bgp neighbors 192.168.255.4 advertised-routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.131.1/32 0.0.0.0                  0    100  32768 i
 *>  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *>  192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 *>  192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 *>  192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Total number of prefixes 12
router01.example.com# show ip bgp neighbors 192.168.255.4 routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 192.168.131.2/32 192.168.255.4            0    100      0 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Displayed 12 routes and 39 total paths
router02 is advertising of course:

Code Select Expand

router02.example.com# show ip bgp neighbors 192.168.255.3 advertised-routes
BGP table version is 47, local router ID is 192.168.255.4, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.131.2/32 0.0.0.0                  0    100  32768 i
 *>  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *>  192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 *>  192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 *>  192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Total number of prefixes 12

router01 config:

Code Select Expand

router01.example.com# show run
Building configuration...

Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router01.example.com
log syslog notifications
!
router bgp 64601
 bgp router-id 192.168.255.3
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart
 neighbor 192.168.128.41 remote-as 64641
 neighbor 192.168.128.41 update-source vtnet1
 neighbor 192.168.128.42 remote-as 64641
 neighbor 192.168.128.42 update-source vtnet1
 neighbor 192.168.128.43 remote-as 64641
 neighbor 192.168.128.43 update-source vtnet1
 neighbor 192.168.128.44 remote-as 64641
 neighbor 192.168.128.44 update-source vtnet1
 neighbor 192.168.128.45 remote-as 64641
 neighbor 192.168.128.45 update-source vtnet1
 neighbor 192.168.128.46 remote-as 64641
 neighbor 192.168.128.46 update-source vtnet1
 neighbor 192.168.255.4 remote-as 64601
 neighbor 192.168.255.4 update-source vtnet0
 !
 address-family ipv4 unicast
  network 192.168.131.1/32
  neighbor 192.168.128.41 activate
  neighbor 192.168.128.41 soft-reconfiguration inbound
  neighbor 192.168.128.42 activate
  neighbor 192.168.128.42 soft-reconfiguration inbound
  neighbor 192.168.128.43 activate
  neighbor 192.168.128.43 soft-reconfiguration inbound
  neighbor 192.168.128.44 activate
  neighbor 192.168.128.44 soft-reconfiguration inbound
  neighbor 192.168.128.45 activate
  neighbor 192.168.128.45 soft-reconfiguration inbound
  neighbor 192.168.128.46 activate
  neighbor 192.168.128.46 soft-reconfiguration inbound
  neighbor 192.168.255.4 activate
  neighbor 192.168.255.4 soft-reconfiguration inbound
 exit-address-family
exit
!
end
router02 config:

Code Select Expand

router02.example.com# show run
Building configuration...

Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router02.example.com
log syslog notifications
!
router bgp 64601
 bgp router-id 192.168.255.4
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart
 neighbor 192.168.128.41 remote-as 64641
 neighbor 192.168.128.41 update-source vtnet1
 neighbor 192.168.128.42 remote-as 64641
 neighbor 192.168.128.42 update-source vtnet1
 neighbor 192.168.128.43 remote-as 64641
 neighbor 192.168.128.43 update-source vtnet1
 neighbor 192.168.128.44 remote-as 64641
 neighbor 192.168.128.44 update-source vtnet1
 neighbor 192.168.128.45 remote-as 64641
 neighbor 192.168.128.45 update-source vtnet1
 neighbor 192.168.128.46 remote-as 64641
 neighbor 192.168.128.46 update-source vtnet1
 neighbor 192.168.255.3 remote-as 64601
 neighbor 192.168.255.3 update-source vtnet0
 !
 address-family ipv4 unicast
  network 192.168.131.2/32
  neighbor 192.168.128.41 activate
  neighbor 192.168.128.41 soft-reconfiguration inbound
  neighbor 192.168.128.42 activate
  neighbor 192.168.128.42 soft-reconfiguration inbound
  neighbor 192.168.128.43 activate
  neighbor 192.168.128.43 soft-reconfiguration inbound
  neighbor 192.168.128.44 activate
  neighbor 192.168.128.44 soft-reconfiguration inbound
  neighbor 192.168.128.45 activate
  neighbor 192.168.128.45 soft-reconfiguration inbound
  neighbor 192.168.128.46 activate
  neighbor 192.168.128.46 soft-reconfiguration inbound
  neighbor 192.168.255.3 activate
  neighbor 192.168.255.3 soft-reconfiguration inbound
 exit-address-family
exit
!
end