When I reboot my secondary router and then my primary router, the primary router seems to have some issue installing the route? I'm specifically trying to get to a loopback ip on the secondary router: 192.168.131.2.
root@router01:~ # route -n get 192.168.131.2
route to: 192.168.131.2
destination: 0.0.0.0
mask: 0.0.0.0
gateway: 1.2.3.4
fib: 0
interface: vtnet3
flags: <UP,GATEWAY,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
Known via "bgp", distance 200, metric 0, best
Last update 00:05:52 ago
* 192.168.255.4, via vtnet0, weight 1
So, it's in FRR's routing table but not the OS routing table? Let's reset the bgp session:
router01.example.com# clear ip bgp 192.168.255.4
router01.example.com#
root@router02:~ # route -n get 192.168.131.2
route to: 192.168.131.2
destination: 192.168.131.2
gateway: 192.168.255.4
fib: 0
interface: vtnet0
flags: <UP,GATEWAY,HOST,DONE,PROTO1>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
Any thoughts on why this might be the case?
There is no FRR routing table, there is only one routing table called GRT (cause OPNsense doesn't support VRFs).
The BGP contains a table/list of prefixes, these prefixes are installed into the routing table under certain conditions:
1. More specific (longest prefix wins)
2. Better AD (lowest AD wins)
3. Protocol specific metric/attributes
So the originator of that route/IP is the Router2 and you advertise it over BGP to Router1?
You say you have problem to install it into the Routing table, meaning it never installs there or it takes time?
Regards,
S.
Quote from: Seimus on October 11, 2025, 01:05:47 PMSo the originator of that route/IP is the Router2 and you advertise it over BGP to Router1?
You say you have problem to install it into the Routing table, meaning it never installs there or it takes time?
Yes, that IP is assigned to a loopback device on router2 and it advertises the /32 over BGP to router1.
If I do my standard upgrade procedure, which is to upgrade router2, reboot router2, upgrade router1, reboot router1, the route never seems to get installed. Last time I waited about 6 weeks before poking it :)
I'm not sure of the specifics, but by saying that it's in the FRR "routing table" I just meant that (as shown above) I could use "show ip route" in vtysh and it would tell me that the best route is what I expect, unlike using "route -n get" from bash.
You mention here upgrade procedure,
So the issue is appearing when you perform upgrade to the latest OPNsense release or you mean in it happens each time you perform reboot?
One very important note, BGP will advertise a prefix only in case it has a route for it installed in the routing table.
What I mean by that is,
Router2 has to have the route towards that destination (loopback) in his route table in order for it to advertise it over BGP.
When this issues happens, can you check
1. if your Router2 advertises that BGP prefix in the BGP table? (not the route table)
show ip bgp neighbors IPneighbors advertised-routes
2. if your Router1 recieves that BGP prefix in the BGP table? (not the routing table)
show ip bgp neighbors IPneighbors routes
https://docs.frrouting.org/en/latest/bgp.html#clicmd-show-bgp-afi-safi-neighbor-PEER-routes-advertised-routes-received-routes-A.B.C.D-M-X-X-X-X-M-detail-json
Another question, as this is actually an interface on R2 you try to advertise.
How do you advertise it? (network statement or connected?)
Can you show your BGP config?
Regards,
S.
I just checked and it happens when I reboot router 1 (which just so happens to be the last thing I do when I upgrade both of them which is why I noticed it in that context).
root@router01:~ # w
7:14PM up 2 mins, 1 user, load averages: 0.13, 0.17, 0.08
USER TTY FROM LOGIN@ IDLE WHAT
me pts/0 192.168.128.147 7:10PM - w
root@router01:~ # route -n get 192.168.131.2
route to: 192.168.131.2
destination: 0.0.0.0
mask: 0.0.0.0
gateway: 1.2.3.4
fib: 0
interface: vtnet3
flags: <UP,GATEWAY,DONE,STATIC>
recvpipe sendpipe ssthresh rtt,msec mtu weight expire
0 0 0 0 1500 1 0
root@router01:~ # vtysh
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 3: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 4: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 5: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 6: SO_RCVBUF set to 8388608 (requested 16777216)
Hello, this is FRRouting (version 10.4.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
Known via "bgp", distance 200, metric 0, best
Last update 00:02:30 ago
* 192.168.255.4, via vtnet0, weight 1
router01.example.com# show ip bgp neighbors 192.168.255.4 advertised-routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 192.168.131.1/32 0.0.0.0 0 100 32768 i
*> 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
* i 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*> 192.168.255.16/32
192.168.128.44 0 100 0 64641 i
* i 192.168.255.16/32
192.168.128.44 0 100 0 64641 i
*> 192.168.255.17/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.17/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*> 192.168.255.19/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.19/32
192.168.128.45 0 100 0 64641 i
*> 192.168.255.20/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.20/32
192.168.128.45 0 100 0 64641 i
*> 192.168.255.21/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.21/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*> 192.168.255.24/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.24/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.25/32
192.168.128.42 0 100 0 64641 i
* i 192.168.255.25/32
192.168.128.42 0 100 0 64641 i
*> 192.168.255.26/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.26/32
192.168.128.45 0 100 0 64641 i
Total number of prefixes 12
router01.example.com# show ip bgp neighbors 192.168.255.4 routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*>i 192.168.131.2/32 192.168.255.4 0 100 0 i
* i 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
* i 192.168.255.16/32
192.168.128.44 0 100 0 64641 i
* i 192.168.255.17/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.19/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.20/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.21/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.24/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.25/32
192.168.128.42 0 100 0 64641 i
* i 192.168.255.26/32
192.168.128.45 0 100 0 64641 i
Displayed 12 routes and 39 total paths
router02 is advertising of course:
router02.example.com# show ip bgp neighbors 192.168.255.3 advertised-routes
BGP table version is 47, local router ID is 192.168.255.4, vrf id 0
Default local pref 100, local AS 64601
Status codes: s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 192.168.131.2/32 0.0.0.0 0 100 32768 i
*> 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*= 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
* i 192.168.255.8/32 192.168.128.46 0 100 0 64641 i
*> 192.168.255.16/32
192.168.128.44 0 100 0 64641 i
* i 192.168.255.16/32
192.168.128.44 0 100 0 64641 i
*> 192.168.255.17/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.17/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.18/32
192.168.128.46 0 100 0 64641 i
*> 192.168.255.19/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.19/32
192.168.128.45 0 100 0 64641 i
*> 192.168.255.20/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.20/32
192.168.128.45 0 100 0 64641 i
*> 192.168.255.21/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.21/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*= 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
* i 192.168.255.23/32
192.168.128.46 0 100 0 64641 i
*> 192.168.255.24/32
192.168.128.41 0 100 0 64641 i
* i 192.168.255.24/32
192.168.128.41 0 100 0 64641 i
*> 192.168.255.25/32
192.168.128.42 0 100 0 64641 i
* i 192.168.255.25/32
192.168.128.42 0 100 0 64641 i
*> 192.168.255.26/32
192.168.128.45 0 100 0 64641 i
* i 192.168.255.26/32
192.168.128.45 0 100 0 64641 i
Total number of prefixes 12
router01 config:
router01.example.com# show run
Building configuration...
Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router01.example.com
log syslog notifications
!
router bgp 64601
bgp router-id 192.168.255.3
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
bgp graceful-restart
neighbor 192.168.128.41 remote-as 64641
neighbor 192.168.128.41 update-source vtnet1
neighbor 192.168.128.42 remote-as 64641
neighbor 192.168.128.42 update-source vtnet1
neighbor 192.168.128.43 remote-as 64641
neighbor 192.168.128.43 update-source vtnet1
neighbor 192.168.128.44 remote-as 64641
neighbor 192.168.128.44 update-source vtnet1
neighbor 192.168.128.45 remote-as 64641
neighbor 192.168.128.45 update-source vtnet1
neighbor 192.168.128.46 remote-as 64641
neighbor 192.168.128.46 update-source vtnet1
neighbor 192.168.255.4 remote-as 64601
neighbor 192.168.255.4 update-source vtnet0
!
address-family ipv4 unicast
network 192.168.131.1/32
neighbor 192.168.128.41 activate
neighbor 192.168.128.41 soft-reconfiguration inbound
neighbor 192.168.128.42 activate
neighbor 192.168.128.42 soft-reconfiguration inbound
neighbor 192.168.128.43 activate
neighbor 192.168.128.43 soft-reconfiguration inbound
neighbor 192.168.128.44 activate
neighbor 192.168.128.44 soft-reconfiguration inbound
neighbor 192.168.128.45 activate
neighbor 192.168.128.45 soft-reconfiguration inbound
neighbor 192.168.128.46 activate
neighbor 192.168.128.46 soft-reconfiguration inbound
neighbor 192.168.255.4 activate
neighbor 192.168.255.4 soft-reconfiguration inbound
exit-address-family
exit
!
end
router02 config:
router02.example.com# show run
Building configuration...
Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router02.example.com
log syslog notifications
!
router bgp 64601
bgp router-id 192.168.255.4
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
bgp graceful-restart
neighbor 192.168.128.41 remote-as 64641
neighbor 192.168.128.41 update-source vtnet1
neighbor 192.168.128.42 remote-as 64641
neighbor 192.168.128.42 update-source vtnet1
neighbor 192.168.128.43 remote-as 64641
neighbor 192.168.128.43 update-source vtnet1
neighbor 192.168.128.44 remote-as 64641
neighbor 192.168.128.44 update-source vtnet1
neighbor 192.168.128.45 remote-as 64641
neighbor 192.168.128.45 update-source vtnet1
neighbor 192.168.128.46 remote-as 64641
neighbor 192.168.128.46 update-source vtnet1
neighbor 192.168.255.3 remote-as 64601
neighbor 192.168.255.3 update-source vtnet0
!
address-family ipv4 unicast
network 192.168.131.2/32
neighbor 192.168.128.41 activate
neighbor 192.168.128.41 soft-reconfiguration inbound
neighbor 192.168.128.42 activate
neighbor 192.168.128.42 soft-reconfiguration inbound
neighbor 192.168.128.43 activate
neighbor 192.168.128.43 soft-reconfiguration inbound
neighbor 192.168.128.44 activate
neighbor 192.168.128.44 soft-reconfiguration inbound
neighbor 192.168.128.45 activate
neighbor 192.168.128.45 soft-reconfiguration inbound
neighbor 192.168.128.46 activate
neighbor 192.168.128.46 soft-reconfiguration inbound
neighbor 192.168.255.3 activate
neighbor 192.168.255.3 soft-reconfiguration inbound
exit-address-family
exit
!
end
Thanks for the detailed outputs!
Can you do one more?
From Router1 when the route is not present in the route table(when you see the issue)
show ip bgp 192.168.131.2
Also does the issues appear as well when you restart just the BGP on the Router1?
Regards,
S.
root@router01:~ # netstat -rn| grep 131
192.168.131.1 link#13 UH lo1
...
router01.example.com# show ip bgp 192.168.131.2
BGP routing table entry for 192.168.131.2/32, version 2
Paths: (1 available, best #1, table default)
Advertised to peers:
192.168.128.41 192.168.128.42 192.168.128.43 192.168.128.44 192.168.128.45 192.168.128.46
Local
192.168.255.4 (metric 1) from 192.168.255.4 (192.168.255.4)
Origin IGP, metric 0, localpref 100, valid, internal, best (First path received)
Last update: Thu Oct 16 19:10:36 2025
root@router01:~ # netstat -rn -f inet| grep 131
192.168.131.1 link#13 UH lo1
root@router01:~ # service frr restart bgpd
Stopping bgpd.
Waiting for PIDS: 62018.
Starting bgpd.
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 3: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 4: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 5: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 6: SO_RCVBUF set to 8388608 (requested 16777216)
[39208|mgmtd] sending configuration
[39306|zebra] sending configuration
[39801|bgpd] sending configuration
Waiting for children to finish applying config...
[40639|watchfrr] sending configuration
[39208|mgmtd] done
[39306|zebra] done
[40639|watchfrr] done
Graceful restart configuration changed, reset all peers to take effect
[39801|bgpd] done
root@router01:~ # netstat -rn -f inet | grep 131
192.168.131.1 link#13 UH lo1
192.168.131.2 192.168.255.4 UGH1 vtnet0
It seems like maybe there's something happening on boot when it first tries to install the route such that it fails and then nothing ever changes so it doesn't try again? Not sure why my other routes seem fine. Is it because they're eBGP? Because the neightbors don't seem to connect as quickly? Because they're different software?
Once again thanks for the outputs.
So your configuration seems reasonable.
The prefix is advertised by router2 and received by router1.
The prefix entry in the BGP table in router1 show proper parameters (w.g doesn't show inaccessible, shows next hop router2 192.168.255.4).
Overall based on the above, It should be installed into the rib (routing table).
If I understand your setup correctly,
iBGP > Router1 (ASN 64601) - (ASN64601) Router2
eBGP > Router1 towards 192.168.128.4X
eBGP > Router2 towards 192.168.128.4X
Here in this setup, you have the route 192.168.131.2 advertised from Router2 via iBGP as well eBGP?
Does the route 192.168.131.2 from iBGP as well eBGP on router1?
-------------
So when you restart the BGP as whole the route is properly installed.
In the first post you showed when restarting BGP towards the router2 the route is installed as well.
What happens when you try to restart the eBGP peers only?
-------------
In regards your questions.
- This looks like, when the prefix is announced 1st time its being received but not actually properly installed.
- eBGP and iBGP have different behavior, they as well affect what kind of route will be installed into the routing table. Usually in case the first 6 attributes are the same between two or more prefixes from two or more neighbors, eBGP has precedence. If a route from Router2 is received over eBGP to a BGP peer that has the same ASN as router2 the "BGP Loop prevention" kicks in. But in this case the update from BGP would drop and you would not see the prefix in the BGP table.
- If a neighbor comes up sooner or later could have effect on which prefix to choose e.g to install, but you are not installing any. This is as well related to BGP Best Path Algorithm and that option is very low on the list.
- By different software you mean different FRR release or cross vendor BGP?
BGP Best Path Algorithm QuoteWeight: Highest
Local Preference: Highest
Network or Aggregate: Locally originated
AS_PATH: Shortest
Origin type: Lowest origin number
Multi-exit discriminator (MED): Lowest
eBGP over iBGP:
IGP metric: Shortest IGP path to BGP next hop
Multiple paths: Highest number of multiple path in the routing table
External paths: Oldest (whatever received first)
Router ID: Lowest
Cluster list: Minimum cluster list length
Neighbor address: Lowest
Regards,
S.