Messages

Thank you for your time looking through this. I agree, it should be installed in the routing table :)

I do have the route advertised via eBGP and iBGP, but the eBGP neighbors are just metallb pods in my kubernetes cluster.

Does the route 192.168.131.2 from iBGP as well eBGP on router1?

I think you accidentally a word here, so I'm not exactly sure what you're asking but I think you want to know if the route arrives at router01 from ebgp and ibgp and the answer is no. The metallb neighbors only advertise LoadBalancerIPs for services available on their nodes.

Code Select Expand

router01.example.com# show ip bgp neighbors 192.168.128.46 received-routes
BGP table version is 36, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *> 192.168.255.8/32 192.168.128.46           0             0 64641 i
 *> 192.168.255.18/32
                    192.168.128.46           0             0 64641 i
 *> 192.168.255.23/32
                    192.168.128.46           0             0 64641 i

Total number of prefixes 3

If I reset all the eBGP connections nothing changes:

Code Select Expand

router01.example.com# clear ip bgp 192.168.128.41
router01.example.com# clear ip bgp 192.168.128.42
router01.example.com# clear ip bgp 192.168.128.43
router01.example.com# clear ip bgp 192.168.128.44
router01.example.com# clear ip bgp 192.168.128.45
router01.example.com# clear ip bgp 192.168.128.46
router01.example.com# show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 192.168.255.3, local AS number 64601 VRF default vrf-id 0
BGP table version 85
RIB entries 25, using 3200 bytes of memory
Peers 7, using 116 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
192.168.128.41  4      64641        16        40       85    0    0 00:00:20            6       13 N/A
192.168.128.42  4      64641        16        35       85    0    0 00:00:04            4       13 N/A
192.168.128.43  4      64641        16        37       85    0    0 00:00:03            3       13 N/A
192.168.128.44  4      64641        16        34       85    0    0 00:00:02            4       13 N/A
192.168.128.45  4      64641        16        37       85    0    0 00:00:01            6       13 N/A
192.168.128.46  4      64641        14        33        0    0    0 00:00:00            0        0 N/A
192.168.255.4   4      64601        17        26       85    0    0 00:09:00           12       12 FRRouting/10.4.1

Total number of neighbors 7
router01.example.com#
root@router01:~ # netstat -rn -f inet| grep 131
192.168.131.1      link#13            UH              lo1


Code Select Expand

root@router01:~ # netstat -rn| grep 131
192.168.131.1      link#13            UH              lo1
...
router01.example.com# show ip bgp 192.168.131.2
BGP routing table entry for 192.168.131.2/32, version 2
Paths: (1 available, best #1, table default)
  Advertised to peers:
  192.168.128.41 192.168.128.42 192.168.128.43 192.168.128.44 192.168.128.45 192.168.128.46
  Local
    192.168.255.4 (metric 1) from 192.168.255.4 (192.168.255.4)
      Origin IGP, metric 0, localpref 100, valid, internal, best (First path received)
      Last update: Thu Oct 16 19:10:36 2025


Code Select Expand

root@router01:~ # netstat -rn -f inet| grep 131
192.168.131.1      link#13            UH              lo1
root@router01:~ # service frr restart bgpd
Stopping bgpd.
Waiting for PIDS: 62018.
Starting bgpd.
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 3: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 4: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 5: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/17 19:54:52 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 6: SO_RCVBUF set to 8388608 (requested 16777216)
[39208|mgmtd] sending configuration
[39306|zebra] sending configuration
[39801|bgpd] sending configuration
Waiting for children to finish applying config...
[40639|watchfrr] sending configuration
[39208|mgmtd] done
[39306|zebra] done
[40639|watchfrr] done
Graceful restart configuration changed, reset all peers to take effect
[39801|bgpd] done
root@router01:~ # netstat -rn -f inet | grep 131
192.168.131.1      link#13            UH              lo1
192.168.131.2      192.168.255.4      UGH1         vtnet0

It seems like maybe there's something happening on boot when it first tries to install the route such that it fails and then nothing ever changes so it doesn't try again? Not sure why my other routes seem fine. Is it because they're eBGP? Because the neightbors don't seem to connect as quickly? Because they're different software?

I just checked and it happens when I reboot router 1 (which just so happens to be the last thing I do when I upgrade both of them which is why I noticed it in that context).

Code Select Expand

root@router01:~ # w
 7:14PM  up 2 mins, 1 user, load averages: 0.13, 0.17, 0.08
USER       TTY      FROM              LOGIN@  IDLE WHAT
me         pts/0    192.168.128.147   7:10PM     - w
root@router01:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 1.2.3.4
        fib: 0
  interface: vtnet3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
root@router01:~ # vtysh
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 3: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 4: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 5: SO_RCVBUF set to 8388608 (requested 16777216)
2025/10/16 19:12:50 [H0DHT-S9KF2][EC 100663299] setsockopt_so_recvbuf: fd 6: SO_RCVBUF set to 8388608 (requested 16777216)

Hello, this is FRRouting (version 10.4.1).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
  Known via "bgp", distance 200, metric 0, best
  Last update 00:02:30 ago
  * 192.168.255.4, via vtnet0, weight 1

router01.example.com# show ip bgp neighbors 192.168.255.4 advertised-routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.131.1/32 0.0.0.0                  0    100  32768 i
 *>  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *>  192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 *>  192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 *>  192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Total number of prefixes 12
router01.example.com# show ip bgp neighbors 192.168.255.4 routes
BGP table version is 39, local router ID is 192.168.255.3, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>i 192.168.131.2/32 192.168.255.4            0    100      0 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Displayed 12 routes and 39 total paths
router02 is advertising of course:

Code Select Expand

router02.example.com# show ip bgp neighbors 192.168.255.3 advertised-routes
BGP table version is 47, local router ID is 192.168.255.4, vrf id 0
Default local pref 100, local AS 64601
Status codes:  s suppressed, d damped, h history, u unsorted, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  192.168.131.2/32 0.0.0.0                  0    100  32768 i
 *>  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *=  192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 * i 192.168.255.8/32 192.168.128.46           0    100      0 64641 i
 *>  192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 * i 192.168.255.16/32
                    192.168.128.44           0    100      0 64641 i
 *>  192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.17/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.18/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.19/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.20/32
                    192.168.128.45           0    100      0 64641 i
 *>  192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.21/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *=  192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 * i 192.168.255.23/32
                    192.168.128.46           0    100      0 64641 i
 *>  192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 * i 192.168.255.24/32
                    192.168.128.41           0    100      0 64641 i
 *>  192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 * i 192.168.255.25/32
                    192.168.128.42           0    100      0 64641 i
 *>  192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i
 * i 192.168.255.26/32
                    192.168.128.45           0    100      0 64641 i

Total number of prefixes 12

router01 config:

Code Select Expand

router01.example.com# show run
Building configuration...

Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router01.example.com
log syslog notifications
!
router bgp 64601
 bgp router-id 192.168.255.3
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart
 neighbor 192.168.128.41 remote-as 64641
 neighbor 192.168.128.41 update-source vtnet1
 neighbor 192.168.128.42 remote-as 64641
 neighbor 192.168.128.42 update-source vtnet1
 neighbor 192.168.128.43 remote-as 64641
 neighbor 192.168.128.43 update-source vtnet1
 neighbor 192.168.128.44 remote-as 64641
 neighbor 192.168.128.44 update-source vtnet1
 neighbor 192.168.128.45 remote-as 64641
 neighbor 192.168.128.45 update-source vtnet1
 neighbor 192.168.128.46 remote-as 64641
 neighbor 192.168.128.46 update-source vtnet1
 neighbor 192.168.255.4 remote-as 64601
 neighbor 192.168.255.4 update-source vtnet0
 !
 address-family ipv4 unicast
  network 192.168.131.1/32
  neighbor 192.168.128.41 activate
  neighbor 192.168.128.41 soft-reconfiguration inbound
  neighbor 192.168.128.42 activate
  neighbor 192.168.128.42 soft-reconfiguration inbound
  neighbor 192.168.128.43 activate
  neighbor 192.168.128.43 soft-reconfiguration inbound
  neighbor 192.168.128.44 activate
  neighbor 192.168.128.44 soft-reconfiguration inbound
  neighbor 192.168.128.45 activate
  neighbor 192.168.128.45 soft-reconfiguration inbound
  neighbor 192.168.128.46 activate
  neighbor 192.168.128.46 soft-reconfiguration inbound
  neighbor 192.168.255.4 activate
  neighbor 192.168.255.4 soft-reconfiguration inbound
 exit-address-family
exit
!
end
router02 config:

Code Select Expand

router02.example.com# show run
Building configuration...

Current configuration:
!
frr version 10.4.1
frr defaults traditional
hostname router02.example.com
log syslog notifications
!
router bgp 64601
 bgp router-id 192.168.255.4
 no bgp ebgp-requires-policy
 no bgp default ipv4-unicast
 bgp graceful-restart
 neighbor 192.168.128.41 remote-as 64641
 neighbor 192.168.128.41 update-source vtnet1
 neighbor 192.168.128.42 remote-as 64641
 neighbor 192.168.128.42 update-source vtnet1
 neighbor 192.168.128.43 remote-as 64641
 neighbor 192.168.128.43 update-source vtnet1
 neighbor 192.168.128.44 remote-as 64641
 neighbor 192.168.128.44 update-source vtnet1
 neighbor 192.168.128.45 remote-as 64641
 neighbor 192.168.128.45 update-source vtnet1
 neighbor 192.168.128.46 remote-as 64641
 neighbor 192.168.128.46 update-source vtnet1
 neighbor 192.168.255.3 remote-as 64601
 neighbor 192.168.255.3 update-source vtnet0
 !
 address-family ipv4 unicast
  network 192.168.131.2/32
  neighbor 192.168.128.41 activate
  neighbor 192.168.128.41 soft-reconfiguration inbound
  neighbor 192.168.128.42 activate
  neighbor 192.168.128.42 soft-reconfiguration inbound
  neighbor 192.168.128.43 activate
  neighbor 192.168.128.43 soft-reconfiguration inbound
  neighbor 192.168.128.44 activate
  neighbor 192.168.128.44 soft-reconfiguration inbound
  neighbor 192.168.128.45 activate
  neighbor 192.168.128.45 soft-reconfiguration inbound
  neighbor 192.168.128.46 activate
  neighbor 192.168.128.46 soft-reconfiguration inbound
  neighbor 192.168.255.3 activate
  neighbor 192.168.255.3 soft-reconfiguration inbound
 exit-address-family
exit
!
end


So the originator of that route/IP is the Router2 and you advertise it over BGP to Router1?
You say you have problem to install it into the Routing table, meaning it never installs there or it takes time?

Yes, that IP is assigned to a loopback device on router2 and it advertises the /32 over BGP to router1.

If I do my standard upgrade procedure, which is to upgrade router2, reboot router2, upgrade router1, reboot router1, the route never seems to get installed. Last time I waited about 6 weeks before poking it :)

I'm not sure of the specifics, but by saying that it's in the FRR "routing table" I just meant that (as shown above) I could use "show ip route" in vtysh and it would tell me that the best route is what I expect, unlike using "route -n get" from bash.

When I reboot my secondary router and then my primary router, the primary router seems to have some issue installing the route? I'm specifically trying to get to a loopback ip on the secondary router: 192.168.131.2.

Code Select Expand

root@router01:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 0.0.0.0
       mask: 0.0.0.0
    gateway: 1.2.3.4
        fib: 0
  interface: vtnet3
      flags: <UP,GATEWAY,DONE,STATIC>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0
 
router01.example.com# show ip route 192.168.131.2
Routing entry for 192.168.131.2/32
  Known via "bgp", distance 200, metric 0, best
  Last update 00:05:52 ago
  * 192.168.255.4, via vtnet0, weight 1

So, it's in FRR's routing table but not the OS routing table? Let's reset the bgp session:

Code Select Expand

router01.example.com# clear ip bgp 192.168.255.4
router01.example.com#

root@router02:~ # route -n get 192.168.131.2
   route to: 192.168.131.2
destination: 192.168.131.2
    gateway: 192.168.255.4
        fib: 0
  interface: vtnet0
      flags: <UP,GATEWAY,HOST,DONE,PROTO1>
 recvpipe  sendpipe  ssthresh  rtt,msec    mtu        weight    expire
       0         0         0         0      1500         1         0

Any thoughts on why this might be the case?

Hi,

I reimaged a computer and now Kea won't give it the ip address that I have reserved for it:

WARN [kea-dhcp4.alloc-engine.0x220add017400] ALLOC_ENGINE_V4_DISCOVER_ADDRESS_CONFLICT [hwtype=1 dc:a6:32:e8:aa:aa], cid=[ff:f8:ce:ff:a1:00:aa:00:00:ab:ff:26:aa:ff:4d:aa:c3:ff:39], tid=0xc87604e6: conflicting reservation for address 192.168.1.41 with existing lease Address: 192.168.1.41

Perhaps because the cid has changed? How do I get this device to pick up the address I have reserved for it?

I've seen people editing the kea-leases4.csv, is there a better way to do this? Is there a recommended way to this for an HA pair?

EDIT: You shouldn't read this, I don't know what happened but I needed to go to System > Firmware > Plugins and install os-frr

I've got an HA Pair of OPNsense vms and I'm pretty sure this used to be working, but now the routing tab (the thing on the left side of the screen below "Firewall" and above "VPN") has disappeared on the backup CARP node. Where did it go? What did I do?

I can run vtysh on my primary node, but not the backup:

```
root@router03:~ # vtysh
vtysh: Command not found.
```

I had both of them talking bgp to each other and my metallb nodes... I upgraded the HA Sync compatibility to 24.7+ and I think FRR was in the list and I synced it before I noticed. I've since removed and rebooted, but it hasn't come back. (In the "Services to synchronize" I've got everything selected but Auth Servers, Backup, FRR, Netsnmp, and Users and Groups).

I've got "Enable CARP Failover" disabled since I want routing running on both my nodes. If I enter "Persistent maintenance mode" on my master and verify failover the tab still doesn't show up.

Any ideas? I'm an OPNsense beginner, so I may have missed something basic.

I recently stood up an OPNsense VM to test it out as a replacement for my increasingly flaky EdgeRouter Lite. My thought was that I could switch to the OPNsense router but keep the EdgeRouter as a backup. I'd set up VRRP and then whenever I was doing maintenance on the OPNsense hypervisor the gateway IPs could just failover and I would retain some basic network functionality.

Unfortunately it looks like VRRP is not available on OPNsense and CARP is not available on EdgeOS. Are there other solutions I could consider? It doesn't seem like CARP has wide availability outside of FreeBSD.