[Solved] GRE tunnel flapping and BGP errors every X minutes on WAN DHCP renewal

[vnxme]

Hi,

My primary ISP provides an IPv4 via DHCP with a 150 300 sec lease time (update: and a 150 sec DHCP renewal interval). Besides, I have an IPv6 provided through a GRE tunnel from a VPS. After an upgrade from 21.7 to 22.7 I am experiencing some issues with this setup.

Every time the IPv4 is renewed, the GRE tunnel goes down and then up again which is reported in console with the following annoying notifications:

Code Select Expand


gre0: link state changed to DOWN
gre0: link state changed to UP


The following 15 lines are added to my general log. Please, note some comments are duplicated (e.g. 'setting IPv6 default route to ...').

Code Select Expand


2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '2aXX:YYYY:1'
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to 2aXX:YYYY:1
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt2
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt1
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt2'
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: creating /tmp/gre0_defaultgwv6 using '2aXX:YYYY:1'
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: removing /tmp/gre0_defaultgwv6
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: setting IPv6 default route to 2aXX:YYYY:1
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv6 default gateway set to opt2
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: skipping IPv4 default route
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: IPv4 default gateway set to opt1
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: entering configure using 'opt2'
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: On (IP address: A.B.C.D) (interface: WAN4[opt1]) (real interface: vtnet1).
2022-08-01T13:43:39 Error opnsense /usr/local/etc/rc.newwanip: IPv4 renewal is starting on 'vtnet1'
2022-08-01T13:41:09 Error opnsense /usr/local/etc/rc.newwanip: ROUTING: keeping current default gateway '2aXX:YYYY:1'


What is more important to me is the following 5 error lines in BGP log:

Code Select Expand


2022-08-01T13:43:39 Error bgpd [EC 100663301] INTERFACE_STATE: Cannot find IF gre0 in VRF 0
2022-08-01T13:43:39 Error bgpd [EC 100663301] INTERFACE_ADDRESS_DEL: Cannot find IF 10 in VRF 0
2022-08-01T13:43:39 Error zebra [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
2022-08-01T13:43:39 Error zebra [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
2022-08-01T13:43:39 Error zebra [EC 100663303] vrf_if_ioctl(SIOCGIFFLAGS) failed: Device not configured
2022-08-01T13:41:09 Error bgpd [EC 100663301] INTERFACE_STATE: Cannot find IF gre0 in VRF 0


I'm pretty sure there was no GRE tunnel flapping on 21.7 update: and confirm 22.1.10_4 is also OK. Could it be a bug of 22.7 which has some underlying mechanics changed?

[franco]

Is there any 22.1 as a datapoint here?


Cheers,
Franco

[vnxme]

@franco Thank you for attention to this topic. I had some issues with 22.1 early this year and decided to skip that version. If it was of any value to you, I could have 22.1 installed and tested on the same config as well.

[franco]

Yes that would help. I don't see anything that isn't supposed to work the way you describe it but I agree that even with a 150 seconds lease time it should try to be a bit more lazy with the reloads (although personally these lease times are very disruptive in an of itself yet we can't always prevent them).


Cheers,
Franco

[vnxme]

@franco Tested 22.1.10_4 with the same config last night:

• No GRE tunnel flapping reported in console
• No entries added to the general log every 2.5 min on WAN DHCP renew
• No BGP errors

Thus, the issue seems to be specific to 22.7.

[franco]

Hey vnxme,

Thanks for testing. It looks like this is actually https://forum.opnsense.org/index.php?topic=29556.msg143071#msg143071 (patch inside)


Cheers,
Franco

[vnxme]

@franco Tested the solution you proposed in this post. After opnsense-patch 7aaa6a263b1 bb4743a7322 the issue is resolved on 22.7_4.

Will you release this patch officially?

[franco]

Yes, in 22.7.1 (likely next week).


Cheers,
Franco

[vnxme]

@franco Good to know. Thank you for your help!