GRE tunnel (over WireGuard) doesn't come up after reboot

[ferryvanaesch]

Hi,

I run a couple of GRE tunnels over WireGuard VPNs. Works fine, except after a reboot. When I look at one of the tunnels after a boot, it looks like this:

root@OPNsense:~ # ifconfig gre1
gre1: flags=8011<UP,POINTOPOINT,MULTICAST> metric 0 mtu 1396
        options=80000<LINKSTATE>
        inet 10.1.11.6 --> 10.1.11.5 netmask 0xfffffffc
        inet6 fe80::20c:29ff:fea3:3bc8%gre1 prefixlen 64 tentative scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: gre

i.e. Link not set up. When, in the OPNsense interface, I go to this GRE tunnel, edit it, hit Save and Apply (without changing any settings), things start working and ifconfig shows me the below:

root@OPNsense:~ # ifconfig gre1
gre1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1396
        options=80000<LINKSTATE>
        tunnel inet 10.1.9.2 --> 10.1.9.1
        inet 10.1.11.6 --> 10.1.11.5 netmask 0xfffffffc
        inet6 fe80::20c:29ff:fea3:3bc8%gre1 prefixlen 64 scopeid 0xa
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: gre

I'm not entirely sure what is happening here. Could it be a timing issue, in that the GRE tunnel can't be set up until routing entries exist after the WireGuard tunnel has negotiated? If so, is there any way I can trigger things after WireGuard has established its link?

Chees,
Ferry.

PS, further to this, some dmesg output from the boot:

Code Select Expand

Trying to mount root from ufs:/dev/gpt/rootfs [rw,noatime]...
random: unblocking device.
VMware memory control driver initialized
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
em2: promiscuous mode enabled
carp: 6@em2: INIT -> BACKUP (initialization complete)
em3: promiscuous mode enabled
carp: 7@em3: INIT -> BACKUP (initialization complete)
em1: promiscuous mode enabled
carp: 5@em1: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em1: 3
ifa_maintain_loopback_route: deletion failed for interface em1: 3
ifa_maintain_loopback_route: deletion failed for interface em1: 3
carp: 5@em1: BACKUP -> INIT (hardware interface up)
carp: 5@em1: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em2: 3
ifa_maintain_loopback_route: deletion failed for interface em2: 3
ifa_maintain_loopback_route: deletion failed for interface em2: 3
carp: 6@em2: BACKUP -> INIT (hardware interface up)
carp: 6@em2: INIT -> BACKUP (initialization complete)
ifa_maintain_loopback_route: deletion failed for interface em3: 3
ifa_maintain_loopback_route: deletion failed for interface em3: 3
ifa_maintain_loopback_route: deletion failed for interface em3: 3
carp: 7@em3: BACKUP -> INIT (hardware interface up)
carp: 7@em3: INIT -> BACKUP (initialization complete)
gre0: link state changed to DOWN
gre1: link state changed to DOWN
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
tun0: link state changed to UP
tun0: changing name to 'wg0'
tun1: link state changed to UP
tun1: changing name to 'wg1'
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
carp: 7@em3: BACKUP -> MASTER (preempting a slower master)
carp: 6@em2: BACKUP -> MASTER (preempting a slower master)
carp: 5@em1: BACKUP -> MASTER (preempting a slower master)
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
Showing both gre0 and gre1 being down.

Then, when I edit, save and apply both the gre tunnels, the following appears:

Code Select Expand

gre0: link state changed to DOWN
gre0: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled
gre1: link state changed to DOWN
gre1: link state changed to UP
pflog0: promiscuous mode disabled
pflog0: promiscuous mode enabled

[mimugmail]

Did you assign the wg interface?

[ferryvanaesch]

Hi,

I did yes, see attached.

[ferryvanaesch]

Made an identical setup using IPSec instead of WireGuard, and after a little while the GRE tunnel does come up automatically, from time to time. (Works after some reboots, not after others)

[mimugmail]

Hm, any messages from the console?

[ferryvanaesch]

No, not a peep.

[ferryvanaesch]

I hacked it temporarily by running a script on Gateway events, so that when the Gateway of the underlying WireGuard tunnel is up, I manually add the right ifconfig tunnel command... Not really ideal but it works.