Bridge interface via VXLAN between sites fail to start when rebooting OPNsense

[pelle]

I have an interface on Site A and another interface on Site B, which I want to have layer 2 connectivity in-between. I add a VXLAN between Site A and Site B. In each OPNsense I bridge the interface with the VXLAN, this way, I have stretched the L2 network between the Site A interface and Site B interface. It works fine when I have it up and running. But each time I reboot the Site A or Site B OPNsense, I have to open VXLAN or the Bridge interface, manually press 'save - > apply' (no change), to have the L2 LAN working between Site A and B interfaces.

If I add another interface on Site A to the bridge, the two interfaces on Site A have no problem to send L2 packets after I restart the OPNsense at Site A. I have no problem using ping between Site A and Site B VXLAN source and dest. IP's after a reboot.

I guess that the bridge starts before the VXLAN has created its remote connection or something like that. This 'mess up' the L2 mac forwarding table in the bridge (or the VXLAN). This problem repeats the same way every time I reboot any of the two OPNsense. Every time I save the VXLAN interface (with no change), everything starts to work right away.

I have tried to cron a "Periodic interface reset" with "vxlan0" as the parameter on each Sites OPNsense, no luck. I also ssh to the box and manually did "ifconfig vxlan0 down" and then "ifconfig vxlan0 up", but it did not work either.

I would be grateful if anyone has a workaround (a corn configuration or maybe a bash script which can be executed at startup) or a suggestion which OPNsense setting I can try to change. I have done some test with different advanced interface setting (on the VXLAN, bridge and interface), but no luck so far.

I'm running the latest OPNsense.

Many thanks for any suggestion.

Regards
- Per Håkansson

[mimugmail]

Just out of couriosity, do you encrypt the traffic between A and B? And a GUESS about your problem, have to ticked "Lock Prevention" in VXLAN interface?

[pelle]

No, I don't encrypt between A & B because both 'sites' are currently inside a single Proxmox. Right now, I only have this as a test environment. It's a single Proxmox, using 4*linux host as end-devices and 6*OPNsense to simulate a real site-to-site environment with VXLAN, Internet and everything. I use it to test out configuration to handle reboots and other problems, like Internet failure and NAT between site A & B. I try to get everything working. The goal is to solve everything before it is put in real production at real sites. But, yes, when I put the VXLAN over the actual Internet, I will for sure have to use encryption between A & B. I will test that to as soon I have got a stable VXLAN configuration.

The "Lock Prevention" doesn't make any difference. The stretched VXLAN is still not working after a reboot.

If I have 'bounced' the VXLAN interface and everything is working, it is no problem if I 'break' the Internet between OPNsense site A & site B. When the Internet starts to work again, my VXLAN also works. The only time I lose connectivity is when I reboot either of the two VXLAN OPNsense.

I found out a command I can run using the shell which solves the problem. It's "configctl service reload all". It might be a bit overkill, but it works. I did try to add the command to crontab with the @reboot. But after a reboot, somehow the line was removed from crontab.

Are there any better way to have OPNsense run a command at startup? I have tried to find a plugin to create a startup script somehow, I have tried to figure out how to use rc.d or rc.local, but I'm too bad in FreeBSD (or Linux) to figure it out. The best would, of course, be if the VXLAN could come up correctly, without me doing some 'ugly' fix running "service reload all". But having the OPNsense doing a "service reload all" just after OPNsense has started, works for me. It will only happen once every reboot, and that will not happen often . . . and the VXLAN function has already been down while OPNsense rebooted, so it does not matter if it's dead a couple of more seconds by the "service reload all" script.

Well, I will keep on trying to find a way to run the "configctl service reload all" at startup. I have tested so many OPNsense settings doing this the 'correct way', but everything I tried failed. So, my last hope is to do an 'ugly fix' by crontab, script or RC. Then, when the VXLAN survive a reboot, I will test out an IPsec/OpenVPN solution for the VXLAN traffic, even if the site is behind NAT. That should work, in theory. We will see which feature I 'hit'

All this is to replace our old solution currently in production. The OPNsense must be as stable (or better) then the current setup. It should also handle more and better encryption and handle more variation than the current solution. But I do like OPNsense, and so far, except for the VXLAN-'bug', it has worked great. I have hope.

Best Regards
- Per Håkansson

[mimugmail]

Better open a report in github/core

[pelle]

Yes, yes, yes . . .  this seems to work great!! Thanks, mimugmail!

In the /usr/local/etc/rc.syshook.d/start directory, I copy the 90-cron.. job to 93-all-service-restart and edit the file to:

   #!/bin/sh

   echo -n "Restart all services: "
   configctl service reload all

This result in a startup output as follows:

   >>> Invoking start script 'all-service-restart'
   Restart all services: OK

. . . and guess what, as soon as the "Restart all services" start to execute, even before the "OK", my VXLAN starts to work This will fix my VXLAN remote bridge 'bug' for now.

Once again, thanks a lot. This is great. I now can go on to test encrypting my site to site VXLAN traffic.

Best regards
- Per Håkansson