LAGG+Openvpn, LAGG not coming up after boot and after ovpn disconnect

[DoubleJ]

After reboot the openvpn connection comes up, but the LAGG -in which the openvpn connections are- does not come up after reboot.

The same problem also occurs when the openvpn connections were disconnected due to bad internet connection. After the openvpn have reconnected, it seems the LAGG doesn't notice this.

This was a bug previously reported in pfsense: https://redmine.pfsense.org/issues/4231
It seems OPNsense (latest dev release) still has this problem?

[franco]

LAGG has some breakage history for both projects, we've amended a few things in the meantime, but I suspect this may be another one of those uncharted territory kind of bugs. No excuse here though, bugs need fixing.

Can you help describe this a bit more so we can make a proper problem report? What I need is a step by step guide to reliably reproduce the issue (setup guide).

[DoubleJ]

Reproduce:

1) Create a site 2 site connection with openvpn (openvpn client in the LAGG will be the test environment); I've created the connection with pre-shared key, udp or tcp (doesn't matter), tap interface, rest is default settings, no ip address info in the openvpn settings. one openvpn connection is enough, behavior with one or multiple is the same. The connection should be up and running, but no pinging, since no ip addresses assigned.
2) create lagg interfaces on both sides. Assign the openvpn connection to the lagg interface (do this for both sides), and choose FAILOVER or ROUNDROBIN (doesn't matter which of the two you choose).
3) Now go to the newly created LAGG interface (do this for both sides again) and assign an ip address in the same subnet (I used 10.0.0.1/24 and 10.0.0.2/24). On the same page you can add and need to add the gateway to the other side of the openvpn tunnel (again do this for both side).

If the firewall rules are set to allow everything, you should be able to ping the tunnel.

Test Case 1: After reboot LAGG interface doesnt come up.
- Reboot the opnsense router with the openvpn client. After reboot the openvpn client is connected, but the LAGG interface is down, so tunnel is not useable. It seems that the LAGG interface is going up before the VPN tunnel is connected, therefore the LAGG interface goes into down mode, and it also seems not to poll the openvpn connection with intervals. (I forgot whether the same behavior was also on the server side, but one can test it easely if the above test environment is created).
To get it started, you have to browse to the assigments/LAGG and edit the LAGG interface. You don't need to change any of the associated interface(s), nor change the mode. The only thing you have to do is to click the save button, and then the LAGG/openvpn client combi should work.

Test Case 2: After openvpn connection has been down LAGG interface does not re-establish a good connection.
-Now in the same test scenario/environment, get the connection up and running.
-Break the openvpn connection; maybe unplug the network cable (in my case the VPN just loses connection over the internet), then reconnect. The openvpn client connects again, but the tunnel is unuseable. (not sure whether in this case the LAGG interface is down (red) or stays green; i forgot).
- to get it running again: you have to do two things: 1) same as situation after reboot: you have to browse to the assigments/LAGG and edit the LAGG interface. You don't need to change any of the associated interface(s), nor change the mode. Just click the save button. 2) navigate to system -> gateways -> all, edit the gateway associated with the LAGG interface, don't change anything, just click the save button (make sure you apply the changes). And the tunnel should be working again.

So there are work arounds, but it is all manual actions after reboot or disconnect.
I didn't test it with bridges.
Maybe the order in which the interfaces start is causing the problem. Maybe some polling mechanism should be implemented?

There's one linux distri that handles the combi multiple openvpn connections with bonding(LAGG in BSD) flawlessly: zeroshell. Maybe it is worth to have a look at it and might give you some ideas.

If any additional (test) help is needed, let me know; for now I just use openvpn without LAGG.   


I hope this helps.

[franco]

[bump]

[mitsos]

Setup:
2 node CARP cluster.
Each member is connected to 2 procurves using a failover LAGG (2 interface) connection. (that's 2xLAGGs each with 2 physical interfaces assigned).

No breakage there. LAGGs come online after reboot and respond correctly when a cable is pulled. What ever is broken, it's specific to the VPN.

[DoubleJ]

Some new info on this. I changed my setup to exclude the LAGG, and just worked with the openvpn.
I discovered that some of the symptoms also came back in this scenario.
When the VPN was disconnected and the gateway (dynamic) went down, it didn't come up again after reconnect.
So I started to play around with the advanced settings for the gateway down functionality. I increased the values (more delay in polling, more polls before marking gateway down) and it seems to be more stable (testing now for 1 day).

I will play around and test some more, then I will try the LAGG again and post the results over here. to be continued...

[franco]

Mentioning gateway monitoring has me worried, it is really not up to the task, see: https://forum.opnsense.org/index.php?topic=1359.0