Weird Problem with Mutliple OpenVPN Clients

[randomwalk]

Hello,

I'm new to OPNsense and have installed 18.1.6.  I'm a previous pfsense user and so far I'm liking OPNsense a lot.  But I'm encountering a weird issue with load balancing multiple OpenVPN clients and I hope you can help.

I have a VPN service that allows 5 concurrent connections.  Historically, I would connect 4 clients from the router, then load balance between the 4 gateways to increase total throughput (I have gigabit fiber so I could actually use the speed from concurrent connections).

The problem is that when I try to do this in OPNsense, I'm not getting any connections (or unstable connections).  I thought maybe I played with the settings too much in the process of getting familiar with OPNsense, so I started from scratch again (reformat, clean reinstall, then update to 18.1.6).  I went step by step to setup the various settings. 

Here is the set up.  So far, I have made minimal changes to the settings (basically, just standard things in the System section).  I have setup the WAN, LAN, OPT1 and OPT2 interfaces, which are the four ethernet ports on my router.  I have NOT made any changes to the Firewall section -- just the standard settings that allow everything through on the LAN.  The internet on the LAN port works fine.

Then I go and set up two OpenVPN servers (I run two servers, one on TCP 443 and one on UDP 80, for me to connect remotely into my home network).  I then set up three OpenVPN clients to my VPN service.  They all connect fine (and to be clear, I connect to my VPN service in such a way that the virtual addresses of the three client connections are in distinct subnets so that they would not share the same gateway).

Here is where the problem starts.  I noticed that I can get internet service on my LAN port only if I have 1 or 2 VPN clients running, but NOT if I have 3 VPN clients running.  I noticed this because I was surfing the web and it stopped when I connected the third client.  I thought it was weird, so I tried to disconnect one of them.  Sure enough, I'm able to surf on the LAN port if I have 2 clients (doesn't matter which two, I have tried all the combinations), but not if I have 3 clients. 

I don't understand why this would happen, especially since right now, I have NOT defined interfaces for the 3 VPN clients (i.e., there are no gateways associated with the three clients), and my firewall rules do NOT attempt to pass any traffic through the VPN client connections.  Also, I have not changed the Unbound settings (the outgoing network interface is the default "all").

In theory, the OpenVPN clients should not affect anything because they're just connected and sitting there, not being used.  But it seems like somehow they do affect something, but I am not sure what.  I do not see any obvious errors.

I appreciate any help you can give me.

[randomwalk]

Ok, now things are weirder.  After I posted the above question, I disabled 1 of 2 remaining OpenVPN clients (recall that I have created 3 client profiles, but previously, the LAN would only work if I have 2 enabled).  Now, the LAN would only get internet service if I have just 1 VPN client enabled.  It would no longer work if I have 2 VPN client enabled -- a few minutes ago, having 2 clients allowed the LAN to work fine.

So I'm kind of baffled.  I literally changed no other settings between my original post and my current post. 

[randomwalk]

Ok, I think I solved my own problem.  I noticed that the VPN client connections inserted entries into the route table for 0.0.0.0/1 that superseded the "default" route to my WAN.  I changed the VPN client settings to "Don't add/remove routes," which solves the issue.  =)

[randomwalk]

Ok, I continue to have issues.  I've now set up everything (I think).  The VPN client connections seem fine, the route table looks reasonable, I have created firewall rules to direct traffic to the load balancing VPN gateway group, I have set up outbound NAT rules (I think this is correct, but am not sure as I never completely understood the NAT page).  I also set the Unbound outgoing network to be the VPN client interfaces.

Using the above set up, I run IP Leak tests and it correctly shows my VPN's IP, and shows that I use DNS servers equal to the VPN's IP.  So everything looks like it's working correct. 

Except, the connections are not consistent.  Sometimes the internet works very quickly.  Sometimes connections time out in the browser.  I don't see any pattern to when connections would time out, nor do I see anything in the logs that would indicate a problem. 

What would cause the set up to work sometimes, but not other times?  The working / not working changes from minute to minute, back and forth. 

[randomwalk]

I think I have narrowed down the issues to either a problem with my NAT settings or firewall rules.  Attached are my NAT and LAN firewall settings.  Unbound is set to use AirVPN1-3 as the outgoing network interface.  If I have disable the firewall rule that redirects LAN traffic to VPN_Gateways, then IP Leaks test will show my real IP, and that my DNS servers are the AirVPN exit nodes.  This indicates that the VPN connections are fine since the DNS queries are going out on the VPN interfaces as expected.

When I turn on the firewall rule that redirects the LAN traffic to the VPN_Gateways, connections will time out.  I have tried various settings for that rule (e.g., changing the destination to be "any," or different versions of the private subnets), but nothing seems to work.  It seems to me that there may be some interaction between the firewall rules and Unbound DNS resolution.  It seems like without the firewall redirect rule, the DNS works fine.  But with the firewall redirect rule, DNS is not resolving?

I would really appreciate any help on these settings.  I want to redirect everything on LAN (except a few IP's that is in the Alias Bypass_VPN) to the VPN_Gateways.