Messages

I recently got a second ISP connection for a backup link. It is a metered connection so I only want to use it in the event of an outage for my main ISP and only for a few of my VLANs. I was following the OPNsense Multi WAN Documentation

• I created a new interface for new new ISP. Primary is WAN_FIBER, secondary is WAN_CABLE.
• I added the new interface as a gateway.
• I enabled gateway switching. (unbound is my DNS)
• I enabled monitoring for both my WAN interfaces. (8.8.8.8 primary and 8.8.4.4 secondary)
• I added a DNS server (same as monitoring IPs) for each interface in the general settings.
• I created a new gateway group (WAN_FAILOVER). WAN_FIBER=Tier 1 and WAN_CABLE=Tier 2.
• I added a new firewall rule for DNS for each interface I want to fail over.
• I updated the inbound firewall rule for the interfaces I want to fail over.

The above seemed to work. When I disconnected the WAN_FIBER connection everything seemed to fail over to WAN_CABLE. The issue is everything failed over, not just the subnets I added the WAN_FAILOVER gateway too. The end goal is to only allow specific subnets to fail over. I have 8 VLANs and I only want to allow 2 of them to fail over (due to the metered connection).

What is the ideal way to achieve this?


EDIT:

I have also noticed that some things don't 'fail back' very well. My site-to-site WireGuard VPN didn't transition back to the Tier1 selection after it was restored.


EDIT2:

The more I mess around with this the more it feels like it is VERY complicated to allow 2 networks to fail over and 6 to not all while allowing LAN access for all 8 networks. I have to add several firewall rules to the 2 networks just to allow access back to the LAN because the way OPNsense handles the gateways.

I was hoping the failing over, would be happening at a higher level and just changing the systems default route, but it looks like it happens on a the interface level.

Its completely pointless discussing anything else until you rule out the hardware issue. The most painless way to diagnose is to download any Linux distro that has live environment like Linux Mint or Fedora. Boot into live environment (do not install) and just try both ports on the NIC by plugging known working internet connection in port 1 and then port 2. See if the link is established on both ports and check link speeds in network manager.

I was quickly arriving at the same conclusion. I will have to find a time to schedule an outage. 

The modem is an Arris TG1682P. However, I have tried connecting my laptop and a switch directly to the port and nothing worked. I know the modem supports 1000Mb, as I connected my laptop directly to it and that is what auto-negotiated.

The interface has a very generic configuration at the moment.

You cannot view this attachment.

I have a system with a 2 port Intel I225-V 2.5Gb card in it. (Interface igc0 and igc1).

I am currently using igc0 as my WAN interface to my 2.5Gb fiber service and it is working great. I recently got a second ISP and I am trying to connect it to igc1. No matter what I do I cannot get the NIC to even light up.

• I have checked the cable by using it in igc0.
• I have tried several devices and several cables, no luck.
• I know it is not a firmware/driver issue as I am currently using an interface on the same card.
• I have the interface enabled, speed set to 1000 (just to be safe).

It feels like the interface is 'administratively down'. The only other explanation is hardware failure, but I find that a bit hard to believe. I've never had one port on a multi port NIC fail personally.

Any other things I should be looking into?

I am getting miserable OpenVPN performance when I connect to my VPN provider via OPNsense compared to when I use my computer behind OPNsense.

I am using Ookla speedtest with the same settings.

Two things come to my mind immediately :
- Ookla Server speeds can vary a lot !!
- Are you connected to the same OpenVPN Server with both and are you sure that Server has the same bandwidth capacity at both times ?

My Laptop using OpenVPN:
200Mb up
240Mb down

Wired or WiFi ?

Reason I am asking is because this makes no sense to me :

OpenVPN .opvn file:

Code Select Expand

tun-mtu 1500


I would expect that value to be lower because now it's equal to Ethernet ?!
Something like 1400 or so would be better I think, but I am not a MTU expert...

And perhaps it gets automatically lowered by the OpenVPN Client Software when using a WiFi connection ?

There are of course a lot of settings in the VPN providers .ovpn file that I cannot configure in OPNsense unfortunately.

Such as ??

1. I was using the exact same VPN settings for both my laptop and OPNsense OpenVPN. The settings file that is the first post...
2. I used the exact same server for both speed test at Ookla. (I also used fast.com as a sanity check)

3. Laptop was wired, but that shouldn't really matter. It was faster than OPNsense regardless of connection method. I just ran it again via WiFi with very similar results.

4. I would agree regarding the MTU settings, however that is what I am provided from my VPN provider.

5. These are the settings that are in the provided .ovpn file that cannot be configured in OPNsense OpenVPN:
(That said, I am not familiar enough with OpenVPN to know whether they matter or not)

Code Select Expand

persist-key
persist-tun
nobind
remote-random
pull
comp-lzo no
route-method exe
route-delay 2
mssfix 1200
verb 3
sndbuf 524288
rcvbuf 524288

Maybe you can try to set the MSS value to 1200 in the interface settings, presuming that you have assigned an interface to the OpenVPN instance.

I didn't know that was an option. That helped a bit. I get get 30-40Mb down and 120Mb up, so that tells me it isn't a CPU issue, but more likely a speed test provider issue limited my download now.

I wish there was a cleaner way to add in the tuneables for OpenVPN in the new OPNsense client.

tun-mtu 1500
fragment 1300
mssfix 1200
Without those enhancements what do you get?

It won't work at all without the fragment 1300, and I cannot set MSS Fix to anything other than enabled/disabled in OPNsense.

However, if I leave TUN MTU blank and MSS Fix unchecked (defaults), I don't get anything different.

It feels like some OPNsense setting somewhere outside of OpenVPN. Like hardware offloading or something. There is no way a simple setting could cause a 90% reduction in speed - right?

I am getting miserable OpenVPN performance when I connect to my VPN provider via OPNsense compared to when I use my computer behind OPNsense. I am using Ookla speedtest with the same settings.

My Laptop using OpenVPN:
200Mb up
240Mb down

OPNsense:
5Mb up
2Mb down

Server:

• Intel i7 6700K
• 16GB Memory
• WAN NIC - Intel i225V
• LAN NIC - Intel x710-DA2

OpenVPN .opvn file:

Code Select Expand

dev tun
fast-io
persist-key
persist-tun
nobind
remote server.com

remote-random
pull
comp-lzo no
tls-client
verify-x509-name Server name-prefix
ns-cert-type server
key-direction 1
route-method exe
route-delay 2
tun-mtu 1500
fragment 1300
mssfix 1200
verb 3
cipher AES-256-GCM
auth SHA512
sndbuf 524288
rcvbuf 524288
auth-user-pass
There are of course a lot of settings in the VPN providers .ovpn file that I cannot configure in OPNsense unfortunately.

What I do have configured in OPNsense to match the config file.

• Auth
• Data cypher
• Options - route-nopull
• Options - fast-io
• TUN device MTU - 1500
• Fragment size 1300
• MSS Fix - checked

What am I missing? I understand OpenVPN isn't as performative as some other protocols, but I should be seeing much better speeds on my hardware even with its poor performance.

I got it.

I needed to set the fragment size in the openVPN configuration. Once I did that everything worked as expected.

Thank you for your support!

I cannot ping anything external. I have tried many known IPs... I am only doing IP addresses to not deal with DNS at all, so DNS resolution isn't an issue at the moment.

The only things I can ping are my local networks and the virtual IPv4 address of the openVPN tunnel.

I am just guessing at openVPN configuration. If the firewall and policies are working (which they appear to be based on the packet captures), there isn't much left. Maybe I have a bad setting so it doesn't pull in the addressing into routing table? I am not sure. All I know is traffic isn't making it back on the openVPN interface.

To investigate run a packet capture. Select all interfaces, at protocol select ICMP and state 8.8.8.8 in the host field. Start the capture and try to ping 8.8.8.8. Then view the capture (less details).
You should see the packets on the wifi interface at least.

Maybe you missed a message I edited because it was a double and I edited after. (message 5)

I did run a packet capture. I ran it on the ExpressVPN interface. I am seeing the traffic there (as expected) so that tells me my firewall rules (at least in the outbound direction) are working fine. The ping is traversing the firewall rules and making it to the ExpressVPN interface. However, I never see an echo reply on the ExpressVPN interface. So that leads me to think it is a routing issue or a openVPN config issue.

If you did enable logging of the rule, you should see the pings in the log, even if you don't get a response. But maybe the rule is not applied.
Note that rule on interface groups and floating quick rules are probed before interface rules. So if any matches to the pings it will be applied.

I don't have any group or floating rules.

Based on the above, it doesn't appear to be a rule issue. It seems to be a routing one. The packets are making it to the correct VPN interface for egress. I just don't see any ingress back on that VPN interface.

I just did a packet capture on the ExpressVPN interface.

I can see my ICMP echo requests going out to the IPv4 address of the openVPN connection. The RFC1918 address.

I don't see any replies coming back on that interface.

Does that point to an issue on the openVPN configuration?

Looks well so far. Should work at least for IPs.
So for testing just ping 1.1.1.1 or 8.8.8.8 from the concerned device.

Check if the ExpressVPN gateway state is online in System: Gateways: Configuration.

Enable the logging of firewall rule and check if it is applied in the live log, after pinging.

I have been trying to ping 8.8.8.8 and nothing.

The gateway is listed as a valid gateway.


When I look in the live firewall view and filter on the IP (192.168.20.75) I don't know how to tell if it is working. When I ping my default gateway I can see the ICMP packet pass. I cannot see the ping to 8.8.8.8 at all. How can I tell that the rule for the gateway is applied?

Interestingly I can see random 'internet' traffic passing from that host, but the host is unable to ping anything. It is almost as if it is getting filtered on the return?

I changed my outbound NAT to hybrid and added a new rule to force all traffic from that one source IP to use the openVPN interface as its gateway.

An outbound NAT rule doesn't force any traffic to anywhere. It just translates the source IP in outbound packets on an interface to any other.
For proper working you have to select the interface as translation address.

Did you add this rule to the OpenVPN interface, which you have created before?

If it's not that, please give more details on your rules.

I am not sure I follow exactly.
These are my two rules I have created:

This rule is on the network (WIRELESS net (192.168.20.0/24)) interface the device (192.168.20.75) I want to take the VPN exists on:


This rule is on the NAT > Outbound manual rules:


Those are the only 2 firewall/NAT rules I created following the youtube guide.