New OPNsense user, trouble with policy-based routing for dual WAN.

[zackboll]

I have been using OPNsense for about a week now, new user that recently switched over from pfsense.

In my previous pfsense setup, I used a dual WAN setup, with all of my traffic going through my primary WAN, with WAN2 as a failover.

I will list my problem at the end, below is my router configuration:

My primary WAN has an IPv4 and IPv6 address.  My secondary WAN2 has just an IPv4 address.  Under System->Gateways->Groups, I created a new IPv4_Failover group with WAN1 as tier1 and WAN2 as tier2.  I have tried adjusting the trigger level and pool options, initially, I trigger was based on packet loss and latency and pool was round robin.  Each of the 3 gateways uses an external DNS server as the IP address to monitor, listed in the next setting.

In System->Settings->General, I have 3 unique DNS servers specified, 1 IPv6 server, 2 IPv4 servers, all google DNS servers for now.  Each server is tied to a WAN gateway.  I have unchecked allow DNS server to be overridden by WAN, and checked Allow default gateway switching.

In Firewall->Rules->LAN, I modified the default rule for IPv4 to go to my IPv4_Failover Gateway group defined earlier (as opposed to default gateway).  For IPv6 traffic, I left it unmodified since I only have a single IPv6 WAN address.

I have been having trouble replicating my previous setup in OPNsense using policy based routing.  The setup appears to kind of work on some systems connected to the LAN, on other systems, I have no WAN access.  On one such problem system, I could not ping the main OPNsense router (192.168.9.1) from my workstation on the LAN (192.168.9.20). As soon as I modified the IPv4 LAN firewall rule to point at the default gateway, instead of my IPv4_Failover GW group, the 192.168.9.1 and internet became available on that machine.

For now, I lowered the priority of my primary WAN GW so that traffic would prefer it, and do not route any LAN traffic to the IPv4_Failover GW group.  If possible, I would like to figure out why my policy based routing is not working as my setup is getting a little more complicated with a 3rd ISP being installed this week.  My final goal will be to load balance some LAN traffic between WAN1 and WAN2, with WAN3 being used as a failover for WAN1 and WAN2.  With policy-based gateway routing, this seemed like it would be easy to accomplish with OPNsense.  I am hoping that I am just missing something simple from my configuration.

Thanks,
Zack

[zackboll]

After reviewing the OPNSense documentation more closely, I am pretty sure what I had missing was a firewall rule to allow DNS traffic to my OPNSense router (192.168.9.1).  I am guessing I will need to add additional rules for WebGUI and SSH access as well?

https://docs.opnsense.org/manual/how-tos/multiwan.html#step-5-add-allow-rule-for-dns-traffic

Thanks,
Zack

[viragomann]

Yes, a policy-routing rule forces all matching traffic to the stated gateway (group).

To allow access to internal services, you need an additional route without a gateway specified and put this above of the policy-routing rule.

Best practice, when using policy-routing rules for common upstream traffic is to create a network alias and add all private network ranges to it. Then use this alias in the policy-routing rule as destination with "invert" checked. So this rule is applied to non-private destinations only and has no impact on local traffic.
But anyway you need additional rules to permit local traffic.

[zackboll]

"Best practice, when using policy-routing rules for common upstream traffic is to create a network alias and add all private network ranges to it. Then use this alias in the policy-routing rule as destination with "invert" checked. So this rule is applied to non-private destinations only and has no impact on local traffic."

Thank you for the feedback, I am going to do some reading tonight and will hopefully accomplish this.

[demyers]

I am guessing I will need to add additional rules for WebGUI and SSH access as well?

There are automatically generated rules for SSH and HTTPS that should take care of these, I think.

[viragomann]

Here is an example for the alias as destination with invert.

[zackboll]

Thanks,

I just ran into another potential issue with my new fiber connection.  I just plugged it in tonight and unfortunately it only provides an IPv4 address.  My Tier 2 ISP provides both an IPv4 and IPv6 address.

I noticed that in GW groups, you cannot mix and match IPv4 WANs with IPv6 WANs which makes sense to me.  Since my Tier 1 ISP does not provide an IPv6 address, do I need to disable IPv6 from my Tier 2 ISP?  My concern is that all my IPv6 traffic will go through my Tier 2 ISP, leaving my Tier 1 ISP (with symmetric Up/Down bandwidth) under utilized.  Is there an alternative to disabling IPv6 on my router (maybe use an IPv6 to IPv4 translator)?  If the best solution is just to disable IPv6, I am fine with that.  I am currently not a Starlink customer and I don't believe that any of my IPv4 WAN addresses are using CGNAT.

Thanks,
Zack

[zackboll]

Thanks for the help, I have my 3 WAN setup with policy-based routing up and running.  I wasn't sure how to handle my tier 1 GW not having an IPv6 address, so I disabled IPv6 on my tier 2 and tier 3 GW.

Zack

[zackboll]

My current work-around for when I need IPv6 connectivity is to tunnel through a VPS that has IPv4 and IPv6 using tailscale.

It would be nice to have IPv6 available all the time for all clients on the network, but I am not sure how to accomplish this with Tier1 WAN being IPv4 only and Tier2 WAN being IPv4/IPv6.

[coffeecup25]

Here is an example for the alias as destination with invert.

Quick question as this has been unclear since someone else mentioned the invert option.

These non-public addresses are non-routable by definition, or so I have been told to believe. So, since it is supposedly impossible for them to travel the internet, why does a rule need to be created to prevent it? If they could travel, everyone with a 192.168.1.1/24 subnet would be in everyone else's 192.168.1.1/24 subnet. Since I create my own subnets, I have no need to prohibit ones that don't exist from going anywhere on my network. Basic firewall operations in every router on earth (NAT / SPI)  keeps the outside from coming in unless a port is open for them - IPS/IDS is for that.

How does that invert with non-public addresses work as it is very unclear to me what it is actually doing.

[coffeecup25]

zackboll, I'm responding to your original post. I have not read most of the rest of the thread as dual WAN is not in my network at this time.

I also recently changed over from pfSense. As this is a hobby for me, I am going from one to the other as I experiment. They are both excellent software routers.

The biggest problem I had with OPNsense initially was that everything is in a different place from pfSense. This is to be expected and normal. But it's a good chance something on your initial setup was missed. Take another look unless you have already done this and are sure they are comparable.

[viragomann]

Quick question as this has been unclear since someone else mentioned the invert option.

In this thread the point is to configure a policy-based routing rule to direct all upstream traffic to an upstream gateway.

This rule must not match traffic to any internal destination, since it forces all traffic to the gateway. Hence accessing this internal resource would be impossible.
The RFC 1918 alias with invert makes sure, that the rule only matches to outside destinations.

In my example I used it to allow some ports going out in a floating rule, which is defined on multiple internal interfaces. I don't want to allow this services to internal IPs though.

These non-public addresses are non-routable by definition, or so I have been told to believe.

Yes, this should be the case, but you cannot be sure.
Therefore it's recommended to additionally block outgoing traffic to private IPs.

Furthermore directing out traffic meant to a private destination can occur quite easily.
Imagine, you have an OpenVPN site to site with the remote network 10.120.0.0/24. OpenVPN adds the proper routes to the systems routing table.
If the VPN is up and running, 10.120.0.0/24 is routed to the remote site. All well.
However, if the VPN goes down, the routes are removed and traffic you want to send to the remote site would go out to the default gateway.
This let the outside know, which IPs you're using inside your network, which is not desirable at all. Apart of this, the packet may contain a payload, which should not go to a public destination.
So just block the traffic private subnets on WAN and you're safe.

[zackboll]

zackboll, I'm responding to your original post. I have not read most of the rest of the thread as dual WAN is not in my network at this time.

I also recently changed over from pfSense. As this is a hobby for me, I am going from one to the other as I experiment. They are both excellent software routers.

The biggest problem I had with OPNsense initially was that everything is in a different place from pfSense. This is to be expected and normal. But it's a good chance something on your initial setup was missed. Take another look unless you have already done this and are sure they are comparable.

Thanks my dual wan policy-based gateway routing is now working after following the suggestion of creating a LAN firewall rule that forwards all traffic to my gateway group that matches the invert (private addresses).

The only thing that I am not really happy about now has to do with my primary ISP being IPv4 only, so I had to downgrade to using only IPv4 on my local network.

In ideal world, what I would like to accomplish is the following with the current limitations of my tier1 ISP:
1.) IPv4 and IPv6 on local network
2.) IPv4 traffic is routed out tier1 WAN
3.) IPv6 traffic is routed out tier1 WAN via IPv4 to cloud server, where it can then be routed as IPv6 (can kind of accomplish this per client with tailscale)
4.) IPv4 and IPv6 traffic routed out tier2 WAN when tier1 WAN fails.