Multi WAN load balancing vlan - Traffic goes always out through default IF

[dash]

Hi,

OpnSense is running in a VM (kvm) under Debian/bookworm. Both WAN are coming in a switch which mark them as VLAN1 (default) for ISP#1 VLAN1002 for ISP#2, the Debian host has interfaces configured in each VLAN and one for the whole traffic. This setup is working since years with Sophos UTM9.

I installed OpnSense v26.1.2 on the same host using same interfaces and VLANs to replace  Sophos in the future. At this time, only using DNAT and Rules (new), outgoing traffic is OK, ipv4 as well as ipv6.

I followed the multi Wan doc for load balancing. Speed being not identical, I gave different priority in System => Gateway => configuration, 250 for the power full ISP#1, 254 for the other one ISP#2. I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway. Default route was automatically setted on ISP#1 on first configuration.

ISP#2 brings an ipv6/48 network, no ipv6 on ISP#1. Both have a public ipv4 address. 

Problem: from an external server I try to connect to a machine in the LAN using ssh. It works with ipv4/ISP#1 ipv6/ISP#2 but not  ipv4/ISP#2. Using tcpdump in OpnSense console, I see the the outgoing traffic from the LAN machine is going out through ISP#1 and not ISP#2 from where the traffic came in. I also tried by giving the same priority in GW configuration, no changes.

Did I miss something knowing that sticky connection is set?

--
Daniel

[dash]

No one? Am I alone with this problem?

[TheSHAD0W]

This may be related to an issue with upgraded setups and multi-WAN in general. https://github.com/opnsense/core/issues/9702
There may be a setting hiding in the config somewhere that's causing the issue; multi-WAN with a fresh install appears to work properly. Don't know if you want to start over and reconfigure everything.

[OPNenthu]

I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway.

I suspect the 'out' rule might be at least one culprit and I don't know why it's needed.

Try:

Interface: LAN
Direction: IN
Source: LAN net
Destination: !LAN net (or whatever 'internet' means on your network)
Gateway: <YOUR_LB_GROUP>

That works for my LB group (I balance two VPN gateways), but I only use them for internet access from my local network.   A LAN rule like this for both IP protocols would take care of load balancing for outbound traffic originating locally.

For ingress, the packets would first enter the WAN firewall interface and get filtered there, then forwarded.   I don't think LAN rules come into play for your external SSH connection if I'm not mistaken, so that LAN 'out' rule wouldn't do what you want.  That's for blocking outbound traffic that originated from LAN (like if you have some internal IPs that should not be allowed out from LAN).

You might be getting into issues with Force Gateway https://docs.opnsense.org/manual/firewall_settings.html#disable-force-gateway or might need to configure policy routing on WAN rules.  Out of my depth though... I haven't tried this.

What seems clear from your description is that it's choosing the default route (your ISP#1) for the IPv4 return traffic, which is the default behavior. 

EDIT: I'm not confident about this part.  Some sources say that the default behavior is to use the same gateway that the packet arrived in on because the gateway is pinned in the state that was created on WAN, but only if 'reply-to' is not disabled (which is the default).

[dash]

Try:

Interface: LAN
Direction: IN
Source: LAN net
Destination: !LAN net (or whatever 'internet' means on your network)
Gateway: <YOUR_LB_GROUP>

No changes.

As stated in another thread, if default route is the vlan1002 interface defined as WAN -tested on a fresh install-, I can't connect anymore from  WAN1 which is vlan1, packets coming out to $IF vlan1002 with the source IP of $IF vlan1. This means that in both csaes default route has always preference of reply-to statement.

I'm surprised that I'm the only one facing this problem.

Thanks for your help

[OPNenthu]

Ok, looked at this again and noticed a few more details.

For debugging I think we can forget about the VLAN tags for your ISP links.  That topology is external to OPNsense and anyway doesn't matter for routing decisions, as long as you have set the VLAN devices correctly in OPNsense for those interfaces.

Coming to the interfaces, I assume you have WAN1 and WAN2 (the name doesn't matter) and you have a LAN.  WAN1 has an IPv4 gateway only.  WAN2 has both an IPv4 and an IPv6 gateway.

Obviously you can only load balance the IPv4 gateways, so you have put those into a group together on Tier 1.  You have set the pool options on the gateway group to either "Default" or "Round Robin with Sticky Address" (they are the same), but Sticky is needed in order to prevent asymmetric routing issues.

Now on your LAN interface rules, you have adjusted your IPv4 rule so that the "Gateway" option uses the LB group.  Your IPv6 rule still uses the default gateway.

The global setting "Firewall->Settings->Advanced->Disable force gateway" is not checked, so this option is enabled globally.

The global setting "Firewall->Settings->Advanced->Disable reply-to" is not checked, so this is also enabled globally.

In your LAN rules, the advanced mode setting "Disable reply-to" is not checked.

In your LAN rules, the advanced mode setting "Reply-to" is not filled (set to None).

Is this correct so far?  This should take care of the load balancing for connections from the inside->out (LAN hosts to internet).

---

Where you are having a problem is with connections from the outside->in (public IP to LAN hosts) when the dest IPv4 is from ISP#2.

I think you will need 3 DNAT rules in order to make this work, but the key thing here is that you should not specify gateway groups anywhere.  For incoming connections, the path is always through the gateway associated with the respective WAN interface so load balancing is out of the picture.

Also, you should not change any of the reply-to or gateway options in the rules.  Leave them default so that OPNsense will automatically track the correct gateway for sending replies.  It's supposed to do this.

DNAT rule #1: Forward SSH from ISP#1 (IPv4)

- Interface: WAN1
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN1 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)


DNAT rule #2: Forward SSH from ISP#2 (IPv4)

- Interface: WAN2
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

DNAT rule #3: Forward SSH from ISP#2 (IPv6)

- Interface: WAN2
- Version: IPv6
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

You can set the firewall rule option to 'Pass' for each of the DNAT rules for simplicity, or you can register or create them manually.  Up to you.

I think this will do it.

[OPNenthu]

And also, make sure to update OPNsense to the latest version because there was a fix in 26.1.2 for the automatic 'reply-to' on rules.  Best to do this before creating any rules.

Maybe you can also combine DNAT rules 2 and 3 if you wish.

[dash]

You have set the pool options on the gateway group to either "Default" or "Round Robin with Sticky Address" (they are the same), but Sticky is needed in order to prevent asymmetric routing issues.

Was default, change it to RR+sticky

Now on your LAN interface rules, you have adjusted your IPv4 rule so that the "Gateway" option uses the LB group.  Your IPv6 rule still uses the default gateway.

IPv6 is working fine, I  have a rule which authorized outgoing traffic.

The global setting "Firewall->Settings->Advanced->Disable force gateway" is not checked, so this option is enabled globally.

The global setting "Firewall->Settings->Advanced->Disable reply-to" is not checked, so this is also enabled globally.

In your LAN rules, the advanced mode setting "Disable reply-to" is not checked.

In your LAN rules, the advanced mode setting "Reply-to" is not filled (set to None).

The two global settings are those you describe. Concerning the LAN rules, I don't find those advanced mode settings. FYI I'm using the Rules[new]

Is this correct so far?  This should take care of the load balancing for connections from the inside->out (LAN hosts to internet).

Which is already working

I think you will need 3 DNAT rules in order to make this work, but the key thing here is that you should not specify gateway groups anywhere.  For incoming connections, the path is always through the gateway associated with the public source IP (the respective WAN interface), so load balancing is out of the picture.

Also, you should not change any of the reply-to or gateway options in the rules.  Leave them default so that OPNsense will automatically track the correct gateway for sending replies.

DNAT rule #1: Forward SSH from ISP#1 (IPv4)

- Interface: WAN1
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN1 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)


DNAT rule #2: Forward SSH from ISP#2 (IPv4)

- Interface: WAN2
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

DNAT rule #3: Forward SSH from ISP#2 (IPv6)

- Interface: WAN2
- Version: IPv6
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

You can set the firewall rule option to 'Pass' for each of the DNAT rules for simplicity, or you can register or create them manually.  Up to you.

For IPv6 I already have a working ssh rule.

I DNAT rule #1 and #2 ,no changes. Output of tcpdump -ni vtnet0_vlan1 host xxx.yyy.252.179 (IP address of WAN2)

16:04:35.632884 IP xxx.yyy.252.179.22 > zzz.aaa.69.107.54242: Flags [S.], seq 3304122593, ack 2683691083, win 64800, options [mss 1452,sackOK,TS val 1780854137 ecr 3771491428,nop,wscale 7], length 0

Thanks for your help

[dash]

And also, make sure to update OPNsense to the latest version because there was a fix in 26.1.2

26.1.4 here

[OPNenthu]

I made a few edits as you were quoting me mostly to correct typos (e.g. source IP->dest IP), but nothing important.

This could be the issue that @TheSHADOW linked to then.  I'm sorry, I cannot test it because I have only one WAN.

[dash]

Already jump in the 9702 issue, no really helpful feedback :(

Many thanks for your help.

[OPNenthu]

I still have a feeling that the culprit is in the configuration, because multi-WAN is something that if it breaks there would be a lot of escalations from business users.

What are your Outbound NAT rules? Automatic?

Concerning the LAN rules, I don't find those advanced mode settings. FYI I'm using the Rules[new]

I am using Rules [new] also.  They are there:

You cannot view this attachment.

You need to switch on the advanced mode toggle.

But these are LAN-side, so they won't help with your issue in this case.

[OPNenthu]

Already jump in the 9702 issue, no really helpful feedback :(

So you tried the suggestion there to create manual firewall rules for the NAT rules with 'reply-to' set to the respective gateway, and this worked?

[dash]

So you tried the suggestion there to create manual firewall rules for the NAT rules with 'reply-to' set to the respective gateway, and this worked?

No, didn't work. I found options in advanced, values are those you told.

I agree with you that it's a misconfiguration or issue as I have no problem with Sophos UTM9 (linux) with the same underlying parameters (interfaces as well as Vlans).

[OPNenthu]

I'm reaching now, but did you reset the state table after making the changes?  Just in case.

Established TCP can take 24h to expire, so that could maybe be short-circuiting our efforts.