Multi WAN load balancing vlan - Traffic goes always out through default IF

[dash]

Hi,

OpnSense is running in a VM (kvm) under Debian/bookworm. Both WAN are coming in a switch which mark them as VLAN1 (default) for ISP#1 VLAN1002 for ISP#2, the Debian host has interfaces configured in each VLAN and one for the whole traffic. This setup is working since years with Sophos UTM9.

I installed OpnSense v26.1.2 on the same host using same interfaces and VLANs to replace  Sophos in the future. At this time, only using DNAT and Rules (new), outgoing traffic is OK, ipv4 as well as ipv6.

I followed the multi Wan doc for load balancing. Speed being not identical, I gave different priority in System => Gateway => configuration, 250 for the power full ISP#1, 254 for the other one ISP#2. I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway. Default route was automatically setted on ISP#1 on first configuration.

ISP#2 brings an ipv6/48 network, no ipv6 on ISP#1. Both have a public ipv4 address. 

Problem: from an external server I try to connect to a machine in the LAN using ssh. It works with ipv4/ISP#1 ipv6/ISP#2 but not  ipv4/ISP#2. Using tcpdump in OpnSense console, I see the the outgoing traffic from the LAN machine is going out through ISP#1 and not ISP#2 from where the traffic came in. I also tried by giving the same priority in GW configuration, no changes.

Did I miss something knowing that sticky connection is set?

--
Daniel

[dash]

No one? Am I alone with this problem?

[TheSHAD0W]

This may be related to an issue with upgraded setups and multi-WAN in general. https://github.com/opnsense/core/issues/9702
There may be a setting hiding in the config somewhere that's causing the issue; multi-WAN with a fresh install appears to work properly. Don't know if you want to start over and reconfigure everything.

[OPNenthu]

I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway.

I suspect the 'out' rule might be at least one culprit and I don't know why it's needed.

Try:

Interface: LAN
Direction: IN
Source: LAN net
Destination: !LAN net (or whatever 'internet' means on your network)
Gateway: <YOUR_LB_GROUP>

That works for my LB group (I balance two VPN gateways), but I only use them for internet access from my local network.   A LAN rule like this for both IP protocols would take care of load balancing for outbound traffic originating locally.

For ingress, the packets would first enter the WAN firewall interface and get filtered there, then forwarded.   I don't think LAN rules come into play for your external SSH connection if I'm not mistaken, so that LAN 'out' rule wouldn't do what you want.  That's for blocking outbound traffic that originated from LAN (like if you have some internal IPs that should not be allowed out from LAN).

You might be getting into issues with Force Gateway https://docs.opnsense.org/manual/firewall_settings.html#disable-force-gateway or might need to configure policy routing on WAN rules.  Out of my depth though... I haven't tried this.

What seems clear from your description is that it's choosing the default route (your ISP#1) for the IPv4 return traffic, which is the default behavior. 

EDIT: I'm not confident about this part.  Some sources say that the default behavior is to use the same gateway that the packet arrived in on because the gateway is pinned in the state that was created on WAN, but only if 'reply-to' is not disabled (which is the default).