Messages

A big thanks to you.

Yes I did. On each modification.

So you tried the suggestion there to create manual firewall rules for the NAT rules with 'reply-to' set to the respective gateway, and this worked?

No, didn't work. I found options in advanced, values are those you told.

I agree with you that it's a misconfiguration or issue as I have no problem with Sophos UTM9 (linux) with the same underlying parameters (interfaces as well as Vlans).

Already jump in the 9702 issue, no really helpful feedback :(

Many thanks for your help.

And also, make sure to update OPNsense to the latest version because there was a fix in 26.1.2

26.1.4 here

You have set the pool options on the gateway group to either "Default" or "Round Robin with Sticky Address" (they are the same), but Sticky is needed in order to prevent asymmetric routing issues.

Was default, change it to RR+sticky

Now on your LAN interface rules, you have adjusted your IPv4 rule so that the "Gateway" option uses the LB group.  Your IPv6 rule still uses the default gateway.

IPv6 is working fine, I  have a rule which authorized outgoing traffic.

The global setting "Firewall->Settings->Advanced->Disable force gateway" is not checked, so this option is enabled globally.

The global setting "Firewall->Settings->Advanced->Disable reply-to" is not checked, so this is also enabled globally.

In your LAN rules, the advanced mode setting "Disable reply-to" is not checked.

In your LAN rules, the advanced mode setting "Reply-to" is not filled (set to None).

The two global settings are those you describe. Concerning the LAN rules, I don't find those advanced mode settings. FYI I'm using the Rules[new]

Is this correct so far?  This should take care of the load balancing for connections from the inside->out (LAN hosts to internet).

Which is already working

I think you will need 3 DNAT rules in order to make this work, but the key thing here is that you should not specify gateway groups anywhere.  For incoming connections, the path is always through the gateway associated with the public source IP (the respective WAN interface), so load balancing is out of the picture.

Also, you should not change any of the reply-to or gateway options in the rules.  Leave them default so that OPNsense will automatically track the correct gateway for sending replies.

DNAT rule #1: Forward SSH from ISP#1 (IPv4)

- Interface: WAN1
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN1 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)


DNAT rule #2: Forward SSH from ISP#2 (IPv4)

- Interface: WAN2
- Version: IPv4
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

DNAT rule #3: Forward SSH from ISP#2 (IPv6)

- Interface: WAN2
- Version: IPv6
- Protocol: TCP
- Source: any
- Source port: any
- Destination: WAN2 address
- Destination Port: 22 (ssh)
- Redirect: <ssh_host>
- Redirect port: 22 (ssh)

You can set the firewall rule option to 'Pass' for each of the DNAT rules for simplicity, or you can register or create them manually.  Up to you.

For IPv6 I already have a working ssh rule.

I DNAT rule #1 and #2 ,no changes. Output of tcpdump -ni vtnet0_vlan1 host xxx.yyy.252.179 (IP address of WAN2)

16:04:35.632884 IP xxx.yyy.252.179.22 > zzz.aaa.69.107.54242: Flags [S.], seq 3304122593, ack 2683691083, win 64800, options [mss 1452,sackOK,TS val 1780854137 ecr 3771491428,nop,wscale 7], length 0

Thanks for your help

Try:

Interface: LAN
Direction: IN
Source: LAN net
Destination: !LAN net (or whatever 'internet' means on your network)
Gateway: <YOUR_LB_GROUP>

No changes.

As stated in another thread, if default route is the vlan1002 interface defined as WAN -tested on a fresh install-, I can't connect anymore from  WAN1 which is vlan1, packets coming out to $IF vlan1002 with the source IP of $IF vlan1. This means that in both csaes default route has always preference of reply-to statement.

I'm surprised that I'm the only one facing this problem.

Thanks for your help

No one? Am I alone with this problem?

I upgraded to 26.1.3. This version correct other problems I faced but still can't connect using ssh on WAN2 with ipv4. Summarize:

. vlan1 WAN1 ipv4: all is good
. vlan1002 WAN2 ipv4 and ipv6: ipv6 OK ipv4 KO.

Problem: incoming ipv4 traffic on WAN2 is going out on WAN1 despite the fact that source IP of outgoing packets is the correct one from WAN2 where packet came in.

Unanswered question are:
. why netstat -rn4 doesn't show vtnet0_vlan1002 as shown below

Internet:
Destination        Gateway            Flags         Netif Expire
default            AAA.BBB.136.254    UGS    vtnet0_vlan1
8.8.8.8            AAA.BBB.136.254    UGHS   vtnet0_vlan1
9.9.9.9            YYY.XXX.252.177    UGHS   vtnet0_vlan1

Name              Mtu Network             Address            Ipkts Ierrs Idrop    Opkts Oerrs  Coll
vtnet0_vlan1        - AAA.BBB.136.0/24    AAA.BBB.136.1       6989     -     -        0     -     -
vtnet0_vlan1002     - YYY.XXX.252.176/28  109.237.252.179     6542     -     -        0     -     -

. why despite the fact that source IP of outgoing packets is the right one (reply-to OK) are those packets delivered to GW of WAN1

. using load balancing for local outgoing packets, should I remove default route which is GW of WAN1?

I did a fresh install from latest 26.1.2 version and then upgrade to 26.1.3, problem doesn't disappear.

Thanks for any hint

I complete informations on https://github.com/opnsense/core/issues/9702

Even with a fresh install, traffic coming from WAN2 failed to go out using the same  GW despite the fact that reply-to seems setted.

Hi,
as others we face a broken Multi Wan. WAN1 is only ipv4 and connected to an upper router in the 192.168.136.0/24 LAN with VLAN 1. WAN2 is directly connected to Internet, ipv4 and ipv6. This connection uses a VLAN 1002. Load balancing is configured for the LAN outgoing traffic, OpnSense version 26.1.2_5-amd64 is running in a kvm VM, ipv6 is  working well.

Problem is that all traffic coming in to ipv4 WAN2 is going out using WAN1 *with the ipv4 source address of WAN2*

Example from an ssh outside connection, src ipv4 being aaa.bbb.ccc.107 to dst ipv4 zzz.yyy.zzz.179:

root@guava:~ # tcpdump -ni vtnet0_vlan1 src zzz.yyy.zzz.179
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet0_vlan1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:24:46.960754 IP xxx.yyy.zzz.179.50022 > aaa.bbb.ccc.107.55504: Flags [S.], seq 114251010, ack 2702443401, win 64800, options [mss 1452,sackOK,TS val 2130106607 ecr 1545017500,nop,wscale 7], length 0

Here is netstat output:

root@guava:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.136.254    UGS    vtnet0_vlan1
8.8.4.4            xxx.yyy.zzz.177    UGHS   vtnet0_vlan1
8.8.8.8            192.168.136.254    UGHS   vtnet0_vlan1
9.9.9.9            zzz.yyy.zzz.177    UGHS   vtnet0_vlan1
10.0.0.0/16        192.168.10.254     UGS          vtnet2
10.1.58.0/24       192.168.10.254     UGS          vtnet2
10.2.67.0/24       192.168.10.254     UGS          vtnet2
10.99.98.0/24      192.168.10.254     UGS          vtnet2
10.99.99.0/24      192.168.10.254     UGS          vtnet2
xxx.yyy.zzz.176/28 link#11            U      vtnet0_vlan1
xxx.yyy.zzz.179    link#4             UHS             lo0
127.0.0.1          link#4             UH              lo0
149.112.112.112    192.168.136.254    UGHS   vtnet0_vlan1
172.31.98.0/24     192.168.10.254     UGS          vtnet2
192.168.10.0/24    link#3             U            vtnet2
192.168.10.1       link#4             UHS             lo0
192.168.12.0/24    link#13            U      vtnet0_vlan2
192.168.12.254     link#4             UHS             lo0
192.168.35.0/24    192.168.10.254     UGS          vtnet2
192.168.67.0/24    192.168.10.254     UGS          vtnet2
192.168.136.0/24   link#8             U      vtnet0_vlan1
192.168.136.1      link#4             UHS             lo0
192.168.210.0/24   link#14            U      vtnet0_vlan2
192.168.210.1      link#4             UHS             lo0

As you can see, there is no vtnet0_vlan1002 interface, insteed xxx.yyy.zzz.176/28 is connected to link#11(?) vtnet0_vlan1 as xxx.yyy.zzz.179 to link#4 interface lo0 !

Also for load balancing tests, [8.8.4.4|9.9.9.9] xxx.yyy.zzz.177 UGHS vtnet0_vlan1 but should be vtnet0_vlan1002

That's wrong and could explain why we can't use WAN2 ipv4 for incoming traffic. Should we open a bug?

I face the same problem, see https://forum.opnsense.org/index.php?topic=50882.0

Add it to github issues

Hi,

OpnSense is running in a VM (kvm) under Debian/bookworm. Both WAN are coming in a switch which mark them as VLAN1 (default) for ISP#1 VLAN1002 for ISP#2, the Debian host has interfaces configured in each VLAN and one for the whole traffic. This setup is working since years with Sophos UTM9.

I installed OpnSense v26.1.2 on the same host using same interfaces and VLANs to replace  Sophos in the future. At this time, only using DNAT and Rules (new), outgoing traffic is OK, ipv4 as well as ipv6.

I followed the multi Wan doc for load balancing. Speed being not identical, I gave different priority in System => Gateway => configuration, 250 for the power full ISP#1, 254 for the other one ISP#2. I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway. Default route was automatically setted on ISP#1 on first configuration.

ISP#2 brings an ipv6/48 network, no ipv6 on ISP#1. Both have a public ipv4 address. 

Problem: from an external server I try to connect to a machine in the LAN using ssh. It works with ipv4/ISP#1 ipv6/ISP#2 but not  ipv4/ISP#2. Using tcpdump in OpnSense console, I see the the outgoing traffic from the LAN machine is going out through ISP#1 and not ISP#2 from where the traffic came in. I also tried by giving the same priority in GW configuration, no changes.

Did I miss something knowing that sticky connection is set?

--
Daniel