Multi Wan broken - Vlan culpit?

[dash]

Hi,
as others we face a broken Multi Wan. WAN1 is only ipv4 and connected to an upper router in the 192.168.136.0/24 LAN with VLAN 1. WAN2 is directly connected to Internet, ipv4 and ipv6. This connection uses a VLAN 1002. Load balancing is configured for the LAN outgoing traffic, OpnSense version 26.1.2_5-amd64 is running in a kvm VM, ipv6 is  working well.

Problem is that all traffic coming in to ipv4 WAN2 is going out using WAN1 *with the ipv4 source address of WAN2*

Example from an ssh outside connection, src ipv4 being aaa.bbb.ccc.107 to dst ipv4 zzz.yyy.zzz.179:

root@guava:~ # tcpdump -ni vtnet0_vlan1 src zzz.yyy.zzz.179
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet0_vlan1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:24:46.960754 IP xxx.yyy.zzz.179.50022 > aaa.bbb.ccc.107.55504: Flags [S.], seq 114251010, ack 2702443401, win 64800, options [mss 1452,sackOK,TS val 2130106607 ecr 1545017500,nop,wscale 7], length 0

Here is netstat output:

root@guava:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.136.254    UGS    vtnet0_vlan1
8.8.4.4            xxx.yyy.zzz.177    UGHS   vtnet0_vlan1
8.8.8.8            192.168.136.254    UGHS   vtnet0_vlan1
9.9.9.9            zzz.yyy.zzz.177    UGHS   vtnet0_vlan1
10.0.0.0/16        192.168.10.254     UGS          vtnet2
10.1.58.0/24       192.168.10.254     UGS          vtnet2
10.2.67.0/24       192.168.10.254     UGS          vtnet2
10.99.98.0/24      192.168.10.254     UGS          vtnet2
10.99.99.0/24      192.168.10.254     UGS          vtnet2
xxx.yyy.zzz.176/28 link#11            U      vtnet0_vlan1
xxx.yyy.zzz.179    link#4             UHS             lo0
127.0.0.1          link#4             UH              lo0
149.112.112.112    192.168.136.254    UGHS   vtnet0_vlan1
172.31.98.0/24     192.168.10.254     UGS          vtnet2
192.168.10.0/24    link#3             U            vtnet2
192.168.10.1       link#4             UHS             lo0
192.168.12.0/24    link#13            U      vtnet0_vlan2
192.168.12.254     link#4             UHS             lo0
192.168.35.0/24    192.168.10.254     UGS          vtnet2
192.168.67.0/24    192.168.10.254     UGS          vtnet2
192.168.136.0/24   link#8             U      vtnet0_vlan1
192.168.136.1      link#4             UHS             lo0
192.168.210.0/24   link#14            U      vtnet0_vlan2
192.168.210.1      link#4             UHS             lo0

As you can see, there is no vtnet0_vlan1002 interface, insteed xxx.yyy.zzz.176/28 is connected to link#11(?) vtnet0_vlan1 as xxx.yyy.zzz.179 to link#4 interface lo0 !

Also for load balancing tests, [8.8.4.4|9.9.9.9] xxx.yyy.zzz.177 UGHS vtnet0_vlan1 but should be vtnet0_vlan1002

That's wrong and could explain why we can't use WAN2 ipv4 for incoming traffic. Should we open a bug?

[TheSHAD0W]

This is apparently an issue with upgrading to the new opnsense version. You can try changing the destination NAT entries to "register rule" and deleting any old rules you had regarding the forwarding. I tested this as working with a fresh install but still haven't gotten my old setup working properly. More info at https://github.com/opnsense/core/issues/9702

[dash]

I complete informations on https://github.com/opnsense/core/issues/9702

Even with a fresh install, traffic coming from WAN2 failed to go out using the same  GW despite the fact that reply-to seems setted.

[dash]

I upgraded to 26.1.3. This version correct other problems I faced but still can't connect using ssh on WAN2 with ipv4. Summarize:

. vlan1 WAN1 ipv4: all is good
. vlan1002 WAN2 ipv4 and ipv6: ipv6 OK ipv4 KO.

Problem: incoming ipv4 traffic on WAN2 is going out on WAN1 despite the fact that source IP of outgoing packets is the correct one from WAN2 where packet came in.

Unanswered question are:
. why netstat -rn4 doesn't show vtnet0_vlan1002 as shown below

Internet:
Destination        Gateway            Flags         Netif Expire
default            AAA.BBB.136.254    UGS    vtnet0_vlan1
8.8.8.8            AAA.BBB.136.254    UGHS   vtnet0_vlan1
9.9.9.9            YYY.XXX.252.177    UGHS   vtnet0_vlan1

Name              Mtu Network             Address            Ipkts Ierrs Idrop    Opkts Oerrs  Coll
vtnet0_vlan1        - AAA.BBB.136.0/24    AAA.BBB.136.1       6989     -     -        0     -     -
vtnet0_vlan1002     - YYY.XXX.252.176/28  109.237.252.179     6542     -     -        0     -     -

. why despite the fact that source IP of outgoing packets is the right one (reply-to OK) are those packets delivered to GW of WAN1

. using load balancing for local outgoing packets, should I remove default route which is GW of WAN1?

I did a fresh install from latest 26.1.2 version and then upgrade to 26.1.3, problem doesn't disappear.

Thanks for any hint