Topics

Hi,
as others we face a broken Multi Wan. WAN1 is only ipv4 and connected to an upper router in the 192.168.136.0/24 LAN with VLAN 1. WAN2 is directly connected to Internet, ipv4 and ipv6. This connection uses a VLAN 1002. Load balancing is configured for the LAN outgoing traffic, OpnSense version 26.1.2_5-amd64 is running in a kvm VM, ipv6 is  working well.

Problem is that all traffic coming in to ipv4 WAN2 is going out using WAN1 *with the ipv4 source address of WAN2*

Example from an ssh outside connection, src ipv4 being aaa.bbb.ccc.107 to dst ipv4 zzz.yyy.zzz.179:

root@guava:~ # tcpdump -ni vtnet0_vlan1 src zzz.yyy.zzz.179
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet0_vlan1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:24:46.960754 IP xxx.yyy.zzz.179.50022 > aaa.bbb.ccc.107.55504: Flags [S.], seq 114251010, ack 2702443401, win 64800, options [mss 1452,sackOK,TS val 2130106607 ecr 1545017500,nop,wscale 7], length 0

Here is netstat output:

root@guava:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags         Netif Expire
default            192.168.136.254    UGS    vtnet0_vlan1
8.8.4.4            xxx.yyy.zzz.177    UGHS   vtnet0_vlan1
8.8.8.8            192.168.136.254    UGHS   vtnet0_vlan1
9.9.9.9            zzz.yyy.zzz.177    UGHS   vtnet0_vlan1
10.0.0.0/16        192.168.10.254     UGS          vtnet2
10.1.58.0/24       192.168.10.254     UGS          vtnet2
10.2.67.0/24       192.168.10.254     UGS          vtnet2
10.99.98.0/24      192.168.10.254     UGS          vtnet2
10.99.99.0/24      192.168.10.254     UGS          vtnet2
xxx.yyy.zzz.176/28 link#11            U      vtnet0_vlan1
xxx.yyy.zzz.179    link#4             UHS             lo0
127.0.0.1          link#4             UH              lo0
149.112.112.112    192.168.136.254    UGHS   vtnet0_vlan1
172.31.98.0/24     192.168.10.254     UGS          vtnet2
192.168.10.0/24    link#3             U            vtnet2
192.168.10.1       link#4             UHS             lo0
192.168.12.0/24    link#13            U      vtnet0_vlan2
192.168.12.254     link#4             UHS             lo0
192.168.35.0/24    192.168.10.254     UGS          vtnet2
192.168.67.0/24    192.168.10.254     UGS          vtnet2
192.168.136.0/24   link#8             U      vtnet0_vlan1
192.168.136.1      link#4             UHS             lo0
192.168.210.0/24   link#14            U      vtnet0_vlan2
192.168.210.1      link#4             UHS             lo0

As you can see, there is no vtnet0_vlan1002 interface, insteed xxx.yyy.zzz.176/28 is connected to link#11(?) vtnet0_vlan1 as xxx.yyy.zzz.179 to link#4 interface lo0 !

Also for load balancing tests, [8.8.4.4|9.9.9.9] xxx.yyy.zzz.177 UGHS vtnet0_vlan1 but should be vtnet0_vlan1002

That's wrong and could explain why we can't use WAN2 ipv4 for incoming traffic. Should we open a bug?

Hi,

OpnSense is running in a VM (kvm) under Debian/bookworm. Both WAN are coming in a switch which mark them as VLAN1 (default) for ISP#1 VLAN1002 for ISP#2, the Debian host has interfaces configured in each VLAN and one for the whole traffic. This setup is working since years with Sophos UTM9.

I installed OpnSense v26.1.2 on the same host using same interfaces and VLANs to replace  Sophos in the future. At this time, only using DNAT and Rules (new), outgoing traffic is OK, ipv4 as well as ipv6.

I followed the multi Wan doc for load balancing. Speed being not identical, I gave different priority in System => Gateway => configuration, 250 for the power full ISP#1, 254 for the other one ISP#2. I create a group Gateway with both GWs on Level1 for load balancing as well as a out rule for LAN net on LAN interface with GW setted to this group Gateway. Default route was automatically setted on ISP#1 on first configuration.

ISP#2 brings an ipv6/48 network, no ipv6 on ISP#1. Both have a public ipv4 address. 

Problem: from an external server I try to connect to a machine in the LAN using ssh. It works with ipv4/ISP#1 ipv6/ISP#2 but not  ipv4/ISP#2. Using tcpdump in OpnSense console, I see the the outgoing traffic from the LAN machine is going out through ISP#1 and not ISP#2 from where the traffic came in. I also tried by giving the same priority in GW configuration, no changes.

Did I miss something knowing that sticky connection is set?

--
Daniel