SOLVED - Packets leaving IP Alias interface have incorrect dest MAC address

[Jaxon]

Hello,

Please see the attached png for a detailed network topology. It's a dual-wan setup where the hosts on the LAN need to reach an aliased IP on one of the WAN CPEs. (Starlink has an app that needs to reach 192.168.100.1)

To accomplish this, I have provisioned a virtual IP on WAN2:

Code: [Select]

root@OPNSense:~ # ifconfig vtnet2
vtnet2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
        ether b6:31:72:6b:66:b6
        inet6 fe80::b431:72ff:fe6b:66b6%vtnet2 prefixlen 64 scopeid 0x3
        inet 100.74.114.223 netmask 0xffc00000 broadcast 100.127.255.255
        inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

and an outbound NAT rule:

Code: [Select]

Outbound NAT Rule:
Interface: WAN2
Source Address: LAN net
Dest Address: 192.168.100.1/32
Translation / target: 192.168.100.2()

In a nutshell, from the firewall I can ping 192.168.100.1, from a host on the LAN, I can not. What is happening is the dest MAC is the wrong one. I have tried static MAC in ARP table, but didn't help.

Eg.

Code: [Select]

root@OPNSense:~ # arp -na
? (100.74.114.223) at b6:31:72:6b:66:b6 on vtnet2 permanent [ethernet]
? (100.127.255.2) at 02:02:00:00:00:02 on vtnet2 expires in 87 seconds [ethernet]
? (192.168.100.1) at 26:12:ac:1a:80:01 on vtnet2 permanent [ethernet]
? (192.168.100.2) at b6:31:72:6b:66:b6 on vtnet2 permanent [ethernet]
Working ping originating from firewall (dest mac is correct: 26:12:ac:1a:80:01)

Code: [Select]

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vtnet2, link-type EN10MB (Ethernet), capture size 262144 bytes
17:44:13.593643 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 54865, seq 0, length 64
        0x0000:  2612 ac1a 8001 b631 726b 66b6 0800 4500  &......1rkf...E.
        0x0010:  0054 01e9 0000 4001 2f6c c0a8 6402 c0a8  .T....@./l..d...
        0x0020:  6401 0800 2b3a d651 0000 0006 a6f7 33e9  d...+:.Q......3.
        0x0030:  308a 0809 0a0b 0c0d 0e0f 1011 1213 1415  0...............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
Broken ping originating from client on LAN (desc mac is incorrect: 02:02:00:00:00:02)

Code: [Select]

17:46:43.691084 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 45857, seq 1, length 64
        0x0000:  0202 0000 0002 b631 726b 66b6 0800 4500  .......1rkf...E.
        0x0010:  0054 5693 4000 3f01 9bc1 c0a8 6402 c0a8  .TV.@.?.....d...
        0x0020:  6401 0800 fc21 b321 0001 d328 3860 0000  d....!.!...(8`..
        0x0030:  0000 7c5f 0200 0000 0000 1011 1213 1415  ..|_............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
        0x0060:  3637



I am hoping to find some help in identifying what it is that I might be doing wrong? Any thoughts are appreciated!

[smyers119]

Is starlink the secondary WAN or primary? or are they load balanced?

[Jaxon]

The gateways are load balanced, both have priority 1 in single gateway settings, and the group gateway has them both set to tier 1.
Not certain if it matters, but the route table shows the Starlink as the default gateway on vtnet2.

The LAN firewall rule "Default allow LAN to any rule " is set to use the "GW_GROUP".

[smyers119]

The gateways are load balanced, both have priority 1 in single gateway settings, and the group gateway has them both set to tier 1.
Not certain if it matters, but the route table shows the Starlink as the default gateway on vtnet2.

The LAN firewall rule "Default allow LAN to any rule " is set to use the "GW_GROUP".

Make a firewall rule on LAN with action PASS

destination 192.168.100.1

gateway starlink

[Jaxon]

Make a firewall rule on LAN with action PASS

destination 192.168.100.1

gateway starlink

Good suggestion, thank you. I had actually already tried that, but the result is the same dest mac being the wrong one.

Code: [Select]

19:31:36.441863 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 32753, seq 18, length 64
        0x0000:  0202 0000 0002 b631 726b 66b6 0800 4500  .......1rkf...E.
        0x0010:  0054 932f 4000 3f01 5f25 c0a8 6402 c0a8  .T./@.?._%..d...
        0x0020:  6401 0800 1418 7ff1 0012 6841 3860 0000  d.........hA8`..
        0x0030:  0000 0270 0200 0000 0000 1011 1213 1415  ...p............
        0x0040:  1617 1819 1a1b 1c1d 1e1f 2021 2223 2425  ...........!"#$%
        0x0050:  2627 2829 2a2b 2c2d 2e2f 3031 3233 3435  &'()*+,-./012345
        0x0060:  3637                                     67

[Jaxon]

I can actually see that the firewall rules are allowing the packets to egress as expected .See next attachment.

[Maurice]

That's a side effect of the policy based routing required for multi WAN. Specifying a gateway in the LAN firewall rule sends all matching packets to the gateway's MAC address. That's what you're seeing.
A "pass LAN to 192.168.100.1" rule with no gateway specified (and a higher priority than the "pass LAN to any" rule) might work.

Cheers

Maurice

[Jaxon]

That's a side effect of the policy based routing required for multi WAN. Specifying a gateway in the LAN firewall rule sends all matching packets to the gateway's MAC address. That's what you're seeing.

Thanks for the response Maurice, yes that makes perfect sense now that you mention it. Although I thought the outbound NAT rule would take care of that. (Obviously not  )

with no gateway specified

How do I do that? Do you mean to choose "Null4 - 127.0.0.1"? It seems that I must always choose a gateway, even if it's the default? (I quickly tried with Null4 and that didn't seem to work)

[Jaxon]

Oh snap! I didn't think that through, I added the suggest rule, and now it's working. Thanks Maurice!

[Maurice]

Yeah, sorry, I meant leaving the gateway in the firewall rule on default, not actually "no gateway". Default means that no particular gateway is enforced and the routing table will be used. Which in this case results in "no gateway".
Glad you figured it out and it works!