OPNsense Forum

Archive => 21.1 Legacy Series => Topic started by: Jaxon on February 25, 2021, 11:58:13 pm

Title: SOLVED - Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 25, 2021, 11:58:13 pm: Hello,

Please see the attached png for a detailed network topology. It's a dual-wan setup where the hosts on the LAN need to reach an aliased IP on one of the WAN CPEs. (Starlink has an app that needs to reach 192.168.100.1)

To accomplish this, I have provisioned a virtual IP on WAN2:
Code: [Select]
root@OPNSense:~ # ifconfig vtnet2 vtnet2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=800a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE> ether b6:31:72:6b:66:b6 inet6 fe80::b431:72ff:fe6b:66b6%vtnet2 prefixlen 64 scopeid 0x3 inet 100.74.114.223 netmask 0xffc00000 broadcast 100.127.255.255 inet 192.168.100.2 netmask 0xffffff00 broadcast 192.168.100.255 media: Ethernet 10Gbase-T <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

and an outbound NAT rule:
Code: [Select]
Outbound NAT Rule: Interface: WAN2 Source Address: LAN net Dest Address: 192.168.100.1/32 Translation / target: 192.168.100.2()

In a nutshell, from the firewall I can ping 192.168.100.1, from a host on the LAN, I can not. What is happening is the dest MAC is the wrong one. I have tried static MAC in ARP table, but didn't help.

Eg.
Code: [Select]
root@OPNSense:~ # arp -na ? (100.74.114.223) at b6:31:72:6b:66:b6 on vtnet2 permanent [ethernet] ? (100.127.255.2) at 02:02:00:00:00:02 on vtnet2 expires in 87 seconds [ethernet] ? (192.168.100.1) at 26:12:ac:1a:80:01 on vtnet2 permanent [ethernet] ? (192.168.100.2) at b6:31:72:6b:66:b6 on vtnet2 permanent [ethernet]
Working ping originating from firewall (dest mac is correct: 26:12:ac:1a:80:01)
Code: [Select]
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vtnet2, link-type EN10MB (Ethernet), capture size 262144 bytes 17:44:13.593643 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 54865, seq 0, length 64 0x0000: 2612 ac1a 8001 b631 726b 66b6 0800 4500 &......1rkf...E. 0x0010: 0054 01e9 0000 4001 2f6c c0a8 6402 c0a8 .T....@./l..d... 0x0020: 6401 0800 2b3a d651 0000 0006 a6f7 33e9 d...+:.Q......3. 0x0030: 308a 0809 0a0b 0c0d 0e0f 1011 1213 1415 0............... 0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$% 0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345
Broken ping originating from client on LAN (desc mac is incorrect: 02:02:00:00:00:02)
Code: [Select]
17:46:43.691084 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 45857, seq 1, length 64 0x0000: 0202 0000 0002 b631 726b 66b6 0800 4500 .......1rkf...E. 0x0010: 0054 5693 4000 3f01 9bc1 c0a8 6402 c0a8 .TV.@.?.....d... 0x0020: 6401 0800 fc21 b321 0001 d328 3860 0000 d....!.!...(8`.. 0x0030: 0000 7c5f 0200 0000 0000 1011 1213 1415 ..|_............ 0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$% 0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345 0x0060: 3637

I am hoping to find some help in identifying what it is that I might be doing wrong? Any thoughts are appreciated!
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: smyers119 on February 26, 2021, 12:19:29 am: Is starlink the secondary WAN or primary? or are they load balanced?
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 26, 2021, 12:59:21 am: The gateways are load balanced, both have priority 1 in single gateway settings, and the group gateway has them both set to tier 1.
Not certain if it matters, but the route table shows the Starlink as the default gateway on vtnet2.

The LAN firewall rule "Default allow LAN to any rule " is set to use the "GW_GROUP".
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: smyers119 on February 26, 2021, 01:24:09 am: Quote from: Jaxon on February 26, 2021, 12:59:21 am
The gateways are load balanced, both have priority 1 in single gateway settings, and the group gateway has them both set to tier 1.
Not certain if it matters, but the route table shows the Starlink as the default gateway on vtnet2.

The LAN firewall rule "Default allow LAN to any rule " is set to use the "GW_GROUP".

Make a firewall rule on LAN with action PASS

destination 192.168.100.1

gateway starlink
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 26, 2021, 01:33:33 am: Quote
Make a firewall rule on LAN with action PASS

destination 192.168.100.1

gateway starlink

Good suggestion, thank you. I had actually already tried that, but the result is the same dest mac being the wrong one.
Code: [Select]
19:31:36.441863 IP 192.168.100.2 > 192.168.100.1: ICMP echo request, id 32753, seq 18, length 64 0x0000: 0202 0000 0002 b631 726b 66b6 0800 4500 .......1rkf...E. 0x0010: 0054 932f 4000 3f01 5f25 c0a8 6402 c0a8 .T./@.?._%..d... 0x0020: 6401 0800 1418 7ff1 0012 6841 3860 0000 d.........hA8`.. 0x0030: 0000 0270 0200 0000 0000 1011 1213 1415 ...p............ 0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425 ...........!"#$% 0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233 3435 &'()*+,-./012345 0x0060: 3637 67
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 26, 2021, 01:38:05 am: I can actually see that the firewall rules are allowing the packets to egress as expected .See next attachment.
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Maurice on February 27, 2021, 03:14:29 am: That's a side effect of the policy based routing required for multi WAN. Specifying a gateway in the LAN firewall rule sends all matching packets to the gateway's MAC address. That's what you're seeing.
A "pass LAN to 192.168.100.1" rule with no gateway specified (and a higher priority than the "pass LAN to any" rule) might work.

Cheers

Maurice
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 27, 2021, 03:40:02 am: Quote
That's a side effect of the policy based routing required for multi WAN. Specifying a gateway in the LAN firewall rule sends all matching packets to the gateway's MAC address. That's what you're seeing.

Thanks for the response Maurice, yes that makes perfect sense now that you mention it. Although I thought the outbound NAT rule would take care of that. (Obviously not :) )

Quote
with no gateway specified

How do I do that? Do you mean to choose "Null4 - 127.0.0.1"? It seems that I must always choose a gateway, even if it's the default? (I quickly tried with Null4 and that didn't seem to work)
Title: Re: Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Jaxon on February 27, 2021, 03:44:32 am: Oh snap! I didn't think that through, I added the suggest rule, and now it's working. Thanks Maurice!
Title: Re: SOLVED - Packets leaving IP Alias interface have incorrect dest MAC address
Post by: Maurice on February 27, 2021, 04:32:28 am: Yeah, sorry, I meant leaving the gateway in the firewall rule on default, not actually "no gateway". Default means that no particular gateway is enforced and the routing table will be used. Which in this case results in "no gateway".
Glad you figured it out and it works!