ICMP-reply from OPNsense fw may have wrong dest MAC-Address.

[salvadordalisdad]

Hi Guys,

TLDR:  ICMP-reply from OPNsense fw may have wrong dest MAC-Address.

I've written this more than a dozen times, and every time has prompted me to do another test.
Always hoping that explaining it to the rubber duck on my shelf will give me the answer!
So deleted & started again.

Intention = replace pfSense with OPNsense, but hit a snag.
Anyway, having installed OPNsense dozens of times, on ESXI 6.7 and tried endless variations, I've finally distilled it down to a simple problem.

This started as an odd occassional "pings stopped from my workstation to the OPNsense, but web gui was fine".
Weird but can't get past it as it might indicate other problems lurking.
So I tested all the variables & eliminated all that I could think of:
It's intermittent.
Might happen after several hours of uptime.
Might be immediately after reboot.
Not because the switch.
Not just one workstation.
Not because the choice of VMXNet or e1000 ethernet device type.
Not because Static IP or DHCP.
Not because VLAN-tagging or esxi-network-adapters for the DMZs
Not because v24, tried v23 and v21 as well.

Packet capture on OPNsense shows ICMP reply getting sent.
However - Layer-2 examination shows it's sending it to the default gateway instead of back to the source mac.
FINALLY I found the right question to ask!!!

This is a FRESH install (maybe number 53 I think) with only TWO interfaces WAN, LAN.
IPAddr on LAN is obtained with DHCP
Arp table from another PC:
  Internet Address      Physical Address      Type
  192.168.4.1           00-0c-29-e9-58-07     dynamic
  192.168.4.18          64-00-6A-78-CB-53     dynamic
  192.168.4.200         00-0c-29-91-81-bc     dynamic

Packet Capture from OPNSense Interfaces - diagnostics - packet capture, filtered for ICMP only = attached.
Frame 1 = 3 = ICMP request from 192.168.4.18 (pc)  to 192.168.4.200 (OPNSense)
Frame 2 = 4 = ICMP Reply from 192.168.4.200 (OPNSense) to   192.168.4.18 (pc)

BUT - the MAC-Address for the targeted ICMP-reply is wrong - it's the mac-address for the pfsense-fw 192.168.4.1
Packet capture from directly attached switch shows exactly the same content.
NB - even during this - the webGUI still works as normal.

Hwhat the heck is going on here, Bobby?
Any suggestions?

[Maurice]

Look up the firewall rule which allows the pings in Firewall: Diagnostics: Statistics: Rules. Does it have a reply-to (gateway) option? If so, you have to explicitly disable reply-to in the rule's advanced features.

Cheers
Maurice

[salvadordalisdad]

Hi Maurice,

Thanks for the quick reply.
It was replying to pings most of yesterday afternoon, but today it's not, and the mac-address in teh ICMP-reply is that of the pfsense internet gateway firewall again.
So I rebooted OPNsense and it's now responding to PING properly.

Attached is the only rule in the table which shows up for a search on "reply-to"
These stats are increasing with or without PINGs running, just using the webGUI.

I don't understand what they mean...
what am I supposed to do with them ?

I tried adding an explicit rule in the firewall rules for ICMP from LAN-net to this-firewall
didn't work, can't seem to place it above all the built-in rules.

Can't think this is what it's supposed to be doing on fresh install, with only 2 interfaces & nothing else configured.
What am I missing?

TIA

[zan]

Click on the rule then advanced features, then pick 'disable' on 'reply-to'.
I think since your LAN gets its address from DHCP it has its gateway set to 192.168.4.1, it uses its gateway's MAC as destination when replying.

[salvadordalisdad]

Hi Zan,

Thanks for the note, but I can't see why it would respond to the original mac-address for 8 hours & then stop, and change the dest-mac-addr to the default gateway, doesn't make sense.

Also, can't edit the built-in rules at all, there's no editing, so no advanced option.
The "allow ICMP" rule I wrote, which didn't fix it, I can edit, the reply-to box offers:
default    /    LAN-DHCP-192.168.4.1     /   null6 / null4
The only one which makes sense is already selected - default.

I'll try again with static IP and no default gateway, but not sure that wil help.

Ta

[zan]

Yea I think its better to use static IP in this case.

Thanks for the note, but I can't see why it would respond to the original mac-address for 8 hours & then stop, and change the dest-mac-addr to the default gateway, doesn't make sense.

Sorry can't help you with this, to me pf sometimes works in a mysterious way.

The "allow ICMP" rule I wrote, which didn't fix it, I can edit, the reply-to box offers:
default    /    LAN-DHCP-192.168.4.1     /   null6 / null4

You don't have 'disable' option?
See attached.

[salvadordalisdad]

Thanks, still testing...

I like the idea of the "disable" for reply-to (although the help is not entirely clear on what it does).
However, although there's now a rule at the top of the list to ALLOW ICMP and disable the reply-to options, it's not being used. Enabled logging on that rule to try & confirm it was being used.
The PING from my workstation, however, is not being logged, so I assume it's hitting a built-in rule over which I have no control, so academic really.

Stil trying a few things, still open to suggestions.

[Maurice]

Attached is the only rule in the table which shows up for a search on "reply-to"

Yes, that's probably the default allow LAN to any rule. As you can see, it has the option reply-to (vmx5 192.168.4.1), which causes all replies to be sent to the MAC address of your pfSense firewall. You need to edit this rule and disable reply-to (which is absolutely possible).

I tried adding an explicit rule in the firewall rules for ICMP from LAN-net to this-firewall
didn't work, can't seem to place it above all the built-in rules.

You don't have to place it above the automatically generated rules, only above the default allow LAN to any rule.

Can't think this is what it's supposed to be doing on fresh install, with only 2 interfaces & nothing else configured. What am I missing?

Your LAN interface uses DHCP and dynamically configures a (default) gateway. This is not a standard setup, LAN interfaces typically have a static configuration and no gateway. Your configuration causes OPNsense to consider this a WAN-type interface, which results in the automatically added reply-to.

[salvadordalisdad]

Hi Maurice,

Your answer is very well written, thanks.
This was the first stage (!) wherby it was just on the LAN with no WAN (only got one IP Addr from my provider, and it's currenrtly being used!).
So a special case, I understand now.
So it worked with a LAN gateway temporarily to get updates & plugins, now I've removed the gateway & routes & it's awaiting a suitable test period to replace the pfsense.
It even got wireguard with mullvad working OK, as well as acme certs & google backups.
Pretty sure I have sufficient config in place to provide same services as old-fw. Fingers crossed.

So I think that's solved the question, sufficiently, thanks.

Do I need to set that advanced - reply-to setting to "disable" at all if it's going to be in a standard lan-wan deployment ?
;-)

[Maurice]

No, for a basic setup, you don't have to mess with reply-to, that's why it's an advanced option.

The automatic reply-to is a bit controversial and there have been heated discussions about it, but it is what it is. Once you know about it, it's not a big deal.