Corrupted ARP entry.

[rambopierce]

Hello,

I never ask for help because I have been in IT for over forty years which means I know everything already    that was a joke.

This one has me stumped.  I am running the following OPNsense version(s):

Versions   OPNsense 23.1.10_1-amd64
FreeBSD 13.1-RELEASE-p7
OpenSSL 1.1.1u 30 May 2023

If I flush the ARP table and allow it to repopulate, the following correct entry will be inserted into the table:

172.18.222.40   dc:a6:32:4a:6b:b5   Raspberry Pi Trading Ltd   em1   lan   PiHole.gopierce.com

The correct hardware address for my PiHole server is dc:a6:32:4a:6b:b5.  Everything works great for hours or days. Then for some unknown reason, I find the following line in the ARP table:

172.18.222.40   dc:a6:32:4a:6b:b8   Raspberry Pi Trading Ltd   em1   lan   PiHole.gopierce.com

Note the ':b8' on the end of the hardware address (it should be ':b5').  That goes nowhere and I can no longer ping 172.18.222.40 from anywhere on my local networks.  Flush the ARP table and can immediatley ping 172.18.222.40 again for hours or days until for some unknown reason the incorrect hardware address appears again in the ARP table.

Any one have a clue as to where or why an ARP table entry would be updated with the incorrect hardware address?

I would prefer to NOT have ARP at all over this mess, but I do not know if there is a way to turn it off.

Many thanks for any help,
Richard

[franco]

Hi,

This is a wild one. If it always ends up corrupting as ":b8" it's likely because that's exactly what's being broadcasted via ARP. I'd do a packet capture and see if you can confirm that... especially who says it's ":b8" rather than ":b5". It's going to be a piece of equipment in your network and worst case the PiHole server as well -- perhaps it has a valid MAC with ":b8"? "b5" - ":b8" are 4 ports... this wouldn't be an uncommon hardware setup.

But just thinking out loud here. 


Cheers,
Franco

[rambopierce]

Thanks Franco.  It is really bizarre, huh?  You have given me an idea though, I'm going to go across my network looking for a hardware address matching the one ending in :b8.  Have no idea how or why it would replace the one that is correct for this IP address.  One hundred percent of the time flushing the ARP table fixes the issue for hours or days.  Hmm... let me check right now, ping 172.18.222.40:

Pinging 172.18.222.40 with 32 bytes of data:
Reply from 172.18.222.40: bytes=32 time<1ms TTL=63
Reply from 172.18.222.40: bytes=32 time<1ms TTL=63
Reply from 172.18.222.40: bytes=32 time<1ms TTL=63
Reply from 172.18.222.40: bytes=32 time<1ms TTL=63

I cleared the ARP table a few days ago and all is well, but it will come back wrong at some point and I just don't know what drives it.

thanks again,
Richard

[franco]

Hi Richard,

> I cleared the ARP table a few days ago and all is well, but it will come back wrong at some point and I just don't know what drives it.

Yep, problems that go away without having solved them will likely return later.

If it happens you could try tcpdump on the offending interface to see if the MAC is active...

https://www.pico.net/kb/how-does-one-filter-mac-addresses-using-tcpdump/

It may give further hints depending on what traffic appears to be generated by that MAC.


Cheers,
Franco

[lilsense]

One thing you can do is re-MAC it to AA:BB:AA:CC:DD and see if that changes. This manual MAC should give you a clue that the PiHole hardware bug is in effect.

[efahl]

Not sure if this will help at all, but I also saw strange ARP behavior on a Pi 4b running just basic Raspian and PiHole...

https://github.com/royhills/arp-scan/issues/56#issuecomment-952374797

Might be something in that discussion that sparks a thought?

[rambopierce]

Resolved and I should have found it a lot sooner.  I used to run this Raspberry Pi on wireless and later moved it to wired.  The following (now incorrect entry) was in the DHCP static lease table (with the static ARP box checked):

   dc:a6:32:4a:6b:b8   172.18.222.40   PiHole   Pi Hole Raspberry Pi   

The correct wired address is dc:a6:32:4a:6b:b5, ending in 5, not 8.  Now, to edit that static lease, you go to Services, DHCPv4, and then the network interface (in my case LAN). The static lease mappings can be edited, deleted at the bottom.

So, what was confusing was, when the Pi would boot, initially and sometimes for days, everything was fine because the Pi had advertised the correct wired address at startup.  The router would use that for a while until whatever random event occurred that caused OPNsense to fallback to it's static lease, ARP entry which was incorrect.

The reason I missed the understanding and fixing the problem sooner was I kept focusing on the ARP table in the OPNsense interface, when I should have spent some time focusing on the DHCP static lease table.

Hope this helps someone else out there that either doesn't know how to manage networks or is sloppy like me.

Thanks to all for your replies and ideas.