IPv4: wrong destination for linux client on other side of transparent bridge

[Uwe@Home]

Hello,

(this is an english copy of https://forum.opnsense.org/index.php?topic=22566.msg107282#msg107282)

following setup with OPNsense 20.7.8 running on an APU4D4:

OPNsense  (LAN)                                                  (LAN)
                 - Switch - Transparent WLan-Bridge - Switch
                 - Client "A"                                        - Client "L" (Linux)
                                                                          - Client "W" (Windows)

Problem: "L" can´t ping "A" BUT "W" can ping "A"  ("A" can ping everything)

What´s important: this setup was perfectly working with Sophos UTM 9 before.

I compared an ICMP-paket and found the following differences:

Request:
  IPv4
    Windows "W": no flags set / Linux "L": don't fragment - bit set
  ICMP
    Windows "W": 32 Byte length (without timestamp) / Linux "L": 48 Bytes (with Timestamp)

Reply:
  IPv4
    Windows "W": no flags set / Linux "L": don't fragment - bit set
    Windows "W": Destination Address = Tp-LinkT_49:33:52 (=> MAC of Access-Point)
    Linux "L": Destination Address = Raspberr_52:ed:3c (=> MAC of Client "L")
  ICMP
    Windows "W": 32 Byte length (without timestamp) / Linux "L": 48 Bytes (with Timestamp)
    Windows "W": response time = 0,088 ms / Linux "L" = 0,154 ms

It seems that OPNsense replaces the MAC-Adress in case "don´t fragment" is set.

What I have tried so far (followed by reboot of OPNsense after each change):
System: Settings: Tunables: net.inet.ip.redirect: 0 => 1
Interfaces => Settings
  ARP Handling: [ ] => [X] Suppress ARP messages
  Hardware TSO: Disable hardware TCP segmentation offload [X] => [ ]
Firewall: Settings: Normalization:  IP Do-Not-Fragment [ ] => [X]
Firewall: Settings: Advanced:
  Network Address Translation: Reflection for port forwards [X] => [ ]
  Network Address Translation: Automatic outbound NAT for Reflection [X] => [ ]
  Disable automatic rules which force local services to use the assigned interface [ ] => [X] 

No Interface has an "upstream gateway" set
Intrusion-Detection ist disabled.

LAN-Interface is configured like this:
  Block private networks: [ ]
  Block bogon networks: [ ]
  MTU: 1500
  MSS: (not set)
  Dynamic gateway policy: [ ] This interface does not require an intermediate system to act as a gateway
  IPv4 Upstream Gateway: Auto-Detect

Is it possible to disabled that "feature" and force OPNsense to always
return the paket to the Source-MAC-Adresse as specified by the initial request?

[Uwe@Home]

Root cause found: in the static (DHCPv4) IP reservations, the real MAC-Address
of the Client was entered instead of the MAC-Address of the Access-Point.

Ping works right after deletion of static entry and flush of ARP-Table.

[Patrick M. Hausen]

In this case the access point is not configured as a transparent bridge. I would try to fix that instead of messing with DHCP.

[Uwe@Home]

In this case the access point is not configured as a transparent bridge

With a Sophos UTM I never every had this issue.
The configuration of the Access-Point has not been modified.
Both Access-Points of the bridge are TP-Link CPE510.
Those Access-Points have no configuration options for "transparency".
The bridge behaves transparent as long as OPNSense sends the pakets back to the origin MAC-Address.

Every now an then, OPNSense uses the wrong MAC-address.
My assumption is that the MAC-address gets replaced during renew of the DHCP lease.

How do I get OPNSense to always recognize the MAC-Address of the outermost header
(the "Bridge-entrance") instead of the MAC-Address of the device behind the bridge?

The OPNSense-tunables are currently set like this - all others have the default-value:
net.inet.icmp.drop_redirect = 1
net.inet.ip.redirect = 1

(Decsribed in here: https://www.freebsdhandbook.com/security/)

[Uwe@Home]

Upgraded to OPNsense 22.1.4 - but it still happend again, today (about 3 weeks later).
Meanwhile I don´t think that it has something to do with DHCP because the lease-timeout
is set to the default-value of 7200 seconds (2 hours)

I solved it again by (temporarily) flushing the ARP-Cache of OPNSense.

Am I the only one with this issue? What the heck is the root-cause?
Is it possible to configure a cron-job to flush the ARP-Table automatically after two weeks?

[Uwe@Home]

Meanwhile I figured out how to create a cron-entry to flush the arp-cache periodically:

SSH => 8:Shell

Code Select Expand


cd /usr/local/opnsense/service/conf/actions.d
vi actions_arp.conf


Code Select Expand


[clear]
command:/usr/sbin/arp -d -a
parameters:
type:script
message:clearing arp cache
description:Clear ARP Cache


Code Select Expand


service configd restart
configctl arp clear
exit


System => Settings => Cron => + => Clear ARP Cache

But it´s sad that no one seems to know the root cause for wrong entries appearing in arp-cache, randomly (between 5 and 30 days).