[solved] Traffic directed to wrong ARP address on WAN subnet

[banym]

Since I am not sure if its a bug or feature this post, maybe others have seen it before:

I have some opnsense firewalls connected to the same /24 WAN subnet.

Firewall A: 212.x.x.1
Firewall B: 212.x.x.2
Router: 212.x.x.254

Code Select Expand



+------------------+
|     Router       |
|     212.x.x.254  |
+--------+---------+
         |
         |
         |
         |
         |             212.x.x.0./24  WAN
         +--------+-------------------------------+----------+
                  |                               |
                  |                               |
         +--------+---------+          +----------+---------+
         |    Firewall-A    |          |     Firewall B     |
         |     212.x.x.1    |          |     212.x.x.2      |
         +--------+---------+          +----------+---------+
                  |                               |
                  |   LAN A                       | LAN B
                  |                               |
                  |                               |
            +-----+-----+                   +-----+------+
            |    PC     |                   |    PC      |
            |    01     |                   |    02      |
            +-----------+                   +------------+



The availability from the LAN of Firewall A to the WAN Interface of Firewall B looks like this:



After doing a traffic capture on Firewall A and B I think I found the problem. Firewall B does not send the Traffic directly back to Firewall A.
The ARP traffic is sent to a combination of IP of Firewall A but with the MAC of Router. I reviewed the ARP table on Firewall B but there the entry was shown correctly.

We replaced hardware and reinstalled but problem persists with multiple installations and different firewalls on the same WAN interface.

For testing I changed:

net.inet.ip.redirect 1
net.inet.icmp.drop_redirect 0

-> no change.

Firewall->Settings->Advanced->Disable force gateway

-> no change

[Fright]

Firewall->Settings->Advanced->Disable force gateway

try to disable reply-to also
(dont forget to kill states after)

[banym]

Thank you, that seems to fix it. I found an older long discussion https://forum.opnsense.org/index.php?topic=15900.0