[Moved to 23.1 Forum] Issue with MultiWAN gateway group and policy based routing

[_richii]

Update: Moved to 23.1 Forum -> https://forum.opnsense.org/index.php?topic=32580.0


Hello,

I set up a OPNsense with 2 WAN interfaces and a gateway group (configured as load balancing) as described in https://docs.opnsense.org/manual/how-tos/multiwan.html.
However, as soon as I activate the firewall rule for policy based routing via the gateway group, I get constant error messages on the hardware console:

arpresolve: can't allocate llinfo for <IP> on igb0
arpresolve: can't allocate llinfo for <IP> on igb1

As far as I can tell the gateway group works, however the hardware console is unusable.
Intresting is that the IP in the error message is always the gateway IP not corresponding to the interface.
For example:

igb0 IP: 86.87.88.90/30
igb0 gateway IP: 86.87.88.89

igb1 IP: 192.168.75.250/24
igb1 gateway IP: 192.168.75.254

arpresolve: can't allocate llinfo for 192.168.75.254 on igb0
arpresolve: can't allocate llinfo for 86.87.88.89 on igb1

Through extensive testing I did find out that the issue has something to to with the Shared forwarding feature under Firewall -> Settings -> Advanced, that allows for traffic shaping while also doing policy based routing.
When I disable it the messages on the hardware console stop.

I did also configure Traffic Shaping rules, but even after disabling them the arpresolve errors occour.
There is no MAC spoofing involved.

Here is the current setup:

OS / Hardware
OPNsense 22.7.11_1-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022
CPU: Intel(R) Pentium(R) Gold G6605 CPU @ 4.30GHz
Mainboard: Supermicro X12STL-IF
Network:
- Onboard: 2x Intel i210 RJ45 1GbE network ports (WAN)
- PCI-E: Mellanox ConnectX-4 Lx with 2x SFP28 25/10/1GbE network ports (LAN)

Interfaces

igb0: WAN_SDSL
- Static IPv4
- IPv6 None
- RFC1918 IP
- Gateway IP in WAN interface subnet

igb1: WAN_DSLHYBRID
- Static IPv4
- IPv6 None
- Public IP
- Gateway IP in WAN interface subnet

Gateways -> Single

WAN_SDSL_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.4.4
- Priority: 254
- Weight: 1

WAN_DSLHYBRID_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.8.8
- Priority: 253
- Weight: 3

Gateways -> Group

WAN_LOADBALANCE
- Tier1: WAN_SDSL_GWv4
- Tier1: WAN_DSLHYBRID_GWv4
- Trigger Level: Member Down
- Pool Options: Default

I also tested different combinatitons of Upstream Gateway, Gateway Priority or even Tiers in the gateway group.
The result is always the same. As soon as traffic is routed via gateway group the arpresolve errors occour.

Does anyone have any ideas to further debug this?

[_richii]

Update

It seems this is not only a cosmetic issue.
When the gateway group is used in firewall rules, clients behind the router experience connection timeouts.
So far this is only reproducible for HTTP/S services in a web browser, but as soon I disable the gateway group the problem goes away.

During the connection timeouts there is no packet loss or high latency on any gateway.
I also tested both gateways as "active" independently from one another.
As soon as the gateway group is not involved anymore it doesn't matter which gateway is active, both are completly stable and produce no connection timeouts on clients.

The next step will be an upgrade to 23.1.1 after it is released.
I will report if this solves the problems.

[_richii]

Update

The issue persists after upgrade to 23.1.1, but I was able to narrow it down further.
It only happens when "shared forwarding" is enabled and both gateways in the gateway group are set to "Tier 1".
It seems to me as soon as load balancing is enabled, there are traffic issues when ipfw2 for traffic shaping is enabled.
I also removed all traffic shaping rules, reset the firewall state table and reset the ource tracking to make shure they are not the source of this.

I probably will open another thread in the 23.1 forum about the specific issue.
When I do I willl update this topic.