Topics

[OS / Hardware]

There seems to be a problem with Load Balancing (2 Gateways on Tier 1) via Gateway Groups and the shared forwarding feature.
As soon as the "Policy Based Routing" Firewall rules with the Load Balancing Gateway Group as a gateway are in place, two things happen:

1. The hardware console is spammed with arpresolve errors:
arpresolve: can't allocate llinfo for <IP> on igb0
arpresolve: can't allocate llinfo for <IP> on igb1

2. There are random(?) timeouts for outgoing traffic:
- First try loading a web page fails
- Second try loading a web page works

When the Gateway Group is is set to Failover, first gateway Tier1 and second gateway Tier 2, there are no problems.
When the shared forwarding feature under Firewall -> Settings -> Advanced ist disabled, there are no problems, too.
But when the feature is disabled, there is no traffic shaping possible for PBR Firewall rules, or at least this is stated in the shared forwarding help text.

This problem also existed in 22.7.11_1, see -> https://forum.opnsense.org/index.php?topic=32374.0 for further information.

Does anybody use the Load Balancing feature of Gateway Groups and can reproduce this?
Or is using shared forwarding and Load Balancing via Gateway Groups mutually exclusive / not supported?

OS / Hardware
OPNsense 23.1.1_2-amd64
FreeBSD 13.1-RELEASE-p6
OpenSSL 1.1.1t 7 Feb 2023
CPU: Intel(R) Pentium(R) Gold G6605 CPU @ 4.30GHz
Mainboard: Supermicro X12STL-IF
Network:
- Onboard: 2x Intel i210 RJ45 1GbE network ports (WAN)
- PCI-E: Mellanox ConnectX-4 Lx with 2x SFP28 25/10/1GbE network ports (LAN)

[Update: Moved to 23.1 Forum -> https://forum.opnsense.org/index.php?topic=32580.0]

Update: Moved to 23.1 Forum -> https://forum.opnsense.org/index.php?topic=32580.0


Hello,

I set up a OPNsense with 2 WAN interfaces and a gateway group (configured as load balancing) as described in https://docs.opnsense.org/manual/how-tos/multiwan.html.
However, as soon as I activate the firewall rule for policy based routing via the gateway group, I get constant error messages on the hardware console:

arpresolve: can't allocate llinfo for <IP> on igb0
arpresolve: can't allocate llinfo for <IP> on igb1

As far as I can tell the gateway group works, however the hardware console is unusable.
Intresting is that the IP in the error message is always the gateway IP not corresponding to the interface.
For example:

igb0 IP: 86.87.88.90/30
igb0 gateway IP: 86.87.88.89

igb1 IP: 192.168.75.250/24
igb1 gateway IP: 192.168.75.254

arpresolve: can't allocate llinfo for 192.168.75.254 on igb0
arpresolve: can't allocate llinfo for 86.87.88.89 on igb1

Through extensive testing I did find out that the issue has something to to with the Shared forwarding feature under Firewall -> Settings -> Advanced, that allows for traffic shaping while also doing policy based routing.
When I disable it the messages on the hardware console stop.

I did also configure Traffic Shaping rules, but even after disabling them the arpresolve errors occour.
There is no MAC spoofing involved.

Here is the current setup:

OS / Hardware
OPNsense 22.7.11_1-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022
CPU: Intel(R) Pentium(R) Gold G6605 CPU @ 4.30GHz
Mainboard: Supermicro X12STL-IF
Network:
- Onboard: 2x Intel i210 RJ45 1GbE network ports (WAN)
- PCI-E: Mellanox ConnectX-4 Lx with 2x SFP28 25/10/1GbE network ports (LAN)

Interfaces

igb0: WAN_SDSL
- Static IPv4
- IPv6 None
- RFC1918 IP
- Gateway IP in WAN interface subnet

igb1: WAN_DSLHYBRID
- Static IPv4
- IPv6 None
- Public IP
- Gateway IP in WAN interface subnet

Gateways -> Single

WAN_SDSL_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.4.4
- Priority: 254
- Weight: 1

WAN_DSLHYBRID_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.8.8
- Priority: 253
- Weight: 3

Gateways -> Group

WAN_LOADBALANCE
- Tier1: WAN_SDSL_GWv4
- Tier1: WAN_DSLHYBRID_GWv4
- Trigger Level: Member Down
- Pool Options: Default

I also tested different combinatitons of Upstream Gateway, Gateway Priority or even Tiers in the gateway group.
The result is always the same. As soon as traffic is routed via gateway group the arpresolve errors occour.

Does anyone have any ideas to further debug this?

After the Unbound custom options were removed in RC2, I tried to apply my custom configuration through the "new" way described here: https://docs.opnsense.org/manual/unbound.html#advanced-configurations

But after a Unbound service restart via GUI all added configuration files in the directory /var/unbound/etc/ got deleted.

I even tried out the template generation variant, even though I don't use a memory file system for /var in my configuration.
After the command "configctl template reload sampleuser/Unbound" the template custom configuration file gets indeed added, but after another Unbound service restart it gets deleted again.

It seems there is a bug that removes all non default configuration files on Unbound service start.