Messages

We tested RSS on a slow PC Engines APU2 device in combination with an IPsec Site to Site VPN.
With enabled RSS, there are some issues with unbound receiving DNS packets through the VPN tunnel:
- unbound has a DNS override for a specific domain, the IP of the authorative server for that override is set to an IP inside the IPsec P2 remote network.
- The outgoing network interface in unbound is set to the interface, the IPsec P2 local network resides in.

With a packet capture we can see the DNS answer packets arriving at the IPsec P2 local network, but unbound does not see them.

With disabled RSS this setup is working without any issues.

[OS / Hardware]

There seems to be a problem with Load Balancing (2 Gateways on Tier 1) via Gateway Groups and the shared forwarding feature.
As soon as the "Policy Based Routing" Firewall rules with the Load Balancing Gateway Group as a gateway are in place, two things happen:

1. The hardware console is spammed with arpresolve errors:
arpresolve: can't allocate llinfo for <IP> on igb0
arpresolve: can't allocate llinfo for <IP> on igb1

2. There are random(?) timeouts for outgoing traffic:
- First try loading a web page fails
- Second try loading a web page works

When the Gateway Group is is set to Failover, first gateway Tier1 and second gateway Tier 2, there are no problems.
When the shared forwarding feature under Firewall -> Settings -> Advanced ist disabled, there are no problems, too.
But when the feature is disabled, there is no traffic shaping possible for PBR Firewall rules, or at least this is stated in the shared forwarding help text.

This problem also existed in 22.7.11_1, see -> https://forum.opnsense.org/index.php?topic=32374.0 for further information.

Does anybody use the Load Balancing feature of Gateway Groups and can reproduce this?
Or is using shared forwarding and Load Balancing via Gateway Groups mutually exclusive / not supported?

OS / Hardware
OPNsense 23.1.1_2-amd64
FreeBSD 13.1-RELEASE-p6
OpenSSL 1.1.1t 7 Feb 2023
CPU: Intel(R) Pentium(R) Gold G6605 CPU @ 4.30GHz
Mainboard: Supermicro X12STL-IF
Network:
- Onboard: 2x Intel i210 RJ45 1GbE network ports (WAN)
- PCI-E: Mellanox ConnectX-4 Lx with 2x SFP28 25/10/1GbE network ports (LAN)

[Update]

Update

The issue persists after upgrade to 23.1.1, but I was able to narrow it down further.
It only happens when "shared forwarding" is enabled and both gateways in the gateway group are set to "Tier 1".
It seems to me as soon as load balancing is enabled, there are traffic issues when ipfw2 for traffic shaping is enabled.
I also removed all traffic shaping rules, reset the firewall state table and reset the ource tracking to make shure they are not the source of this.

I probably will open another thread in the 23.1 forum about the specific issue.
When I do I willl update this topic.

[Update]

Update

It seems this is not only a cosmetic issue.
When the gateway group is used in firewall rules, clients behind the router experience connection timeouts.
So far this is only reproducible for HTTP/S services in a web browser, but as soon I disable the gateway group the problem goes away.

During the connection timeouts there is no packet loss or high latency on any gateway.
I also tested both gateways as "active" independently from one another.
As soon as the gateway group is not involved anymore it doesn't matter which gateway is active, both are completly stable and produce no connection timeouts on clients.

The next step will be an upgrade to 23.1.1 after it is released.
I will report if this solves the problems.

[Update: Moved to 23.1 Forum -> https://forum.opnsense.org/index.php?topic=32580.0]

Update: Moved to 23.1 Forum -> https://forum.opnsense.org/index.php?topic=32580.0


Hello,

I set up a OPNsense with 2 WAN interfaces and a gateway group (configured as load balancing) as described in https://docs.opnsense.org/manual/how-tos/multiwan.html.
However, as soon as I activate the firewall rule for policy based routing via the gateway group, I get constant error messages on the hardware console:

arpresolve: can't allocate llinfo for <IP> on igb0
arpresolve: can't allocate llinfo for <IP> on igb1

As far as I can tell the gateway group works, however the hardware console is unusable.
Intresting is that the IP in the error message is always the gateway IP not corresponding to the interface.
For example:

igb0 IP: 86.87.88.90/30
igb0 gateway IP: 86.87.88.89

igb1 IP: 192.168.75.250/24
igb1 gateway IP: 192.168.75.254

arpresolve: can't allocate llinfo for 192.168.75.254 on igb0
arpresolve: can't allocate llinfo for 86.87.88.89 on igb1

Through extensive testing I did find out that the issue has something to to with the Shared forwarding feature under Firewall -> Settings -> Advanced, that allows for traffic shaping while also doing policy based routing.
When I disable it the messages on the hardware console stop.

I did also configure Traffic Shaping rules, but even after disabling them the arpresolve errors occour.
There is no MAC spoofing involved.

Here is the current setup:

OS / Hardware
OPNsense 22.7.11_1-amd64
FreeBSD 13.1-RELEASE-p5
OpenSSL 1.1.1s 1 Nov 2022
CPU: Intel(R) Pentium(R) Gold G6605 CPU @ 4.30GHz
Mainboard: Supermicro X12STL-IF
Network:
- Onboard: 2x Intel i210 RJ45 1GbE network ports (WAN)
- PCI-E: Mellanox ConnectX-4 Lx with 2x SFP28 25/10/1GbE network ports (LAN)

Interfaces

igb0: WAN_SDSL
- Static IPv4
- IPv6 None
- RFC1918 IP
- Gateway IP in WAN interface subnet

igb1: WAN_DSLHYBRID
- Static IPv4
- IPv6 None
- Public IP
- Gateway IP in WAN interface subnet

Gateways -> Single

WAN_SDSL_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.4.4
- Priority: 254
- Weight: 1

WAN_DSLHYBRID_GWv4
- Upstream Gateway: Unchecked
- Far Gateway: Unchecked
- Disable Gateway Monitoring: Unchecked
- Monitor IP: 8.8.8.8
- Priority: 253
- Weight: 3

Gateways -> Group

WAN_LOADBALANCE
- Tier1: WAN_SDSL_GWv4
- Tier1: WAN_DSLHYBRID_GWv4
- Trigger Level: Member Down
- Pool Options: Default

I also tested different combinatitons of Upstream Gateway, Gateway Priority or even Tiers in the gateway group.
The result is always the same. As soon as traffic is routed via gateway group the arpresolve errors occour.

Does anyone have any ideas to further debug this?

Oh, it seems I missed that hint in the release notes.
It is working now, Thank You!

May I suggest a versioning in the Sphinx docs at docs.opnsense.org?
That way you could select the development release there and be apprised of all changes that comes with it without having to check out Github for documentation changes.

After the Unbound custom options were removed in RC2, I tried to apply my custom configuration through the "new" way described here: https://docs.opnsense.org/manual/unbound.html#advanced-configurations

But after a Unbound service restart via GUI all added configuration files in the directory /var/unbound/etc/ got deleted.

I even tried out the template generation variant, even though I don't use a memory file system for /var in my configuration.
After the command "configctl template reload sampleuser/Unbound" the template custom configuration file gets indeed added, but after another Unbound service restart it gets deleted again.

It seems there is a bug that removes all non default configuration files on Unbound service start.