Messages

Hi,

I am seeing what looks like a PF/gateway initialization issue after boot on OPNsense 26.7.2_2.

Environment
OPNsense 26.7.2_2

FreeBSD 15.1-RELEASE-p2
amd64

WAN uses an Intel ix interface with a static IPv4 gateway.

The firewall also has several internal interfaces/VLANs with rules that explicitly use the WAN gateway for Internet traffic.

Problem

Shortly after boot, the kernel repeatedly logs:

arpresolve: can't allocate llinfo for 0.0.0.0 on ix0

While this is happening, the routing table itself is correct.

For example:

default            <WAN_GATEWAY>       UGS      ix0

and:

route -n get 0.0.0.0

route to: 0.0.0.0
destination: 0.0.0.0
mask: 0.0.0.0
gateway: <WAN_GATEWAY>
interface: ix0
flags: <UP,GATEWAY,DONE,STATIC>

The currently loaded PF rules are also correct and contain:

route-to (ix0 <WAN_GATEWAY>)

There are no active rules containing:

route-to (ix0 0.0.0.0)

However, some PF states created shortly after boot contain:

route-to: 0.0.0.0@ix0

For example:

all udp <INTERNAL_HOST_A>:514 <- <INTERNAL_HOST_B>:514
    route-to: 0.0.0.0@ix0

I also found affected states for normal Internet traffic, so this is not limited to one internal destination.

Examples included HTTPS traffic to public IP addresses such as:

1.1.1.1:443
8.8.8.8:443

In the diagnostic capture there were several dozen states containing:

route-to: 0.0.0.0@ix0
Timing

The bad states appear very early after boot.

In one capture, system uptime was only a few minutes and the affected PF states were almost the same age, indicating they had been created roughly during the first minute after startup.

The arpresolve errors started afterward and continued while those states existed.

Test performed

Once the firewall was fully booted, I flushed the state table:

pfctl -F states

No firewall rules, aliases, gateways, NAT configuration, or interfaces were changed.

Immediately afterward:

pfctl -ss -vv | grep 'route-to: 0\.0\.0\.0@ix0'

returned no results.

New states were then recreated automatically by normal traffic and all of them used the correct gateway:

route-to: <WAN_GATEWAY>@ix0

For example:

route-to: <WAN_GATEWAY>@ix0
route-to: <WAN_GATEWAY>@ix0
route-to: <WAN_GATEWAY>@ix0

The kernel arpresolve messages also stopped. The last recorded message remained at the same kernel uptime timestamp even several minutes after the state flush.

I suspect there may be a boot-time ordering/race condition where some policy-routed states are created before the WAN gateway information is fully available to PF.

Has anyone else seen PF states being created with route-to: 0.0.0.0 during early boot on 26.7.x?

Hi,

Thanks for your reply but I think I resolved it. After many hours of tshooting and reconfig here are some "lessons learned" for me that might help:

1. Disable any ISP/IDS services. It was CrowdSec that was triggering that "Permission denied".
2. Make sure the interface naming/assignment is exactly as it was prior to upgrade. I had to create dummy vlans so that opt9 would be assigned to my desired interface so that things would start to work again. (maybe there are better ways but hey, it worked!)
3. Double check fw rules, they might seem fine but a Gateway in a random lan/vlan might be set to "none" and the weirdness start.
4. WireGuard should be called WierdGuard, why it worked sometimes and sometimes not, why sometimes it was using my dns and sometimes none at all or why every time I rebooted opnsense something got fixed and something else broke, I could not figure it out. 

Seems like network goblins keep following me even after i leave the office.

Maybe these are common knowledge for you but for me it was a nice learning session. As of now everything seems to be back in order.

regarding "Notice kernel <7>[4458] arpresolve: can't allocate llinfo for 192.168.3.5 on ix0" it might have been related to some misconfigured FW rule that made WAN believe that client was connected to it in L2, but did not have time to investigate further as i haven't noticed it anymore in the logs (FW rule tinkering might have fixed it).

Hi there,

After upgrading to 26.1.6 from 26.1.5 i noticed "Notice kernel <7>[4458] arpresolve: can't allocate llinfo for 192.168.3.5 on ix0" (ix0 is my wan interface and 192.168.3.5 is a client in vlan3), resetting the firewall state table cleared these errors.

Meanwhile there were other issues also: HAproxy was down, re0 interface that is connected directly to a client was down, when trying to ping it I got "ping: sendto: Permission denied".

At this point (after hours of tshooting :) ) i thought OPNsense is broken and did a fresh install, loaded the config file but GUI says check interfaces (I know re0 is known to cause issues so I removed it) removed the the <interface> part in the config file, clean installed the os again, loaded the modified config file and assigned the interfaces again.

I installed the re0 nic again, assigned it, and waited for config to finish (plugins to be isntalled and so on) and voila its broken again, the same issue appeared. Also in Gateways the IP is different from the one the WAN interface has received the the ISP dhcp.

What am i missing here?

I'd appreciate some help.

Thanks!