Messages

Same problem here, but on a CARP pair, and in our case it ended in a kernel panic.

Setup: 26.7.7, two VMs on Hyper-V (2019, SET, hn), CARP + pfsync, NetFlow on LAN/GUEST/WAN.

Since the day the HA pair went live, unknown unicast (a printer MAC that had aged out of the switches) has been bounced between the two nodes with decreasing TTL until it expired. That meant short floods several times an hour, with hn0: RXBUF ack failed during them. Yesterday the master panicked:

Code Select Expand

Fatal trap 12: page fault while in kernel mode
m_copydata+0xfa
hn_encap+0x4c7
hn_xmit+0xedI turned off NetFlow on both nodes and changed nothing else. No floods and no hn errors since.

This is FreeBSD Bug 295415 (netgraph skips the M_PROMISC check, and CARP keeps the NIC promiscuous). The same loop was reported in opnsense/src #279, which was closed as "Timeout." It would be good to reopen that, or at least warn in the GUI when NetFlow is enabled on CARP interfaces. Until then: no NetFlow on CARP interfaces.

Hi again!

26.7.7 and I got a stable guest on Hyper-V again! KUDOS!
26.7.6 the bootloops was fixed but lots of other crashes... 😁

Brgs,

[hv_hid_probe()]

Final update for now:

The panic backtrace points to hv_hid_probe() page fault on cpu3, triggered during device_do_deferred_actions() at boot. This is reproducible with 8 vCPU and occurs regardless of OPNsense configuration, a clean default install with no custom tunables shows the same behaviour.

The test VM without RSS tunables also boot loops after repeated reboots, which rules out RSS as the root cause. It is a race condition, consistent with Monviech's 10%/90% observation.

26.7.4 does not appear to address this.

Would it be appropriate to open a FreeBSD-level issue, or add to the existing https://github.com/opnsense/src/issues/297? Happy to provide the backtrace and any additional data that would help.

[net.isr.maxthreads=-1]

LLM generated:

Compared loader.conf between the two systems. Key differences:

Code Select Expand

                              Boot loop system    Test VM (OK)
net.inet.rss.enabled          1                   not set (=0)
net.inet.rss.bits             3                   not set
net.isr.dispatch              deferred            not set
net.isr.maxthreads            -1                  not set

RSS is the most notable difference. The test VM has none of those set, the production system has RSS enabled with 8 queues and net.isr.maxthreads=-1, which increases the number of active threads at boot and likely widens the race window for the hv_hid_probe page fault.
Next test: disable RSS (net.inet.rss.enabled=0, remove net.inet.rss.bits and net.isr.maxthreads) on the production system and retry with 8 vCPU.

Can't test right now, family is online 😁

But have rebooted the newly installed 27.7.3_11 a couple of times and it also boot looped now. Interesting.
Behavior, it loops a couple of times and then manage to do a complete boot.

Hi again!

Updated again, to _11, this update didn't require a reboot but did one anyway.
Ended up in a bootloop. Tried kern.panic_reboot_wait_time=-1 too, but that didn't work, no stop to see what actually happen.

So, a MacGyver solution was to record the console 🙂

Got this (extracted from video frames):

Code Select Expand

panic: page fault
cpuid = 3

hv_hid_probe() at hv_hid_probe/frame 0xfffffe0070bd8a90
device_probe() at device_probe+0x71
bus_generic_driver_added() at bus_generic_driver_added+0x67
devclass_driver_added() at devclass_driver_added+0x29
device_do_deferred_actions() at device_do_deferred_actions+0x3b
devctl2_ioctl() at devctl2_ioctl+0x211
devfs_ioctl() at devfs_ioctl+0xcb
vn_ioctl() at vn_ioctl+0xc8
kern_ioctl() at kern_ioctl+0x286
sys_ioctl() at sys_ioctl+0x101
amd64_syscall() at amd64_syscall+0x126
fast_syscall_common() at fast_syscall_common+0xf8
To add:

• Looks like it always manage to boot OK when vCPU=1
• vCPU=8 sometimes manage to boot but it can take some time before it succeeds...

Thanks guys for followup.

You want me to test anything out here?

DNS setup:

Unbound: enabled, port 8053, active on LAN and WG interfaces, forwards to Quad9 via DoT.
AdGuard Home (os-adguardhome-maxit): port 53, uses Unbound on 127.0.0.1:8053 as upstream for internal zones.
Dnsmasq: port 53053, DHCP hostname resolution only.

So Unbound is active and running at boot, which fits Franco's description of the nullfs trigger.

Would it add any value if I run the test (mount -r -t nullfs /usr/local/lib/python3.13 /mnt) on the running 26.7.3_8 system? I still have the 26.7.2 snapshot available if a confirmed reproduction on a clean upgrade attempt would be more useful.

Hi again!

Think we maybe have to file this under strange

I set only one vCPU on the VM and ran the upgrade. It booted and works. Changed it back to 8 vCPU. And booted as it should into 26.7.3_8...

Still got a snapshot of 26.7.2 if we should investigate this further. I'm willing to revert and see if the problem can be reproduced.
I tried two times from 26.7.2 without success.

But, I did something different (without thinking about it...). The successful upgrade when I had set it to 1 vCPU was from the console, option 12. The previous two attempts when vCPU was set to 8 was done from UI. That shouldn't make any difference how the upgrade process is done, or?
 

Some more info.

Tried 26.7.3_8, showed the same problem.

Updated another system which went well:
Intel i7-8559U, 3 vCPU, I219-LM, Windows Server 2025

Claudes take on the information provided on the troublesome upgrade:
Both are Hyper-V Gen 2, Windows Server 2025, synthetic NICs (vmbus/hn), static RAM.

Worth noting: the AMD guest ran 26.7.2 (FreeBSD 15.1-p2) stably for weeks with the same 8-vCPU and RSS configuration. The boot loop only appeared with 26.7.3 (p3), which suggests a regression introduced in p2 → p3 rather than a pre-existing Hyper-V SMP issue. This also makes the Issue #297 link less likely to be the root cause, since that was reported against RC1 (p1) and should have manifested earlier.

Next planned test: disable RSS (net.inet.rss.enabled=0) before retrying the upgrade, to isolate whether RSS or vCPU count is the deciding factor.

Can't test this right now, but can give it a shoot on Sunday. That and the vCPU=1, but that should have shown for 26.7 already as I understand it.

Host:
Windows Server 2025 Standard (Build 26100), Hyper-V Generation 2 guest.

Host CPU:
AMD64 Family 23 Model 113 Stepping 0 (Zen 2 / Ryzen 7 3700X, ~3573 MHz). BIOS AMI P4.10 (2021-05-18).

Guest config:
8 vCPU, 5 GB static RAM, 6 synthetic NICs (vmbus/hn), VLANs handled via Hyper-V Access mode per adapter. No SR-IOV assigned to guest.

Relevant tunables active at time of upgrade:

debug.kassert.warn_only = 1
net.inet.rss.enabled = 1
net.inet.rss.bits = 3 (8 RSS queues matching 8 vCPUs)
net.isr.maxthreads = -1

I'm going to check your link, looks interesting. Trying to get a chance to turn it of and change vCPU to one.

Upgraded a Hyper-V VM (gen 2, synthetic NIC) from 26.7.2 to 26.7.3 and hit a restart loop. No panic output captured -- reverted to snapshot before I could gather logs.

Running: Windows Server Hyper-V, OPNsense amd64, VLANs on synthetic adapters (hn0--hn5).

Has anyone else on Hyper-V seen this with 26.7.3?

Was able to get this, it just flashes by... and then restart. Reverted back a snapshot of .2 to get it running again.

Hi again!

Don't know if this is frowned upon or maybe worth much for others, but this solved it for me and may give other an idea how to get it going with WireGuard and hostname.

Some LLM vibe-coding...

This script ensures that all required services are running and that the hostname is resolvable before it triggers a WireGuard restart via pluginctl. It also includes a "pre-flight check" so it doesn't restart anything if the tunnel is already up.

It grabs what it can from the config.conf. A warning is that it assumes that a pingable IP through the tunnel is .1 from one of the .0/24 allowed networks in the peer configuration.

Add:
Put it here, and make an +x on it.

Code Select Expand

root@fwa:~ # cd /usr/local/etc/rc.syshook.d/start
root@fwa:/usr/local/etc/rc.syshook.d/start # ls
10-newwanip             25-syslog               90-carp                 90-openvpn              95-beep
20-freebsd              50-ntopng               90-cron                 90-sysctl               99-wireguard-dns-fix
root@fwa:/usr/local/etc/rc.syshook.d/start # chmod +x 99-wireguard-dns-fix

PEER_NAME: Change the to match your peer name.
REQUIRED_SERVICES: Adjust Services that should be running.

Script:

Code Select Expand

#!/bin/sh

# Header: Dynamic Wireguard Startup Sync for OPNsense 26.1
# Description: Extracts peer data from config.xml and ensures DNS stack is ready.
# Note: This script will "man-guess" that .1 is a reachable gateway address if a .0 network is found.
# Optimization: Checks if the tunnel is already up before performing any actions.
# Logging: Outputs to system log with tag "man-guess-wg" and priority "user.notice".

# --- Configuration ---
PEER_NAME="wg_peer"
REQUIRED_SERVICES="unbound dnsmasq AdGuardHome"
CONFIG_PATH="/conf/config.xml"
LOG_TAG="man-guess-wg"
MAX_RETRIES=20
SLEEP_INTERVAL=5

log_msg() {
    # Output to stdout and OPNsense System Log (General)
    echo "$1"
    logger -p user.notice -t "$LOG_TAG" "$1"
}

# 1. Extract the Endpoint (serveraddress)
CHECK_HOST=$(xmllint --xpath "string(//wireguard/client/clients/client[name='$PEER_NAME']/serveraddress)" $CONFIG_PATH 2>/dev/null)

# 2. Extract the first entry from Allowed IPs (tunneladdress)
RAW_IP=$(xmllint --xpath "string(//wireguard/client/clients/client[name='$PEER_NAME']/tunneladdress)" $CONFIG_PATH 2>/dev/null | cut -d',' -f1 | cut -d'/' -f1)

# Logic: Convert network address (ending in .0) to gateway (.1) for ping test.
# This part is a confident man-guess that .1 is the responding address.
TUNNEL_IP=$(echo "$RAW_IP" | sed 's/\.0$/.1/')

# Validation: Stop if configuration data is missing
if [ -z "$CHECK_HOST" ] || [ -z "$RAW_IP" ]; then
    log_msg "ERROR: Extraction failed for '$PEER_NAME'. Check if the name matches in config.xml."
    exit 1
fi

# --- INITIAL CHECK ---
# If the tunnel is already reachable, we exit immediately to avoid unnecessary restarts.
if ping -c 1 -t 2 "$TUNNEL_IP" > /dev/null 2>&1; then
    log_msg "STATUS: Tunnel to $TUNNEL_IP is already UP. No action taken."
    exit 0
fi

check_process() {
    pgrep -f "$1" > /dev/null
    return $?
}

log_msg "STATUS: Tunnel is DOWN. Initiating sync for peer: $PEER_NAME (Target: $CHECK_HOST)"

for i in $(seq 1 $MAX_RETRIES); do
    SERVICES_READY=true
   
    # Verify that all DNS-related services are running
    for SERVICE in $REQUIRED_SERVICES; do
        if ! check_process "$SERVICE"; then
            SERVICES_READY=false
            break
        fi
    done
   
    if [ "$SERVICES_READY" = true ]; then
        # Check if the dynamic hostname can be resolved yet
        if host "$CHECK_HOST" > /dev/null 2>&1; then
            log_msg "DNS OK: $CHECK_HOST resolved. Restarting Wireguard via pluginctl."

            # Official OPNsense 26.1 method for plugin service management
            /usr/local/sbin/pluginctl -s wireguard restart
           
            # Allow time for handshake and routing table updates
            sleep 15
           
            if ping -c 1 -t 3 "$TUNNEL_IP" > /dev/null 2>&1; then
                log_msg "SUCCESS: Connectivity to $TUNNEL_IP confirmed."
                exit 0
            fi
        fi
    fi
   
    # Log status every 5th attempt to keep the system log clean
    if [ $((i % 5)) -eq 0 ]; then
        log_msg "WAIT: Attempt $i/$MAX_RETRIES - Dependencies or DNS not ready."
    fi
   
    sleep $SLEEP_INTERVAL
done

log_msg "FATAL: Timeout reached. Connectivity could not be verified for $PEER_NAME."
exit 1


[/usr/local/opnsense/scripts/wireguard/wg-service-control.php: The command </usr/bin/wg syncconf 'wg1' '/usr/local/etc/wireguard/wg1.conf'> returned exit code 1 and the output was "Name does not resolve: `my-very-nice-dyn-hostname:55120' Configuration parsing error"]

Hi on my self and others!

Still got this problem it looks like. I resorted to IP-adress, but it would be really nice if it is possible to get it working with a peer hostname instead.
Any way to resolve this?
Make WireGuard wait for DNS-service and maybe be a bit more stubborn on resolving the the name? Make it wait for a while before starting?

from the log:
/usr/local/opnsense/scripts/wireguard/wg-service-control.php: The command </usr/bin/wg syncconf 'wg1' '/usr/local/etc/wireguard/wg1.conf'> returned exit code 1 and the output was "Name does not resolve: `my-very-nice-dyn-hostname:55120' Configuration parsing error"


Brgs,

Awesome!

And such responsiveness here, KUDOS!

I had a go at the source to check what I could find, but with limited knowledge I gave in... Had a small idea to contribute with more than just "the Apply button is moving, please fix" 😁

Keep up the amazing work!

Hi everyone,

To start with, OPNsense is awesome! After years of procrastination, I finally migrated from pfSense to a fully "pimped" OPNsense setup. From DNS redirects, DoH blocking, multi-site WireGuard, 6 VLANs, working forward and reverse lookups. It has been a blast to configure!

What I noticed. It's a "luxury problem":
The Apply button! Hitting a moving target is... 😁
Currently, when the notification banner appears to remind you to apply changes, it causes a layout shift. This means the Apply button moves just as you are about to click it. I think this may improve the experience while, as I've done, work intensively in the UI.

And, some times I had unapplied settings. Maybe a notification at the top that there are lingerings stuff that needs an Apply?

FI: Mainly using Firefox, but same-same behavior in Edge.

Brgs!

Sorry for the noise. Found posts on the subject. Will be removed later down the road...

Brgs,