OPNsense Forum

Hi all,

I want to share a problem that took us weeks to track down, because it looked exactly like a bad ISP line. The cause turned out to be a bug in miniupnpd as shipped with OPNsense 26.7. It is already fixed upstream, but not yet in OPNsense. If you run

UPnP with PCP/NAT-PMP enabled, you are probably affected to some degree.

Environment

• OPNsense 26.7.6_3, FreeBSD 15.1-RELEASE-p4
• miniupnpd-2.3.9_3,1 / os-upnp-1.9_1, UPnP IGD + PCP/NAT-PMP enabled, default deny with allow rules for three internal subnets
• PPPoE WAN (1000/500 fiber) plus a second WAN (5G), Intel ix/igb NICs, Xeon E3, 32 GB RAM
• About 60 internal hosts running ZeroTier, which request port mappings via NAT-PMP all the time

Symptoms

• Packet loss of 1–7 % and jitter of 30–140 ms on the WAN gateway, RTT spikes of up to 2 seconds
• It builds up slowly over 4–9 days, then is suddenly fine again after a reboot or a PPPoE reconnect
• Load was low (about 20 % of the line), CPU 90 % idle, all NIC error counters at 0
• A second OPNsense box on the same ISP network was perfect at the same time → we blamed the line and opened tickets with the ISP...

What gave it away
The second WAN (5G) on the same box showed the same sawtooth pattern in jitter. Pinging a LAN host on the same switch (base RTT 0.04 ms) from the firewall showed stalls of up to 450 ms, at exactly the same moments as on both WANs. Those three paths have only one thing in common: the firewall itself.

Narrowing it down (all read-only):

• Ping to 127.0.0.1 (lo0, pf skipped): max 1 ms → the network stack itself is fine
• Scheduler oversleep test (sleep 5 ms in a loop): no stalls → not a general CPU or SMI problem
• pf ioctl probe (pfctl -t <table> -T test every 40 ms): no blocking → not a pf lock
• Only the physical NIC paths (ix0, ix1, igb1) stall

Then I sampled the kernel stacks of the NIC taskqueue threads with procstat -kk 0 every 20 ms, while a ping ran in parallel. Whenever the if_io_tqg_* threads were busy, they were almost always here:

pf_match_translation_rule < pf_get_translation < pf_test_rule < pf_test < pf_check_out < pfil_mbuf_fwd < ip_tryforward

That is pf scanning the NAT rules linearly for every new connection.

The cause
# pfctl -sn | head -1
nat-anchor "miniupnpd" all          <-- first rule, evaluated for EVERY new connection

# pfctl -a miniupnpd -sn | grep -c '^nat'
10747                                <-- after 5 days uptime

# wc -l /var/run/miniupnpd.leases
66                                   <-- actual active mappings

The anchor was full of orphaned rules like this one:
nat quick on pppoe0 inet proto udp from 10.0.140.69 port = 26592 to any keep state label "NAT-PMP 26592 UDP" rtable 0 -> x.x.x.x port 26592

and the log showed thousands of these lines per day (12,500–14,500 per day for us):
miniupnpd: could not find nat rule to delete iport=22062 addr=0a008c45

So when a NAT-PMP mapping expires, miniupnpd removes the rdr rule but never the nat rule. Each client accumulated hundreds of dead rules, and the anchor grew by thousands of rules per day. The anchor sits at position 1 of the NAT ruleset, so every new connection of every host has to walk through all of them, including hosts that never use UPnP. This runs inside the NIC taskqueue threads. PPPoE traffic cannot be RSS-hashed and lands entirely on rx queue 0 / one thread. So in bursts of new connections that thread saturates, and everything else queued on that core waits. That explains the stalls on LAN and the second WAN as well.

The "fix" we saw every few days (reboot, PPPoE reconnect) simply restarted miniupnpd, which flushes the anchor.

Root cause in the code (already fixed upstream)
In miniupnpd 2.3.9, pf/obsdrdr.c, delete_nat_rule(), the libpfctl code path queries the rules with PF_PASS. On FreeBSD that is the FILTER ruleset, but the nat rules are created in the NAT ruleset. So the lookup never finds the rule.

Fixed upstream in commit 463ec13 (https://github.com/miniupnp/miniupnp/commit/463ec13a4b2f0a7549b2f57482e560c47a936348) ("miniupnpd/pf: fix the ruleset scanned by delete_nat_rule() with libpfctl"). This is included in miniupnpd 2.3.11 (released 2026-08-11). OPNsense / FreeBSD ports still ship 2.3.9. I have asked for an update on GitHub: (link to your issue)

Result after disabling PCP/NAT-PMP (60 s, ping every 10 ms from the firewall)
                         before                 after
LAN host, max            377 ms, 32 stalls      1 ms, 0 stalls
2nd WAN router, max      235 ms, 20 stalls      17 ms, 0 stalls
WAN gateway (dpinger)    jitter 30-140 ms       16 ms RTT, 0.3 ms jitter, 0 % loss
miniupnpd anchor         >6,000 nat rules       133 rules (= leases x rdr+nat)

How to check if you are affected
pfctl -a miniupnpd -sn | grep -c '^nat'      # compare with the next line
wc -l /var/run/miniupnpd.leases
grep -c "could not find nat rule" /var/log/miniupnpd/latest.log
If the nat count is far above the number of leases, you have the leak. Note: pfctl -a miniupnpd -sn often returns truncated output while miniupnpd is writing to the anchor. Run it a few times and take the maximum.

Workaround
[list=1]

• Services → UPnP IGD & PCP → uncheck PCP/NAT-PMP → Save. This restarts miniupnpd and flushes the anchor. Mappings created via UPnP IGD are deleted correctly. For us only the NAT-PMP path leaked.
• Optional safety net: a small watchdog that restarts miniupnpd when the anchor gets too big. Leases are restored from the lease file on restart.

Watchdog script (/usr/local/bin/upnp-watchdog.sh, chmod 755):
#!/bin/sh
# Restart miniupnpd when its nat anchor holds far more rules than active leases
LEASES=/var/run/miniupnpd.leases
STAMP=/var/run/upnp-watchdog.last
pgrep -x miniupnpd > /dev/null || exit 0
nat=0
for i in 1 2 3; do   # listing may be truncated while miniupnpd writes -> take max
  n=$(pfctl -a miniupnpd -sn 2>/dev/null | grep -c '^nat')
  [ "$n" -gt "$nat" ] && nat=$n
done
leases=$(wc -l < $LEASES 2>/dev/null | tr -d ' '); leases=${leases:-0}
limit=$((leases * 2 + 20))
[ "$nat" -le "$limit" ] && exit 0
if [ -f $STAMP ] && [ $(( $(date +%s) - $(cat $STAMP) )) -lt 3600 ]; then exit 0; fi
logger -t upnp-watchdog "nat=$nat leases=$leases limit=$limit -> restarting miniupnpd"
/usr/local/sbin/pluginctl -s miniupnpd restart > /dev/null 2>&1
date +%s > $STAMP

configd action (/usr/local/opnsense/service/conf/actions.d/actions_upnpwatch.conf):
[check]
command:/usr/local/bin/upnp-watchdog.sh
parameters:
type:script
message:UPnP anchor watchdog check
description:UPnP Watchdog: clean up orphaned NAT rules
Then run service configd restart and add a cron job under System → Settings → Cron (command "UPnP Watchdog", every 15 minutes). Note: the script and the action file live outside config.xml, so you have to recreate them after a reinstall.

Takeaways

• If your line "degrades over days and recovers after a reboot", check the firewall before calling your ISP.
• A second WAN on the same box that shows the same jitter pattern is a strong hint that the box is the problem.
• procstat -kk sampling is a very effective poor man's profiler on OPNsense when dtrace isn't loaded.

Hope this saves someone a few weeks. Happy to answer questions.

Thank god for AI.  In the past someone would have simply asked

a) to include "Fixed upstream in commit 463ec13", or
b) to ask why version 2.3.11 wasn't in OPNsense yet.

Since a) is new we could have done that but since b) makes more sense the answer was already mentioned here:

https://forum.opnsense.org/index.php?topic=52419.msg275754#msg275754

Don't take this wrong way. I just think walls of text get in the way of human communication and effectiveness.


Cheers,
Franco

Fair point, thanks Franco. Didn't see that thread.
If ports stays slow: 463ec13 is a one-line PATCHFILES addition on top of 2.3.9 – happy to test a package.
Workaround for others meanwhile: disable PCP/NAT-PMP.

I'm not in the office next week. Let's try to pick it up when I'm back.


Cheers,
Franco