Menu

Show posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Show posts Menu

Messages - Mann-IT

#1
Hi Franco,

quick follow-up: FreeBSD ports got net/miniupnpd 2.3.11 on Oct 16 (rcmcdonald91), which includes the delete_nat_rule() fix (463ec13). opnsense/ports and 26.7.8 are still on 2.3.9_3.

Any chance to get it into 26.7.9? Happy to test right away - NAT-PMP is still off here as a workaround.

Thanks!
#2
HI,

Mostly servers/VMs, all ZeroTier members. Thanks for the hint, but we'll stay with UPnP for now:

- With PCP/NAT-PMP off the problem is gone: the anchor stays at ~23 rules (= leases), no stalls, 0.3 ms jitter.
- UPnP IGD deletes its rules correctly. Only the NAT-PMP path leaks.
- Static port/EIM outbound NAT would remove the anchor, but gives no measurable gain at 23 rules,
  and changes NAT behaviour for every host on a multi-tenant box.
- Once miniupnpd gets 463ec13 we simply re-enable NAT-PMP.

Grretings Mario
#3
Fair point, thanks Franco. Didn't see that thread.
If ports stays slow: 463ec13 is a one-line PATCHFILES addition on top of 2.3.9 – happy to test a package.
Workaround for others meanwhile: disable PCP/NAT-PMP.
#4
    Hi all,

    I want to share a problem that took us weeks to track down, because it looked exactly like a bad ISP line. The cause turned out to be a bug in miniupnpd as shipped with OPNsense 26.7. It is already fixed upstream, but not yet in OPNsense. If you run
UPnP with PCP/NAT-PMP enabled, you are probably affected to some degree.

Environment
  • OPNsense 26.7.6_3, FreeBSD 15.1-RELEASE-p4
  • miniupnpd-2.3.9_3,1 / os-upnp-1.9_1, UPnP IGD + PCP/NAT-PMP enabled, default deny with allow rules for three internal subnets
  • PPPoE WAN (1000/500 fiber) plus a second WAN (5G), Intel ix/igb NICs, Xeon E3, 32 GB RAM
  • About 60 internal hosts running ZeroTier, which request port mappings via NAT-PMP all the time

Symptoms
  • Packet loss of 1–7 % and jitter of 30–140 ms on the WAN gateway, RTT spikes of up to 2 seconds
  • It builds up slowly over 4–9 days, then is suddenly fine again after a reboot or a PPPoE reconnect
  • Load was low (about 20 % of the line), CPU 90 % idle, all NIC error counters at 0
  • A second OPNsense box on the same ISP network was perfect at the same time → we blamed the line and opened tickets with the ISP...

What gave it away
The second WAN (5G) on the same box showed the same sawtooth pattern in jitter. Pinging a LAN host on the same switch (base RTT 0.04 ms) from the firewall showed stalls of up to 450 ms, at exactly the same moments as on both WANs. Those three paths have only one thing in common: the firewall itself.

Narrowing it down (all read-only):
  • Ping to 127.0.0.1 (lo0, pf skipped): max 1 ms → the network stack itself is fine
  • Scheduler oversleep test (sleep 5 ms in a loop): no stalls → not a general CPU or SMI problem
  • pf ioctl probe (pfctl -t <table> -T test every 40 ms): no blocking → not a pf lock
  • Only the physical NIC paths (ix0, ix1, igb1) stall

Then I sampled the kernel stacks of the NIC taskqueue threads with procstat -kk 0 every 20 ms, while a ping ran in parallel. Whenever the if_io_tqg_* threads were busy, they were almost always here:

pf_match_translation_rule < pf_get_translation < pf_test_rule < pf_test < pf_check_out < pfil_mbuf_fwd < ip_tryforward

That is pf scanning the NAT rules linearly for every new connection.

The cause
# pfctl -sn | head -1
nat-anchor "miniupnpd" all          <-- first rule, evaluated for EVERY new connection

# pfctl -a miniupnpd -sn | grep -c '^nat'
10747                                <-- after 5 days uptime

# wc -l /var/run/miniupnpd.leases
66                                   <-- actual active mappings

The anchor was full of orphaned rules like this one:
nat quick on pppoe0 inet proto udp from 10.0.140.69 port = 26592 to any keep state label "NAT-PMP 26592 UDP" rtable 0 -> x.x.x.x port 26592

and the log showed thousands of these lines per day (12,500–14,500 per day for us):
miniupnpd: could not find nat rule to delete iport=22062 addr=0a008c45

So when a NAT-PMP mapping expires, miniupnpd removes the rdr rule but never the nat rule. Each client accumulated hundreds of dead rules, and the anchor grew by thousands of rules per day. The anchor sits at position 1 of the NAT ruleset, so every new connection of every host has to walk through all of them, including hosts that never use UPnP. This runs inside the NIC taskqueue threads. PPPoE traffic cannot be RSS-hashed and lands entirely on rx queue 0 / one thread. So in bursts of new connections that thread saturates, and everything else queued on that core waits. That explains the stalls on LAN and the second WAN as well.

The "fix" we saw every few days (reboot, PPPoE reconnect) simply restarted miniupnpd, which flushes the anchor.

Root cause in the code (already fixed upstream)
In miniupnpd 2.3.9, pf/obsdrdr.c, delete_nat_rule(), the libpfctl code path queries the rules with PF_PASS. On FreeBSD that is the FILTER ruleset, but the nat rules are created in the NAT ruleset. So the lookup never finds the rule.

Fixed upstream in commit 463ec13 ("miniupnpd/pf: fix the ruleset scanned by delete_nat_rule() with libpfctl"). This is included in miniupnpd 2.3.11 (released 2026-08-11). OPNsense / FreeBSD ports still ship 2.3.9. I have asked for an update on GitHub: (link to your issue)

Result after disabling PCP/NAT-PMP (60 s, ping every 10 ms from the firewall)
                         before                 after
LAN host, max            377 ms, 32 stalls      1 ms, 0 stalls
2nd WAN router, max      235 ms, 20 stalls      17 ms, 0 stalls
WAN gateway (dpinger)    jitter 30-140 ms       16 ms RTT, 0.3 ms jitter, 0 % loss
miniupnpd anchor         >6,000 nat rules       133 rules (= leases x rdr+nat)

How to check if you are affected
pfctl -a miniupnpd -sn | grep -c '^nat'      # compare with the next line
wc -l /var/run/miniupnpd.leases
grep -c "could not find nat rule" /var/log/miniupnpd/latest.log
If the nat count is far above the number of leases, you have the leak. Note: pfctl -a miniupnpd -sn often returns truncated output while miniupnpd is writing to the anchor. Run it a few times and take the maximum.

Workaround
[list=1]
  • Services → UPnP IGD & PCP → uncheck PCP/NAT-PMP → Save. This restarts miniupnpd and flushes the anchor. Mappings created via UPnP IGD are deleted correctly. For us only the NAT-PMP path leaked.
  • Optional safety net: a small watchdog that restarts miniupnpd when the anchor gets too big. Leases are restored from the lease file on restart.

Watchdog script (/usr/local/bin/upnp-watchdog.sh, chmod 755):
#!/bin/sh
# Restart miniupnpd when its nat anchor holds far more rules than active leases
LEASES=/var/run/miniupnpd.leases
STAMP=/var/run/upnp-watchdog.last
pgrep -x miniupnpd > /dev/null || exit 0
nat=0
for i in 1 2 3; do   # listing may be truncated while miniupnpd writes -> take max
  n=$(pfctl -a miniupnpd -sn 2>/dev/null | grep -c '^nat')
  [ "$n" -gt "$nat" ] && nat=$n
done
leases=$(wc -l < $LEASES 2>/dev/null | tr -d ' '); leases=${leases:-0}
limit=$((leases * 2 + 20))
[ "$nat" -le "$limit" ] && exit 0
if [ -f $STAMP ] && [ $(( $(date +%s) - $(cat $STAMP) )) -lt 3600 ]; then exit 0; fi
logger -t upnp-watchdog "nat=$nat leases=$leases limit=$limit -> restarting miniupnpd"
/usr/local/sbin/pluginctl -s miniupnpd restart > /dev/null 2>&1
date +%s > $STAMP

configd action (/usr/local/opnsense/service/conf/actions.d/actions_upnpwatch.conf):
[check]
command:/usr/local/bin/upnp-watchdog.sh
parameters:
type:script
message:UPnP anchor watchdog check
description:UPnP Watchdog: clean up orphaned NAT rules
Then run service configd restart and add a cron job under System → Settings → Cron (command "UPnP Watchdog", every 15 minutes). Note: the script and the action file live outside config.xml, so you have to recreate them after a reinstall.

Takeaways
  • If your line "degrades over days and recovers after a reboot", check the firewall before calling your ISP.
  • A second WAN on the same box that shows the same jitter pattern is a strong hint that the box is the problem.
  • procstat -kk sampling is a very effective poor man's profiler on OPNsense when dtrace isn't loaded.

Hope this saves someone a few weeks. Happy to answer questions.
#5
I do this on 24.7.4

and it help!!!!

root@OPNsense:~ # opnsense-patch 1dba25fed8
Fetched 1dba25fed8 via https://github.com/opnsense/core
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|From 1dba25fed8686f865a9425bd08c04a01075c94e1 Mon Sep 17 00:00:00 2001
|From: Franco Fichtner <franco@opnsense.org>
|Date: Fri, 7 Jun 2024 22:22:02 +0200
|Subject: [PATCH] interfaces: force regeneration of link-local on spoofed MAC;
| closes #4430
|
|(cherry picked from commit a2ac1999f37ee98da22b6edd42c430c8dbb6534b)
|(cherry picked from commit 7669567944ec26ffea088a636482e04b0e9912d6)
|---
| src/etc/inc/interfaces.inc | 35 +++++++++++++++++++++++++++++++----
| 1 file changed, 31 insertions(+), 4 deletions(-)
|
|diff --git a/src/etc/inc/interfaces.inc b/src/etc/inc/interfaces.inc
|index 1b672e7f50..152155dafa 100644
|--- a/src/etc/inc/interfaces.inc
|+++ b/src/etc/inc/interfaces.inc
--------------------------
Patching file etc/inc/interfaces.inc using Plan A...
Reversed (or previously applied) patch detected!  Assuming -R.Hunk #1 succeeded at 2316 (offset 32 lines).
done
All patches have been applied successfully.  Have a nice day.
#6
HI,

sorry I do not unterstand what do to?


Greetings Mario
#7
HI,

after upgrade to 24.7.4 my zerotier connection are death.
Move back to 24.7.3 they are working.
What is wrong?

Greetings Mario
#8
German - Deutsch / APON/XGS-PON SFP Modul
August 06, 2024, 03:08:08 PM
HI,

ich möchte den Glasfaseranschluß direkt mit der OpnSense verbinden also direkt ohne ONT.
Jetzt bräuchet ich APON zum Ende des Jahres XGS-PON.
Hat das jemand schon gemacht? Welche Netzwerkkarte und Modul habt ihr benutzt?

Gruß Mario
#9
Yes auch geschaut und zurückgesetzt, nach 8 Stunden hat die Opnsene die 3cx wieder durchgelassen.
Ohne das ich was gemachte hatte.
#10
Ist alles auf hyper-v dann würde alle VM nicht laufen.
Wie geschrieben wenn ich die Ip ändere kommt die 3cx wieder ins Internet.
Ändere ich es zurück geht nix.
#11
Der FQDN geht ja, DNS geht ja auch. Kann auch im gleichen Segment alles anpingen.
Aber wenn ich die OPNsense anpinge kommt nur 100% lost.
Es scheint so als würde die Opnsense die IP sperren. Mich wundert ja nur warum selbst in der Liveansicht nicht zu sehen ist, kein Ping kein gar nichts.
#12
German - Deutsch / OpnSense sperrt ip Adresse ausgehend
October 23, 2023, 05:36:33 PM
HI,

habe gerade ein Phänomen was absolut verrückt ist.
3cx hinter Opsense Laufzeit mehr als 5 Jahre ohne Problem.
Nun seit heute 10 Uhr etwa kommt die 3cx nicht mehr ins Internet.
DNS geht aber, IP ist auch nicht doppelt belegt. Eingehend geht alles.
Ändere ich von der 192.168.1.3 auf die 192.168.1.6 geht es wieder. Ist auch keine Lösung da ich alle Telefone mal anlangen müsst.
Ping sagt dann auch das die 192.168.1.3 frei ist.
Ändere ich wieder zurück geht wieder nix. Liveansicht sagt auch nix.
Bin gerade extrem ratlos, das gleiche Problem hatte ich mal vor 3 Monaten aber bei 8 Nebenstellen war das mit der IP schneller die Lösung.

Gruß Mario
#13
German - Deutsch / Re: RDS FARM NAT
September 19, 2023, 10:14:35 PM
Den Gedanken hatte ich auch schon. Aber es ins gesamt 3 User die darüber kommen. Alle anderen Intern + Zerotier VPN gehen immer perfekt.

Was mich wundert, ich sehe am RD das er auf Server 1 mich schieben will aber im Livelog der Sense sehe ich er geht auf Server 2 los und das verstehe ich nicht.
#14
German - Deutsch / Re: RDS FARM NAT
September 19, 2023, 08:49:31 PM
Mist zu früh gefreut, war Zufall das er immer auf richtige raus gekommen ist.
Ich stehe gerade völlig auf Schlauch, warum der nicht auf die verschiedenen RDP Server nicht verbindet?
Laut Livelog kommt connection zustande aber es läuft dann in einen Timeout.
#15
German - Deutsch / Re: RDS FARM NAT
September 14, 2023, 09:16:37 PM
Nachtrag, habe es zum laufen gebracht.
In den Alias habe ich in der Hosts statt der IP die davor definierten TS1-3 als Objekt eingefügt und dann ging es.