OPNsense Forum

English Forums => 25.7, 25.10 Series => Topic started by: hina on December 09, 2025, 04:30:19 PM

Title: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 09, 2025, 04:30:19 PM
Hi all,

I have this issue where Dnsmasq will stop after some time, and checking dmesg gives me the following.
Could there be something I can change to fix this?

System specs:
Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz (6 cores, 6 threads)
16GB ram
Chelsio T6225-CR

[49244] 764.926056 [4335] netmap_transmit           cc1 full hwcur 1951 hwtail 1951 qlen 2047
[49244] 764.933502 [4335] netmap_transmit           cc1 full hwcur 1951 hwtail 1951 qlen 2047
[278270] 790.680905 [4335] netmap_transmit           cc1 full hwcur 575 hwtail 575 qlen 2047
[278270] 790.688179 [4335] netmap_transmit           cc1 full hwcur 575 hwtail 575 qlen 2047
[548912] 433.149610 [4335] netmap_transmit           cc1 full hwcur 1733 hwtail 338 qlen 1394
[548912] 433.156943 [4335] netmap_transmit           cc1 full hwcur 1733 hwtail 338 qlen 1394
[548921] 441.701315 [4335] netmap_transmit           cc1 full hwcur 479 hwtail 1367 qlen 1159
[548921] 441.708648 [4335] netmap_transmit           cc1 full hwcur 479 hwtail 1367 qlen 1159
[548949] 469.313550 [4335] netmap_transmit           cc1 full hwcur 834 hwtail 1484 qlen 1397
[548949] 469.320885 [4335] netmap_transmit           cc1 full hwcur 834 hwtail 1484 qlen 1397
[717560] 080.906436 [4335] netmap_transmit           cc1 full hwcur 335 hwtail 520 qlen 1862
[717560] 080.913694 [4335] netmap_transmit           cc1 full hwcur 335 hwtail 520 qlen 1862
[748371] 891.599329 [4335] netmap_transmit           cc1 full hwcur 23 hwtail 1151 qlen 919
[748371] 891.608863 [4335] netmap_transmit           cc1 full hwcur 1151 hwtail 22 qlen 1128
[852487] swap_pager: out of swap space
[852487] swp_pager_getswapspace(8): failed
[856087] swp_pager_getswapspace(27): failed
[859687] swp_pager_getswapspace(8): failed
[874085] swap_pager: out of swap space
[874085] swp_pager_getswapspace(31): failed
[888485] swp_pager_getswapspace(20): failed
[892087] swp_pager_getswapspace(30): failed
[895687] swap_pager: out of swap space
[895687] swp_pager_getswapspace(8): failed
[913687] swp_pager_getswapspace(9): failed
[917284] swp_pager_getswapspace(32): failed
[924487] swp_pager_getswapspace(10): failed
[925199] swp_pager_getswapspace(28): failed
[938887] swp_pager_getswapspace(23): failed
[942484] swp_pager_getswapspace(17): failed
[942487] swap_pager: out of swap space
[942487] swp_pager_getswapspace(11): failed
[946085] swp_pager_getswapspace(5): failed
[946085] swp_pager_getswapspace(17): failed
[946087] swap_pager: out of swap space
[946087] swp_pager_getswapspace(32): failed
[949684] swp_pager_getswapspace(30): failed
[964086] swp_pager_getswapspace(32): failed
[971281] swp_pager_getswapspace(17): failed
[971285] swap_pager: out of swap space
[971285] swp_pager_getswapspace(19): failed
[971285] swp_pager_getswapspace(8): failed
[971289] pid 33489 (dnsmasq), jid 0, uid 65534, was killed: failed to reclaim memory
[1008339] arp: 192.168.1.80 moved from 46:f5:ef:xx:xx:xx to 44:6f:f8:xx:xx:xx on cc1
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Monviech (Cedrik) on December 09, 2025, 04:51:40 PM
Which process did run away with all RAM and SWAP space?

You can check via when the RAM is starting to look full. But in hindsight you might not find out which process it was.

ps -faxdvvv

You would need a monitoring system to see RAM usage increase over time and store it if you don't want to watch it all day long.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 09, 2025, 05:47:13 PM
Looks like eastpect (zenarmour) and java, currently the ram is sitting at 51.16% usage (8GB used, 3GB ARC)
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 09, 2025, 05:49:29 PM
I'll try going to 25.7.9 too
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Patrick M. Hausen on December 09, 2025, 05:50:33 PM
Quote from: hina on December 09, 2025, 05:47:13 PMjava

Are you running Elasticsearch locally on your OPNsense?
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 09, 2025, 05:55:53 PM
Yes, I'm running elasticsearch locally
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Patrick M. Hausen on December 09, 2025, 06:16:29 PM
Elasticsearch is a known memory hog and probably the culprit. The default configuration is 32 G fixed allocation. I don't know if ZA changes anything about the ES setup.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 10, 2025, 03:01:08 AM
Had a similar issue today.

[759718] swap_pager: out of swap space
[759718] swp_pager_getswapspace(7): failed
[760739] pid 88184 (dnsmasq), jid 0, uid 65534, was killed: failed to reclaim memory

I restarted dnsmasq - right now ooks like it's currently consuming > 1G memory.  Surprised it's that large.  Any idea if that footprint for dnsmasq is normal? I just switched over from ISC dhcp yesterday so don't need to induce instability ...

last pid: 63926;  load averages:  0.62,  0.46,  0.40                                            up 9+01:02:06  21:00:08
95 processes:  2 running, 93 sleeping
CPU: 16.7% user,  0.0% nice,  1.0% system,  0.1% interrupt, 82.2% idle
Mem: 453M Active, 1960M Inact, 10M Laundry, 2537M Wired, 104K Buf, 2895M Free
ARC: 1252M Total, 308M MFU, 847M MRU, 26M Anon, 7437K Header, 63M Other
     1077M Compressed, 3074M Uncompressed, 2.86:1 Ratio
Swap: 8192M Total, 338M Used, 7854M Free, 4% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 2049 nobody        1  20    0  1194M  1022M select   3   4:23   0.01% dnsmasq
25353 root          5  20    0   789M   535M kqread   0  55:23   0.00% python3.11
20699 unbound       4  20    0   754M   405M kqread   0  16:43   0.04% unbound
32880 root          1  20    0    89M    40M nanslp   2  95:37   0.75% php
38754 root          1  21    0    67M    39M accept   3   0:00   0.00% php-cgi
38980 root          1  24    0    66M    38M accept   1   0:01   0.00% php-cgi
39173 root          1  20    0    63M    36M accept   2   0:01   0.00% php-cgi
34530 root          1  20    0   132M    36M accept   3   0:49   0.00% python3.11
94077 root         12  20    0  1322M    35M uwait    2   1:49   0.00% tailscaled
85867 root          1  20    0    66M    35M accept   1   0:00   0.00% php-cgi
40453 root          1  47    0    60M    34M CPU2     2 484:29  64.90% python3.11
85405 root          1  20    0    61M    33M accept   0   0:00   0.00% php-cgi
85656 root          1  20    0    57M    30M accept   3   0:00   0.00% php-cgi
39215 root          1  66    0    46M    29M nanslp   1   0:00   0.00% python3.11
85460 root          1  26    0    57M    29M accept   2   0:00   0.00% php-cgi
68339 root          1  26    0    70M    28M accept   3   0:01   0.00% php-cgi
69129 root          1  20    0    66M    25M accept   3   0:01   0.00% php-cgi
39250 root          1  29    0    53M    24M accept   3   0:00   0.00% php-cgi
39218 root          1  28    0    53M    24M accept   1   0:00   0.00% php-cgi
38386 root          1  31    0    53M    24M wait     3   0:00   0.00% php-cgi
16645 root          1  20    0    66M    24M accept   2   0:01   0.00% php-cgi
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 11, 2025, 03:50:15 AM
For sure having an issue with dnsmasq.  Restarted the process not 24 hours ago, and its memory consumption continues to balloon.

The process has a 9.5gb memory size, 5.4gb resident.

We have some kind of a memory leak.  About to restart the process - when (not if) it crashes my entire network takes a nosedive with no DHCP or local zone DNS service.

Any suggestions or ideas?

Opnsense 25.7.9

edit: Just restarted the process; now the process size is 17MB; resident less than 5MB.  This is a lot more in line with what I'd expect.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 11, 2025, 04:02:18 AM
The restarted process has been running for 12 minutes .. memory size has ballooned to 98MB already (67MB resident) ...

Yikes.

Don't see any errors in the dnsmasq log - just the usual DHCPREQUEST/DHCPACK/RTR-SOLICIT/RTR-ADVERT etc.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Monviech (Cedrik) on December 11, 2025, 05:59:39 AM
Thats really weird mine stays at around below 10mb RAM used.

Here:
40886 nobody        1  20    0    17M  4920K select   2  36:51   0.00% dnsmasq

What kinda configuration do you run with it? If you share maybe we can see something.

/usr/local/etc/dnsmasq.conf

Also how many leases in total?
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 11, 2025, 04:27:13 PM
Good morning!  Until I can figure this out I setup a cron job to restart the dnsmasq process hourly so the network doesn't fold in on itself. :-)

I've got somewhere around 120 leases or so; it's not extraordinary by any means.

As soon as I'm back at my desk, I'll post my dnsmasq.conf.  Nothing extraordinary or terribly unusual.  I do have DNS listening on port 5353, as I'm using Unbound as the primary DNS.  (Unbound forwards my local zones to dnsmasq) - but very common config.

Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Monviech (Cedrik) on December 11, 2025, 04:31:00 PM
Hey there, thanks. Just want to see if anything catches my eye.

There seems to be a new dnsmasq RC in the making which also has a memory leak related bug fixed. Unsure if that's the one thats interesting for us:
https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=5846f749e5d878b6b5f7c20f6975bc96b95e4aae

https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=summary
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: franco on December 11, 2025, 04:38:56 PM
To be a bit more clear: we'll try to see if 2.92 RC3 behaves normally memory leaks aside and provide a test package here if it looks ok some time next week.


Cheers,
Franco
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 11, 2025, 06:26:17 PM
Some extra data points on my current system

/usr/local/etc/dnsmasq.conf
# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#

rebind-localhost-ok
stop-dns-rebind

port=53053

# If you want dnsmasq to listen for DHCP and DNS requests only on
# specified interfaces (and the loopback) give the name of the
# interface (eg eth0) here.
# Repeat the line for more than one interface.
interface=cc1,cc0,wg0



dhcp-fqdn
domain=redacted.dev
# This tells dnsmasq that a domain is local and it may answer queries from /etc/hosts
# or DHCP but should never forward queries on that domain to any upstream servers.
local=/redacted.dev/

dhcp-authoritative


enable-ra





# Never forward addresses in the non-routed address spaces.
bogus-priv

server=/vpn.redacted.dev/9.9.9.9
rebind-domain-ok=/vpn.redacted.dev/








# host entries flushed via dnsmasq_watcher.py [isc] and a dump of the static reservations
addn-hosts=/var/etc/dnsmasq-hosts
addn-hosts=/var/etc/dnsmasq-leases

dns-forward-max=5000
cache-size=10000
local-ttl=1

conf-dir=/usr/local/etc/dnsmasq.conf.d,*.conf

dhcp-range=tag:cc1,192.168.1.20,192.168.1.254,255.255.255.0,86400

domain=redacted.dev,cc1
dhcp-range=tag:cc1,::,::ffff:ffff:ffff:ffff,constructor:cc1,slaac,ra-names,64,86400

domain=redacted.dev,cc1
ra-param=cc1,60,1200


dhcp-host=id:01:ff:ff:ff:ff:ff:ff,ff:ff:ff:ff:ff:ff,192.168.1.10,[::10],sx3206hpp
dhcp-host=id:01:ee:ee:ee:ee:ee:ee,ee:ee:ee:ee:ee:ee,192.168.1.11,[::11],sx3008f

dhcp-option=tag:cc1,option6:23,[::]
dhcp-option=42,0.0.0.0
dhcp-option=tag:cc1,option6:23,[fc00::ffff:ffff:ffff:ffff]
dhcp-option=option6:56,[fc00::ffff:ffff:ffff:ffff]

# default dns mapped to this server (0.0.0.0)
dhcp-option=6,0.0.0.0


no-ident

top:
last pid: 91378;  load averages:  0.27,  0.25,  0.24                                           up 14+15:07:47  01:19:47
88 processes:  1 running, 87 sleeping
CPU:  0.6% user,  0.0% nice,  1.0% system,  0.0% interrupt, 98.4% idle
Mem: 3285M Active, 3868M Inact, 1071M Laundry, 6049M Wired, 236K Buf, 1420M Free
ARC: 2832M Total, 821M MFU, 1754M MRU, 1034K Anon, 33M Header, 222M Other
     2369M Compressed, 6794M Uncompressed, 2.87:1 Ratio
Swap: 8192M Total, 1802M Used, 6389M Free, 22% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
93702 root         13  20  -20    15G  1278M nanslp   1  26.8H   4.95% eastpect
38891 unbound       6  20    0  1400M  1136M kqread   4   0:16   0.77% unbound
41424 root          1  20    0    79M    35M nanslp   4 125:38   0.65% php
  648 elasticsea  102  68    0  8027M  3253M uwait    3 186:11   0.61% java
 6572 root         14  68  -20  1268M    40M uwait    2  10:44   0.45% ipdrstreamer
 5523 root          3  20    0    49M    12M kqread   5  19:28   0.11% syslog-ng
28181 nobody        1  20    0  2414M  2194M select   5   2:40   0.09% dnsmasq
85048 root          1  20    0    15M  3452K CPU4     4   0:00   0.08% top
51227 root          1  20    0    13M  1324K select   3   4:26   0.06% powerd
41303 root          1  20    0    14M  2332K bpf      2   1:19   0.05% filterlog
31704 _lldpd        1  20    0    87M  1632K kqread   4   2:40   0.03% lldpd
60850 root          1  20    0    28M    14M select   0   0:19   0.01% python3.11
60766 root          1  20    0    27M    13M select   3   0:18   0.01% python3.11
39442 root          7  20    0   276M   179M kqread   2   0:31   0.01% python3.11
37372 root          4  68    0    14M  2328K uwait    3   0:14   0.01% dpinger
80470 root          4  68    0    14M  1520K uwait    4   1:38   0.01% dpinger
94062 _flowd        1  20    0    13M  1700K select   2   1:01   0.01% flowd
78208 root          1  20    0    53M    32M nanslp   1 263:22   0.01% python3.11
16355 root          1  20    0    14M  1652K select   1   0:43   0.01% rtsold
82496 root          1  20    0    20M  7992K select   4   0:00   0.01% sshd-session
49305 root         17  68    0    98M    13M sigwai   3   0:28   0.01% charon

ps -faxdvvv
  PID STAT         TIME  SL  RE  PAGEIN      VSZ     RSS LIM  TSIZ  %CPU %MEM COMMAND
    0 DLs      59:45.91   1 127       0        0    2176   -     0   0.0  0.0 [kernel]
    1 ILs       0:00.11 127 127      36    12324     740   -   652   0.0  0.0 - /sbin/init
88920 S<        0:33.84   0 127      23    37428    8108   - 11968   0.0  0.0 |-- /usr/local/zenarmor//bin/eastpect -D
 6555 I<        0:00.00 127 127      11    37428    8344   - 11968   0.0  0.1 | |-- eastpect: Eastpect Streamer Instanc
 6572 I<       10:45.19 127 127     320  1298168   41252   - 13044   0.0  0.2 | | `-- ipdrstreamer /usr/local/zenarmor/
93702 S<     1607:46.27   0 127   98930 15861768 1308484   - 11968  41.8  7.9 | `-- eastpect: Eastpect Instance 0 (east
38891 Ss        0:16.97   5 127       0  1434064 1163380   -  1028   0.0  7.0 |-- /usr/local/sbin/unbound -c /var/unbou
20345 Is        0:00.03 127 127      68    20316    5712   -   312   0.0  0.0 |-- sshd: /usr/local/sbin/sshd [listener]
82287 Is        0:00.02 127 127       0    20868    7692   -   712   0.0  0.0 | `-- sshd-session: root [priv] (sshd-ses
82496 S         0:00.03   0 127       0    20868    7992   -   712   0.0  0.0 |   `-- sshd-session: root@pts/0 (sshd-se
82599 Is        0:00.00 127 127       0    14312    2596   -   120   0.0  0.0 |     `-- /bin/sh /usr/local/sbin/opnsens
84876 S         0:00.02   0 127       0    14792    3424   -   324   0.0  0.0 |       `-- /bin/csh
82883 R+        0:00.00 127   0       0    14384    2764   -    24   0.0  0.0 |         `-- ps -faxdvvv
28181 S         2:40.19   0 127       0  2471776 2249864   -   312   0.0 13.6 |-- /usr/local/sbin/dnsmasq -x /var/run/d
79321 S         0:01.56   1 127      92    26684    2488   -   220   0.0  0.0 |-- /usr/local/sbin/chronyd -f /usr/local
79399 I         0:00.09 127 127       0    22200     348   -   220   0.0  0.0 | `-- /usr/local/sbin/chronyd -f /usr/loc
94896 S         0:06.28   0 127       0    23852   10128   -   220   0.0  0.1 |-- /usr/local/sbin/lighttpd -f /usr/loca
95033 Is        0:00.04 127 127       0   189040   30428   -  2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
96080 I         0:00.63 127 127       0   197892   37992   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96082 I         0:00.56 127 127       0   197380   37368   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96424 I         0:00.00 127 127       0   189040   30428   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96782 I         0:00.37 127 127       0   192688   37136   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
97148 I         0:00.00 127 127       0   189040   30428   -  2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
95159 Is        0:00.04 127 127       0   189040   30412   -  2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
96276 I         0:00.63 127 127       0   192688   37432   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96644 I         0:00.64 127 127       0   192688   37248   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96869 I         0:00.63 127 127       0   201012   37724   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
97335 I         0:00.00 127 127       0   189040   30416   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
97568 I         0:00.00 127 127       0   189040   30416   -  2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
95208 Is        0:00.04 127 127       0   189040   30428   -  2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
30187 I         0:00.62 127 127       0   197252   37536   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
60132 I         0:00.62 127 127       0   197252   37480   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
96858 I         0:00.00 127 127       0   189040   30432   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
97268 I         0:00.00 127 127       0   189040   30432   -  2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
97419 I         0:00.00 127 127       0   189040   30432   -  2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
95284 Is        0:00.04 127 127       0   189040   30420   -  2928   0.0  0.2 | `-- /usr/local/bin/php-cgi
24696 I         0:00.40 127 127       0   196784   37712   -  2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
69250 I         0:00.59 127 127       0   196916   37620   -  2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
83937 I         0:00.25 127 127       0   196916   37412   -  2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
96325 I         0:00.01 127 127       0   192688   34200   -  2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
97597 I         0:00.00 127 127       0   189040   30420   -  2928   0.0  0.2 |   `-- /usr/local/bin/php-cgi
96675 S         0:28.55   0 127       0    15656    1892   -   220   0.0  0.0 |-- /usr/local/sbin/lighttpd -f /var/etc/
11809 Is        0:00.03 127 127       2    14076    1648   -    64   0.0  0.0 |-- dhclient: system.syslog (dhclient)
12798 Is        0:00.10 127 127       8    14076    1696   -    64   0.0  0.0 |-- dhclient: cc0 [priv] (dhclient)
15763 SCs       0:08.41  11 127      62    14080    1832   -    64   0.0  0.0 |-- dhclient: cc0 (dhclient)
31364 Is        0:00.55  54 127       1    38832     884   -   108   0.0  0.0 |-- lldpd: monitor. (lldpd)
31704 S         2:40.40  10 127      21    89008    1632   -   108   0.0  0.0 | `-- lldpd: no neighbor. (lldpd)
44021 Ss        3:32.87   2 127       0    14120    2712   -   116   0.0  0.0 |-- /usr/local/sbin/miniupnpd -f /var/etc
16651 Is        0:00.23 127 127     100    13800    1748   -    96   0.0  0.0 |-- /usr/local/sbin/dhcp6c -c /var/etc/dh
94031 Is        0:00.02 127 127      38    13748     264   -    76   0.0  0.0 |-- flowd: monitor (flowd)
94062 Ss        1:01.24   0 127      23    13748    1700   -    76   0.0  0.0 | `-- flowd: net (flowd)
 1088 Ss        0:00.47  16 127      47    15340    2944   -    36   0.0  0.0 |-- /sbin/devd
16017 SCs       0:38.79  11 127      26    13924     916   -    28   0.0  0.0 |-- /usr/sbin/rtsold -aiu -p /var/run/rts
16144 Is        0:00.00 127 127       0    13924    1428   -    28   0.0  0.0 |-- rtsold: rtsold.llflags (rtsold)
16233 Is        0:00.00 127 127       0    13924    1428   -    28   0.0  0.0 |-- rtsold: rtsold.script (rtsold)
16339 Is        0:00.00 127 127       0    13924    1428   -    28   0.0  0.0 |-- rtsold: rtsold.sendmsg (rtsold)
16355 Ss        0:43.49  11 127       5    13924    1652   -    28   0.0  0.0 |-- rtsold: system.syslog (rtsold)
60251 Ss        0:03.38   1 127       0    13852    2248   -    28   0.0  0.0 |-- /usr/sbin/cron -s
77766 S         0:00.00   2   2       0    14108    2344   -    28   0.0  0.0 | `-- cron: running job (cron)
78035 Ss        0:00.00   2   2       0    14028    2224   -    24   0.0  0.0 |   `-- /usr/local/bin/flock -n -E 0 -o /
78245 S         0:00.03   1   2       0    65780   36028   -  2944   0.2  0.2 |     `-- /usr/local/bin/php /usr/local/o
80904 S         0:00.00   1   1       0    19876    6428   -   152   0.2  0.0 |       `-- /usr/local/sbin/ntpq -c rv
64826 Is+       0:00.00 127 127       0    13780    1272   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv0
64908 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv1
64995 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv2
65010 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv3
65078 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv4
65097 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv5
65239 Is+       0:00.00 127 127       0    13780    1268   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv6
65323 Is+       0:00.00 127 127       0    13780    1272   -    20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv7
37372 Is        0:14.51 127 127       0    14504    2328   -    16   0.0  0.0 |-- /usr/local/bin/dpinger -f -S -r 0 -i
80470 Is        1:37.91 127 127       0    14504    1520   -    16   0.0  0.0 |-- /usr/local/bin/dpinger -f -S -r 0 -i
 5513 IW        0:00.00 127 127       0    24496       4   -     8   0.0  0.0 |-- /usr/local/sbin/syslog-ng -f /usr/loc
 5523 Ss       19:28.05   1 127     211    50112   12524   -     8   0.0  0.1 | `-- /usr/local/sbin/syslog-ng -f /usr/l
60766 S         0:18.50   0 127       0    27368   13740   -     4   0.0  0.1 |   |-- /usr/local/bin/python3 /usr/local
60850 S         0:18.86   0 127       0    28392   14120   -     4   0.0  0.1 |   `-- /usr/local/bin/python3 /usr/local
39340 Is        0:00.00 127 127       0    13764    1960   -     8   0.0  0.0 |-- daemon: /usr/local/opnsense/scripts/u
39442 S         0:31.64   5 127       0   282372  182920   -     4   0.0  1.1 | `-- /usr/local/bin/python3 /usr/local/o
41303 Ss        1:18.80   0 127       2    14044    2332   -     8   0.0  0.0 |-- /usr/local/sbin/filterlog -i pflog0 -
41403 IWs       0:00.00 127 127       0    13764       4   -     8   0.0  0.0 |-- daemon: /usr/local/opnsense/scripts/r
41424 S       125:39.78   0 127      54    81144   35848   -  2944   0.1  0.2 | `-- /usr/local/bin/php /usr/local/opnse
49280 IWs       0:00.00 127 127       0    13764       4   -     8   0.0  0.0 |-- daemon: /usr/local/libexec/ipsec/char
49305 I         0:27.80 127 127     531   100264   13140   -     8   0.0  0.1 | `-- /usr/local/libexec/ipsec/charon --u
51227 Ss        4:26.29   0 127       0    13756    1324   -    12   0.0  0.0 |-- /usr/sbin/powerd -b hadp -a hadp -n h
68707 IWs       0:00.00 127 127       0    13764       4   -     8   0.0  0.0 |-- daemon: /usr/local/bin/samplicate[687
68788 S         0:19.82   0 127       0    13660     112   -    12   0.0  0.0 | `-- /usr/local/bin/samplicate -s 127.0.
44754 Is        0:00.34 127 127       0    40396   16488   -     4   0.0  0.1 |-- /usr/local/bin/python3 /usr/local/opn
44825 S         0:46.35   1 127       0    99512   44800   -     4   0.0  0.3 | `-- /usr/local/bin/python3 /usr/local/o
31478 I         0:00.00 127 127       0    14312    2476   -   120   0.0  0.0 |   `-- /bin/sh /usr/local/opnsense/scrip
31668 I         0:00.00 127 127       0    14312    2508   -   120   0.0  0.0 |     `-- /bin/sh /usr/local/opnsense/scr
32856 IC        0:00.00 127 127       0    13648    1824   -     8   0.0  0.0 |       `-- sleep 41053
78208 Ss      263:23.79   0 127      49    53760   32660   -     4   0.2  0.2 |-- /usr/local/bin/python3 /usr/local/opn
80229 Ss        0:36.91   1 127       9    63136    8948   -     4   0.0  0.1 |-- /usr/local/bin/python3 /usr/local/opn
  648 I       186:13.42 127 127 2075350  8219332 3328836   -     4   0.0 20.1 `-- /usr/local/openjdk17/bin/java -Des.ne
    2 WL      242:10.55 127 127       0        0      96   -     0   0.0  0.0 - [clock]
    3 DL        0:02.01 127 127       0        0     112   -     0   0.0  0.0 - [crypto]
    4 DL        0:00.00 127 127       0        0      64   -     0   0.0  0.0 - [cam]
    5 DL        0:00.00 127 127       0        0      16   -     0   0.0  0.0 - [busdma]
    6 DL        2:46.17   0 127       0        0    1088   -     0   0.0  0.0 - [zfskern]
    7 DL        7:31.24   0 127       0        0      16   -     0   0.0  0.0 - [pf purge]
    8 DL        4:21.15   0 127       0        0      16   -     0   0.0  0.0 - [rand_harvestq]
    9 DL       11:52.38   0 127       0        0      48   -     0   0.0  0.0 - [pagedaemon]
   10 DL        0:00.00 127 127       0        0      16   -     0   0.0  0.0 - [audit]
   11 RNL  122299:12.88 127 127       0        0      96   -     0 562.1  0.0 - [idle]
   12 WL      259:18.92 127 127       0        0     624   -     0   1.4  0.0 - [intr]
   13 DL        0:00.01 127 127       0        0      48   -     0   0.0  0.0 - [geom]
   14 DL        0:00.00 127 127       0        0      16   -     0   0.0  0.0 - [sequencer 00]
   15 DL        0:23.62 127 127       0        0      80   -     0   0.0  0.0 - [usb]
   16 DL        0:16.59   1 127       0        0      16   -     0   0.0  0.0 - [acpi_thermal]
   17 DL        0:01.36 127 127       0        0      16   -     0   0.0  0.0 - [vmdaemon]
   18 DL        1:04.16   0 127       0        0     144   -     0   0.0  0.0 - [bufdaemon]
   19 DL        0:06.17   0 127       0        0      16   -     0   0.0  0.0 - [vnlru]
   20 DL        0:16.72   0 127       0        0      16   -     0   0.0  0.0 - [syncer]
   32 DL        0:00.20  26 127       0        0      16   -     0   0.0  0.0 - [aiod1]
   33 DL        0:00.19  27 127       0        0      16   -     0   0.0  0.0 - [aiod2]
   34 DL        0:00.19  21 127       0        0      16   -     0   0.0  0.0 - [aiod3]
   35 DL        0:00.19  20 127       0        0      16   -     0   0.0  0.0 - [aiod4]
65513 DL        0:10.74  11 127       0        0      96   -     0   0.0  0.0 - [ng_queue]
88414 DL        0:03.65   3 127       0        0      16   -     0   0.0  0.0 - [md43]
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Monviech (Cedrik) on December 11, 2025, 08:21:17 PM
What are those cc interfaces? Looks kinda weird.

What NIC hardware is that? The Chelsio T6225-CR?

Ive never seen that used ever, maybe there is some weird sideffect, especially if netmap is also running on them.

Can you try to deactivate any netmap drivers that attach to these interfaces by disabling zenarmor/intrusion detection and see if dnsmasq still inflates RAM?
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 11, 2025, 08:42:37 PM
Quote from: hina on December 11, 2025, 06:26:17 PMSome extra data points on my current system

Thank you!  Here's my dnsmasq config; I removed the static reservations since they are repetitive and I didn't feel like masking all of the MAC addresses ... that's just pure laziness on my part.

Nothing crazy in here I don't think!

EDIT: Oh, FYI, the hardware is a DEC2752, vanilla standard ...  the internal interfaces are an 802.3ad active-active LAGG across ax0/ax1;  WAN is on igc0.

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file
#

rebind-localhost-ok
stop-dns-rebind

port=53053

# If you want dnsmasq to listen for DHCP and DNS requests only on
# specified interfaces (and the loopback) give the name of the
# interface (eg eth0) here.
# Repeat the line for more than one interface.
interface=lagg0_vlan77,lagg0_vlan99,lagg0,lagg0_vlan91



dhcp-fqdn
domain=home
# This tells dnsmasq that a domain is local and it may answer queries from /etc/hosts
# or DHCP but should never forward queries on that domain to any upstream servers.
local=/home/
local=/dmz/
local=/winhome/
local=/guest/




# On systems which support it, dnsmasq binds the wildcard address,
# even when it is listening on only some interfaces. It then discards
# requests that it shouldn't reply to. This has the advantage of
# working even when interfaces come and go and change address. If you
# want dnsmasq to really bind only the interfaces it is listening on,
# uncomment this option. About the only time you may need this is when
# running another nameserver on the same machine.
bind-interfaces




# Never forward addresses in the non-routed address spaces.
bogus-priv

server=/plex.direct/8.8.8.8
rebind-domain-ok=/plex.direct/

# By  default,  dnsmasq  will  send queries to any of the upstream
# servers it knows about and tries to favour servers to are  known
# to  be  up.  Uncommenting this forces dnsmasq to try each query
# with  each  server  strictly  in  the  order  they   appear   in
# /etc/resolv.conf
strict-order

# Never forward to servers in /etc/resolv.conf
no-resolv






# host entries flushed via dnsmasq_watcher.py [isc] and a dump of the static reservations
addn-hosts=/var/etc/dnsmasq-hosts
addn-hosts=/var/etc/dnsmasq-leases

dns-forward-max=5000
cache-size=10000
local-ttl=1

conf-dir=/usr/local/etc/dnsmasq.conf.d,*.conf

dhcp-range=tag:lagg0_vlan77,192.168.77.50,192.168.77.200,255.255.255.0,86400

domain=dmz,lagg0_vlan77
dhcp-range=tag:lagg0_vlan91,192.168.91.20,192.168.91.50,255.255.255.0,86400

domain=winhome,lagg0_vlan91
dhcp-range=tag:lagg0_vlan99,192.168.99.100,192.168.99.190,255.255.255.0,86400

domain=guest,lagg0_vlan99
dhcp-range=tag:lagg0,192.168.0.100,192.168.0.245,255.255.252.0,86400

domain=home,lagg0
dhcp-range=tag:lagg0,192.168.1.100,192.168.1.250,255.255.252.0,86400

domain=home,lagg0
dhcp-range=tag:lagg0,::,constructor:lagg0,ra-names,ra-stateless,64,86400

domain=home,lagg0
ra-param=lagg0,60,1200

dhcp-range=tag:lagg0_vlan77,::,constructor:lagg0_vlan77,ra-stateless,64,86400

domain=dmz,lagg0_vlan77
ra-param=lagg0_vlan77,60,1200

dhcp-range=tag:lagg0_vlan91,::,constructor:lagg0_vlan91,ra-names,ra-stateless,64,86400

domain=winhome,lagg0_vlan91
ra-param=lagg0_vlan91,60,1200

dhcp-range=tag:lagg0_vlan99,::,constructor:lagg0_vlan99,ra-stateless,64,86400

domain=guest,lagg0_vlan99
ra-param=lagg0_vlan99,60,1200


===== 8< 24 static reservations, all in the same format ... >8 =====
dhcp-host=08:00:20:00:00:00,192.168.0.3,sun-microsystems-forver
===== 8< 24 static reservations, all in the same format ... >8 =====

dhcp-option-force=tag:lagg0_vlan99,3,192.168.99.1
dhcp-option-force=tag:lagg0_vlan99,6,192.168.99.1
dhcp-option-force=tag:lagg0_vlan91,3,192.168.91.1
dhcp-option-force=tag:lagg0_vlan91,6,208.67.222.222,208.67.220.200
dhcp-option-force=tag:lagg0,1,255.255.252.0
dhcp-option-force=tag:lagg0,6,192.168.0.1
dhcp-option-force=tag:lagg0,3,192.168.0.1

# default dns mapped to this server (0.0.0.0)
dhcp-option=6,0.0.0.0


no-ident

Looks like your dnsmasq has a large memory footprint as well - keep an eye on it and see if it continues to grow.

I saw the same post about the dnsmasq 2.92RC3 but didn't have time last night to dig into the patch; seems it's related to parsing some static host files which "shouldn't" be the issue here, but who knows!  Worth checking out.

Thanks so much for digging into this ... thankfully the hourly restart is getting me through but it's certainly not ideal.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: Monviech (Cedrik) on December 11, 2025, 09:51:18 PM
Nothing weird I can see in the config.

All features I have tested and ran for months without crashes.

Strange.

I could only recommend slowly disabling features until it doesnt happen anymore.

E.g. first disable RA/DHCPv6...check, then DHCPv4... check

That could get to the bottom of this.


------

Also since the known memory leak seems to refer to reading hostnames from files, maybe check if dnsmasq_watcher.py is running?

I refer to this part

# host entries flushed via dnsmasq_watcher.py [isc] and a dump of the static reservations
addn-hosts=/var/etc/dnsmasq-hosts
addn-hosts=/var/etc/dnsmasq-leases
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 13, 2025, 04:58:58 PM
Yes that cc0/cc1 interface belongs to Chelsio T6225-CR.

I'll test by disabling netmap features first, then ra/dhcpv6 then finally dhcpv4.
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: hina on December 13, 2025, 05:03:12 PM
Just disabled netmap, haven't restarted dnsmasq

top
last pid: 95284;  load averages:  0.37,  0.30,  0.27                                           up 16+13:49:42  00:01:42
93 processes:  1 running, 92 sleeping
CPU:  0.1% user,  0.0% nice,  0.2% system,  0.3% interrupt, 99.4% idle
Mem: 1052M Active, 4761M Inact, 120M Laundry, 4839M Wired, 216K Buf, 4928M Free
ARC: 1538M Total, 197M MFU, 1095M MRU, 5164K Anon, 19M Header, 222M Other
     1128M Compressed, 3861M Uncompressed, 3.42:1 Ratio
Swap: 8192M Total, 60M Used, 8132M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
41424 root          1  20    0    79M    34M nanslp   4 142:17   0.60% php
75839 root          1  20    0    15M  3224K CPU3     3   0:00   0.07% top
 5523 root          3  20    0    49M    12M kqread   4  21:57   0.07% syslog-ng
28181 nobody        1  20    0  4074M  3982M select   5   4:58   0.06% dnsmasq
41303 root          1  20    0    14M  2024K bpf      1   2:23   0.04% filterlog
31704 _lldpd        1  20    0    87M  1780K kqread   4   2:58   0.03% lldpd
38714 unbound       6  20    0  1402M  1138M kqread   0   0:14   0.01% unbound
60766 root          1  20    0    27M    13M select   0   0:36   0.01% python3.11
60850 root          1  20    0    28M    14M select   0   0:37   0.01% python3.11
51227 root          1  20    0    13M  1324K select   4   5:01   0.01% powerd
78208 root          1  20    0    53M    32M nanslp   4 295:39   0.01% python3.11
80470 root          4  68    0    14M  1520K uwait    4   1:50   0.01% dpinger
94062 _flowd        1  20    0    13M  1704K select   4   1:07   0.01% flowd
29146 root          1  20    0    20M  7876K select   1   0:00   0.01% sshd-session
37372 root          4  68    0    14M  2112K uwait    3   0:27   0.00% dpinger
52530 root          1  20    0    23M  8904K kqread   1   0:00   0.00% lighttpd
96675 root          1  20    0    15M  1884K kqread   5   0:32   0.00% lighttpd
68788 nobody        1  20    0    13M   112K sbwait   2   0:22   0.00% samplicate
44021 root          1  20    0    14M  2300K select   2   6:29   0.00% miniupnpd
44825 root          3  31    0   100M    46M accept   5   1:26   0.00% python3.11
16355 root          1  20    0    14M  1648K select   5   0:48   0.00% rtsold

ps -faxdvvv
  PID STAT         TIME  SL  RE PAGEIN     VSZ     RSS LIM TSIZ  %CPU %MEM COMMAND
    0 DLs      67:03.08   9 127      0       0    1920   -    0   0.0  0.0 [kernel]
    1 ILs       0:00.12  89 127     36   12324     740   -  652   0.0  0.0 - /sbin/init
38714 Ss        0:14.34   3 127      0 1435984 1165016   - 1028   0.0  7.0 |-- /usr/local/sbin/unbound -c /var/unbound/
20345 Is        0:00.03 127 127     68   20316    5412   -  312   0.0  0.0 |-- sshd: /usr/local/sbin/sshd [listener] 0
29071 Is        0:00.02 127 127      2   20868    7572   -  712   0.0  0.0 | `-- sshd-session: root [priv] (sshd-sessio
29146 S         0:00.05   0 127      0   20868    7876   -  712   0.0  0.0 |   `-- sshd-session: root@pts/0 (sshd-sessi
29168 Is        0:00.00 127 127      0   14312    2264   -  120   0.0  0.0 |     `-- /bin/sh /usr/local/sbin/opnsense-s
30919 S         0:00.01   0 127      6   17352    3624   -  324   0.0  0.0 |       `-- /bin/csh
50468 R+        0:00.00 127   0      0   14384    2664   -   24   0.0  0.0 |         `-- ps -faxdvvv
28181 S         4:58.02   0 127      0 4171616 4077972   -  312   0.0 24.7 |-- /usr/local/sbin/dnsmasq -x /var/run/dnsm
79321 S         0:01.76  10 127     92   26684    2448   -  220   0.0  0.0 |-- /usr/local/sbin/chronyd -f /usr/local/et
79399 I         0:00.10 127 127      0   22200     336   -  220   0.0  0.0 | `-- /usr/local/sbin/chronyd -f /usr/local/
52530 S         0:00.05   0 127      0   23852    8904   -  220   0.0  0.1 |-- /usr/local/sbin/lighttpd -f /usr/local/e
52655 Is        0:00.04 127 127      0  189040   30192   - 2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
52966 I         0:00.07  29 127      0  192688   36804   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53150 I         0:00.04  81 127      0  196916   36200   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53486 I         0:00.00 127 127      0  189040   30196   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53781 I         0:00.00 127 127      0  189040   30196   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54113 I         0:00.00 127 127      0  189040   30196   - 2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
52686 Is        0:00.03 127 127      0  189040   30216   - 2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
52943 I         0:00.05  29 127      0  192688   36628   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53279 I         0:00.03  50 127      0  192688   35868   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53521 I         0:00.00 127 127      0  189040   30216   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53787 I         0:00.00 127 127      0  189040   30216   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54139 I         0:00.00 127 127      0  189040   30216   - 2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
52783 Is        0:00.03 127 127      0  189040   30184   - 2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
53026 I         0:00.07 127 127     14  196784   39040   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53474 I         0:00.13  28 127      2  196848   39140   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53682 I         0:00.12  29 127     13  196784   38716   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54064 I         0:00.06 127 127     13  192688   37560   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54374 I         0:00.05 127 127     13  192688   37544   - 2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
52838 Is        0:00.03 127 127      0  189040   30220   - 2928   0.0  0.2 | |-- /usr/local/bin/php-cgi
53377 I         0:00.17 127 127     15  196848   39148   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53625 I         0:00.14 105 127     32  196784   39572   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
53915 I         0:00.14 104 127     28  192688   37760   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54275 I         0:00.17  29 127     26  196848   39632   - 2928   0.0  0.2 | | |-- /usr/local/bin/php-cgi
54478 I         0:00.12  28 127     26  196784   38488   - 2928   0.0  0.2 | | `-- /usr/local/bin/php-cgi
71038 Is        0:00.02 103 103      0  189040   30208   - 2928   0.0  0.2 | `-- /usr/local/bin/php-cgi
71147 S         0:00.14  14 103      0  192688   36804   - 2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
71295 I         0:00.02 103 103      0  192688   33996   - 2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
71406 I         0:00.01 103 103      0  192688   33504   - 2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
71446 S         0:00.12  14 103      0  192688   36508   - 2928   0.0  0.2 |   |-- /usr/local/bin/php-cgi
71479 I         0:00.00 103 103      0  189040   30208   - 2928   0.0  0.2 |   `-- /usr/local/bin/php-cgi
96675 S         0:32.35   0 127      0   15656    1884   -  220   0.0  0.0 |-- /usr/local/sbin/lighttpd -f /var/etc/lig
11809 Is        0:00.04 127 127      2   14076    1648   -   64   0.0  0.0 |-- dhclient: system.syslog (dhclient)
12798 Is        0:00.11 127 127     11   14076    1708   -   64   0.0  0.0 |-- dhclient: cc0 [priv] (dhclient)
15763 SCs       0:09.29   1 127     62   14080    1800   -   64   0.0  0.0 |-- dhclient: cc0 (dhclient)
31364 Is        0:00.62 127 127      1   38832     832   -  108   0.0  0.0 |-- lldpd: monitor. (lldpd)
31704 S         2:58.16   0 127     21   89008    1780   -  108   0.0  0.0 | `-- lldpd: no neighbor. (lldpd)
44021 Ss        6:28.59   1 127      0   14120    2300   -  116   0.0  0.0 |-- /usr/local/sbin/miniupnpd -f /var/etc/mi
16651 Is        0:00.26 127 127    100   13800    1680   -   96   0.0  0.0 |-- /usr/local/sbin/dhcp6c -c /var/etc/dhcp6
94031 Is        0:00.02 127 127     38   13748     252   -   76   0.0  0.0 |-- flowd: monitor (flowd)
94062 Ss        1:06.76   3 127     24   13748    1704   -   76   0.0  0.0 | `-- flowd: net (flowd)
 1088 Ss        0:00.53  18 127     47   15340    2616   -   36   0.0  0.0 |-- /sbin/devd
16017 SCs       0:42.95   1 127     26   13924     880   -   28   0.0  0.0 |-- /usr/sbin/rtsold -aiu -p /var/run/rtsold
16144 Is        0:00.00 127 127      0   13924    1428   -   28   0.0  0.0 |-- rtsold: rtsold.llflags (rtsold)
16233 Is        0:00.00 127 127      0   13924    1428   -   28   0.0  0.0 |-- rtsold: rtsold.script (rtsold)
16339 Is        0:00.00 127 127      0   13924    1428   -   28   0.0  0.0 |-- rtsold: rtsold.sendmsg (rtsold)
16355 Ss        0:48.08   1 127      5   13924    1648   -   28   0.0  0.0 |-- rtsold: system.syslog (rtsold)
60251 Ss        0:06.55   4 127      0   13852    2024   -   28   0.0  0.0 |-- /usr/sbin/cron -s
31196 I         0:00.00 127 127      0   13852    2032   -   28   0.0  0.0 | `-- cron: running job (cron)
31790 Is        0:00.14 127 127      0   46780   30820   -    4   0.0  0.2 |   `-- /usr/local/bin/python3 /usr/local/op
64826 Is+       0:00.00 127 127      0   13780    1272   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv0
64908 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv1
64995 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv2
65010 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv3
65078 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv4
65097 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv5
65239 Is+       0:00.00 127 127      0   13780    1268   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv6
65323 Is+       0:00.00 127 127      0   13780    1272   -   20   0.0  0.0 |-- /usr/libexec/getty Pc ttyv7
37372 Is        0:27.11 127 127      0   14504    2112   -   16   0.0  0.0 |-- /usr/local/bin/dpinger -f -S -r 0 -i WAN
80470 Is        1:50.28 127 127      0   14504    1520   -   16   0.0  0.0 |-- /usr/local/bin/dpinger -f -S -r 0 -i WAN
 5513 IW        0:00.00 127 127      0   24496       4   -    8   0.0  0.0 |-- /usr/local/sbin/syslog-ng -f /usr/local/
 5523 Ss       21:57.10   0 127    211   50112   11952   -    8   0.0  0.1 | `-- /usr/local/sbin/syslog-ng -f /usr/loca
60766 S         0:36.10   0 127      0   27368   13276   -    4   0.0  0.1 |   |-- /usr/local/bin/python3 /usr/local/sb
60850 S         0:36.58   0 127      0   28392   13880   -    4   0.0  0.1 |   `-- /usr/local/bin/python3 /usr/local/op
39244 Is        0:00.00 127 127      0   13764    1852   -    8   0.0  0.0 |-- daemon: /usr/local/opnsense/scripts/unbo
39264 S         0:07.20   3 127      0  224004  133228   -    4   0.0  0.8 | `-- /usr/local/bin/python3 /usr/local/opns
41303 Ss        2:23.20   0 127      2   14044    2024   -    8   0.0  0.0 |-- /usr/local/sbin/filterlog -i pflog0 -p /
41403 IWs       0:00.00 127 127      0   13764       4   -    8   0.0  0.0 |-- daemon: /usr/local/opnsense/scripts/rout
41424 S       142:16.72   0 127     54   81144   35104   - 2944   0.1  0.2 | `-- /usr/local/bin/php /usr/local/opnsense
49280 IWs       0:00.00 127 127      0   13764       4   -    8   0.0  0.0 |-- daemon: /usr/local/libexec/ipsec/charon[
49305 I         0:31.51 127 127    536  100264   13224   -    8   0.0  0.1 | `-- /usr/local/libexec/ipsec/charon --use-
51227 Ss        5:01.13   0 127      0   13756    1324   -   12   0.0  0.0 |-- /usr/sbin/powerd -b hadp -a hadp -n hadp
68707 IWs       0:00.00 127 127      0   13764       4   -    8   0.0  0.0 |-- daemon: /usr/local/bin/samplicate[68788]
68788 S         0:21.62   3 127      0   13660     112   -   12   0.0  0.0 | `-- /usr/local/bin/samplicate -s 127.0.0.1
44754 Is        0:00.34 127 127      0   40396   16260   -    4   0.0  0.1 |-- /usr/local/bin/python3 /usr/local/opnsen
44825 S         1:26.12  10 127      0  102200   46936   -    4   0.0  0.3 | `-- /usr/local/bin/python3 /usr/local/opns
31768 I         0:00.00 127 127      0   14312    2164   -  120   0.0  0.0 |   `-- /bin/sh /usr/local/opnsense/scripts/
32086 I         0:00.00 127 127      1   14312    2164   -  120   0.0  0.0 |     `-- /bin/sh /usr/local/opnsense/script
34048 IC        0:00.00 127 127      0   13648    1724   -    8   0.0  0.0 |       `-- sleep 8191
78208 Ss      295:39.37   0 127     49   53760   32660   -    4   4.6  0.2 |-- /usr/local/bin/python3 /usr/local/opnsen
80229 Ss        0:41.37   1 127      9   63136    8940   -    4   0.0  0.1 `-- /usr/local/bin/python3 /usr/local/opnsen
    2 WL      273:09.29 127 127      0       0      96   -    0   0.4  0.0 - [clock]
    3 DL        0:02.01 127 127      0       0     112   -    0   0.0  0.0 - [crypto]
    4 DL        0:00.00 127 127      0       0      64   -    0   0.0  0.0 - [cam]
    5 DL        0:00.00 127 127      0       0      16   -    0   0.0  0.0 - [busdma]
    6 DL        3:06.21   0 127      0       0    1088   -    0   0.0  0.0 - [zfskern]
    7 DL        8:30.02   0 127      0       0      16   -    0   0.0  0.0 - [pf purge]
    8 DL        4:55.37   0 127      0       0      16   -    0   0.0  0.0 - [rand_harvestq]
    9 DL       13:25.21   0 127      0       0      48   -    0   0.0  0.0 - [pagedaemon]
   10 DL        0:00.00 127 127      0       0      16   -    0   0.0  0.0 - [audit]
   11 RNL  138597:21.26 127 127      0       0      96   -    0 595.6  0.0 - [idle]
   12 WL      290:44.15 127 127      0       0     624   -    0   0.2  0.0 - [intr]
   13 DL        0:00.01 127 127      0       0      48   -    0   0.0  0.0 - [geom]
   14 DL        0:00.00 127 127      0       0      16   -    0   0.0  0.0 - [sequencer 00]
   15 DL        0:26.76 127 127      0       0      80   -    0   0.0  0.0 - [usb]
   16 DL        0:18.80   5 127      0       0      16   -    0   0.0  0.0 - [acpi_thermal]
   17 DL        0:01.36 127 127      0       0      16   -    0   0.0  0.0 - [vmdaemon]
   18 DL        1:12.48   0 127      0       0     144   -    0   0.0  0.0 - [bufdaemon]
   19 DL        0:07.04   0 127      0       0      16   -    0   0.0  0.0 - [vnlru]
   20 DL        0:18.93   0 127      0       0      16   -    0   0.0  0.0 - [syncer]
   32 DL        0:00.22   6 127      0       0      16   -    0   0.0  0.0 - [aiod1]
   33 DL        0:00.21   5 127      0       0      16   -    0   0.0  0.0 - [aiod2]
   34 DL        0:00.22   0 127      0       0      16   -    0   0.0  0.0 - [aiod3]
   35 DL        0:00.21  27 127      0       0      16   -    0   0.0  0.0 - [aiod4]
65513 DL        0:11.71   8 127      0       0      96   -    0   0.0  0.0 - [ng_queue]
88414 DL        0:04.12 116 127      0       0      16   -    0   0.0  0.0 - [md43]

Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 27, 2025, 04:00:17 AM
Just following up that I still see a memory leak in dnsmasq even after a reboot and an update to 25.7.10.

The cronjob which periodically restarts the process is keeping it from being an issue, but I still am trying to track down the leak.

Unfortunately it appears our base FreeBSD doesn't have proper dtrace support; it's mostly broken.

I also tried digging in with gdb but hit a deadend there too; can't really dig into jemalloc that way.

Anyway, looking forward to trying dnsmasq $version++ where we know there are memory leak fixes.  No urgency since the crontab process restart is doing its thing -- seems to be a slow leak.

For fun, here's a procstat -v.  You can see two memory regions that continue to grow.

[root@dmurphy-gw /usr/local/sbin]# procstat -v 21656
  PID              START                END PRT  RES PRES REF SHD FLAG  TP PATH
21656           0x200000           0x216000 r--   22  102   4   1 CN--- vn /usr/local/sbin/dnsmasq
21656           0x216000           0x264000 r-x   78  102   4   1 CN--- vn /usr/local/sbin/dnsmasq
21656           0x264000           0x265000 r--    1  102   4   1 CN--- vn /usr/local/sbin/dnsmasq
21656           0x265000           0x266000 r--    1    1   1   0 CN--- sw
21656           0x266000           0x268000 rw-    2    0   1   0 C---- vn /usr/local/sbin/dnsmasq
21656           0x268000           0x269000 rw-    1    1   1   0 C---- sw
21656        0x800e5b000        0x820e3b000 ---    0    0   0   0 ----- gd
21656        0x820e3b000        0x820e5b000 rw-    5    5   1   0 C--D- sw
21656        0x8219d8000        0x8219d9000 r-x    1    1  99   0 ----- ph
21656        0x8229c4000        0x8229ea000 r--   31   60  20  10 CN--- vn /usr/local/lib/libnettle.so.8.11
21656        0x8229ea000        0x822a16000 r-x   28   60  20  10 CN--- vn /usr/local/lib/libnettle.so.8.11
21656        0x822a16000        0x822a19000 r--    3    0   1   0 CN--- vn /usr/local/lib/libnettle.so.8.11
21656        0x822a19000        0x822a1a000 rw-    1    0   1   0 CN--- vn /usr/local/lib/libnettle.so.8.11
21656        0x823729000        0x82375b000 r--   17   39   4   2 CN--- vn /usr/local/lib/libhogweed.so.6.11
21656        0x82375b000        0x823771000 r-x   22   39   4   2 CN--- vn /usr/local/lib/libhogweed.so.6.11
21656        0x823771000        0x823772000 r--    1    0   1   0 CN--- vn /usr/local/lib/libhogweed.so.6.11
21656        0x823772000        0x823774000 rw-    2    0   1   0 CN--- vn /usr/local/lib/libhogweed.so.6.11
21656        0x8245aa000        0x8245cc000 r--   24   66   4   2 CN--- vn /usr/local/lib/libgmp.so.10.5.0
21656        0x8245cc000        0x824626000 r-x   42   66   4   2 CN--- vn /usr/local/lib/libgmp.so.10.5.0
21656        0x824626000        0x824627000 r--    1    0   1   0 CN--- vn /usr/local/lib/libgmp.so.10.5.0
21656        0x824627000        0x824629000 rw-    2    0   1   0 CN--- vn /usr/local/lib/libgmp.so.10.5.0
21656        0x824ede000        0x824f5d000 r--  121  451 308 114 CN--- vn /lib/libc.so.7
21656        0x824f5d000        0x825099000 r-x  316  451 308 114 CN--- vn /lib/libc.so.7
21656        0x825099000        0x8250a2000 r--    9    0   1   0 CN--- vn /lib/libc.so.7
21656        0x8250a2000        0x8250a9000 rw-    7    0   1   0 C---- vn /lib/libc.so.7
21656        0x8250a9000        0x8251ca000 rw-   19   19   1   0 C---- sw
21656     0x1259f7200000     0x1259f7221000 rw-   28   28   1   0 C---- sw
21656     0x1259f7400000     0x1259f7e00000 rw-  840  840   1   0 C---- sw
21656     0x1259f7e00000     0x125a00500000 rw- 31625 31625   1   0 ----- sw
21656     0x125a00600000     0x125a0a200000 rw- 29677 29677   1   0 ----- sw

21656     0x19660f171000     0x19660f177000 r--    6   30 251  57 CN--- vn /libexec/ld-elf.so.1
21656     0x19660f177000     0x19660f18e000 r-x   23   30 251  57 CN--- vn /libexec/ld-elf.so.1
21656     0x19660f18e000     0x19660f18f000 r--    1    0   1   0 CN--- vn /libexec/ld-elf.so.1
21656     0x19660f18f000     0x19660f190000 r--    1    1   1   0 CN--- sw
21656     0x19660f190000     0x19660f192000 rw-    2    2   1   0 C---- sw
21656     0x7fffffffe000     0x7ffffffff000 ---    0    0   0   0 ----- gd

Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: franco on December 27, 2025, 02:12:40 PM
> Just following up that I still see a memory leak in dnsmasq even after a reboot and an update to 25.7.10.

I don't think it's surprising given the fact that the binary did not change.


Cheers,
Franco
Title: Re: Dnsmasq stops after swap_pager: out of swap space
Post by: dmurphy on December 27, 2025, 05:22:18 PM
Quote from: franco on December 27, 2025, 02:12:40 PM> Just following up that I still see a memory leak in dnsmasq even after a reboot and an update to 25.7.10.

I don't think it's surprising given the fact that the binary did not change.

Exactly the expected behavior.  Was curious if the reboot and any of the netmap changes might make any difference, but appears not.

We'll see what happens when dnsmasq $version++ hits.  If it continues to drip memory, I'll spin up a dev machine, replicate the config and get dtrace running against it.

Again, not a big deal as recycling the process occasionally solves the practical issue, but now I want to know what I'm doing wrong to trip it up. :-)

Happy new year!!