Dnsmasq stops after swap_pager: out of swap space

[hina]

Hi all,

I have this issue where Dnsmasq will stop after some time, and checking dmesg gives me the following.
Could there be something I can change to fix this?

System specs:
Intel(R) Core(TM) i5-8500 CPU @ 3.00GHz (6 cores, 6 threads)
16GB ram
Chelsio T6225-CR

Code Select Expand

[49244] 764.926056 [4335] netmap_transmit           cc1 full hwcur 1951 hwtail 1951 qlen 2047
[49244] 764.933502 [4335] netmap_transmit           cc1 full hwcur 1951 hwtail 1951 qlen 2047
[278270] 790.680905 [4335] netmap_transmit           cc1 full hwcur 575 hwtail 575 qlen 2047
[278270] 790.688179 [4335] netmap_transmit           cc1 full hwcur 575 hwtail 575 qlen 2047
[548912] 433.149610 [4335] netmap_transmit           cc1 full hwcur 1733 hwtail 338 qlen 1394
[548912] 433.156943 [4335] netmap_transmit           cc1 full hwcur 1733 hwtail 338 qlen 1394
[548921] 441.701315 [4335] netmap_transmit           cc1 full hwcur 479 hwtail 1367 qlen 1159
[548921] 441.708648 [4335] netmap_transmit           cc1 full hwcur 479 hwtail 1367 qlen 1159
[548949] 469.313550 [4335] netmap_transmit           cc1 full hwcur 834 hwtail 1484 qlen 1397
[548949] 469.320885 [4335] netmap_transmit           cc1 full hwcur 834 hwtail 1484 qlen 1397
[717560] 080.906436 [4335] netmap_transmit           cc1 full hwcur 335 hwtail 520 qlen 1862
[717560] 080.913694 [4335] netmap_transmit           cc1 full hwcur 335 hwtail 520 qlen 1862
[748371] 891.599329 [4335] netmap_transmit           cc1 full hwcur 23 hwtail 1151 qlen 919
[748371] 891.608863 [4335] netmap_transmit           cc1 full hwcur 1151 hwtail 22 qlen 1128
[852487] swap_pager: out of swap space
[852487] swp_pager_getswapspace(8): failed
[856087] swp_pager_getswapspace(27): failed
[859687] swp_pager_getswapspace(8): failed
[874085] swap_pager: out of swap space
[874085] swp_pager_getswapspace(31): failed
[888485] swp_pager_getswapspace(20): failed
[892087] swp_pager_getswapspace(30): failed
[895687] swap_pager: out of swap space
[895687] swp_pager_getswapspace(8): failed
[913687] swp_pager_getswapspace(9): failed
[917284] swp_pager_getswapspace(32): failed
[924487] swp_pager_getswapspace(10): failed
[925199] swp_pager_getswapspace(28): failed
[938887] swp_pager_getswapspace(23): failed
[942484] swp_pager_getswapspace(17): failed
[942487] swap_pager: out of swap space
[942487] swp_pager_getswapspace(11): failed
[946085] swp_pager_getswapspace(5): failed
[946085] swp_pager_getswapspace(17): failed
[946087] swap_pager: out of swap space
[946087] swp_pager_getswapspace(32): failed
[949684] swp_pager_getswapspace(30): failed
[964086] swp_pager_getswapspace(32): failed
[971281] swp_pager_getswapspace(17): failed
[971285] swap_pager: out of swap space
[971285] swp_pager_getswapspace(19): failed
[971285] swp_pager_getswapspace(8): failed
[971289] pid 33489 (dnsmasq), jid 0, uid 65534, was killed: failed to reclaim memory
[1008339] arp: 192.168.1.80 moved from 46:f5:ef:xx:xx:xx to 44:6f:f8:xx:xx:xx on cc1


[Monviech (Cedrik)]

Which process did run away with all RAM and SWAP space?

You can check via when the RAM is starting to look full. But in hindsight you might not find out which process it was.

ps -faxdvvv

You would need a monitoring system to see RAM usage increase over time and store it if you don't want to watch it all day long.

[hina]

Looks like eastpect (zenarmour) and java, currently the ram is sitting at 51.16% usage (8GB used, 3GB ARC)

[hina]

I'll try going to 25.7.9 too

[Patrick M. Hausen]

java

Are you running Elasticsearch locally on your OPNsense?

[hina]

Yes, I'm running elasticsearch locally

[Patrick M. Hausen]

Elasticsearch is a known memory hog and probably the culprit. The default configuration is 32 G fixed allocation. I don't know if ZA changes anything about the ES setup.

[dmurphy]

Had a similar issue today.

Code Select Expand

[759718] swap_pager: out of swap space
[759718] swp_pager_getswapspace(7): failed
[760739] pid 88184 (dnsmasq), jid 0, uid 65534, was killed: failed to reclaim memory
I restarted dnsmasq - right now ooks like it's currently consuming > 1G memory.  Surprised it's that large.  Any idea if that footprint for dnsmasq is normal? I just switched over from ISC dhcp yesterday so don't need to induce instability ...

Code Select Expand

last pid: 63926;  load averages:  0.62,  0.46,  0.40                                            up 9+01:02:06  21:00:08
95 processes:  2 running, 93 sleeping
CPU: 16.7% user,  0.0% nice,  1.0% system,  0.1% interrupt, 82.2% idle
Mem: 453M Active, 1960M Inact, 10M Laundry, 2537M Wired, 104K Buf, 2895M Free
ARC: 1252M Total, 308M MFU, 847M MRU, 26M Anon, 7437K Header, 63M Other
     1077M Compressed, 3074M Uncompressed, 2.86:1 Ratio
Swap: 8192M Total, 338M Used, 7854M Free, 4% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 2049 nobody        1  20    0  1194M  1022M select   3   4:23   0.01% dnsmasq
25353 root          5  20    0   789M   535M kqread   0  55:23   0.00% python3.11
20699 unbound       4  20    0   754M   405M kqread   0  16:43   0.04% unbound
32880 root          1  20    0    89M    40M nanslp   2  95:37   0.75% php
38754 root          1  21    0    67M    39M accept   3   0:00   0.00% php-cgi
38980 root          1  24    0    66M    38M accept   1   0:01   0.00% php-cgi
39173 root          1  20    0    63M    36M accept   2   0:01   0.00% php-cgi
34530 root          1  20    0   132M    36M accept   3   0:49   0.00% python3.11
94077 root         12  20    0  1322M    35M uwait    2   1:49   0.00% tailscaled
85867 root          1  20    0    66M    35M accept   1   0:00   0.00% php-cgi
40453 root          1  47    0    60M    34M CPU2     2 484:29  64.90% python3.11
85405 root          1  20    0    61M    33M accept   0   0:00   0.00% php-cgi
85656 root          1  20    0    57M    30M accept   3   0:00   0.00% php-cgi
39215 root          1  66    0    46M    29M nanslp   1   0:00   0.00% python3.11
85460 root          1  26    0    57M    29M accept   2   0:00   0.00% php-cgi
68339 root          1  26    0    70M    28M accept   3   0:01   0.00% php-cgi
69129 root          1  20    0    66M    25M accept   3   0:01   0.00% php-cgi
39250 root          1  29    0    53M    24M accept   3   0:00   0.00% php-cgi
39218 root          1  28    0    53M    24M accept   1   0:00   0.00% php-cgi
38386 root          1  31    0    53M    24M wait     3   0:00   0.00% php-cgi
16645 root          1  20    0    66M    24M accept   2   0:01   0.00% php-cgi