16.1.16 i386 no. of States exploding - 100% CPU

Started by chemlud, June 10, 2016, 04:42:15 PM

Previous topic - Next topic
Print
Go Down Pages1
User actions
June 10, 2016, 04:42:15 PM
Hy!

On my infamous installation with a wifi stick as WAN interface (stable otherwise) in the last 2-3 days I had two occasions with slow response of GUI. In the Dashboard I saw incredibly high numbers of states (only one client attached, normally 70-120 states) in the rang of 60,000 to 80,000 (no joke, see attached) and 100% cpu.

I tried to have a look at the states in the GUI, but gave up after some minutes, as the statistics never came up (100% cpu?). A reboot resolved the storm.

Any idea what's going on here? :-)
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 10, 2016, 06:37:00 PM #1
...2 h after the last restart I had more than 1000 states, most from the the WAN IP to the DNS servers configured in Settings on port 53.

What's going on here?
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 10, 2016, 06:52:28 PM #2
This is new, definitely due to traffic on this interface. Do you have a capture to see what kind of packets seem to hit your WAN?
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT
June 10, 2016, 06:58:15 PM #3 Last Edit: June 10, 2016, 07:02:15 PM by chemlud
..started a capture for WAN with port 53 10 min ago, for the moment everything is calm...

How to get this through to you? Is there a download option in the GUI for the file? :-)
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 10, 2016, 07:02:01 PM #4
You can download from the GUI, yes, if not big via mail (franco@********.org) or just a text dump of the packets, contents is probably overrated.

Yes, view capture and text dump from the GUI is enough, best via mail.
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT
June 10, 2016, 10:42:42 PM #5 Last Edit: June 10, 2016, 11:01:06 PM by chemlud
Hi!

I will send you the capture, believe it or not: It's all MS telemetry DNS requests. Although I'm running an openSUSE Tumbleweed (OK, it's a dual boot machine with a Win 7 pro partition, but the system actually running is the Linux).

And the firewall has a laaaaarge alias and blocks MS telemetry.

I can't believe it...

PS:

Reminds me of this here:

https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 11, 2016, 01:49:34 PM #6
I removed duckduckgo from the search bar of Firefox, since then only the usual suspects:

incoming.telemetry.mozilla.org
self.repair.mozilla.org
conncheck.opensuse.org
safebrowsing-cache.google.com
clients1.google.com

in the DNS requests, no. of states below 20, for the time beeing...

tbc...
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 27, 2016, 10:08:46 PM #7
Hi again!

since this afternoon states exploding, no reboot helps, even prepared a fresh nano i386 CF card and did all updates


between 60.000 and 98.000 states (hard limit), 100% CPU all the time, this time nothing interesting on the only client on LAN (opensuse tumbleweed, last), found this here on the box:

Code Select
netstat -an

Active Internet connections (including servers)                                                 
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)                       
tcp4       0      0 127.0.0.1.953          *.*                    LISTEN                         
tcp6       0      0 *.53                   *.*                    LISTEN                         
tcp4       0      0 *.53                   *.*                    LISTEN                         
tcp6       0      0 *.80                   *.*                    LISTEN                         
tcp4       0      0 *.80                   *.*                    LISTEN                         
tcp6       0      0 *.443                  *.*                    LISTEN                         
tcp4       0      0 *.443                  *.*                    LISTEN                         
udp4       0      0 127.0.0.1.46370        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.18929        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.57403        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.24359        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.32846        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.39936        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.18748        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.40545        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.48400        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.52512        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.33374        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.28454        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.12186        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.44876        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.61900        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.45838        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.26292        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.19759        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.49251        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.49908        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.3318         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.5440         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.25242        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.48846        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.16498        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.46852        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.3954         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.62756        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.53663        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.30357        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.51436        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.52407        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.24613        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.20193        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.42795        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.37845        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.9890         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.40923        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.60502        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.17225        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.22265        127.0.0.1.53                                         
udp4      72      0 10.0.2.4.46472         *.*                                                   
udp4      72      0 10.0.2.4.17334         *.*                                                   
udp4     106      0 10.0.2.4.59534         *.*                                                   
udp4     124      0 10.0.2.4.48529         *.*                                                   
udp4       0      0 127.0.0.1.26742        127.0.0.1.53                                         
udp4       0      0 10.0.2.4.2282          *.*                                                   
udp4       0      0 10.0.2.4.123           *.*                                                   
udp6       0      0 fe80::7edd:90ff:.123   *.*                                                   
udp4       0      0 10.19.29.2.123         *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                                                   
udp4       0      0 10.76.6.2.123          *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                                                   
udp6       0      0 fe80::1%lo0.123        *.*                                                   
udp6       0      0 ::1.123                *.*                                                   
udp4       0      0 127.0.0.1.123          *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                   
udp4       0      0 192.168.192.1.123      *.*                   
udp4       0      0 *.123                  *.*                   
udp6       0      0 *.123                  *.*                   
udp4       0      0 *.67                   *.*                   
udp6       0      0 *.25596                *.*                   
udp4       0      0 *.39639                *.*                   
udp6       0      0 *.53                   *.*                   
udp4    2616      0 *.53                   *.*                   
udp4       0      0 10.0.2.4.24200         *.*                   
udp4       0      0 *.*                    *.*                   
udp4       0      0 *.514                  *.*                   
udp6       0      0 *.514                  *.*                   
Some udp sockets may have been created or deleted.
icm4       0      0 *.*                    *.*                   
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c6fad158 stream      0      0        0 c6fa70ac        0        0
c6fa70ac stream      0      0        0 c6fad158        0        0
c6fa7000 stream      0      0        0 c6fa6ec8        0        0
c6fa6ec8 stream      0      0        0 c6fa7000        0        0
c6face1c stream      0      0 c70c311c        0        0        0 /tmp/php-fastcgi.socket-1
c6facec8 stream      0      0 c70c3354        0        0        0 /tmp/php-fastcgi.socket-0
c6fa7408 stream      0      0 c7013354        0        0        0 /var/etc/openvpn/client2.sock
c6fa7204 stream      0      0 c70c17c4        0        0        0 /var/etc/openvpn/client1.sock
c6fa7968 stream      0      0 c6faf11c        0        0        0 /var/run/devd.pipe
c6fad408 stream      0      0 c6f99b18        0        0        0 /var/run/configd.socket
c6fabd70 dgram       0      0        0 c6fad35c        0 c6faca14
c6faca14 dgram       0      0        0 c6fad35c        0 c6facd70
c6faccc4 dgram       0      0        0 c6fa76b8        0        0
c6facd70 dgram       0      0        0 c6fad35c        0 c6fad204
c6fad204 dgram       0      0        0 c6fad35c        0 c6fad000
c6fad000 dgram       0      0        0 c6fad35c        0 c6fa7764
c6fa7764 dgram       0      0        0 c6fad35c        0 c6fa72b0
c6fa72b0 dgram       0      0        0 c6fad35c        0 c6fa7158
c6fa7158 dgram       0      0        0 c6fad35c        0 c6fad2b0
c6fad2b0 dgram       0      0        0 c6fad35c        0 c6fa735c
c6fa735c dgram       0      0        0 c6fad35c        0 c6fa7560
c6fa74b4 dgram       0      0 c70c2d50        0        0        0 /var/run/wpa_supplicant/run0_w1
c6fa7560 dgram       0      0        0 c6fad35c        0        0
c6fa76b8 dgram       0      0 c700f354        0 c6faccc4        0 /var/dhcpd/var/run/log
c6fad35c dgram       0      0 c700f470        0 c6fabd70        0 /var/run/logpriv
c6fa7810 dgram       0      0 c700f58c        0        0        0 /var/run/log
c6fa78bc seqpac      0      0 c6faf000        0        0        0 /var/run/devd.seqpacket.pipe


Firewall: Diagnostics: States Summary

never finishes, the CPU is apparently to busy.

Any ideas what's going on here? Reboot helps nothing...


Had one error with the old CF-card

Code Select
Fatal trap 12: page fault while in kernel mode                                 
cpuid = 0; apic id = 00                                                         
fault virtual address   = 0x0                                                   
fault code              = supervisor read, page not present                     
instruction pointer     = 0x20:0x0                                             
stack pointer           = 0x28:0xe7306a04                                       
frame pointer           = 0x28:0xe7306a38                                       
code segment            = base 0x0, limit 0xfffff, type 0x1b                   

processor eflags        = interrupt enabled, resume, IOPL = 0                   
current process         = 23620 (ifconfig)                                     
[ thread pid 23620 tid 100201 ]                                                 
Stopped at      0:KDB: reentering                                               
KDB: stack backtrace:                                                           
      db_trace_self_wrapper(c13c7029,e7306440,c11d4ba2,c1d4939c,c65b2f22,...) a8
kdb_backtrace(c13c70a6,c1e27fa0,e73065bc,c12317a3,c11d4b23,...) at kdb_backtrac0
kdb_reenter(c11d4b23,c1d49374,12,18,e73064e8,...) at kdb_reenter+0x31/frame 0xe0
trap(e73065c8) at trap+0x53/frame 0xe73065bc                                   
calltrap() at calltrap+0x6/frame 0xe73065bc                                     
--- trap 0xc, eip = 0xc121b821, esp = 0xe7306608, ebp = 0xe7306658 ---         
db_read_bytes(0,1,e730666c,0,c127ea6d,...) at db_read_bytes+0x41/frame 0xe730668
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x29/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704                               
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770               
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8               
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8     
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878               
trap(e73069c4) at trap+0x674/frame 0xe73069b8                                   
calltrap() at calltrap+0x6/frame 0xe73069b8                                     
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---                   
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
*** error reading from address 0 ***
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e730666c,0,c,0,...) at db_trace_self_wrapper+0x20
kdb_backtrace(c13c70a6,c,e7306680,c053ed3a,c127e371,...) at kdb_backtrace+0x30/8
kdb_reenter(c127e371,0,0,0,c127ea6d,...) at kdb_reenter+0x31/frame 0xe7306658
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x4a/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
db>
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 28, 2016, 07:06:32 AM #8
Hi chemlud,

Coincidentally, there are reports popping up regarding:

https://github.com/opnsense/src/commit/eda123ca76a90bb0151a662a85a2a8bd20ef75b4

Which was always broken on 10.2 and did not appear to be a problem before. There was only one concise crash report regarding this, and it was only a few weeks go, when the fix was not yet found.

I'll prepare a new kernel for that reason with 16.1.18 in the hopes that this will help here too, but I'm not entirely sure.

Also see: https://redmine.pfsense.org/issues/6499


Cheers,
Franco
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT
June 28, 2016, 08:18:46 AM #9
...this redmine thing looks especially plausible! Will be interesting to see if the patch improves behaviour of my box... :-D
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 28, 2016, 08:23:57 AM #10
I'll have a version to test around noon and let you know? :)
"AI has absolutely reduced the cost of creating technical debt." -- ChatGPT
June 28, 2016, 08:33:24 AM #11
Perfect, I'll prepare a fresh CF-card, just to be on the safe side....
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
June 28, 2016, 02:59:29 PM #12
...all tried to no avail! I'm more and more convinced that it is (in part?) related to some kind of problem involving opensuse. Major updates 1-2 days ago and the problem was back. This morning starting from 19 states, after 1 h more than 500 states.

Booted to Win 7, after 1 hour still only 14 states (identical use of tunnels/internet)...

Strange! But would explain why 10.3/bug fix above does not help.
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
Print
Go Up Pages1
User actions