OPNsense
  • Home
  • Help
  • Search
  • Login
  • Register

  • OPNsense Forum »
  • Archive »
  • 16.1 Legacy Series »
  • 16.1.16 i386 no. of States exploding - 100% CPU
« previous next »
  • Print
Pages: [1]

Author Topic: 16.1.16 i386 no. of States exploding - 100% CPU  (Read 10812 times)

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
16.1.16 i386 no. of States exploding - 100% CPU
« on: June 10, 2016, 04:42:15 pm »
Hy!

On my infamous installation with a wifi stick as WAN interface (stable otherwise) in the last 2-3 days I had two occasions with slow response of GUI. In the Dashboard I saw incredibly high numbers of states (only one client attached, normally 70-120 states) in the rang of 60,000 to 80,000 (no joke, see attached) and 100% cpu.

I tried to have a look at the states in the GUI, but gave up after some minutes, as the statistics never came up (100% cpu?). A reboot resolved the storm.

Any idea what's going on here? :-)
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #1 on: June 10, 2016, 06:37:00 pm »
...2 h after the last restart I had more than 1000 states, most from the the WAN IP to the DNS servers configured in Settings on port 53.

What's going on here?
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 17660
  • Karma: 1611
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #2 on: June 10, 2016, 06:52:28 pm »
This is new, definitely due to traffic on this interface. Do you have a capture to see what kind of packets seem to hit your WAN?
Logged

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #3 on: June 10, 2016, 06:58:15 pm »
..started a capture for WAN with port 53 10 min ago, for the moment everything is calm...

How to get this through to you? Is there a download option in the GUI for the file? :-)
« Last Edit: June 10, 2016, 07:02:15 pm by chemlud »
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 17660
  • Karma: 1611
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #4 on: June 10, 2016, 07:02:01 pm »
You can download from the GUI, yes, if not big via mail (franco@********.org) or just a text dump of the packets, contents is probably overrated.

Yes, view capture and text dump from the GUI is enough, best via mail.
Logged

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #5 on: June 10, 2016, 10:42:42 pm »
Hi!

I will send you the capture, believe it or not: It's all MS telemetry DNS requests. Although I'm running an openSUSE Tumbleweed (OK, it's a dual boot machine with a Win 7 pro partition, but the system actually running is the Linux).

And the firewall has a laaaaarge alias and blocks MS telemetry.

I can't believe it...

PS:

Reminds me of this here:

https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844
« Last Edit: June 10, 2016, 11:01:06 pm by chemlud »
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #6 on: June 11, 2016, 01:49:34 pm »
I removed duckduckgo from the search bar of Firefox, since then only the usual suspects:

incoming.telemetry.mozilla.org
self.repair.mozilla.org
conncheck.opensuse.org
safebrowsing-cache.google.com
clients1.google.com

in the DNS requests, no. of states below 20, for the time beeing...

tbc...
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #7 on: June 27, 2016, 10:08:46 pm »
Hi again!

since this afternoon states exploding, no reboot helps, even prepared a fresh nano i386 CF card and did all updates


between 60.000 and 98.000 states (hard limit), 100% CPU all the time, this time nothing interesting on the only client on LAN (opensuse tumbleweed, last), found this here on the box:

Code: [Select]
netstat -an

Active Internet connections (including servers)                                                 
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)                       
tcp4       0      0 127.0.0.1.953          *.*                    LISTEN                         
tcp6       0      0 *.53                   *.*                    LISTEN                         
tcp4       0      0 *.53                   *.*                    LISTEN                         
tcp6       0      0 *.80                   *.*                    LISTEN                         
tcp4       0      0 *.80                   *.*                    LISTEN                         
tcp6       0      0 *.443                  *.*                    LISTEN                         
tcp4       0      0 *.443                  *.*                    LISTEN                         
udp4       0      0 127.0.0.1.46370        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.18929        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.57403        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.24359        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.32846        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.39936        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.18748        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.40545        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.48400        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.52512        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.33374        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.28454        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.12186        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.44876        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.61900        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.45838        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.26292        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.19759        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.49251        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.49908        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.3318         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.5440         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.25242        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.48846        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.16498        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.46852        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.3954         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.62756        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.53663        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.30357        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.51436        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.52407        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.24613        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.20193        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.42795        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.37845        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.9890         127.0.0.1.53                                         
udp4       0      0 127.0.0.1.40923        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.60502        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.17225        127.0.0.1.53                                         
udp4       0      0 127.0.0.1.22265        127.0.0.1.53                                         
udp4      72      0 10.0.2.4.46472         *.*                                                   
udp4      72      0 10.0.2.4.17334         *.*                                                   
udp4     106      0 10.0.2.4.59534         *.*                                                   
udp4     124      0 10.0.2.4.48529         *.*                                                   
udp4       0      0 127.0.0.1.26742        127.0.0.1.53                                         
udp4       0      0 10.0.2.4.2282          *.*                                                   
udp4       0      0 10.0.2.4.123           *.*                                                   
udp6       0      0 fe80::7edd:90ff:.123   *.*                                                   
udp4       0      0 10.19.29.2.123         *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                                                   
udp4       0      0 10.76.6.2.123          *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                                                   
udp6       0      0 fe80::1%lo0.123        *.*                                                   
udp6       0      0 ::1.123                *.*                                                   
udp4       0      0 127.0.0.1.123          *.*                                                   
udp6       0      0 fe80::a298:5ff:f.123   *.*                   
udp4       0      0 192.168.192.1.123      *.*                   
udp4       0      0 *.123                  *.*                   
udp6       0      0 *.123                  *.*                   
udp4       0      0 *.67                   *.*                   
udp6       0      0 *.25596                *.*                   
udp4       0      0 *.39639                *.*                   
udp6       0      0 *.53                   *.*                   
udp4    2616      0 *.53                   *.*                   
udp4       0      0 10.0.2.4.24200         *.*                   
udp4       0      0 *.*                    *.*                   
udp4       0      0 *.514                  *.*                   
udp6       0      0 *.514                  *.*                   
Some udp sockets may have been created or deleted.
icm4       0      0 *.*                    *.*                   
Active UNIX domain sockets
Address  Type   Recv-Q Send-Q    Inode     Conn     Refs  Nextref Addr
c6fad158 stream      0      0        0 c6fa70ac        0        0
c6fa70ac stream      0      0        0 c6fad158        0        0
c6fa7000 stream      0      0        0 c6fa6ec8        0        0
c6fa6ec8 stream      0      0        0 c6fa7000        0        0
c6face1c stream      0      0 c70c311c        0        0        0 /tmp/php-fastcgi.socket-1
c6facec8 stream      0      0 c70c3354        0        0        0 /tmp/php-fastcgi.socket-0
c6fa7408 stream      0      0 c7013354        0        0        0 /var/etc/openvpn/client2.sock
c6fa7204 stream      0      0 c70c17c4        0        0        0 /var/etc/openvpn/client1.sock
c6fa7968 stream      0      0 c6faf11c        0        0        0 /var/run/devd.pipe
c6fad408 stream      0      0 c6f99b18        0        0        0 /var/run/configd.socket
c6fabd70 dgram       0      0        0 c6fad35c        0 c6faca14
c6faca14 dgram       0      0        0 c6fad35c        0 c6facd70
c6faccc4 dgram       0      0        0 c6fa76b8        0        0
c6facd70 dgram       0      0        0 c6fad35c        0 c6fad204
c6fad204 dgram       0      0        0 c6fad35c        0 c6fad000
c6fad000 dgram       0      0        0 c6fad35c        0 c6fa7764
c6fa7764 dgram       0      0        0 c6fad35c        0 c6fa72b0
c6fa72b0 dgram       0      0        0 c6fad35c        0 c6fa7158
c6fa7158 dgram       0      0        0 c6fad35c        0 c6fad2b0
c6fad2b0 dgram       0      0        0 c6fad35c        0 c6fa735c
c6fa735c dgram       0      0        0 c6fad35c        0 c6fa7560
c6fa74b4 dgram       0      0 c70c2d50        0        0        0 /var/run/wpa_supplicant/run0_w1
c6fa7560 dgram       0      0        0 c6fad35c        0        0
c6fa76b8 dgram       0      0 c700f354        0 c6faccc4        0 /var/dhcpd/var/run/log
c6fad35c dgram       0      0 c700f470        0 c6fabd70        0 /var/run/logpriv
c6fa7810 dgram       0      0 c700f58c        0        0        0 /var/run/log
c6fa78bc seqpac      0      0 c6faf000        0        0        0 /var/run/devd.seqpacket.pipe

Firewall: Diagnostics: States Summary

never finishes, the CPU is apparently to busy.

Any ideas what's going on here? Reboot helps nothing...


Had one error with the old CF-card

Code: [Select]
Fatal trap 12: page fault while in kernel mode                                 
cpuid = 0; apic id = 00                                                         
fault virtual address   = 0x0                                                   
fault code              = supervisor read, page not present                     
instruction pointer     = 0x20:0x0                                             
stack pointer           = 0x28:0xe7306a04                                       
frame pointer           = 0x28:0xe7306a38                                       
code segment            = base 0x0, limit 0xfffff, type 0x1b                   

processor eflags        = interrupt enabled, resume, IOPL = 0                   
current process         = 23620 (ifconfig)                                     
[ thread pid 23620 tid 100201 ]                                                 
Stopped at      0:KDB: reentering                                               
KDB: stack backtrace:                                                           
      db_trace_self_wrapper(c13c7029,e7306440,c11d4ba2,c1d4939c,c65b2f22,...) a8
kdb_backtrace(c13c70a6,c1e27fa0,e73065bc,c12317a3,c11d4b23,...) at kdb_backtrac0
kdb_reenter(c11d4b23,c1d49374,12,18,e73064e8,...) at kdb_reenter+0x31/frame 0xe0
trap(e73065c8) at trap+0x53/frame 0xe73065bc                                   
calltrap() at calltrap+0x6/frame 0xe73065bc                                     
--- trap 0xc, eip = 0xc121b821, esp = 0xe7306608, ebp = 0xe7306658 ---         
db_read_bytes(0,1,e730666c,0,c127ea6d,...) at db_read_bytes+0x41/frame 0xe730668
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x29/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704                               
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770               
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8               
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8     
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878               
trap(e73069c4) at trap+0x674/frame 0xe73069b8                                   
calltrap() at calltrap+0x6/frame 0xe73069b8                                     
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---                   
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
*** error reading from address 0 ***
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e730666c,0,c,0,...) at db_trace_self_wrapper+0x20
kdb_backtrace(c13c70a6,c,e7306680,c053ed3a,c127e371,...) at kdb_backtrace+0x30/8
kdb_reenter(c127e371,0,0,0,c127ea6d,...) at kdb_reenter+0x31/frame 0xe7306658
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x4a/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
db>
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 17660
  • Karma: 1611
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #8 on: June 28, 2016, 07:06:32 am »
Hi chemlud,

Coincidentally, there are reports popping up regarding:

https://github.com/opnsense/src/commit/eda123ca76a90bb0151a662a85a2a8bd20ef75b4

Which was always broken on 10.2 and did not appear to be a problem before. There was only one concise crash report regarding this, and it was only a few weeks go, when the fix was not yet found.

I'll prepare a new kernel for that reason with 16.1.18 in the hopes that this will help here too, but I'm not entirely sure.

Also see: https://redmine.pfsense.org/issues/6499


Cheers,
Franco
Logged

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #9 on: June 28, 2016, 08:18:46 am »
...this redmine thing looks especially plausible! Will be interesting to see if the patch improves behaviour of my box... :-D
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

franco

  • Administrator
  • Hero Member
  • *****
  • Posts: 17660
  • Karma: 1611
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #10 on: June 28, 2016, 08:23:57 am »
I'll have a version to test around noon and let you know? :)
Logged

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #11 on: June 28, 2016, 08:33:24 am »
Perfect, I'll prepare a fresh CF-card, just to be on the safe side....
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....

chemlud

  • Hero Member
  • *****
  • Posts: 2485
  • Karma: 112
    • View Profile
Re: 16.1.16 i386 no. of States exploding - 100% CPU
« Reply #12 on: June 28, 2016, 02:59:29 pm »
...all tried to no avail! I'm more and more convinced that it is (in part?) related to some kind of problem involving opensuse. Major updates 1-2 days ago and the problem was back. This morning starting from 19 states, after 1 h more than 500 states.

Booted to Win 7, after 1 hour still only 14 states (identical use of tunnels/internet)...

Strange! But would explain why 10.3/bug fix above does not help.
Logged
kind regards
chemlud
____
"The price of reliability is the pursuit of the utmost simplicity."
C.A.R. Hoare

felix eichhorns premium katzenfutter mit der extraportion energie

A router is not a switch - A router is not a switch - A router is not a switch - A rou....
  • Print
Pages: [1]
« previous next »
  • OPNsense Forum »
  • Archive »
  • 16.1 Legacy Series »
  • 16.1.16 i386 no. of States exploding - 100% CPU
 

OPNsense is an OSS project © Deciso B.V. 2015 - 2024 All rights reserved
  • SMF 2.0.19 | SMF © 2021, Simple Machines
    Privacy Policy     | XHTML | RSS | WAP2