OPNsense Forum
Archive => 16.1 Legacy Series => Topic started by: chemlud on June 10, 2016, 04:42:15 pm
-
Hy!
On my infamous installation with a wifi stick as WAN interface (stable otherwise) in the last 2-3 days I had two occasions with slow response of GUI. In the Dashboard I saw incredibly high numbers of states (only one client attached, normally 70-120 states) in the rang of 60,000 to 80,000 (no joke, see attached) and 100% cpu.
I tried to have a look at the states in the GUI, but gave up after some minutes, as the statistics never came up (100% cpu?). A reboot resolved the storm.
Any idea what's going on here? :-)
-
...2 h after the last restart I had more than 1000 states, most from the the WAN IP to the DNS servers configured in Settings on port 53.
What's going on here?
-
This is new, definitely due to traffic on this interface. Do you have a capture to see what kind of packets seem to hit your WAN?
-
..started a capture for WAN with port 53 10 min ago, for the moment everything is calm...
How to get this through to you? Is there a download option in the GUI for the file? :-)
-
You can download from the GUI, yes, if not big via mail (franco@********.org) or just a text dump of the packets, contents is probably overrated.
Yes, view capture and text dump from the GUI is enough, best via mail.
-
Hi!
I will send you the capture, believe it or not: It's all MS telemetry DNS requests. Although I'm running an openSUSE Tumbleweed (OK, it's a dual boot machine with a Win 7 pro partition, but the system actually running is the Linux).
And the firewall has a laaaaarge alias and blocks MS telemetry.
I can't believe it...
PS:
Reminds me of this here:
https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844 (https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844)
-
I removed duckduckgo from the search bar of Firefox, since then only the usual suspects:
incoming.telemetry.mozilla.org
self.repair.mozilla.org
conncheck.opensuse.org
safebrowsing-cache.google.com
clients1.google.com
in the DNS requests, no. of states below 20, for the time beeing...
tbc...
-
Hi again!
since this afternoon states exploding, no reboot helps, even prepared a fresh nano i386 CF card and did all updates
between 60.000 and 98.000 states (hard limit), 100% CPU all the time, this time nothing interesting on the only client on LAN (opensuse tumbleweed, last), found this here on the box:
netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 127.0.0.1.953 *.* LISTEN
tcp6 0 0 *.53 *.* LISTEN
tcp4 0 0 *.53 *.* LISTEN
tcp6 0 0 *.80 *.* LISTEN
tcp4 0 0 *.80 *.* LISTEN
tcp6 0 0 *.443 *.* LISTEN
tcp4 0 0 *.443 *.* LISTEN
udp4 0 0 127.0.0.1.46370 127.0.0.1.53
udp4 0 0 127.0.0.1.18929 127.0.0.1.53
udp4 0 0 127.0.0.1.57403 127.0.0.1.53
udp4 0 0 127.0.0.1.24359 127.0.0.1.53
udp4 0 0 127.0.0.1.32846 127.0.0.1.53
udp4 0 0 127.0.0.1.39936 127.0.0.1.53
udp4 0 0 127.0.0.1.18748 127.0.0.1.53
udp4 0 0 127.0.0.1.40545 127.0.0.1.53
udp4 0 0 127.0.0.1.48400 127.0.0.1.53
udp4 0 0 127.0.0.1.52512 127.0.0.1.53
udp4 0 0 127.0.0.1.33374 127.0.0.1.53
udp4 0 0 127.0.0.1.28454 127.0.0.1.53
udp4 0 0 127.0.0.1.12186 127.0.0.1.53
udp4 0 0 127.0.0.1.44876 127.0.0.1.53
udp4 0 0 127.0.0.1.61900 127.0.0.1.53
udp4 0 0 127.0.0.1.45838 127.0.0.1.53
udp4 0 0 127.0.0.1.26292 127.0.0.1.53
udp4 0 0 127.0.0.1.19759 127.0.0.1.53
udp4 0 0 127.0.0.1.49251 127.0.0.1.53
udp4 0 0 127.0.0.1.49908 127.0.0.1.53
udp4 0 0 127.0.0.1.3318 127.0.0.1.53
udp4 0 0 127.0.0.1.5440 127.0.0.1.53
udp4 0 0 127.0.0.1.25242 127.0.0.1.53
udp4 0 0 127.0.0.1.48846 127.0.0.1.53
udp4 0 0 127.0.0.1.16498 127.0.0.1.53
udp4 0 0 127.0.0.1.46852 127.0.0.1.53
udp4 0 0 127.0.0.1.3954 127.0.0.1.53
udp4 0 0 127.0.0.1.62756 127.0.0.1.53
udp4 0 0 127.0.0.1.53663 127.0.0.1.53
udp4 0 0 127.0.0.1.30357 127.0.0.1.53
udp4 0 0 127.0.0.1.51436 127.0.0.1.53
udp4 0 0 127.0.0.1.52407 127.0.0.1.53
udp4 0 0 127.0.0.1.24613 127.0.0.1.53
udp4 0 0 127.0.0.1.20193 127.0.0.1.53
udp4 0 0 127.0.0.1.42795 127.0.0.1.53
udp4 0 0 127.0.0.1.37845 127.0.0.1.53
udp4 0 0 127.0.0.1.9890 127.0.0.1.53
udp4 0 0 127.0.0.1.40923 127.0.0.1.53
udp4 0 0 127.0.0.1.60502 127.0.0.1.53
udp4 0 0 127.0.0.1.17225 127.0.0.1.53
udp4 0 0 127.0.0.1.22265 127.0.0.1.53
udp4 72 0 10.0.2.4.46472 *.*
udp4 72 0 10.0.2.4.17334 *.*
udp4 106 0 10.0.2.4.59534 *.*
udp4 124 0 10.0.2.4.48529 *.*
udp4 0 0 127.0.0.1.26742 127.0.0.1.53
udp4 0 0 10.0.2.4.2282 *.*
udp4 0 0 10.0.2.4.123 *.*
udp6 0 0 fe80::7edd:90ff:.123 *.*
udp4 0 0 10.19.29.2.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp4 0 0 10.76.6.2.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp6 0 0 fe80::1%lo0.123 *.*
udp6 0 0 ::1.123 *.*
udp4 0 0 127.0.0.1.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp4 0 0 192.168.192.1.123 *.*
udp4 0 0 *.123 *.*
udp6 0 0 *.123 *.*
udp4 0 0 *.67 *.*
udp6 0 0 *.25596 *.*
udp4 0 0 *.39639 *.*
udp6 0 0 *.53 *.*
udp4 2616 0 *.53 *.*
udp4 0 0 10.0.2.4.24200 *.*
udp4 0 0 *.* *.*
udp4 0 0 *.514 *.*
udp6 0 0 *.514 *.*
Some udp sockets may have been created or deleted.
icm4 0 0 *.* *.*
Active UNIX domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
c6fad158 stream 0 0 0 c6fa70ac 0 0
c6fa70ac stream 0 0 0 c6fad158 0 0
c6fa7000 stream 0 0 0 c6fa6ec8 0 0
c6fa6ec8 stream 0 0 0 c6fa7000 0 0
c6face1c stream 0 0 c70c311c 0 0 0 /tmp/php-fastcgi.socket-1
c6facec8 stream 0 0 c70c3354 0 0 0 /tmp/php-fastcgi.socket-0
c6fa7408 stream 0 0 c7013354 0 0 0 /var/etc/openvpn/client2.sock
c6fa7204 stream 0 0 c70c17c4 0 0 0 /var/etc/openvpn/client1.sock
c6fa7968 stream 0 0 c6faf11c 0 0 0 /var/run/devd.pipe
c6fad408 stream 0 0 c6f99b18 0 0 0 /var/run/configd.socket
c6fabd70 dgram 0 0 0 c6fad35c 0 c6faca14
c6faca14 dgram 0 0 0 c6fad35c 0 c6facd70
c6faccc4 dgram 0 0 0 c6fa76b8 0 0
c6facd70 dgram 0 0 0 c6fad35c 0 c6fad204
c6fad204 dgram 0 0 0 c6fad35c 0 c6fad000
c6fad000 dgram 0 0 0 c6fad35c 0 c6fa7764
c6fa7764 dgram 0 0 0 c6fad35c 0 c6fa72b0
c6fa72b0 dgram 0 0 0 c6fad35c 0 c6fa7158
c6fa7158 dgram 0 0 0 c6fad35c 0 c6fad2b0
c6fad2b0 dgram 0 0 0 c6fad35c 0 c6fa735c
c6fa735c dgram 0 0 0 c6fad35c 0 c6fa7560
c6fa74b4 dgram 0 0 c70c2d50 0 0 0 /var/run/wpa_supplicant/run0_w1
c6fa7560 dgram 0 0 0 c6fad35c 0 0
c6fa76b8 dgram 0 0 c700f354 0 c6faccc4 0 /var/dhcpd/var/run/log
c6fad35c dgram 0 0 c700f470 0 c6fabd70 0 /var/run/logpriv
c6fa7810 dgram 0 0 c700f58c 0 0 0 /var/run/log
c6fa78bc seqpac 0 0 c6faf000 0 0 0 /var/run/devd.seqpacket.pipe
Firewall: Diagnostics: States Summary
never finishes, the CPU is apparently to busy.
Any ideas what's going on here? Reboot helps nothing...
Had one error with the old CF-card
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0x0
stack pointer = 0x28:0xe7306a04
frame pointer = 0x28:0xe7306a38
code segment = base 0x0, limit 0xfffff, type 0x1b
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 23620 (ifconfig)
[ thread pid 23620 tid 100201 ]
Stopped at 0:KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e7306440,c11d4ba2,c1d4939c,c65b2f22,...) a8
kdb_backtrace(c13c70a6,c1e27fa0,e73065bc,c12317a3,c11d4b23,...) at kdb_backtrac0
kdb_reenter(c11d4b23,c1d49374,12,18,e73064e8,...) at kdb_reenter+0x31/frame 0xe0
trap(e73065c8) at trap+0x53/frame 0xe73065bc
calltrap() at calltrap+0x6/frame 0xe73065bc
--- trap 0xc, eip = 0xc121b821, esp = 0xe7306608, ebp = 0xe7306658 ---
db_read_bytes(0,1,e730666c,0,c127ea6d,...) at db_read_bytes+0x41/frame 0xe730668
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x29/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
*** error reading from address 0 ***
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e730666c,0,c,0,...) at db_trace_self_wrapper+0x20
kdb_backtrace(c13c70a6,c,e7306680,c053ed3a,c127e371,...) at kdb_backtrace+0x30/8
kdb_reenter(c127e371,0,0,0,c127ea6d,...) at kdb_reenter+0x31/frame 0xe7306658
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x4a/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
db>
-
Hi chemlud,
Coincidentally, there are reports popping up regarding:
https://github.com/opnsense/src/commit/eda123ca76a90bb0151a662a85a2a8bd20ef75b4
Which was always broken on 10.2 and did not appear to be a problem before. There was only one concise crash report regarding this, and it was only a few weeks go, when the fix was not yet found.
I'll prepare a new kernel for that reason with 16.1.18 in the hopes that this will help here too, but I'm not entirely sure.
Also see: https://redmine.pfsense.org/issues/6499
Cheers,
Franco
-
...this redmine thing looks especially plausible! Will be interesting to see if the patch improves behaviour of my box... :-D
-
I'll have a version to test around noon and let you know? :)
-
Perfect, I'll prepare a fresh CF-card, just to be on the safe side....
-
...all tried to no avail! I'm more and more convinced that it is (in part?) related to some kind of problem involving opensuse. Major updates 1-2 days ago and the problem was back. This morning starting from 19 states, after 1 h more than 500 states.
Booted to Win 7, after 1 hour still only 14 states (identical use of tunnels/internet)...
Strange! But would explain why 10.3/bug fix above does not help.