Hy!
On my infamous installation with a wifi stick as WAN interface (stable otherwise) in the last 2-3 days I had two occasions with slow response of GUI. In the Dashboard I saw incredibly high numbers of states (only one client attached, normally 70-120 states) in the rang of 60,000 to 80,000 (no joke, see attached) and 100% cpu.
I tried to have a look at the states in the GUI, but gave up after some minutes, as the statistics never came up (100% cpu?). A reboot resolved the storm.
Any idea what's going on here? :-)
...2 h after the last restart I had more than 1000 states, most from the the WAN IP to the DNS servers configured in Settings on port 53.
What's going on here?
This is new, definitely due to traffic on this interface. Do you have a capture to see what kind of packets seem to hit your WAN?
..started a capture for WAN with port 53 10 min ago, for the moment everything is calm...
How to get this through to you? Is there a download option in the GUI for the file? :-)
You can download from the GUI, yes, if not big via mail (franco@********.org) or just a text dump of the packets, contents is probably overrated.
Yes, view capture and text dump from the GUI is enough, best via mail.
Hi!
I will send you the capture, believe it or not: It's all MS telemetry DNS requests. Although I'm running an openSUSE Tumbleweed (OK, it's a dual boot machine with a Win 7 pro partition, but the system actually running is the Linux).
And the firewall has a laaaaarge alias and blocks MS telemetry.
I can't believe it...
PS:
Reminds me of this here:
https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844 (https://forum.pfsense.org/index.php?topic=98087.msg577844#msg577844)
I removed duckduckgo from the search bar of Firefox, since then only the usual suspects:
incoming.telemetry.mozilla.org
self.repair.mozilla.org
conncheck.opensuse.org
safebrowsing-cache.google.com
clients1.google.com
in the DNS requests, no. of states below 20, for the time beeing...
tbc...
Hi again!
since this afternoon states exploding, no reboot helps, even prepared a fresh nano i386 CF card and did all updates
between 60.000 and 98.000 states (hard limit), 100% CPU all the time, this time nothing interesting on the only client on LAN (opensuse tumbleweed, last), found this here on the box:
netstat -an
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp4 0 0 127.0.0.1.953 *.* LISTEN
tcp6 0 0 *.53 *.* LISTEN
tcp4 0 0 *.53 *.* LISTEN
tcp6 0 0 *.80 *.* LISTEN
tcp4 0 0 *.80 *.* LISTEN
tcp6 0 0 *.443 *.* LISTEN
tcp4 0 0 *.443 *.* LISTEN
udp4 0 0 127.0.0.1.46370 127.0.0.1.53
udp4 0 0 127.0.0.1.18929 127.0.0.1.53
udp4 0 0 127.0.0.1.57403 127.0.0.1.53
udp4 0 0 127.0.0.1.24359 127.0.0.1.53
udp4 0 0 127.0.0.1.32846 127.0.0.1.53
udp4 0 0 127.0.0.1.39936 127.0.0.1.53
udp4 0 0 127.0.0.1.18748 127.0.0.1.53
udp4 0 0 127.0.0.1.40545 127.0.0.1.53
udp4 0 0 127.0.0.1.48400 127.0.0.1.53
udp4 0 0 127.0.0.1.52512 127.0.0.1.53
udp4 0 0 127.0.0.1.33374 127.0.0.1.53
udp4 0 0 127.0.0.1.28454 127.0.0.1.53
udp4 0 0 127.0.0.1.12186 127.0.0.1.53
udp4 0 0 127.0.0.1.44876 127.0.0.1.53
udp4 0 0 127.0.0.1.61900 127.0.0.1.53
udp4 0 0 127.0.0.1.45838 127.0.0.1.53
udp4 0 0 127.0.0.1.26292 127.0.0.1.53
udp4 0 0 127.0.0.1.19759 127.0.0.1.53
udp4 0 0 127.0.0.1.49251 127.0.0.1.53
udp4 0 0 127.0.0.1.49908 127.0.0.1.53
udp4 0 0 127.0.0.1.3318 127.0.0.1.53
udp4 0 0 127.0.0.1.5440 127.0.0.1.53
udp4 0 0 127.0.0.1.25242 127.0.0.1.53
udp4 0 0 127.0.0.1.48846 127.0.0.1.53
udp4 0 0 127.0.0.1.16498 127.0.0.1.53
udp4 0 0 127.0.0.1.46852 127.0.0.1.53
udp4 0 0 127.0.0.1.3954 127.0.0.1.53
udp4 0 0 127.0.0.1.62756 127.0.0.1.53
udp4 0 0 127.0.0.1.53663 127.0.0.1.53
udp4 0 0 127.0.0.1.30357 127.0.0.1.53
udp4 0 0 127.0.0.1.51436 127.0.0.1.53
udp4 0 0 127.0.0.1.52407 127.0.0.1.53
udp4 0 0 127.0.0.1.24613 127.0.0.1.53
udp4 0 0 127.0.0.1.20193 127.0.0.1.53
udp4 0 0 127.0.0.1.42795 127.0.0.1.53
udp4 0 0 127.0.0.1.37845 127.0.0.1.53
udp4 0 0 127.0.0.1.9890 127.0.0.1.53
udp4 0 0 127.0.0.1.40923 127.0.0.1.53
udp4 0 0 127.0.0.1.60502 127.0.0.1.53
udp4 0 0 127.0.0.1.17225 127.0.0.1.53
udp4 0 0 127.0.0.1.22265 127.0.0.1.53
udp4 72 0 10.0.2.4.46472 *.*
udp4 72 0 10.0.2.4.17334 *.*
udp4 106 0 10.0.2.4.59534 *.*
udp4 124 0 10.0.2.4.48529 *.*
udp4 0 0 127.0.0.1.26742 127.0.0.1.53
udp4 0 0 10.0.2.4.2282 *.*
udp4 0 0 10.0.2.4.123 *.*
udp6 0 0 fe80::7edd:90ff:.123 *.*
udp4 0 0 10.19.29.2.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp4 0 0 10.76.6.2.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp6 0 0 fe80::1%lo0.123 *.*
udp6 0 0 ::1.123 *.*
udp4 0 0 127.0.0.1.123 *.*
udp6 0 0 fe80::a298:5ff:f.123 *.*
udp4 0 0 192.168.192.1.123 *.*
udp4 0 0 *.123 *.*
udp6 0 0 *.123 *.*
udp4 0 0 *.67 *.*
udp6 0 0 *.25596 *.*
udp4 0 0 *.39639 *.*
udp6 0 0 *.53 *.*
udp4 2616 0 *.53 *.*
udp4 0 0 10.0.2.4.24200 *.*
udp4 0 0 *.* *.*
udp4 0 0 *.514 *.*
udp6 0 0 *.514 *.*
Some udp sockets may have been created or deleted.
icm4 0 0 *.* *.*
Active UNIX domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
c6fad158 stream 0 0 0 c6fa70ac 0 0
c6fa70ac stream 0 0 0 c6fad158 0 0
c6fa7000 stream 0 0 0 c6fa6ec8 0 0
c6fa6ec8 stream 0 0 0 c6fa7000 0 0
c6face1c stream 0 0 c70c311c 0 0 0 /tmp/php-fastcgi.socket-1
c6facec8 stream 0 0 c70c3354 0 0 0 /tmp/php-fastcgi.socket-0
c6fa7408 stream 0 0 c7013354 0 0 0 /var/etc/openvpn/client2.sock
c6fa7204 stream 0 0 c70c17c4 0 0 0 /var/etc/openvpn/client1.sock
c6fa7968 stream 0 0 c6faf11c 0 0 0 /var/run/devd.pipe
c6fad408 stream 0 0 c6f99b18 0 0 0 /var/run/configd.socket
c6fabd70 dgram 0 0 0 c6fad35c 0 c6faca14
c6faca14 dgram 0 0 0 c6fad35c 0 c6facd70
c6faccc4 dgram 0 0 0 c6fa76b8 0 0
c6facd70 dgram 0 0 0 c6fad35c 0 c6fad204
c6fad204 dgram 0 0 0 c6fad35c 0 c6fad000
c6fad000 dgram 0 0 0 c6fad35c 0 c6fa7764
c6fa7764 dgram 0 0 0 c6fad35c 0 c6fa72b0
c6fa72b0 dgram 0 0 0 c6fad35c 0 c6fa7158
c6fa7158 dgram 0 0 0 c6fad35c 0 c6fad2b0
c6fad2b0 dgram 0 0 0 c6fad35c 0 c6fa735c
c6fa735c dgram 0 0 0 c6fad35c 0 c6fa7560
c6fa74b4 dgram 0 0 c70c2d50 0 0 0 /var/run/wpa_supplicant/run0_w1
c6fa7560 dgram 0 0 0 c6fad35c 0 0
c6fa76b8 dgram 0 0 c700f354 0 c6faccc4 0 /var/dhcpd/var/run/log
c6fad35c dgram 0 0 c700f470 0 c6fabd70 0 /var/run/logpriv
c6fa7810 dgram 0 0 c700f58c 0 0 0 /var/run/log
c6fa78bc seqpac 0 0 c6faf000 0 0 0 /var/run/devd.seqpacket.pipe
Firewall: Diagnostics: States Summary
never finishes, the CPU is apparently to busy.
Any ideas what's going on here? Reboot helps nothing...
Had one error with the old CF-card
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0x0
stack pointer = 0x28:0xe7306a04
frame pointer = 0x28:0xe7306a38
code segment = base 0x0, limit 0xfffff, type 0x1b
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 23620 (ifconfig)
[ thread pid 23620 tid 100201 ]
Stopped at 0:KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e7306440,c11d4ba2,c1d4939c,c65b2f22,...) a8
kdb_backtrace(c13c70a6,c1e27fa0,e73065bc,c12317a3,c11d4b23,...) at kdb_backtrac0
kdb_reenter(c11d4b23,c1d49374,12,18,e73064e8,...) at kdb_reenter+0x31/frame 0xe0
trap(e73065c8) at trap+0x53/frame 0xe73065bc
calltrap() at calltrap+0x6/frame 0xe73065bc
--- trap 0xc, eip = 0xc121b821, esp = 0xe7306608, ebp = 0xe7306658 ---
db_read_bytes(0,1,e730666c,0,c127ea6d,...) at db_read_bytes+0x41/frame 0xe730668
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x29/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
*** error reading from address 0 ***
KDB: reentering
KDB: stack backtrace:
db_trace_self_wrapper(c13c7029,e730666c,0,c,0,...) at db_trace_self_wrapper+0x20
kdb_backtrace(c13c70a6,c,e7306680,c053ed3a,c127e371,...) at kdb_backtrace+0x30/8
kdb_reenter(c127e371,0,0,0,c127ea6d,...) at kdb_reenter+0x31/frame 0xe7306658
db_get_value(0,1,0,c0542314,c127ed64,...) at db_get_value+0x4a/frame 0xe7306680
db_disasm(0,1) at db_disasm+0x2a/frame 0xe7306704
db_print_loc_and_inst(0,c695a34c,e7306738,e7306744,c083d1f7,...) at db_print_lo8
db_trap(c,0,b5159e7a,246,1,...) at db_trap+0xc3/frame 0xe7306770
kdb_trap(c,0,e73069c4,1,1,...) at kdb_trap+0x114/frame 0xe73067a8
trap_fatal(c6877064,40aa53,0,0,0,...) at trap_fatal+0x2cc/frame 0xe73067f8
trap_pfault(0,0,0,0,0,...) at trap_pfault+0x355/frame 0xe7306878
trap(e73069c4) at trap+0x674/frame 0xe73069b8
calltrap() at calltrap+0x6/frame 0xe73069b8
--- trap 0xc, eip = 0, esp = 0xe7306a04, ebp = 0xe7306a38 ---
uart_sab82532_class(c6f49400,c01c697b,e7306b80,c6f49400,c74c9620,...) at 0/fram8
ifioctl(c74016a0,c01c697b,e7306b80,c74c9620,c0fb3011,...) at ifioctl+0x1528/fra4
soo_ioctl(c6de9070,c01c697b,e7306b80,c6d92e80,c74c9620,...) at soo_ioctl+0x2e8/0
kern_ioctl(c74c9620,8,c01c697b,e7306b80,c715776c,...) at kern_ioctl+0x258/frame8
sys_ioctl(c74c9620,e7306ca8,c9203954,c6f21930,2,...) at sys_ioctl+0x122/frame 08
syscall(e7306ce8) at syscall+0x4a6/frame 0xe7306cdc
Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xe7306cdc
--- syscall (54, FreeBSD ELF32, sys_ioctl), eip = 0x2a40d65f, esp = 0xbfbe0e04,-
db>
Hi chemlud,
Coincidentally, there are reports popping up regarding:
https://github.com/opnsense/src/commit/eda123ca76a90bb0151a662a85a2a8bd20ef75b4
Which was always broken on 10.2 and did not appear to be a problem before. There was only one concise crash report regarding this, and it was only a few weeks go, when the fix was not yet found.
I'll prepare a new kernel for that reason with 16.1.18 in the hopes that this will help here too, but I'm not entirely sure.
Also see: https://redmine.pfsense.org/issues/6499
Cheers,
Franco
...this redmine thing looks especially plausible! Will be interesting to see if the patch improves behaviour of my box... :-D
I'll have a version to test around noon and let you know? :)
Perfect, I'll prepare a fresh CF-card, just to be on the safe side....
...all tried to no avail! I'm more and more convinced that it is (in part?) related to some kind of problem involving opensuse. Major updates 1-2 days ago and the problem was back. This morning starting from 19 states, after 1 h more than 500 states.
Booted to Win 7, after 1 hour still only 14 states (identical use of tunnels/internet)...
Strange! But would explain why 10.3/bug fix above does not help.