On 10/03/2013 15:00, Jan Včelák wrote:
Hello Anand,
I ran the non-debug version of the server, and
when it shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
# gdb -p `pidof knotd`
...
(gdb) info thread
...
(gdb) thread apply all backtrace
...
(gdb) detach
(gdb) quit
I saw it happen again, so I got some more gdb output:
(gdb) info thread
30 Thread 0x7f7c4cd2a700 (LWP 30601) 0x00007f7c4de7b4ed in write ()
from /lib64/libpthread.so.0
29 Thread 0x7f7c4ce2b700 (LWP 30602) 0x00007f7c4e08bd3b in
update_counter_and_wait () from /usr/lib64/liburcu.so.1
28 Thread 0x7f7c4cf2c700 (LWP 30603) 0x00007f7c4de79b1b in
pthread_once () from /lib64/libpthread.so.0
27 Thread 0x7f7c4d02d700 (LWP 30604) 0x00007f7c4de7b4ed in write ()
from /lib64/libpthread.so.0
26 Thread 0x7f7c4d12e700 (LWP 30605) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
25 Thread 0x7f7c4512e700 (LWP 30606) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
24 Thread 0x7f7c4d22f700 (LWP 30607) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
23 Thread 0x7f7c4ec41700 (LWP 30608) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
22 Thread 0x7f7c4cc29700 (LWP 30609) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
21 Thread 0x7f7c4cb28700 (LWP 30610) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
20 Thread 0x7f7c4ca27700 (LWP 30611) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
19 Thread 0x7f7c4c926700 (LWP 30612) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
18 Thread 0x7f7c4c825700 (LWP 30613) 0x00007f7c4de7b84d in accept ()
from /lib64/libpthread.so.0
17 Thread 0x7f7c4c724700 (LWP 30614) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
16 Thread 0x7f7c4c623700 (LWP 30615) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
15 Thread 0x7f7c4c522700 (LWP 30616) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
14 Thread 0x7f7c4c421700 (LWP 30617) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
13 Thread 0x7f7c4c320700 (LWP 30618) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
12 Thread 0x7f7c4c21f700 (LWP 30619) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
11 Thread 0x7f7c4c11e700 (LWP 30620) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
10 Thread 0x7f7c47fff700 (LWP 30621) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
9 Thread 0x7f7c47efe700 (LWP 30622) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
8 Thread 0x7f7c47dfd700 (LWP 30623) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
7 Thread 0x7f7c47cfc700 (LWP 30624) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
6 Thread 0x7f7c47bfb700 (LWP 30625) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
5 Thread 0x7f7c47afa700 (LWP 30626) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
4 Thread 0x7f7c479f9700 (LWP 30627) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
3 Thread 0x7f7c478f8700 (LWP 30628) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
2 Thread 0x7f7c477f7700 (LWP 30629) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
* 1 Thread 0x7f7c4ec42880 (LWP 30481) 0x00007f7c4d936f0e in pselect ()
from /lib64/libc.so.6
(gdb) thread apply all backtrace
Thread 30 (Thread 0x7f7c4cd2a700 (LWP 30601)):
#0 0x00007f7c4de7b4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040ea69 in ?? ()
#4 0x000000000040c105 in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f7c4ce2b700 (LWP 30602)):
#0 0x00007f7c4e08bd3b in update_counter_and_wait () from
/usr/lib64/liburcu.so.1
#1 0x00007f7c4e08c36d in synchronize_rcu_mb () from /usr/lib64/liburcu.so.1
#2 0x000000000042d85f in ?? ()
#3 0x000000000040bf1b in ?? ()
#4 0x000000000040cb00 in xfr_worker ()
#5 0x0000000000443e7e in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f7c4cf2c700 (LWP 30603)):
#0 0x00007f7c4de79b1b in pthread_once () from /lib64/libpthread.so.0
#1 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#2 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#3 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007f7c4d8cf394 in _int_malloc () from /lib64/libc.so.6
#5 0x00007f7c4d8cfdd6 in calloc () from /lib64/libc.so.6
#6 0x00007f7c4ea3fd0f in _dl_new_object () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f7c4ea3c19e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8 0x00007f7c4ea3d35a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f7c4ea479b4 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x00007f7c4ea4746a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x00007f7c4d97bb40 in do_dlopen () from /lib64/libc.so.6
#13 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x00007f7c4d97bc97 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x00007f7c4d954095 in init () from /lib64/libc.so.6
#16 0x00007f7c4de79b23 in pthread_once () from /lib64/libpthread.so.0
#17 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#18 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#19 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#20 0x00007f7c4d8ce443 in _int_free () from /lib64/libc.so.6
#21 0x0000000000417605 in ?? ()
#22 0x0000000000417d10 in ?? ()
#23 0x0000000000417ef2 in ?? ()
#24 0x0000000000418023 in node_dump_text ()
#25 0x000000000043784f in ?? ()
#26 0x0000000000437846 in ?? ()
#27 0x00000000004379c5 in ?? ()
#28 0x000000000043292a in ?? ()
#29 0x0000000000416d5e in ?? ()
#30 0x000000000040e3c6 in ?? ()
#31 0x000000000040e7e8 in ?? ()
#32 0x000000000040bec8 in ?? ()
#33 0x000000000040cb00 in xfr_worker ()
#34 0x0000000000443e7e in ?? ()
#35 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#36 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f7c4d02d700 (LWP 30604)):
#0 0x00007f7c4de7b4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040de33 in ?? ()
#4 0x00000000004091df in ?? ()
#5 0x0000000000443e7e in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f7c4d12e700 (LWP 30605)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f7c4512e700 (LWP 30606)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f7c4d22f700 (LWP 30607)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f7c4ec41700 (LWP 30608)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f7c4cc29700 (LWP 30609)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f7c4cb28700 (LWP 30610)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f7c4ca27700 (LWP 30611)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f7c4c926700 (LWP 30612)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f7c4c825700 (LWP 30613)):
#0 0x00007f7c4de7b84d in accept () from /lib64/libpthread.so.0
#1 0x0000000000409def in ?? ()
#2 0x0000000000409f84 in tcp_loop_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f7c4c724700 (LWP 30614)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f7c4c623700 (LWP 30615)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f7c4c522700 (LWP 30616)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f7c4c421700 (LWP 30617)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f7c4c320700 (LWP 30618)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f7c4c21f700 (LWP 30619)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f7c4c11e700 (LWP 30620)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f7c47fff700 (LWP 30621)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f7c47efe700 (LWP 30622)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f7c47dfd700 (LWP 30623)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f7c47cfc700 (LWP 30624)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f7c47bfb700 (LWP 30625)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f7c47afa700 (LWP 30626)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f7c479f9700 (LWP 30627)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f7c478f8700 (LWP 30628)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f7c477f7700 (LWP 30629)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7c4ec42880 (LWP 30481)):
#0 0x00007f7c4d936f0e in pselect () from /lib64/libc.so.6
#1 0x000000000040782e in ?? ()
#2 0x0000000000404e54 in ?? ()
#3 0x00007f7c4d874cdd in __libc_start_main () from /lib64/libc.so.6
#4 0x0000000000404829 in ?? ()
#5 0x00007fffa571cb68 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000003 in ?? ()
#8 0x00007fffa571cf0e in ?? ()
#9 0x00007fffa571cf1e in ?? ()
#10 0x00007fffa571cf21 in ?? ()
#11 0x0000000000000000 in ?? ()