9 Bře
2013
9 Bře
'13
13:28
I thought I'd enable debugging code to try and figure out why the server
goes into a loop. So I added the following to configure:
--enable-debug=server,zones,xfr,packet,dname,rr,ns,hash,compiler,stash
However, Knot doesn't compile now. The compile fails with:
...
...
libknot/packet/packet.c: In function 'knot_packet_parse_question':
libknot/packet/packet.c:311: error: 'bp' undeclared (first use in this
function)
libknot/packet/packet.c:311: error: (Each undeclared identifier is
reported only once
libknot/packet/packet.c:311: error: for each function it appears in.)
If I remove the packet debug option, I can compile Knot again.
Regards,
Anand
9 Bře
9 Bře
15:31
Hello and thanks for all the valuable feedback. Debugging messages might not help you in
this case, because they tend to generate a lot of data, especially when Knot is loading
16G worth of zones, so it could take a lot of time until the issue occurs. If you could
attach a debugger to knotd process when it gets stuck and let us know what is doing at the
moment. Perhaps using Poor Man's Profiler might be the best option. Thanks again for
your feedback. Best regards, Jan.
Anand Buddhdev <anandb(a)ripe.net> wrote:
I thought I'd enable debugging code to try and
figure out why the
server
goes into a loop. So I added the following to configure:
--enable-debug=server,zones,xfr,packet,dname,rr,ns,hash,compiler,stash
However, Knot doesn't compile now. The compile fails with:
...
...
libknot/packet/packet.c: In function 'knot_packet_parse_question':
libknot/packet/packet.c:311: error: 'bp' undeclared (first use in this
function)
libknot/packet/packet.c:311: error: (Each undeclared identifier is
reported only once
libknot/packet/packet.c:311: error: for each function it appears in.)
If I remove the packet debug option, I can compile Knot again.
Regards,
Anand
_______________________________________________
knot-dns-users mailing list
knot-dns-users(a)lists.nic.cz
https://lists.nic.cz/cgi-bin/mailman/listinfo/knot-dns-users
--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.
16:33
On 09/03/2013 15:31, Jan Kadlec wrote:
Hello and thanks for all the valuable feedback.
Debugging messages
might not help you in this case, because they tend to generate a lot of
data, especially when Knot is loading 16G worth of zones, so it could
take a lot of time until the issue occurs. If you could attach a
debugger to knotd process when it gets stuck and let us know what is
doing at the moment. Perhaps using Poor Man's Profiler might be the best
option. Thanks again for your feedback. Best regards, Jan.
Hi Jan,
I ran the non-debug version of the server, and when it shot up to 100%
and stopped logging, I attached to it with strace:
# strace -p 20919
Process 20919 attached - interrupt to quit
pselect6(83, [82], NULL, NULL, NULL, {NULL, 8}
That's all it gave. Nothing more.
Then I sent the process a TERM signal from another terminal, and it got
unwedged:
# strace -p 20919
Process 20919 attached - interrupt to quit
pselect6(83, [82], NULL, NULL, NULL, {NULL, 8}) = ? ERESTARTNOHAND (To
be restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
rt_sigreturn(0xf) = -1 EINTR (Interrupted system call)
rt_sigprocmask(SIG_BLOCK, [], NULL, 8) = 0
write(6, "\0", 1) = 1
close(6) = 0
write(9, "\0", 1) = 1
close(9) = 0
write(12, "\0", 1) = 1
close(12) = 0
write(14, "2013-03-09T15:25:46.939857-00:00"..., 52) = 52
The server continued writing into the log file, and continued to XFR
zones. I expected it to shutdown with a TERM signal, but it appeared to
ignore the signal and continue working. It only stopped with a second
TERM signal.
Regards,
Anand
10 Bře
10 Bře
15:00
Hello Anand,
I ran the non-debug version of the server, and when it
shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
# gdb -p `pidof knotd`
...
(gdb) info thread
...
(gdb) thread apply all backtrace
...
(gdb) detach
(gdb) quit
Thanks and Regards,
Jan V.
21:49
On 10/03/2013 15:00, Jan Včelák wrote:
Hello Anand,
Okay, here's the output from gdb, as you requested. Does it help?
(gdb) info thread
30 Thread 0x7fc2acd08700 (LWP 28899) 0x00007fc2adf5a4ed in write ()
from /lib64/libpthread.so.0
29 Thread 0x7fc2ace09700 (LWP 28900) 0x00007fc2adf5a054 in
__lll_lock_wait () from /lib64/libpthread.so.0
28 Thread 0x7fc2acf0a700 (LWP 28901) 0x00007fc2ae16ad3b in
update_counter_and_wait () from /usr/lib64/liburcu.so.1
27 Thread 0x7fc2ad00b700 (LWP 28902) 0x00007fc2adf5a4ed in write ()
from /lib64/libpthread.so.0
26 Thread 0x7fc2ad10c700 (LWP 28903) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
25 Thread 0x7fc2ad20d700 (LWP 28904) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
24 Thread 0x7fc2ad30e700 (LWP 28905) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
23 Thread 0x7fc2aed20700 (LWP 28906) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
22 Thread 0x7fc2acc07700 (LWP 28907) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
21 Thread 0x7fc2acb06700 (LWP 28908) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
20 Thread 0x7fc2aca05700 (LWP 28909) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
19 Thread 0x7fc2ac904700 (LWP 28910) 0x00007fc2adf5a9c3 in recvfrom
() from /lib64/libpthread.so.0
18 Thread 0x7fc2ac803700 (LWP 28911) 0x00007fc2adf5a84d in accept ()
from /lib64/libpthread.so.0
17 Thread 0x7fc2ac702700 (LWP 28912) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
16 Thread 0x7fc2ac601700 (LWP 28913) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
15 Thread 0x7fc2ac500700 (LWP 28914) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
14 Thread 0x7fc2ac3ff700 (LWP 28915) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
13 Thread 0x7fc2ac2fe700 (LWP 28916) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
12 Thread 0x7fc2ac1fd700 (LWP 28917) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
11 Thread 0x7fc2a7fff700 (LWP 28918) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
10 Thread 0x7fc2a7efe700 (LWP 28919) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
9 Thread 0x7fc2a7dfd700 (LWP 28920) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
8 Thread 0x7fc2a7cfc700 (LWP 28921) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
7 Thread 0x7fc2a7bfb700 (LWP 28922) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
6 Thread 0x7fc2a7afa700 (LWP 28923) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
5 Thread 0x7fc2a79f9700 (LWP 28924) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
4 Thread 0x7fc2a78f8700 (LWP 28925) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
3 Thread 0x7fc2a77f7700 (LWP 28926) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
2 Thread 0x7fc2a76f6700 (LWP 28927) 0x00007fc2ada1d713 in epoll_wait
() from /lib64/libc.so.6
* 1 Thread 0x7fc2aed21880 (LWP 28759) 0x00007fc2ada15f0e in pselect ()
from /lib64/libc.so.6
Thread 30 (Thread 0x7fc2acd08700 (LWP 28899)):
#0 0x00007fc2adf5a4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040ea69 in ?? ()
#4 0x000000000040c105 in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7fc2ace09700 (LWP 28900)):
#0 0x00007fc2adf5a054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007fc2adf55388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007fc2adf55257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007fc2ae16b260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007fc2ae16b359 in synchronize_rcu_mb () from /usr/lib64/liburcu.so.1
#5 0x000000000042d85f in ?? ()
#6 0x000000000040bf1b in ?? ()
#7 0x000000000040cb00 in xfr_worker ()
#8 0x0000000000443e7e in ?? ()
#9 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7fc2acf0a700 (LWP 28901)):
#0 0x00007fc2ae16ad3b in update_counter_and_wait () from
/usr/lib64/liburcu.so.1
#1 0x00007fc2ae16b36d in synchronize_rcu_mb () from /usr/lib64/liburcu.so.1
#2 0x000000000042d85f in ?? ()
#3 0x0000000000410f3e in ?? ()
#4 0x000000000040be37 in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7fc2ad00b700 (LWP 28902)):
#0 0x00007fc2adf5a4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040dd30 in ?? ()
#4 0x00000000004091df in ?? ()
#5 0x0000000000443e7e in ?? ()
#6 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7fc2ad10c700 (LWP 28903)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7fc2ad20d700 (LWP 28904)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7fc2ad30e700 (LWP 28905)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7fc2aed20700 (LWP 28906)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7fc2acc07700 (LWP 28907)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7fc2acb06700 (LWP 28908)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7fc2aca05700 (LWP 28909)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7fc2ac904700 (LWP 28910)):
#0 0x00007fc2adf5a9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7fc2ac803700 (LWP 28911)):
#0 0x00007fc2adf5a84d in accept () from /lib64/libpthread.so.0
#1 0x0000000000409def in ?? ()
#2 0x0000000000409f84 in tcp_loop_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7fc2ac702700 (LWP 28912)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7fc2ac601700 (LWP 28913)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fc2ac500700 (LWP 28914)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fc2ac3ff700 (LWP 28915)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fc2ac2fe700 (LWP 28916)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fc2ac1fd700 (LWP 28917)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fc2a7fff700 (LWP 28918)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fc2a7efe700 (LWP 28919)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fc2a7dfd700 (LWP 28920)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fc2a7cfc700 (LWP 28921)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7fc2a7bfb700 (LWP 28922)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fc2a7afa700 (LWP 28923)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fc2a79f9700 (LWP 28924)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fc2a78f8700 (LWP 28925)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7fc2a77f7700 (LWP 28926)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fc2a76f6700 (LWP 28927)):
#0 0x00007fc2ada1d713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007fc2adf53851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fc2ada1d11d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7fc2aed21880 (LWP 28759)):
#0 0x00007fc2ada15f0e in pselect () from /lib64/libc.so.6
#1 0x000000000040782e in ?? ()
#2 0x0000000000404e54 in ?? ()
#3 0x00007fc2ad953cdd in __libc_start_main () from /lib64/libc.so.6
#4 0x0000000000404829 in ?? ()
#5 0x00007fff8d492e78 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000003 in ?? ()
#8 0x00007fff8d494f0e in ?? ()
#9 0x00007fff8d494f1e in ?? ()
#10 0x00007fff8d494f21 in ?? ()
#11 0x0000000000000000 in ?? ()
I ran the non-debug version of the server, and
when it shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
# gdb -p `pidof knotd`
...
(gdb) info thread
...
(gdb) thread apply all backtrace
...
(gdb) detach
(gdb) quit
22:21
On 10/03/2013 15:00, Jan Včelák wrote:
Hello Anand,
I saw it happen again, so I got some more gdb output:
(gdb) info thread
30 Thread 0x7f7c4cd2a700 (LWP 30601) 0x00007f7c4de7b4ed in write ()
from /lib64/libpthread.so.0
29 Thread 0x7f7c4ce2b700 (LWP 30602) 0x00007f7c4e08bd3b in
update_counter_and_wait () from /usr/lib64/liburcu.so.1
28 Thread 0x7f7c4cf2c700 (LWP 30603) 0x00007f7c4de79b1b in
pthread_once () from /lib64/libpthread.so.0
27 Thread 0x7f7c4d02d700 (LWP 30604) 0x00007f7c4de7b4ed in write ()
from /lib64/libpthread.so.0
26 Thread 0x7f7c4d12e700 (LWP 30605) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
25 Thread 0x7f7c4512e700 (LWP 30606) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
24 Thread 0x7f7c4d22f700 (LWP 30607) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
23 Thread 0x7f7c4ec41700 (LWP 30608) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
22 Thread 0x7f7c4cc29700 (LWP 30609) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
21 Thread 0x7f7c4cb28700 (LWP 30610) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
20 Thread 0x7f7c4ca27700 (LWP 30611) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
19 Thread 0x7f7c4c926700 (LWP 30612) 0x00007f7c4de7b9c3 in recvfrom
() from /lib64/libpthread.so.0
18 Thread 0x7f7c4c825700 (LWP 30613) 0x00007f7c4de7b84d in accept ()
from /lib64/libpthread.so.0
17 Thread 0x7f7c4c724700 (LWP 30614) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
16 Thread 0x7f7c4c623700 (LWP 30615) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
15 Thread 0x7f7c4c522700 (LWP 30616) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
14 Thread 0x7f7c4c421700 (LWP 30617) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
13 Thread 0x7f7c4c320700 (LWP 30618) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
12 Thread 0x7f7c4c21f700 (LWP 30619) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
11 Thread 0x7f7c4c11e700 (LWP 30620) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
10 Thread 0x7f7c47fff700 (LWP 30621) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
9 Thread 0x7f7c47efe700 (LWP 30622) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
8 Thread 0x7f7c47dfd700 (LWP 30623) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
7 Thread 0x7f7c47cfc700 (LWP 30624) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
6 Thread 0x7f7c47bfb700 (LWP 30625) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
5 Thread 0x7f7c47afa700 (LWP 30626) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
4 Thread 0x7f7c479f9700 (LWP 30627) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
3 Thread 0x7f7c478f8700 (LWP 30628) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
2 Thread 0x7f7c477f7700 (LWP 30629) 0x00007f7c4d93e713 in epoll_wait
() from /lib64/libc.so.6
* 1 Thread 0x7f7c4ec42880 (LWP 30481) 0x00007f7c4d936f0e in pselect ()
from /lib64/libc.so.6
(gdb) thread apply all backtrace
Thread 30 (Thread 0x7f7c4cd2a700 (LWP 30601)):
#0 0x00007f7c4de7b4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040ea69 in ?? ()
#4 0x000000000040c105 in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f7c4ce2b700 (LWP 30602)):
#0 0x00007f7c4e08bd3b in update_counter_and_wait () from
/usr/lib64/liburcu.so.1
#1 0x00007f7c4e08c36d in synchronize_rcu_mb () from /usr/lib64/liburcu.so.1
#2 0x000000000042d85f in ?? ()
#3 0x000000000040bf1b in ?? ()
#4 0x000000000040cb00 in xfr_worker ()
#5 0x0000000000443e7e in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f7c4cf2c700 (LWP 30603)):
#0 0x00007f7c4de79b1b in pthread_once () from /lib64/libpthread.so.0
#1 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#2 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#3 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007f7c4d8cf394 in _int_malloc () from /lib64/libc.so.6
#5 0x00007f7c4d8cfdd6 in calloc () from /lib64/libc.so.6
#6 0x00007f7c4ea3fd0f in _dl_new_object () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f7c4ea3c19e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8 0x00007f7c4ea3d35a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f7c4ea479b4 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x00007f7c4ea4746a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x00007f7c4d97bb40 in do_dlopen () from /lib64/libc.so.6
#13 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x00007f7c4d97bc97 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x00007f7c4d954095 in init () from /lib64/libc.so.6
#16 0x00007f7c4de79b23 in pthread_once () from /lib64/libpthread.so.0
#17 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#18 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#19 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#20 0x00007f7c4d8ce443 in _int_free () from /lib64/libc.so.6
#21 0x0000000000417605 in ?? ()
#22 0x0000000000417d10 in ?? ()
#23 0x0000000000417ef2 in ?? ()
#24 0x0000000000418023 in node_dump_text ()
#25 0x000000000043784f in ?? ()
#26 0x0000000000437846 in ?? ()
#27 0x00000000004379c5 in ?? ()
#28 0x000000000043292a in ?? ()
#29 0x0000000000416d5e in ?? ()
#30 0x000000000040e3c6 in ?? ()
#31 0x000000000040e7e8 in ?? ()
#32 0x000000000040bec8 in ?? ()
#33 0x000000000040cb00 in xfr_worker ()
#34 0x0000000000443e7e in ?? ()
#35 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#36 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f7c4d02d700 (LWP 30604)):
#0 0x00007f7c4de7b4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040de33 in ?? ()
#4 0x00000000004091df in ?? ()
#5 0x0000000000443e7e in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f7c4d12e700 (LWP 30605)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f7c4512e700 (LWP 30606)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f7c4d22f700 (LWP 30607)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f7c4ec41700 (LWP 30608)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f7c4cc29700 (LWP 30609)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f7c4cb28700 (LWP 30610)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f7c4ca27700 (LWP 30611)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f7c4c926700 (LWP 30612)):
#0 0x00007f7c4de7b9c3 in recvfrom () from /lib64/libpthread.so.0
#1 0x00000000004097d4 in ?? ()
#2 0x00000000004092aa in udp_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f7c4c825700 (LWP 30613)):
#0 0x00007f7c4de7b84d in accept () from /lib64/libpthread.so.0
#1 0x0000000000409def in ?? ()
#2 0x0000000000409f84 in tcp_loop_master ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f7c4c724700 (LWP 30614)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f7c4c623700 (LWP 30615)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f7c4c522700 (LWP 30616)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f7c4c421700 (LWP 30617)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f7c4c320700 (LWP 30618)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f7c4c21f700 (LWP 30619)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f7c4c11e700 (LWP 30620)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f7c47fff700 (LWP 30621)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f7c47efe700 (LWP 30622)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f7c47dfd700 (LWP 30623)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f7c47cfc700 (LWP 30624)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f7c47bfb700 (LWP 30625)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f7c47afa700 (LWP 30626)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f7c479f9700 (LWP 30627)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f7c478f8700 (LWP 30628)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f7c477f7700 (LWP 30629)):
#0 0x00007f7c4d93e713 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043a9eb in ?? ()
#2 0x000000000040a0bd in tcp_loop_worker ()
#3 0x0000000000443e7e in ?? ()
#4 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7c4ec42880 (LWP 30481)):
#0 0x00007f7c4d936f0e in pselect () from /lib64/libc.so.6
#1 0x000000000040782e in ?? ()
#2 0x0000000000404e54 in ?? ()
#3 0x00007f7c4d874cdd in __libc_start_main () from /lib64/libc.so.6
#4 0x0000000000404829 in ?? ()
#5 0x00007fffa571cb68 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000003 in ?? ()
#8 0x00007fffa571cf0e in ?? ()
#9 0x00007fffa571cf1e in ?? ()
#10 0x00007fffa571cf21 in ?? ()
#11 0x0000000000000000 in ?? ()
I ran the non-debug version of the server, and
when it shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
# gdb -p `pidof knotd`
...
(gdb) info thread
...
(gdb) thread apply all backtrace
...
(gdb) detach
(gdb) quit
22:49
On 10/03/2013 15:00, Jan Včelák wrote:
Hello Anand,
And a third message. After the server go stuck I sent it a TERM signal.
It logged:
2013-03-10T21:39:50.526502-00:00 Stopping server...
But didn't actually die. So I attached to it with gdb again, and this is
the output:
(gdb) info thread
30 Thread 0x7f7c4cd2a700 (LWP 30601) 0x00007f7c4de7b4ed in write ()
from /lib64/libpthread.so.0
29 Thread 0x7f7c4ce2b700 (LWP 30602) 0x00007f7c4d93aae9 in syscall ()
from /lib64/libc.so.6
28 Thread 0x7f7c4cf2c700 (LWP 30603) 0x00007f7c4de79b1b in
pthread_once () from /lib64/libpthread.so.0
27 Thread 0x7f7c4d02d700 (LWP 30604) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
26 Thread 0x7f7c4d12e700 (LWP 30605) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
25 Thread 0x7f7c4512e700 (LWP 30606) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
24 Thread 0x7f7c4d22f700 (LWP 30607) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
23 Thread 0x7f7c4ec41700 (LWP 30608) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
22 Thread 0x7f7c4cc29700 (LWP 30609) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
21 Thread 0x7f7c4cb28700 (LWP 30610) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
20 Thread 0x7f7c4ca27700 (LWP 30611) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
19 Thread 0x7f7c4c926700 (LWP 30612) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
18 Thread 0x7f7c4c825700 (LWP 30613) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
17 Thread 0x7f7c4c724700 (LWP 30614) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
16 Thread 0x7f7c4c623700 (LWP 30615) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
15 Thread 0x7f7c4c522700 (LWP 30616) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
14 Thread 0x7f7c4c421700 (LWP 30617) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
13 Thread 0x7f7c4c320700 (LWP 30618) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
12 Thread 0x7f7c4c21f700 (LWP 30619) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
11 Thread 0x7f7c4c11e700 (LWP 30620) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
10 Thread 0x7f7c47fff700 (LWP 30621) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
9 Thread 0x7f7c47efe700 (LWP 30622) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
8 Thread 0x7f7c47dfd700 (LWP 30623) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
7 Thread 0x7f7c47cfc700 (LWP 30624) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
6 Thread 0x7f7c47bfb700 (LWP 30625) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
5 Thread 0x7f7c47afa700 (LWP 30626) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
4 Thread 0x7f7c479f9700 (LWP 30627) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
3 Thread 0x7f7c478f8700 (LWP 30628) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
2 Thread 0x7f7c477f7700 (LWP 30629) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
* 1 Thread 0x7f7c4ec42880 (LWP 30481) 0x00007f7c4de7b054 in
__lll_lock_wait () from /lib64/libpthread.so.0
(gdb) thread apply all backtrace
Thread 30 (Thread 0x7f7c4cd2a700 (LWP 30601)):
#0 0x00007f7c4de7b4ed in write () from /lib64/libpthread.so.0
#1 0x0000000000439d1a in ?? ()
#2 0x000000000040acbb in ?? ()
#3 0x000000000040ea69 in ?? ()
#4 0x000000000040c105 in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f7c4ce2b700 (LWP 30602)):
#0 0x00007f7c4d93aae9 in syscall () from /lib64/libc.so.6
#1 0x00007f7c4e08bde3 in update_counter_and_wait () from
/usr/lib64/liburcu.so.1
#2 0x00007f7c4e08c36d in synchronize_rcu_mb () from /usr/lib64/liburcu.so.1
#3 0x000000000042d85f in ?? ()
#4 0x000000000040bf1b in ?? ()
#5 0x000000000040cb00 in xfr_worker ()
#6 0x0000000000443e7e in ?? ()
#7 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f7c4cf2c700 (LWP 30603)):
#0 0x00007f7c4de79b1b in pthread_once () from /lib64/libpthread.so.0
#1 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#2 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#3 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#4 0x00007f7c4d8cf394 in _int_malloc () from /lib64/libc.so.6
#5 0x00007f7c4d8cfdd6 in calloc () from /lib64/libc.so.6
#6 0x00007f7c4ea3fd0f in _dl_new_object () from /lib64/ld-linux-x86-64.so.2
#7 0x00007f7c4ea3c19e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8 0x00007f7c4ea3d35a in _dl_map_object () from /lib64/ld-linux-x86-64.so.2
#9 0x00007f7c4ea479b4 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x00007f7c4ea4746a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x00007f7c4d97bb40 in do_dlopen () from /lib64/libc.so.6
#13 0x00007f7c4ea43196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x00007f7c4d97bc97 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x00007f7c4d954095 in init () from /lib64/libc.so.6
#16 0x00007f7c4de79b23 in pthread_once () from /lib64/libpthread.so.0
#17 0x00007f7c4d954194 in backtrace () from /lib64/libc.so.6
#18 0x00007f7c4d8c5ffb in __libc_message () from /lib64/libc.so.6
#19 0x00007f7c4d8cb916 in malloc_printerr () from /lib64/libc.so.6
#20 0x00007f7c4d8ce443 in _int_free () from /lib64/libc.so.6
#21 0x0000000000417605 in ?? ()
#22 0x0000000000417d10 in ?? ()
#23 0x0000000000417ef2 in ?? ()
#24 0x0000000000418023 in node_dump_text ()
#25 0x000000000043784f in ?? ()
#26 0x0000000000437846 in ?? ()
#27 0x00000000004379c5 in ?? ()
#28 0x000000000043292a in ?? ()
#29 0x0000000000416d5e in ?? ()
#30 0x000000000040e3c6 in ?? ()
#31 0x000000000040e7e8 in ?? ()
#32 0x000000000040bec8 in ?? ()
#33 0x000000000040cb00 in xfr_worker ()
#34 0x0000000000443e7e in ?? ()
#35 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#36 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f7c4d02d700 (LWP 30604)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f7c4d12e700 (LWP 30605)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f7c4512e700 (LWP 30606)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f7c4d22f700 (LWP 30607)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f7c4ec41700 (LWP 30608)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f7c4cc29700 (LWP 30609)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f7c4cb28700 (LWP 30610)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f7c4ca27700 (LWP 30611)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f7c4c926700 (LWP 30612)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f7c4c825700 (LWP 30613)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f7c4c724700 (LWP 30614)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f7c4c623700 (LWP 30615)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f7c4c522700 (LWP 30616)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f7c4c421700 (LWP 30617)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f7c4c320700 (LWP 30618)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f7c4c21f700 (LWP 30619)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f7c4c11e700 (LWP 30620)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f7c47fff700 (LWP 30621)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f7c47efe700 (LWP 30622)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f7c47dfd700 (LWP 30623)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x7f7c47cfc700 (LWP 30624)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f7c47bfb700 (LWP 30625)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f7c47afa700 (LWP 30626)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f7c479f9700 (LWP 30627)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f7c478f8700 (LWP 30628)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f7c477f7700 (LWP 30629)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de76388 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4e08c260 in ?? () from /usr/lib64/liburcu.so.1
#4 0x00007f7c4e08c959 in rcu_unregister_thread_mb () from
/usr/lib64/liburcu.so.1
#5 0x0000000000443eea in ?? ()
#6 0x00007f7c4de74851 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f7c4d93e11d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7c4ec42880 (LWP 30481)):
#0 0x00007f7c4de7b054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x00007f7c4de763a3 in _L_lock_892 () from /lib64/libpthread.so.0
#2 0x00007f7c4de76287 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00007f7c4ea43a4c in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#4 0x00007f7c4d88bdb2 in exit () from /lib64/libc.so.6
#5 0x00000000004051f8 in ?? ()
#6 <signal handler called>
#7 0x00007f7c4de7843c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#8 0x000000000044308c in ?? ()
#9 0x0000000000408b08 in ?? ()
#10 0x0000000000405008 in ?? ()
#11 0x00007f7c4d874cdd in __libc_start_main () from /lib64/libc.so.6
#12 0x0000000000404829 in ?? ()
#13 0x00007fffa571cb68 in ?? ()
#14 0x000000000000001c in ?? ()
#15 0x0000000000000003 in ?? ()
#16 0x00007fffa571cf0e in ?? ()
#17 0x00007fffa571cf1e in ?? ()
#18 0x00007fffa571cf21 in ?? ()
#19 0x0000000000000000 in ?? ()
I ran the non-debug version of the server, and
when it shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
# gdb -p `pidof knotd`
...
(gdb) info thread
...
(gdb) thread apply all backtrace
...
(gdb) detach
(gdb) quit 
11 Bře
11 Bře
15:41
Hi, the debug build is fixed in the latest git. Thanks for reporting it.
Marek
On 9 March 2013 13:28, Anand Buddhdev <anandb(a)ripe.net> wrote:
I thought I'd enable debugging code to try and
figure out why the server
goes into a loop. So I added the following to configure:
--enable-debug=server,zones,xfr,packet,dname,rr,ns,hash,compiler,stash
However, Knot doesn't compile now. The compile fails with:
...
...
libknot/packet/packet.c: In function 'knot_packet_parse_question':
libknot/packet/packet.c:311: error: 'bp' undeclared (first use in this
function)
libknot/packet/packet.c:311: error: (Each undeclared identifier is
reported only once
libknot/packet/packet.c:311: error: for each function it appears in.)
If I remove the packet debug option, I can compile Knot again.
Regards,
Anand
_______________________________________________
knot-dns-users mailing list
knot-dns-users(a)lists.nic.cz
https://lists.nic.cz/cgi-bin/mailman/listinfo/knot-dns-users
12 Bře
12 Bře
11:11
Hello Anand,
Okay, here's the output from gdb, as you requested. Does it help?
this looks either as a problem in RCU library, or we are doing something
wrong with it in Knot.
Please, can you give us some information about the machine? Operating
system, libc version, versions of linked libraries. And some hint on
reproducing...
Jan
I ran the non-debug version of the server, and when it
shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
13:02
On 12/03/2013 11:11, Jan Včelák wrote:
Hello Anand,
Okay, here's the output from gdb, as you requested. Does it help?
this looks either as a problem in RCU library, or we are doing something
wrong with it in Knot.
Please, can you give us some information about the machine? Operating
system, libc version, versions of linked libraries. And some hint on
reproducing...
Hello Jan,
This is a CentOS 6.3 system, x86_64 architecture.
$ uname -a
Linux ams3.authdns.ripe.net 2.6.32-279.19.1.el6.x86_64 #1 SMP Wed Dec 19
07:05:20 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ $ rpm -q glibc
glibc-2.12-1.80.el6_3.7.x86_64
glibc-2.12-1.80.el6_3.7.i686
$ rpm -q userspace-rcu
userspace-rcu-0.7.3-1.el6.x86_64
Other libraries linked to this copy of Knot:
$ ldd /usr/sbin/knotd
linux-vdso.so.1 => (0x00007fff37fb0000)
libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f851b03c000)
librt.so.1 => /lib64/librt.so.1 (0x00007f851ae34000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f851ac2f000)
liburcu.so.1 => /usr/lib64/liburcu.so.1 (0x00007f851aa2a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f851a80d000)
libm.so.6 => /lib64/libm.so.6 (0x00007f851a588000)
libc.so.6 => /lib64/libc.so.6 (0x00007f851a1f5000)
libz.so.1 => /lib64/libz.so.1 (0x00007f8519fdf000)
/lib64/ld-linux-x86-64.so.2 (0x00007f851b3e2000)
liburcu-common.so.1 => /usr/lib64/liburcu-common.so.1 (0x00007f8519ddc000)
The server has 16 GB of RAM, and NO SWAP. This is a deliberate decision.
If the server runs out of memory, we'd like to know immediately, rather
than let it begin a slow spiral of death.
I do't know what triggers the behaviour. However, what I did was to
configure it with over 5000 slave zones, which would cause Knot to try
and use more memory. The Linux OOM killer comes along and kills Knot.
Since Knot is run from upstart, it is automatically restarted, so it
tries to load all its zones. And when this happens, it sometimes locks
up. The memory usage is close to 16GB at this point, so perhaps there is
something causing this when the server memory is close to 100% usage.
Please let me know if I can supply any more information.
Regards,
Anand
I ran the non-debug version of the server, and when it
shot up to 100%
and stopped logging, I attached to it with strace:
please, can you also try with GDB? The output might be more useful.
4085
days inactive
4088
days old
9 comments
4 participants
participants (4)
-
Anand Buddhdev -
Jan Kadlec -
Jan Včelák -
Marek Vavruša