The first 6 frames of the backtrace are missing the debugging symbols -- no file name and line -- do you get the same for all traces?
Also, some more details if you have them: can it be reproduced? how quick is ending up in deadlock? do you get any error messages in the syslog printed by dialog module? any event_route from dialog module?
Cheers, Daniel
On 10/08/15 10:14, Pawel Kuzak wrote:
No, we are not using any other module on top of dialog, also not cnxcc.
Here 3 backtraces out of 12 deadlocked processes (if you need all of them please tell :-) ):
(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596947 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:49789654123@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: sip:XXX.XXX.XXX.XXX;lr=on\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bK8081.1d9a97b13c3c42328b572ca0ddd297aa."..., len=1542, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566
(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596947 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:49321654987@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: sip:XXX.XXX.XXX.XXX;lr=on\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bKa2d3.8d04ff9e19f9b1e4e9cbcb9bbae4cf41"..., len=1555, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566
(gdb) bt #0 0x00007f7e98cfd8e9 in syscall () from /lib/libc.so.6 #1 0x00007f7e9359146a in ?? () from /usr/lib/kamailio/modules/dialog.so #2 0x00007f7e9359616d in ?? () from /usr/lib/kamailio/modules/dialog.so #3 0x00007f7e93596767 in get_dlg () from /usr/lib/kamailio/modules/dialog.so #4 0x00007f7e9358d8a2 in dlg_get_msg_dialog () from /usr/lib/kamailio/modules/dialog.so #5 0x00007f7e935ac77f in pv_get_dlg_variable () from /usr/lib/kamailio/modules/dialog.so #6 0x00000000004a046c in pv_get_spec_value (msg=0x7f7e98346208, sp=0x7f7e97fb30d0, value=0x7fffb0805630) at pvapi.c:1266 #7 0x00000000004d063a in rv_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rv=0x7f7e97fb30c8, cache=0x0) at rvalue.c:1859 #8 0x00000000004d06ed in int_rve_defined (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb30c0) at rvalue.c:1895 #9 0x00000000004d14e5 in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805968, rve=0x7f7e97fb3fe0) at rvalue.c:2117 #10 0x00000000004d0adc in rval_expr_eval_int (h=0x7fffb0806510, msg=0x7f7e98346208, res=0x7fffb0805dd8, rve=0x7f7e97fb4e20) at rvalue.c:1960 #11 0x000000000041c912 in do_action (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1075 #12 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97fc1418, msg=0x7f7e98346208) at action.c:1599 #13 0x000000000041a9cc in do_action (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:715 #14 0x0000000000425b01 in run_actions (h=0x7fffb0806510, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1599 #15 0x00000000004261fb in run_actions_safe (h=0x7fffb0807520, a=0x7f7e97d203b8, msg=0x7f7e98346208) at action.c:1664 #16 0x00000000004cd997 in rval_get_int (h=0x7fffb0807520, msg=0x7f7e98346208, i=0x7fffb0806978, rv=0x7f7e97d20500, cache=0x0) at rvalue.c:924 #17 0x00000000004d078b in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806978, rve=0x7f7e97d204f8) at rvalue.c:1918 #18 0x00000000004d09a2 in rval_expr_eval_int (h=0x7fffb0807520, msg=0x7f7e98346208, res=0x7fffb0806de8, rve=0x7f7e97d20c18) at rvalue.c:1926 #19 0x000000000041c912 in do_action (h=0x7fffb0807520, a=0x7f7e97d23890, msg=0x7f7e98346208) at action.c:1075 #20 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97d20198, msg=0x7f7e98346208) at action.c:1599 #21 0x000000000041cb82 in do_action (h=0x7fffb0807520, a=0x7f7e97dce4a8, msg=0x7f7e98346208) at action.c:1090 #22 0x0000000000425b01 in run_actions (h=0x7fffb0807520, a=0x7f7e97cb6bb8, msg=0x7f7e98346208) at action.c:1599 #23 0x00000000004262c3 in run_top_route (a=0x7f7e97cb6bb8, msg=0x7f7e98346208, c=0x0) at action.c:1685 #24 0x00000000004ac5d5 in receive_msg ( buf=0x937360 "INVITE sip:491234567@XXX.XXX.XXX.XXX:5060 SIP/2.0\r\nRecord-Route: sip:XXX.XXX.XXX.XXX;lr=on\r\nVia: SIP/2.0/UDP XXX.XXX.XXX.XXX;branch=z9hG4bK41dd.793c6bda464296fb439170efecc7c405."..., len=1542, rcv_info=0x7fffb08077e0) at receive.c:212 #25 0x000000000054808d in udp_rcv_loop () at udp_server.c:536 #26 0x0000000000472f6d in main_loop () at main.c:1620 #27 0x000000000047692e in main (argc=13, argv=0x7fffb0807b28) at main.c:2566
Thank you for you help.
Paul
On 07.08.2015 14:12, Daniel-Constantin Mierla wrote:
Hello,
are you using cnxcc or other module on top of dialog?
The reason was cnxcc, as it apparently does some lower level dialog ops that involve locking in case of a dialog being already locked.
Have you got the backtrace in case of the lock you encountered to see what situations ended there?
Cheers, Daniel
On 07/08/15 13:14, Pawel Kuzak wrote:
Hi Daniel,
We are testing Kamailio 4.1.8 and we are running into a deadlock in the dialog module. I therefore searched the git commits if something has been done to the locking in dialog module and I found this commit. I wanted to ask you if there was a specific motivation for you to change this. I mean in the previous locking (e.g. like it is still in 4.1.8), do you see potential scenarios which could lead to a deadlock? Obviously there is one, but at a first glance checking the code acquiring and releasing locks looks fine. Of course we will now give 4.2 a try. I am just curious, that's why I am asking :-) .
Regards, Paul
On 16.06.2015 15:18, Daniel-Constantin Mierla wrote:
Module: kamailio Branch: master Commit: 9c3ea838b31039ac067e17d519df67b64b0dada1 URL: https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b6...
Author: Daniel-Constantin Mierla miconda@gmail.com Committer: Daniel-Constantin Mierla miconda@gmail.com Date: 2015-06-16T15:18:27+02:00
dialog: re-entrant mutex for dialogs hash table slots
- changed from a lock set usage to per slot lock field
Modified: modules/dialog/dlg_hash.c Modified: modules/dialog/dlg_hash.h
Diff: https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b6...
Patch: https://github.com/kamailio/kamailio/commit/9c3ea838b31039ac067e17d519df67b6...
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev