Can you install the debug symbols? Without them is hard to track the issue, because there is no direct call of pthread_mutex_destroy() from inside rpc_reply(). --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-217843053

Testing now with debug symbols, and wait for coredump. --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-218080526

Got a core dump with debug symbols. ``` May 10 09:56:32 <local0.crit> voip-integration-cust1 kamailio[58953]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 18 May 10 09:56:32 <kern.info> voip-integration-cust1 kernel: pid 58887 (kamailio), uid 986: exited on signal 6 (core dumped) May 10 09:56:32 <local0.alert> voip-integration-cust1 kamailio[58882]: ALERT: <core> [main.c:739]: handle_sigs(): child process 58887 exited by a signal 6 May 10 09:56:32 <local0.alert> voip-integration-cust1 kamailio[58882]: ALERT: <core> [main.c:742]: handle_sigs(): core was generated ``` ``` (gdb) bt full #0 0x000000080111735a in thr_kill () from /lib/libc.so.7 No symbol table info available. #1 0x0000000801117346 in raise () from /lib/libc.so.7 No symbol table info available. #2 0x00000008011172c9 in abort () from /lib/libc.so.7 No symbol table info available. #3 0x0000000800e4695a in pthread_attr_getaffinity_np () from /lib/libthr.so.3 No symbol table info available. #4 0x0000000800e42046 in pthread_mutex_destroy () from /lib/libthr.so.3 No symbol table info available. #5 0x00000008053477d5 in _lock (s=0x807bf0ee0) at lock.h:83 No locals. #6 0x0000000805328038 in tm_reply_mutex_lock (t=0x807bf0d20) at t_reply.c:2840 mypid = 58887 #7 0x0000000805338cc2 in reply_received (p_msg=0x8020ab268) at t_reply.c:2443 msg_status = 200 last_uac_status = 0 ack = 0x313a632e65766965 <Address 0x313a632e65766965 out of bounds> ack_len = 1667592795 branch = 0 reply_status = 1868774432 onreply_route = 0 cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 1563899960}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 1563899960}}}} uac = (struct ua_client *) 0x807bf0f30 t = (struct cell *) 0x807bf0d20 lack_dst = {send_sock = 0xc7521c, to = {s = {sa_len = 0 '\0', sa_family = 0 '\0', sa_data = "\000\000\t\000\000\000S\233\226\000\000\000\000"}, sin = {sin_len = 0 '\0', sin_family = 0 '\0', sin_port = 0, sin_addr = {s_addr = 9}, sin_zero = "S\233\226\000\000\000\000"}, sin6 = {sin6_len = 0 '\0', sin6_family = 0 '\0', sin6_port = 0, sin6_flowinfo = 9, sin6_addr = {__u6_addr = { __u6_addr8 = "S\233\226\000\000\000\000\000S\233\226\000\000\000\000", __u6_addr16 = {39763, 150, 0, 0, 39763, 150, 0, 0}, __u6_addr32 = {9870163, 0, 9870163, 0}}}, sin6_scope_id = 0}}, id = 135, proto = -108 '\224', send_flags = {f = 143 '\217', blst_imask = 65 'A'}} backup_user_from = (avp_list_t *) 0xc1fb80 backup_user_to = (avp_list_t *) 0xc1fb88 backup_domain_from = (avp_list_t *) 0xc1fb90 backup_domain_to = (avp_list_t *) 0xc1fb98 backup_uri_from = (avp_list_t *) 0xc1fb70 backup_uri_to = (avp_list_t *) 0xc1fb78 backup_xavps = (sr_xavp_t **) 0xc201a8 replies_locked = 0 branch_ret = 1 prev_branch = 0 blst_503_timeout = 1 hf = (struct hdr_field *) 0x972520 onsend_params = {req = 0x45, rpl = 0x7fffffffa4b0, param = 0x8011910dc, code = -23536, flags = 32767, branch = 0, t_rbuf = 0x8013d8b58, dst = 0x7fffffffa400, send_buf = { s = 0x8013f02c0 "�\213��\025\v�6V�\224�O#\034��4&\2301v\224�/J����H��l\022t\231\027aD\213", len = -23632}} ctx = {rec_lev = 540029216, run_flags = 893008176, last_retcode = 942815798, jmp_env = {{_jb = {4050243973957839201, 4991149722077444152, 8026325070018729282, 8530231501994747250, 6716613293192195696, 7232623180324020282, 7809093798452294501, 7214830551494325097, 7956023736295256933, 7791351009094672487, 734218887359656809, 9905464}}}} #8 0x000000000053f8ab in do_forward_reply (msg=0x8020ab268, mode=0) at forward.c:747 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_len = 0 '\0', sa_family = 0 '\0', sa_data = '\0' <repeats 13 times>}, sin = {sin_len = 0 '\0', sin_family = 0 '\0', sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_len = 0 '\0', sin6_family = 0 '\0', sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\0', send_flags = {f = 0 '\0', blst_imask = 0 '\0'}} new_len = 1 r = 1 ip = {af = 9873292, len = 0, u = {addrl = {9872972, 4296596}, addr32 = {9872972, 0, 4296596, 0}, addr16 = {42572, 150, 0, 0, 36756, 65, 0, 0}, addr = "L�\226\000\000\000\000\000\224\217A\000\000\000\000"}} s = 0x969b53 "DEBUG" len = 0 #9 0x000000000053f10a in forward_reply (msg=0x8020ab268) at forward.c:849 No locals. #10 0x000000000045e9e4 in receive_msg ( buf=0xc75210 "SIP/2.0 200 Keepalive\r\nVia: SIP/2.0/UDP 5.159.56.122;branch=z9hG4bK1c19.5b6b0e92", '0' <repeats 24 times>, ".0\r\nTo: <sip:5.159.56.120>;tag=b27e1a1d33761e85846fc98f5f3a7e58.f814\r\nFrom: <sip:dispatcher@voip"..., len=391, rcv_info=0x7fffffffb9d8) at receive.c:299 msg = (struct sip_msg *) 0x8020ab268 ctx = {rec_lev = 895, run_flags = 7, last_retcode = 14882400, jmp_env = {{_jb = {140737488336144, 34359738368, 34372540416, 0, 140737488349968, 140737488350104, 129697256, 0, 140737488349968, 140737488350104, 140737488349976, 15}}}} ret = 8 inb = { s = 0xc75210 "SIP/2.0 200 Keepalive\r\nVia: SIP/2.0/UDP 5.159.56.122;branch=z9hG4bK1c19.5b6b0e92", '0' <repeats 24 times>, ".0\r\nTo: <sip:5.159.56.120>;tag=b27e1a1d33761e85846fc98f5f3a7e58.f814\r\nFrom: <sip:dispatcher@voip"..., len = 391} netinfo = {data = {s = 0x800c35000 "z�P�", len = -19464}, dst = 0x7fffffffb4a0, rcv = 0x7fffffffeb10} #11 0x000000000061eb63 in udp_rcv_loop () at udp_server.c:495 buf = "SIP/2.0 200 Keepalive\r\nVia: SIP/2.0/UDP 5.159.56.122;branch=z9hG4bK1c19.5b6b0e92", '0' <repeats 24 times>, ".0\r\nTo: <sip:5.159.56.120>;tag=b27e1a1d33761e85846fc98f5f3a7e58.f814\r\nFrom: <sip:dispatcher@voip-integration-l"... len = 391 tmp = 0x5 <Address 0x5 out of bounds> from = (union sockaddr_union *) 0x80209e120 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {2016976645, 0}, addr32 = {2016976645, 0, 0, 0}, addr16 = {40709, 30776, 0, 0, 0, 0, 0, 0}, addr = "\005\2378x", '\0' <repeats 11 times>}}, dst_ip = { af = 2, len = 4, u = {addrl = {2050531077, 0}, addr32 = {2050531077, 0, 0, 0}, addr16 = {40709, 31288, 0, 0, 0, 0, 0, 0}, addr = "\005\2378z", '\0' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_len = 16 '\020', sa_family = 2 '\002', sa_data = "\023�\005\2378x\000\000\000\000\000\000\000"}, sin = {sin_len = 16 '\020', sin_family = 2 '\002', sin_port = 50195, sin_addr = {s_addr = 2016976645}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_len = 16 '\020', sin6_family = 2 '\002', sin6_port = 50195, sin6_flowinfo = 2016976645, sin6_addr = {__u6_addr = {__u6_addr8 = '\0' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x8020239f8, proto = 1 '\001'} #12 0x000000000051420d in main_loop () at main.c:1600 i = 4 pid = 0 si = (struct socket_info *) 0x8020239f8 si_desc = "udp receiver child=4 sock=5.159.56.122:5060\000\b\000\000\000\b", '\0' <repeats 15 times>, "\020����\177\000\000\230����\177\000\000\030����\177\000\000\017\000\000\000\000\000\000\000P����\177\000\000�\210w\000\000\000\000\000\b", '\0' <repeats 11 times>, "\003\000\000" nrprocs = 16 woneinit = 1 #13 0x0000000000521f05 in main (argc=15, argv=0x7fffffffeb18) at main.c:2616 cfg_stream = (FILE *) 0x801401170 c = -1 r = 0 tmp = 0x7fffffffede9 "" tmp_len = 32767 port = -5224 proto = 32767 options = 0x9751e5 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 4082520776 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = (struct name_lst *) 0xf p = 0x7fffffffeb18 "�����\177" st = {st_dev = 90, st_ino = 161220, st_mode = 16877, st_nlink = 2, st_uid = 986, st_gid = 986, st_rdev = 345223, st_atim = {tv_sec = 1462193365, tv_nsec = 44093000}, st_mtim = {tv_sec = 1462865509, tv_nsec = 14843000}, st_ctim = {tv_sec = 1462865509, tv_nsec = 14843000}, st_size = 512, st_blocks = 8, st_blksize = 32768, st_flags = 0, st_gen = 0, st_lspare = 0, st_birthtim = {tv_sec = 1462193365, tv_nsec = 43126000}} ``` --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-218087650

Something strange around, because the backtrace is showing a lead to lock_get(lock), which defined to pthread_mutex_lock(lock), but the trace then points to pthread_mutex_destroy(). Have you compiled the sources on the machine running kamailio? --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-218593181

After the kamailio of one of the machines which had pinged each other from dispatcher module was replace with fresh compiled source from master from github, the problem just stops appearing. Both machines now keep running even just one has updated kamailio... I will keep an eye on it. --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-218762412

Closed #604. --- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#event-658740284

Open a new issue, this is more than one year old and the version was 4.4.0, you have a different one. Add the backtraces from gdb. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/604#issuecomment-322973129