We started a Kamailio handling about 30k registrations with dmq_usrloc module enabled (only sending the traffic to one other host). It crashes almost right away.
That's what the log says for the instance. You see, there are not only about 45 seconds from starting up to crashing. ``` root@pleisse:~# grep '[35833]' /var/log/kamailio/proxy.log Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:384]: init_avp_params(): using AVP for TM fr_timer is deprecated, use t_set_fr(...) instead Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:416]: init_avp_params(): using AVP for TM fr_inv_timer is deprecated, use t_set_fr(...) instead Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6 Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting ``` We first thought, it is a problem with shm, so we doubled it (from 512 to 1024M), but the problem stays. Those are the last lines from the log. ``` Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35909]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35910]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser /parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6 Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting ```
This is the core dump of process 35833: ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done. done. [New LWP 35833]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 6, Aborted. #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35833)): #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441 #3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623 #4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643 #5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252 #6 destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248 #7 0x00007f3cc3af3a6a in dmq_send_message (peer=peer@entry=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, node=node@entry=0x7f3c89c16880, resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:258 #8 0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, except=except@entry=0x0, resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:167 #9 0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward@entry=1) at notification_peer.c:562 #10 0x00007f3cc3aee88f in destroy () at dmq.c:331 #11 0x000000000056e872 in destroy_modules () at sr_module.c:805 #12 0x00000000004a410d in cleanup (show_status=show_status@entry=1) at main.c:513 #13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655 #14 0x00000000004a66e8 in handle_sigs () at main.c:746 #15 0x00000000004aa976 in main_loop () at main.c:1701 #16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547 #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441 f = <optimized out> size = <optimized out> next = <optimized out> prev = <optimized out> __FUNCTION__ = "qm_free" #3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623 ptr = <optimized out> #4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643 No locals. #5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252 No locals. #6 destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248 No locals. #7 0x00007f3cc3af3a6a in dmq_send_message (peer=peer@entry=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, node=node@entry=0x7f3c89c16880, resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:258 uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd77d0, body = 0x7f3cca9992f0, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024, cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34646c8, callid = 0x0} str_hdr = {s = 0x7f3cca711348 "Max-Forwards: 1\r\nContent-Type: text/plain\r\n", len = 43} from = {s = 0x7f3cca999368 "sip:notification_peer@172.20.40.5:5060ain\r\n", len = 38} to = {s = 0x7f3cca999408 "sip:notification_peer@217.116.118.19:5060", len = 41} cb_param = <optimized out> result = <optimized out> len = <optimized out> __FUNCTION__ = "dmq_send_message" #8 0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, except=except@entry=0x0, resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:167 node = 0x7f3c89c16880 __FUNCTION__ = "bcast_dmq_message" #9 0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward@entry=1) at notification_peer.c:562 body = 0x7f3cca9992f0 ret = <optimized out> __FUNCTION__ = "request_nodelist" #10 0x00007f3cc3aee88f in destroy () at dmq.c:331 __FUNCTION__ = "destroy" #11 0x000000000056e872 in destroy_modules () at sr_module.c:805 t = <optimized out> foo = 0x7f3cca4d5f20 __FUNCTION__ = "destroy_modules" #12 0x00000000004a410d in cleanup (show_status=show_status@entry=1) at main.c:513 memlog = <optimized out> __FUNCTION__ = "cleanup" #13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655 No locals. #14 0x00000000004a66e8 in handle_sigs () at main.c:746 chld = 0 chld_status = 134 memlog = <optimized out> __FUNCTION__ = "handle_sigs" #15 0x00000000004aa976 in main_loop () at main.c:1701 i = <optimized out> pid = <optimized out> si = <optimized out> si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000" nrprocs = <optimized out> __FUNCTION__ = "main_loop" #16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffd03cd9f15 "" tmp_len = 0 port = 1 proto = 1 options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 3232125741 rfd = 4 debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = 0x21 p = <optimized out> st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630}, __unused = {0, 0, 0}} __FUNCTION__ = "main" ```
There is also a core dump of PID 35911: ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done. done. [New LWP 35911]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 6, Aborted. #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35911)): #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441 #3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623 #4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643 #5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252 #6 destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248 #7 0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>, content_type=<optimized out>) at dmq_funcs.c:258 #8 0x00007f3cc38e00b2 in usrloc_dmq_send (body=body@entry=0x7ffd03cd7540, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:261 #9 0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action@entry=1, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:505 #10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node@entry=0x7f3c89c16880) at usrloc_sync.c:211 #11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398 #12 0x00007f3cc3b0710b in worker_loop (id=id@entry=2) at worker.c:105 #13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293 #14 child_init (rank=<optimized out>) at dmq.c:279 #15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank@entry=0) at sr_module.c:915 #16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank@entry=0) at sr_module.c:912 #17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank@entry=0) at sr_module.c:912 #18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank@entry=0) at sr_module.c:912 #19 0x000000000056e91b in init_child (rank=rank@entry=0) at sr_module.c:941 #20 0x00000000004aa78c in main_loop () at main.c:1650 #21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547 #0 0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441 f = <optimized out> size = <optimized out> next = <optimized out> prev = <optimized out> __FUNCTION__ = "qm_free" #3 0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623 ptr = <optimized out> #4 shm_free_params (_p=<optimized out>) at parser/parse_param.c:643 No locals. #5 0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252 No locals. #6 destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248 No locals. #7 0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>, content_type=<optimized out>) at dmq_funcs.c:258 uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd73c0, body = 0x7ffd03cd7540, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024, cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34844c8, callid = 0x0} str_hdr = {s = 0x7f3cca713a20 "Max-Forwards: 1\r\nContent-Type: application/json\r\n", len = 49} from = {s = 0x7f3cca713918 "sip:usrloc@172.20.40.5:50600/UDP\300\300\300\300", len = 27} to = {s = 0x7f3cca713998 "sip:usrloc@217.116.118.19:506047698f4.9e\300\300\300\300", len = 30} cb_param = <optimized out> result = <optimized out> len = <optimized out> __FUNCTION__ = "dmq_send_message" #8 0x00007f3cc38e00b2 in usrloc_dmq_send (body=body@entry=0x7ffd03cd7540, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:261 __FUNCTION__ = "usrloc_dmq_send" #9 0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action@entry=1, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:505 jdoc = {root = 0xd71160, flags = 0, buf = { s = 0xd70070 "{"action":1,"aor":"2241474e0@sipgate.de","ruid":"uloc-564b246e-7416-dec","c":"sip:2241474@192.168.0.140:5060","received":"sip:87.79.157.233:1027","path":"<sip:217.10.79.9;lr;received=sip:87.79.157.233"..., len = 414}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>} flags = <optimized out> __FUNCTION__ = "usrloc_dmq_send_contact" #10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node@entry=0x7f3c89c16880) at usrloc_sync.c:211 rval = <optimized out> len = <optimized out> buf = 0x7f3cca9c8080 cp = 0x7f3ccab62a49 ruid = {s = 0x7f3ccab62a2f "uloc-564b246e-7416-dec\337\321z\205\036", len = 22} aorhash = <optimized out> send_sock = <optimized out> flags = <optimized out> aor = {s = 0x7f3c8608a528 "2241474e0@sipgate.de", len = 20} r = 0x7f3c8608a488 _d = 0x7f3c837257a0 ptr = 0x7f3c8608a5a0 res = <optimized out> __FUNCTION__ = "usrloc_get_all_ucontact" #11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398 content_length = <optimized out> body = <optimized out> jdoc = {root = 0xd69490, flags = 0, buf = {s = 0x7f3c8eb7f378 "{"action":3}", len = 12}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>} it = <optimized out> ci = {ruid = {s = 0x0, len = 0}, c = 0x7ffd03cd7750, received = {s = 0x0, len = 0}, path = 0x7ffd03cd7760, expires = 0, q = 0, callid = 0x7ffd03cd7770, cseq = 0, flags = 2, cflags = 0, user_agent = 0x7ffd03cd7780, sock = 0x0, methods = 0, instance = {s = 0x0, len = 0}, reg_id = 0, server_id = 0, tcpconn_id = -1, keepalive = 0, xavp = 0x0, last_modified = 0} action = <optimized out> expires = <optimized out> cseq = <optimized out> flags = <optimized out> cflags = <optimized out> q = <optimized out> last_modified = <optimized out> methods = <optimized out> reg_id = <optimized out> aor = {s = <optimized out>, len = 0} ruid = <optimized out> c = {s = 0x0, len = 0} received = <optimized out> path = {s = 0x0, len = 0} callid = {s = 0x0, len = 0} user_agent = {s = 0x0, len = 0} instance = <optimized out> __FUNCTION__ = "usrloc_dmq_handle_msg" #12 0x00007f3cc3b0710b in worker_loop (id=id@entry=2) at worker.c:105 worker = 0x7f3c837c9560 current_job = 0x7f3c8eb7bdd8 peer_response = {resp_code = 0, content_type = {s = 0x0, len = 0}, reason = {s = 0x0, len = 0}, body = {s = 0x0, len = 0}} ret_value = <optimized out> not_parsed = 1 dmq_node = 0x7f3c89c16880 __FUNCTION__ = "worker_loop" #13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293 i = <optimized out> newpid = <optimized out> #14 child_init (rank=<optimized out>) at dmq.c:279 No locals. #15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank@entry=0) at sr_module.c:915 __FUNCTION__ = "init_mod_child" #16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank@entry=0) at sr_module.c:912 __FUNCTION__ = "init_mod_child" #17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank@entry=0) at sr_module.c:912 __FUNCTION__ = "init_mod_child" #18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank@entry=0) at sr_module.c:912 __FUNCTION__ = "init_mod_child" #19 0x000000000056e91b in init_child (rank=rank@entry=0) at sr_module.c:941 No locals. #20 0x00000000004aa78c in main_loop () at main.c:1650 i = <optimized out> pid = <optimized out> si = <optimized out> si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000" nrprocs = <optimized out> __FUNCTION__ = "main_loop" #21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffd03cd9f15 "" tmp_len = 0 port = 1 proto = 1 options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 3232125741 rfd = 4 debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = 0x21 p = <optimized out> st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630}, __unused = {0, 0, 0}} __FUNCTION__ = "main" ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/399