[kamailio] DMQ under high load kills Kamailio (#399) - sr-dev

17 Nov 2015


      We started a Kamailio handling about 30k registrations with dmq_usrloc module enabled (only sending the traffic to one other host). It crashes almost right away.
That's what the log says for the instance. You see, there are not only about 45 seconds from starting up to crashing. 
```
root@pleisse:~# grep '[35833]' /var/log/kamailio/proxy.log
Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:384]: init_avp_params(): using AVP for TM fr_timer is deprecated, use t_set_fr(...) instead
Nov 17 14:46:40 pleisse /usr/sbin/kamailio[35833]: WARNING: tm [t_funcs.c:416]: init_avp_params(): using AVP for TM fr_inv_timer is deprecated, use t_set_fr(...) instead
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
```
We first thought, it is a problem with shm, so we doubled it (from 512 to 1024M), but the problem stays.
Those are the last lines from the log.
```
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35838]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35912]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: <core> [sip_msg_clone.c:497]: sip_msg_shm_clone(): cannot allocate memory
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [worker.c:164]: add_dmq_job(): error cloning sip message
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35871]: ERROR: dmq [message.c:67]: dmq_handle_message(): failed to add dmq job
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35909]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35910]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:22 pleisse /usr/sbin/kamailio[35911]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser
/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:728]: handle_sigs(): child process 35911 exited by a signal 6
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: tm [uac.c:296]: t_uac_prepare(): t_uac: short of cell shmem
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: ERROR: dmq [dmq_funcs.c:243]: dmq_send_message(): error in tmb.t_request_within
Nov 17 14:47:24 pleisse /usr/sbin/kamailio[35833]: : <core> [mem/q_malloc.c:453]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f3c896c11c0), called from <core>: parser/parse_param.c: do_free_params(623), first free <core>: parser/parse_param.c: do_free_params(623) - aborting
```
This is the core dump of process 35833:
```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done.
done.
[New LWP 35833]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'.
Program terminated with signal 6, Aborted.
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35833)):
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
#3  0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
#4  shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
#5  0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252
#6  destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248
#7  0x00007f3cc3af3a6a in dmq_send_message (peer=peer@entry=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, node=node@entry=0x7f3c89c16880,
    resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:258
#8  0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, except=except@entry=0x0, resp_cback=resp_cback@entry=0x7f3cc3d11650,
    max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:167
#9  0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward@entry=1) at notification_peer.c:562
#10 0x00007f3cc3aee88f in destroy () at dmq.c:331
#11 0x000000000056e872 in destroy_modules () at sr_module.c:805
#12 0x00000000004a410d in cleanup (show_status=show_status@entry=1) at main.c:513
#13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655
#14 0x00000000004a66e8 in handle_sigs () at main.c:746
#15 0x00000000004aa976 in main_loop () at main.c:1701
#16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
        f = <optimized out>
        size = <optimized out>
        next = <optimized out>
        prev = <optimized out>
        __FUNCTION__ = "qm_free"
#3  0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
        ptr = <optimized out>
#4  shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
No locals.
#5  0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:252
No locals.
#6  destroy_dmq_node (node=0x7f3cc3484c98, shm=<optimized out>) at dmqnode.c:248
No locals.
#7  0x00007f3cc3af3a6a in dmq_send_message (peer=peer@entry=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, node=node@entry=0x7f3c89c16880,
    resp_cback=resp_cback@entry=0x7f3cc3d11650, max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:258
        uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd77d0, body = 0x7f3cca9992f0, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024,
          cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34646c8, callid = 0x0}
        str_hdr = {s = 0x7f3cca711348 "Max-Forwards: 1\r\nContent-Type: text/plain\r\n", len = 43}
        from = {s = 0x7f3cca999368 "sip:notification_peer@172.20.40.5:5060ain\r\n", len = 38}
        to = {s = 0x7f3cca999408 "sip:notification_peer@217.116.118.19:5060", len = 41}
        cb_param = <optimized out>
        result = <optimized out>
        len = <optimized out>
        __FUNCTION__ = "dmq_send_message"
#8  0x00007f3cc3af49b5 in bcast_dmq_message (peer=0x7f3c837c9658, body=body@entry=0x7f3cca9992f0, except=except@entry=0x0, resp_cback=resp_cback@entry=0x7f3cc3d11650,
    max_forwards=max_forwards@entry=1, content_type=content_type@entry=0x7f3cc3d11660) at dmq_funcs.c:167
        node = 0x7f3c89c16880
        __FUNCTION__ = "bcast_dmq_message"
#9  0x00007f3cc3b044a2 in request_nodelist (node=<optimized out>, forward=forward@entry=1) at notification_peer.c:562
        body = 0x7f3cca9992f0
        ret = <optimized out>
        __FUNCTION__ = "request_nodelist"
#10 0x00007f3cc3aee88f in destroy () at dmq.c:331
        __FUNCTION__ = "destroy"
#11 0x000000000056e872 in destroy_modules () at sr_module.c:805
        t = <optimized out>
        foo = 0x7f3cca4d5f20
        __FUNCTION__ = "destroy_modules"
#12 0x00000000004a410d in cleanup (show_status=show_status@entry=1) at main.c:513
        memlog = <optimized out>
        __FUNCTION__ = "cleanup"
#13 0x00000000004a4d98 in shutdown_children (show_status=1, sig=15) at main.c:655
No locals.
#14 0x00000000004a66e8 in handle_sigs () at main.c:746
        chld = 0
        chld_status = 134
        memlog = <optimized out>
        __FUNCTION__ = "handle_sigs"
#15 0x00000000004aa976 in main_loop () at main.c:1701
        i = <optimized out>
        pid = <optimized out>
        si = <optimized out>
        si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000"
        nrprocs = <optimized out>
        __FUNCTION__ = "main_loop"
#16 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7ffd03cd9f15 ""
        tmp_len = 0
        port = 1
        proto = 1
        options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
        ret = -1
        seed = 3232125741
        rfd = 4
        debug_save = <optimized out>
        debug_flag = <optimized out>
        dont_fork_cnt = <optimized out>
        n_lst = 0x21
        p = <optimized out>
        st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0,
          st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630},
          __unused = {0, 0, 0}}
        __FUNCTION__ = "main"
```
There is also a core dump of PID 35911:
```
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/c3/a4fc0cef8da17048bce2dc080e543509e39d37.debug...done.
done.
[New LWP 35911]
warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'.
Program terminated with signal 6, Aborted.
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7f3ccce2e700 (LWP 35911)):
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
#3  0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
#4  shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
#5  0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252
#6  destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248
#7  0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>,
    content_type=<optimized out>) at dmq_funcs.c:258
#8  0x00007f3cc38e00b2 in usrloc_dmq_send (body=body@entry=0x7ffd03cd7540, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:261
#9  0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action@entry=1, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:505
#10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node@entry=0x7f3c89c16880) at usrloc_sync.c:211
#11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398
#12 0x00007f3cc3b0710b in worker_loop (id=id@entry=2) at worker.c:105
#13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293
#14 child_init (rank=<optimized out>) at dmq.c:279
#15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank@entry=0) at sr_module.c:915
#16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank@entry=0) at sr_module.c:912
#17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank@entry=0) at sr_module.c:912
#18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank@entry=0) at sr_module.c:912
#19 0x000000000056e91b in init_child (rank=rank@entry=0) at sr_module.c:941
#20 0x00000000004aa78c in main_loop () at main.c:1650
#21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
#0  0x00007f3ccc4a6165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007f3ccc4a93e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x00000000005f29ae in qm_free (qm=<optimized out>, p=<optimized out>, file=<optimized out>, func=<optimized out>, line=623) at mem/q_malloc.c:441
        f = <optimized out>
        size = <optimized out>
        next = <optimized out>
        prev = <optimized out>
        __FUNCTION__ = "qm_free"
#3  0x0000000000624dd8 in do_free_params (_shm=1, _p=0x0) at parser/parse_param.c:623
        ptr = <optimized out>
#4  shm_free_params (_p=<optimized out>) at parser/parse_param.c:643
No locals.
#5  0x00007f3cc3af95c9 in destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:252
No locals.
#6  destroy_dmq_node (node=0x7f3cc3484218, shm=<optimized out>) at dmqnode.c:248
No locals.
#7  0x00007f3cc3af3a6a in dmq_send_message (peer=<optimized out>, body=0x7ffd03cd7540, node=<optimized out>, resp_cback=<optimized out>, max_forwards=<optimized out>,
    content_type=<optimized out>) at dmq_funcs.c:258
        uac_r = {method = 0x7f3cc3d11340, headers = 0x7ffd03cd73c0, body = 0x7ffd03cd7540, ssock = 0x7f3cc3d116e0, dialog = 0x0, cb_flags = 1024,
          cb = 0x7f3cc3af1e70 <dmq_tm_callback>, cbp = 0x7f3cc34844c8, callid = 0x0}
        str_hdr = {s = 0x7f3cca713a20 "Max-Forwards: 1\r\nContent-Type: application/json\r\n", len = 49}
        from = {s = 0x7f3cca713918 "sip:usrloc@172.20.40.5:50600/UDP\300\300\300\300", len = 27}
        to = {s = 0x7f3cca713998 "sip:usrloc@217.116.118.19:506047698f4.9e\300\300\300\300", len = 30}
        cb_param = <optimized out>
        result = <optimized out>
        len = <optimized out>
        __FUNCTION__ = "dmq_send_message"
#8  0x00007f3cc38e00b2 in usrloc_dmq_send (body=body@entry=0x7ffd03cd7540, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:261
        __FUNCTION__ = "usrloc_dmq_send"
#9  0x00007f3cc38e13fe in usrloc_dmq_send_contact (ptr=<optimized out>, aor=..., action=action@entry=1, node=node@entry=0x7f3c89c16880) at usrloc_sync.c:505
        jdoc = {root = 0xd71160, flags = 0, buf = {
            s = 0xd70070 "{"action":1,"aor":"2241474e0@sipgate.de","ruid":"uloc-564b246e-7416-dec","c":"sip:2241474@192.168.0.140:5060","received":"sip:87.79.157.233:1027","path":"<sip:217.10.79.9;lr;received=sip:87.79.157.233"..., len = 414}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>}
        flags = <optimized out>
        __FUNCTION__ = "usrloc_dmq_send_contact"
#10 0x00007f3cc38e1bfc in usrloc_get_all_ucontact (node=node@entry=0x7f3c89c16880) at usrloc_sync.c:211
        rval = <optimized out>
        len = <optimized out>
        buf = 0x7f3cca9c8080
        cp = 0x7f3ccab62a49
        ruid = {s = 0x7f3ccab62a2f "uloc-564b246e-7416-dec\337\321z\205\036", len = 22}
        aorhash = <optimized out>
        send_sock = <optimized out>
        flags = <optimized out>
        aor = {s = 0x7f3c8608a528 "2241474e0@sipgate.de", len = 20}
        r = 0x7f3c8608a488
        _d = 0x7f3c837257a0
        ptr = 0x7f3c8608a5a0
        res = <optimized out>
        __FUNCTION__ = "usrloc_get_all_ucontact"
#11 0x00007f3cc38e2e96 in usrloc_dmq_handle_msg (msg=<optimized out>, resp=0x7ffd03cd7880, node=0x7f3c89c16880) at usrloc_sync.c:398
        content_length = <optimized out>
        body = <optimized out>
        jdoc = {root = 0xd69490, flags = 0, buf = {s = 0x7f3c8eb7f378 "{"action":3}", len = 12}, malloc_fn = 0x7f3ccc4eea00 <malloc>, free_fn = 0x7f3ccc4ee920 <free>}
        it = <optimized out>
        ci = {ruid = {s = 0x0, len = 0}, c = 0x7ffd03cd7750, received = {s = 0x0, len = 0}, path = 0x7ffd03cd7760, expires = 0, q = 0, callid = 0x7ffd03cd7770, cseq = 0,
          flags = 2, cflags = 0, user_agent = 0x7ffd03cd7780, sock = 0x0, methods = 0, instance = {s = 0x0, len = 0}, reg_id = 0, server_id = 0, tcpconn_id = -1, keepalive = 0,
          xavp = 0x0, last_modified = 0}
        action = <optimized out>
        expires = <optimized out>
        cseq = <optimized out>
        flags = <optimized out>
        cflags = <optimized out>
        q = <optimized out>
        last_modified = <optimized out>
        methods = <optimized out>
        reg_id = <optimized out>
        aor = {s = <optimized out>, len = 0}
        ruid = <optimized out>
        c = {s = 0x0, len = 0}
        received = <optimized out>
        path = {s = 0x0, len = 0}
        callid = {s = 0x0, len = 0}
        user_agent = {s = 0x0, len = 0}
        instance = <optimized out>
        __FUNCTION__ = "usrloc_dmq_handle_msg"
#12 0x00007f3cc3b0710b in worker_loop (id=id@entry=2) at worker.c:105
        worker = 0x7f3c837c9560
        current_job = 0x7f3c8eb7bdd8
        peer_response = {resp_code = 0, content_type = {s = 0x0, len = 0}, reason = {s = 0x0, len = 0}, body = {s = 0x0, len = 0}}
        ret_value = <optimized out>
        not_parsed = 1
        dmq_node = 0x7f3c89c16880
        __FUNCTION__ = "worker_loop"
#13 0x00007f3cc3aee1e8 in child_init (rank=<optimized out>) at dmq.c:293
        i = <optimized out>
        newpid = <optimized out>
#14 child_init (rank=<optimized out>) at dmq.c:279
No locals.
#15 0x000000000056d4f8 in init_mod_child (m=0x7f3cca4d6b38, rank=rank@entry=0) at sr_module.c:915
        __FUNCTION__ = "init_mod_child"
#16 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7030, rank=rank@entry=0) at sr_module.c:912
        __FUNCTION__ = "init_mod_child"
#17 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7308, rank=rank@entry=0) at sr_module.c:912
        __FUNCTION__ = "init_mod_child"
#18 0x000000000056d442 in init_mod_child (m=0x7f3cca4d7930, rank=rank@entry=0) at sr_module.c:912
        __FUNCTION__ = "init_mod_child"
#19 0x000000000056e91b in init_child (rank=rank@entry=0) at sr_module.c:941
No locals.
#20 0x00000000004aa78c in main_loop () at main.c:1650
        i = <optimized out>
        pid = <optimized out>
        si = <optimized out>
        si_desc = "udp receiver child=31 sock=172.20.40.5:5060\000\000\000\000\000\030\216\231\312<\177\000\000\016\000\000\000\000\000\000\000\003\000\000\000\000\000\000\000(\245K\203<\177\000\000\001\000\000\000\000\000\000\000\000Bp\367\353\301<~ \000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\310\222\241\000\000\000\000\000\300\222\241\000\000\000\000"
        nrprocs = <optimized out>
        __FUNCTION__ = "main_loop"
#21 0x0000000000427292 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2547
        cfg_stream = <optimized out>
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7ffd03cd9f15 ""
        tmp_len = 0
        port = 1
        proto = 1
        options = 0x6baf50 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
        ret = -1
        seed = 3232125741
        rfd = 4
        debug_save = <optimized out>
        debug_flag = <optimized out>
        dont_fork_cnt = <optimized out>
        n_lst = 0x21
        p = <optimized out>
        st = {st_dev = 14, st_ino = 23606, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 200, st_blksize = 4096, st_blocks = 0,
          st_atim = {tv_sec = 1447757500, tv_nsec = 873437536}, st_mtim = {tv_sec = 1447767996, tv_nsec = 445431681}, st_ctim = {tv_sec = 1447768000, tv_nsec = 253521630},
          __unused = {0, 0, 0}}
        __FUNCTION__ = "main"
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/399