#### Use Case Description:
I'm trying to log "user last register status" using function `jsonrpc_notification` from module `jsonrpc-c`
##### invoke in kamailio.cfg: `jsonrpc_notification("last_reg", "{'user': '$au', 'data':['$Ts', '$au', '$ct', '$ua', '$sel(received.proto_ip_port)']}");`
**according to [doc](http://kamailio.org/docs/modules/4.3.x/modules/jsonrpc-c.html#idp2463944):** jsonrpc_notification is *asynchronous delayed job* and *do not receive response*
notifications do not receive a response. Script processing continues in the usual fashion as soon as the notification has been sent.
The rpc server on localhost is implemented with Node.js which never return responses and never close connections ( it only support jsonrpc notification )
#### Observation and Test:
##### Crash occurs I found kamailio crashed periodically about 2-3 days. ( back trace attached at the end)
**settings:** 1. the amount of total registered users are about 15k ~ 50k; register expiration is 1200 Secs. 2. kamailio start with parameter pkg 1G and shm 5G. 3. I had patched jsonrpc_io.c from #268
**json crash log:** ``` Aug 18 22:12:04 SIP-4 kernel: [50538943.565484] kamailio[84670]: segfault at 0 ip 00007f424a3ea3bf sp 00007fff8b3eab60 error 4 in libjson.so.0.1.0[7f424a3e6000+9000] Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core> [main.c:728]: handle_sigs(): child process 84670 exited by a signal 11 Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated ```
sometimes it would not crash, but result in allocation failure. no new connections can be accept.
``` Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core> [tcp_main.c:959]: tcpconn_new(): mem. allocation failure Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core> [tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket ```
##### Test
use `sip-p` to start a stress test with RPS 2000 register and observing results both from `kamcmd core.shmmem` and `top` command
Kamailio died quickly along with depleting shm memory. `top` shows kamailio SHR consumed 4.8G `kamcmd core.shmmem` shows **free** is less than 1MB
##### the more `jsonrpc_notification` call we've made, the quicker kamailio out of memory.
#### MEMDBG
switch on memdbg and shutdown kamailio to get mem status. see Memory still-in-use summary (shm): `shm_strdup` and `jsonrpc_notification` from jsonrpc_request.c
``` Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: ----------------------------- Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : <core> [main.c:569]: cleanup(): Memory still-in-use summary (shm): Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: summarizing all alloc'ed. fragments: Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 4608 bytes from dialog: dlg_var.c: new_dlg_var(97) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 2016 bytes from dialog: dlg_var.c: new_dlg_var(106) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96 size= 1152 bytes from dialog: dlg_var.c: new_dlg_var(116) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40 size= 10568 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40 size= 5520 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 993 size= 7960 bytes from dialog: dlg_handlers.c: dlg_get_iuid_shm_clone(279) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=997818 size= 113547928 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=498909 size= 40377480 bytes from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: ----------------------------- ```
#### crash on jsonrpc back trace
``` #0 0x00007f424a3ea3bf in json_tokener_parse_ex () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #1 0x00007f424a3ebda6 in json_tokener_parse_verbose () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #2 0x00007f424a3ebdfe in json_tokener_parse () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #3 0x00007f424a5f4e1a in cmd_pipe_cb (fd=6, event=2, arg=0x7fff8b3eae60) at jsonrpc_io.c:152 cmd = 0x7f4304a93cf8 __FUNCTION__ = "cmd_pipe_cb" params = 0x1bc0aa0 payload = 0x3985b4d6 req = 0x304a9c5 json = 0x7fff8b3ead00 "0\f\274\001" ns = 0x7f4391ebdc60 "298:{ "jsonrpc": "2.0", "method": "last_reg", "params": { "user": "597438910", "data": [ "1439907102", "597438910", "sip:597438910@211.246.212.119:55430;transport=tcp;+sip.instance=\"<urn:uuid:f1016"... bytes = 140735529528624 g = 0x7f424a1ac2f8 sent = 0 #4 0x00007f424a1af254 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 No symbol table info available. #5 0x00007f424a5f45ef in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7f4391aba578 "10.1.3.9:5013,3 10.1.3.10:5013,2 10.1.3.11:5013,1") at jsonrpc_io.c:92 __FUNCTION__ = "jsonrpc_io_child_process" pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x1bb9dc0}, ev_next = {tqe_next = 0x7f4391ebda20, tqe_prev = 0x1bc0c00}, ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6, ev_base = 0x1bc0aa0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x1bc0fd0}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x1bc0fd0}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 130, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec = 6631108320}, ev_callback = 0x7f424a5f4aa7 <cmd_pipe_cb>, ev_arg = 0x7fff8b3eae60} #6 0x00007f424a5fd50b in child_init (rank=0) at jsonrpc_mod.c:135 pid = 0 #7 0x0000000000596a41 in init_mod_child (m=0x7f4391ab0db8, rank=0) at sr_module.c:898 __FUNCTION__ = "init_mod_child" #8 0x0000000000596779 in init_mod_child (m=0x7f4391ab1110, rank=0) at sr_module.c:895 __FUNCTION__ = "init_mod_child" #9 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924 No locals. #10 0x00000000004aafd3 in main_loop () at main.c:1650 i = 24 pid = 84647 si = 0x0 si_desc = "udp receiver child=23 sock=38.106.58.4:5063\000n\000\000\000~/q\000\001\000\000\000\260\344\213OB\177\000\000\020\261>\213\377\177\000\000\032\264N\000\000\000\000\000`\261>\213\377\177\000\000\000\224\351\221C\177\000\000 \333\330OB\177\000\000\260\344\213OB\177\000\000$!\314\a\000\000\000\000\260\210A\000\001\000\000" nrprocs = 24 __FUNCTION__ = "main_loop" #11 0x00000000004b0b5b in main (argc=13, argv=0x7fff8b3eb3d8) at main.c:2533 cfg_stream = 0x1b1a010 c = -1 r = 0 tmp = 0x7fff8b3ecd59 "" tmp_len = 32579 port = -767281296 proto = 0 options = 0x707db8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 189487301 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x76 p = 0x7fff8b3eb2de "" __FUNCTION__ = "main" quit
```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296
I pushed a patch to fix the leak -- it is in master branch 79459d3f87731309641794a2b881ee1cacc5e1f2
If you test it and report it solves the leak, I will backport.
For the crash, not sure it is related, can you get the output in gdb for the next commands?
``` frame 4 p *cmd ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-134593966
I'm afraid that core files has been deleted. Anyway, after test your patch, it seems there are still have issues.
#### So, let's from the ground up again:
I will attach **two** back trace below, one is crashed **BEFORE your patch** that I reproduced, the other one is crashed **AFTER your patch**
both tested under `kamailio-4.3.1 stable` with only recompiled `jsonrpc-c.so` if you need any further info, just let me know which corefiles and the gdb command.
I will keep corefiles this time : )
#### bt_BEFORE_patch.corefile
``` #0 0x00007fdc4c1e03bf in json_tokener_parse_ex () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #1 0x00007fdc4c1e1da6 in json_tokener_parse_verbose () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #2 0x00007fdc4c1e1dfe in json_tokener_parse () from /lib/x86_64-linux-gnu/libjson.so.0 No symbol table info available. #3 0x00007fdc4c3eae1a in cmd_pipe_cb (fd=6, event=2, arg=0x7ffff51d9750) at jsonrpc_io.c:152 cmd = 0x7fdc71690608 __FUNCTION__ = "cmd_pipe_cb" params = 0x15d4bf0 payload = 0x2ea971e6 req = 0x1c3115d json = 0x7ffff51d95f0 "\200M]\001" ns = 0x7fdc73cce438 "201:{ "jsonrpc": "2.0", "method": "last_reg", "params": { "user": "5978002", "data": [ "1440567940", "5978002", "sip:5978002@38.106.57.69:5060;transport=TCP", "SIPp\/Linux", "tcp:38.106.57.68:5003" "... bytes = 140737305744928 g = 0x7fdc4bfa22f8 sent = 0 #4 0x00007fdc4bfa5254 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 No symbol table info available. #5 0x00007fdc4c3ea5ef in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7fdc738b16e8 "10.1.4.3:5013,3 10.1.4.1:5013,2 10.1.4.2:5013,1") at jsonrpc_io.c:93 __FUNCTION__ = "jsonrpc_io_child_process" pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x15cdf10}, ev_next = {tqe_next = 0x7fdc73cce1f8, tqe_prev = 0x15d4d50}, ev_timeout_pos = {ev_next_with_common_timeout = { tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6, ev_base = 0x15d4bf0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x15d5120}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x15d5120}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 130, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec = 8407324624}, ev_callback = 0x7fdc4c3eaaa7 <cmd_pipe_cb+8>, ev_arg = 0x7ffff51d9750} #6 0x00007fdc4c3f350b in child_init (rank=0) at jsonrpc_mod.c:130 pid = 0 #7 0x0000000000596a41 in init_mod_child (m=0x7fdc738a7db8, rank=0) at sr_module.c:898 __FUNCTION__ = "init_mod_child" #8 0x0000000000596779 in init_mod_child (m=0x7fdc738a8110, rank=0) at sr_module.c:895 __FUNCTION__ = "init_mod_child" #9 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924 No locals. #10 0x00000000004aafd3 in main_loop () at main.c:1650 i = 24 pid = 106880 si = 0x0 si_desc = "udp receiver child=23 sock=38.106.57.68:5003\000\000\000\000~/q\000\001\000\000\000\260TkQ\334\177\000\000\000\232\035\365\377\177\000\000\032\264N\000\000\000\000\000P\232\035\365\377\177\000\000\270:\312s\334\177\000\000\310K\270Q\334\177\000\000\260TkQ\334\177\000\000N\354\070p\000\000\000\000\260\210A\000\001\000\000" nrprocs = 24 __FUNCTION__ = "main_loop" #11 0x00000000004b0b5b in main (argc=13, argv=0x7ffff51d9cc8) at main.c:2533 cfg_stream = 0x152e010 c = -1 r = 0 tmp = 0x7ffff51db7c1 "" tmp_len = 32732 port = -1809602704 proto = 0 options = 0x707db8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 1532920115 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x76 p = 0x7ffff51d9bce "" __FUNCTION__ = "main" quit ```
#### bt_AFTER_patch.corefile
``` #0 0x00007fe61e702165 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007fe61e7053e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x0000000000620cc2 in fm_free (qm=0x7fe5dc50a000, p=0x112dca4002a, file=0x7fe5d7254d6e "jsonrpc-c: jsonrpc_io.c", func=0x7fe5d725710c "free_pipe_cmd", line=551) at mem/f_malloc.c:564 f = 0x7fe5dca4a198 __FUNCTION__ = "fm_free" #3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551 __FUNCTION__ = "free_pipe_cmd" #4 0x00007fe5d72499d1 in cmd_pipe_cb (fd=6, event=2, arg=0x7fff85af5370) at jsonrpc_io.c:241 cmd = 0x7fe5dca4a1d0 __FUNCTION__ = "cmd_pipe_cb" params = 0x2328a40 payload = 0x23285b0 req = 0x0 json = 0x2329170 "" ns = 0x7fe5feb2b438 "201:{ "jsonrpc": "2.0", "method": "last_reg", "params": { "user": "5978002", "data": [ "1440568494", "5978002", "sip:5978002@38.106.57.69:5060;transport=TCP", "SIPp\/Linux", "tcp:38.106.57.68:5003" "... bytes = 206 g = 0x7fe5feac9a98 sent = 1 #5 0x00007fe5d6e02254 in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5 No symbol table info available. #6 0x00007fe5d72475e7 in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7fe5fe70e6e8 "10.1.4.3:5013,3 10.1.4.1:5013,2 10.1.4.2:5013,1") at jsonrpc_io.c:91 __FUNCTION__ = "jsonrpc_io_child_process" pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x2320f10}, ev_next = {tqe_next = 0x7fe5feb2b1f8, tqe_prev = 0x2327d50}, ev_timeout_pos = {ev_next_with_common_timeout = { tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6, ev_base = 0x2327bf0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x2328120}, ev_timeout = {tv_sec = 0, tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x2328120}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 130, ev_pri = 0 '\000', ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec = 6537827312}, ev_callback = 0x7fe5d7247a9f <cmd_pipe_cb>, ev_arg = 0x7fff85af5370} #7 0x00007fe5d7250547 in child_init (rank=0) at jsonrpc_mod.c:135 pid = 0 #8 0x0000000000596a41 in init_mod_child (m=0x7fe5fe704db8, rank=0) at sr_module.c:898 __FUNCTION__ = "init_mod_child" #9 0x0000000000596779 in init_mod_child (m=0x7fe5fe705110, rank=0) at sr_module.c:895 __FUNCTION__ = "init_mod_child" #10 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924 No locals. #11 0x00000000004aafd3 in main_loop () at main.c:1650 i = 24 pid = 108881 si = 0x0 si_desc = "udp receiver child=23 sock=38.106.57.68:5003\000\000\000\000~/q\000\001\000\000\000\260$Q\334\345\177\000\000 V\257\205\377\177\000\000\032\264N\000\000\000\000\000pV\257\205\377\177\000\000\270\n\260\376\345\177\000\000\310\033\236\334\345\177\000\000\260$Q\334\345\177\000\000ZI\301\026\000\000\000\000\260\210A\000\001\000\000" nrprocs = 24 __FUNCTION__ = "main_loop" #12 0x00000000004b0b5b in main (argc=13, argv=0x7fff85af58e8) at main.c:2533 cfg_stream = 0x2281010 c = -1 r = 0 tmp = 0x7fff85af77c1 "" tmp_len = 32742 port = 520714096 proto = 0 options = 0x707db8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 1048170556 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x76 p = 0x7fff85af57ee "" __FUNCTION__ = "main" quit ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-134866858
#### bt_BEFORE_patch.corefile with frame3 p *cmd
**Addition gdb command**
``` frame 3 p *cmd ```
**Output**
``` #3 0x00007fdc4c3eae1a in cmd_pipe_cb (fd=6, event=2, arg=0x7ffff51d9750) at jsonrpc_io.c:152 152 json_object *payload = NULL; $1 = {method = 0x0, params = 0x0, cb_route = 0x0, err_route = 0x0, t_hash = 0, t_label = 0, notify_only = 1, cb_pv = 0x0, msg = 0x0} ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-134869099
Attach **kamailio syslog** when crash:
#### BEFORE_patch.syslog
``` Aug 26 13:45:42 SIP-B03 kernel: [29523793.119992] kamailio[106903]: segfault at 0 ip 00007fdc4c1e03bf sp 00007ffff51d9450 error 4 in libjson.so.0.1.0[7fdc4c1dc000+9000] Aug 26 13:45:42 SIP-B03 /home/pkg/kamailio/sbin/kamailio[106853]: ALERT: <core> [main.c:728]: handle_sigs(): child process 106903 exited by a signal 11 ```
#### AFTER_patch.syslog
``` Aug 26 13:53:49 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108633]: : <core> [mem/f_malloc.c:562]: fm_free(): BUG: fm_free: bad pointer 0x7fc30000000b (out of memory block!), called from jsonrpc-c: jsonrpc_io.c: free_pipe_cmd(553) - aborting Aug 26 13:53:50 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108583]: ALERT: <core> [main.c:728]: handle_sigs(): child process 108633 exited by a signal 6 ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-134872654
Can you get what is at modules/jsonrpc-c/jsonrpc_io.c line 551?
You applied the patch to your version and I cannot match in the source code.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-134951398
oh, sorry for the chaos.
here it is:
#### bt_AFTER_patch.corefile
``` (gdb) frame 3 #3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551 551 shm_free(cmd->method); (gdb) l 546 547 548 void free_pipe_cmd(struct jsonrpc_pipe_cmd *cmd) 549 { 550 if (cmd->method) 551 shm_free(cmd->method); 552 if (cmd->params) 553 shm_free(cmd->params); 554 if (cmd->cb_route) 555 shm_free(cmd->cb_route); ```
#### below is my current diff between patched jsonrpc_io.c and 4.3.1 stable ( jsonrpc_io.c.orig )
``` --- jsonrpc_io.c.orig 2015-07-20 21:16:25.000000000 +0800 +++ jsonrpc_io.c 2015-08-26 13:25:24.628835993 +0800 @@ -85,8 +85,7 @@
if (!connect_servers(server_group)) { - LM_ERR("failed to connect to any servers\n"); - return -1; + LM_WARN("failed to connect to any servers\n"); }
event_dispatch(); @@ -163,7 +162,7 @@
if (!payload) { LM_ERR("Failed to build jsonrpc_request_t (method: %s, params: %s)\n", cmd->method, cmd->params); - return; + goto error; } char *json = (char*)json_object_get_string(payload);
@@ -201,7 +200,7 @@
if (timerfd == -1) { LM_ERR("Could not create timerfd."); - return; + goto error; }
req->timerfd = timerfd; @@ -215,7 +214,7 @@ if (timerfd_settime(timerfd, 0, itime, NULL) == -1) { LM_ERR("Could not set timer."); - return; + goto error; } pkg_free(itime); struct event *timer_ev = pkg_malloc(sizeof(struct event)); @@ -223,7 +222,7 @@ event_set(timer_ev, timerfd, EV_READ, timeout_cb, req); if(event_add(timer_ev, NULL) == -1) { LM_ERR("event_add failed while setting request timer (%s).", strerror(errno)); - return; + goto error; } req->timer_ev = timer_ev; } else if (!sent) { @@ -237,6 +236,10 @@
pkg_free(ns); json_object_put(payload); + if (cmd->notify_only) free_pipe_cmd(cmd); +error: + if (cmd->notify_only) free_pipe_cmd(cmd); + return; }
void socket_cb(int fd, short event, void *arg) @@ -338,6 +341,7 @@
struct jsonrpc_server *server = pkg_malloc(sizeof(struct jsonrpc_server)); CHECK_MALLOC(server); + memset(server, 0, sizeof(struct jsonrpc_server)); char *h = pkg_malloc(strlen(host)+1); CHECK_MALLOC(h);
@@ -365,6 +369,7 @@
selected_group = pkg_malloc(sizeof(struct jsonrpc_server_group)); CHECK_MALLOC(selected_group); + memset(selected_group, 0, sizeof(struct jsonrpc_server_group)); selected_group->priority = priority; selected_group->next_server = server;
```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135039373
Hi, I think your problem is at line 238 ``` @@ -237,6 +236,10 @@
pkg_free(ns); json_object_put(payload); + if (cmd->notify_only) free_pipe_cmd(cmd); +error: + if (cmd->notify_only) free_pipe_cmd(cmd); + return; } ``` You are calling free_pipe_cmd before and after the error: label
Hugh
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135437733
Are the backtrace and the syslog messages from the same case? One is reporting a crash at line 551 and the other at 553.
``` (gdb) frame 3 #3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551 ```
``` Aug 26 13:53:49 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108633]: : <core> [mem/f_malloc.c:562]: fm_free(): BUG: fm_free: bad pointer 0x7fc30000000b (out of memory block!), called from jsonrpc-c: jsonrpc_io.c: free_pipe_cmd(553) - aborting ```
Also, rather interesting that pointer fields of the cmd are null (0x0), but the free is reporting a non-NULL pointer:
``` $1 = {method = 0x0, params = 0x0, cb_route = 0x0, err_route = 0x0, t_hash = 0, t_label = 0, notify_only = 1, cb_pv = 0x0, msg = 0x0} ```
Can you reproduce this crash easily?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135439537
@que273 great hit, thanks! It is missing a return there ...
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135440046
Pushed a commit for it, including some other cleanup operations.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135443860
I will recompile and test it from the latest master branch ec0ba37b9f2b6c60303227ea0527d69896f2930e
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135450282
I compiled brand new kamailio, and use my kamailio.cfg
``` $> git branch -v * master ec0ba37 jsonrpc-c: proper error handling for pipe cmd callback
$> kamcmd core.info { version: kamailio 4.4.0-dev2 id: ec0ba3 compiler: gcc 4.7.2 compiled: 22:33:06 Aug 27 2015 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES } ```
#### Strange Result
* **It NEVER crash** * use `kamcmd core.shmmem` shows still consumed shmmem and did not free * when shmmem free reaches 0 ( or near 0 ), it frees all memory suddenly, and free mem restore max. * when restored all mem, the sipp shows: `Error on TCP connection, remote peer probably closed the socket: Connection reset by peer.` and the test finished. * in the mean time, in kamailio syslog, it only shows:
``` ERROR: <core> [tcp_read.c:271]: tcp_read_data(): error reading: Connection reset by peer (104) ERROR: <core> [tcp_read.c:1296]: tcp_read_req(): ERROR: tcp_read_req: error reading ```
seems like all fd were gone.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135462739
Can you get the shm summary again, when shm free is very low?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135701765
#### This is initial state kamailio just restart
``` root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 531530384 used: 5295024 real_used: 5340528 max_used: 5340528 fragments: 2 } ```
#### begin test:
issue command `kamcmd.shmmem` within every 0.5 secs
p.s. during the test, fragments sometimes up to 4~5000, but reduced quickly.
``` root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 2352 used: 534157344 real_used: 536868560 max_used: 536870912 fragments: 26 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 1936 used: 534157760 real_used: 536868976 max_used: 536870912 fragments: 31 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 1648 used: 534158216 real_used: 536869264 max_used: 536870912 fragments: 18 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 696 used: 534159224 real_used: 536870216 max_used: 536870912 fragments: 13 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 336 used: 534159584 real_used: 536870576 max_used: 536870912 fragments: 6 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 424 used: 534159440 real_used: 536870488 max_used: 536870912 fragments: 11 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 1456 used: 534158464 real_used: 536869456 max_used: 536870912 fragments: 22 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 280 used: 534159640 real_used: 536870632 max_used: 536870912 fragments: 8 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 720 used: 534159200 real_used: 536870192 max_used: 536870912 fragments: 11 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 472 used: 534159560 real_used: 536870440 max_used: 536870912 fragments: 6 }
======APPROXIMATELY BREAK HERE======
root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 1714168 used: 532426880 real_used: 535156744 max_used: 536870912 fragments: 111 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 3521856 used: 530602896 real_used: 533349056 max_used: 536870912 fragments: 213 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 5294736 used: 528814000 real_used: 531576176 max_used: 536870912 fragments: 320 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 7121304 used: 526971528 real_used: 529749608 max_used: 536870912 fragments: 436 } root@SIP-B03:~# kamcmd core.shmmem { total: 536870912 free: 8955368 used: 525124920 real_used: 527915544 max_used: 536870912 fragments: 545 }
```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135775378
I was looking for the summary printed in syslog, like you attached in a comment before, eg:
``` Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: ----------------------------- Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : <core> [main.c:569]: cleanup(): Memory still-in-use summary (shm): Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: summarizing all alloc'ed. fragments: ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135776679
There are lots of
``` Aug 28 23:04:05 SIP-B03 /home/pkg/kamailio/sbin/kamailio[35260]: : <core> [mem/f_malloc.c:431]: fm_malloc(): fm_malloc(0x7f3b170e0000, 64) called from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135) ``` but no statistics, since it will not shutdown or SIGUSR1.
I test under:
``` debug = 0 memdbg = -2 memlog = -2 ```
is anything else I can help?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135804167
I played some human reaction game with kamailio, try to issue `kamailio stop` when I saw shm go down near 0 to get the shm status.
hope this helps. I can play more games if you need : )
### FIRST: missed the point, syslog AFTER kamailio just free shm
**below is time sequence where I stop kamailio** ``` root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 616576 used: 266686528 real_used: 267818880 max_used: 267920976 fragments: 1321 } root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 435008 used: 266839704 real_used: 268000448 max_used: 268001752 fragments: 554 }
===== kamailio free shm between this two messages. =====
root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 1076152 used: 266186912 real_used: 267359304 max_used: 268031368 fragments: 487 } root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 3740208 used: 263494632 real_used: 264695248 max_used: 268031368 fragments: 484 }
===== MANUALLY STOP KAMAILIO HERE ===== ===== AFTER kamailio free shm when shm go down near 0 ===== ``` `kamailio stop`
**below is `Memory status (shm)`, but no `Memory still-in-use summary (shm)` shows in syslog**
``` Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : <core> [main.c:565]: cleanup(): Memory status (shm): Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: fm_status (0x7ff92bb1d000): Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: heap size= 268435456 Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: used= 530664, used+overhead=1748304, free=266687152 Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: max used (+overhead)= 268031368 Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: dumping free list: Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1 fragments no.: 30, unused: 0#012#011#011 bucket size: 8 - 8 (first 8) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2 fragments no.: 14, unused: 0#012#011#011 bucket size: 16 - 16 (first 16) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 3 fragments no.: 21, unused: 0#012#011#011 bucket size: 24 - 24 (first 24) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 4 fragments no.: 7, unused: 0#012#011#011 bucket size: 32 - 32 (first 32) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 5 fragments no.: 7, unused: 0#012#011#011 bucket size: 40 - 40 (first 40) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 6 fragments no.: 6, unused: 0#012#011#011 bucket size: 48 - 48 (first 48) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 8 fragments no.: 24, unused: 0#012#011#011 bucket size: 64 - 64 (first 64) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 10 fragments no.: 1, unused: 0#012#011#011 bucket size: 80 - 80 (first 80) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 11 fragments no.: 1, unused: 0#012#011#011 bucket size: 88 - 88 (first 88) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 12 fragments no.: 33, unused: 0#012#011#011 bucket size: 96 - 96 (first 96) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 13 fragments no.: 31, unused: 0#012#011#011 bucket size: 104 - 104 (first 104) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 14 fragments no.: 5, unused: 0#012#011#011 bucket size: 112 - 112 (first 112) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 29 fragments no.: 2, unused: 0#012#011#011 bucket size: 232 - 232 (first 232) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 32 fragments no.: 1, unused: 0#012#011#011 bucket size: 256 - 256 (first 256) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 35 fragments no.: 1, unused: 0#012#011#011 bucket size: 280 - 280 (first 280) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 40 fragments no.: 1, unused: 0#012#011#011 bucket size: 320 - 320 (first 320) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 86 fragments no.: 1, unused: 0#012#011#011 bucket size: 688 - 688 (first 688) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 98 fragments no.: 1, unused: 0#012#011#011 bucket size: 784 - 784 (first 784) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 99 fragments no.: 4, unused: 0#012#011#011 bucket size: 792 - 792 (first 792) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 100 fragments no.: 4, unused: 0#012#011#011 bucket size: 800 - 800 (first 800) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 101 fragments no.: 11, unused: 0#012#011#011 bucket size: 808 - 808 (first 808) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 102 fragments no.: 5, unused: 0#012#011#011 bucket size: 816 - 816 (first 816) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 103 fragments no.: 4, unused: 0#012#011#011 bucket size: 824 - 824 (first 824) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 104 fragments no.: 7, unused: 0#012#011#011 bucket size: 832 - 832 (first 832) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 105 fragments no.: 2, unused: 0#012#011#011 bucket size: 840 - 840 (first 840) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 106 fragments no.: 5, unused: 0#012#011#011 bucket size: 848 - 848 (first 848) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 107 fragments no.: 3, unused: 0#012#011#011 bucket size: 856 - 856 (first 856) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 108 fragments no.: 2, unused: 0#012#011#011 bucket size: 864 - 864 (first 864) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 109 fragments no.: 3, unused: 0#012#011#011 bucket size: 872 - 872 (first 872) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 110 fragments no.: 3, unused: 0#012#011#011 bucket size: 880 - 880 (first 880) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 111 fragments no.: 5, unused: 0#012#011#011 bucket size: 888 - 888 (first 888) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 112 fragments no.: 4, unused: 0#012#011#011 bucket size: 896 - 896 (first 896) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 113 fragments no.: 4, unused: 0#012#011#011 bucket size: 904 - 904 (first 904) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 114 fragments no.: 3, unused: 0#012#011#011 bucket size: 912 - 912 (first 912) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 115 fragments no.: 3, unused: 0#012#011#011 bucket size: 920 - 920 (first 920) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 116 fragments no.: 1, unused: 0#012#011#011 bucket size: 928 - 928 (first 928) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 117 fragments no.: 4, unused: 0#012#011#011 bucket size: 936 - 936 (first 936) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 119 fragments no.: 2, unused: 0#012#011#011 bucket size: 952 - 952 (first 952) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 120 fragments no.: 3, unused: 0#012#011#011 bucket size: 960 - 960 (first 960) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 122 fragments no.: 5, unused: 0#012#011#011 bucket size: 976 - 976 (first 976) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 123 fragments no.: 1, unused: 0#012#011#011 bucket size: 984 - 984 (first 984) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 124 fragments no.: 2, unused: 0#012#011#011 bucket size: 992 - 992 (first 992) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 125 fragments no.: 1, unused: 0#012#011#011 bucket size: 1000 - 1000 (first 1000) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 126 fragments no.: 2, unused: 0#012#011#011 bucket size: 1008 - 1008 (first 1008) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 127 fragments no.: 4, unused: 0#012#011#011 bucket size: 1016 - 1016 (first 1016) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 128 fragments no.: 5, unused: 0#012#011#011 bucket size: 1024 - 1024 (first 1024) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 129 fragments no.: 3, unused: 0#012#011#011 bucket size: 1032 - 1032 (first 1032) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 130 fragments no.: 4, unused: 0#012#011#011 bucket size: 1040 - 1040 (first 1040) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 132 fragments no.: 2, unused: 0#012#011#011 bucket size: 1056 - 1056 (first 1056) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 134 fragments no.: 3, unused: 0#012#011#011 bucket size: 1072 - 1072 (first 1072) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 135 fragments no.: 1, unused: 0#012#011#011 bucket size: 1080 - 1080 (first 1080) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 136 fragments no.: 3, unused: 0#012#011#011 bucket size: 1088 - 1088 (first 1088) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 137 fragments no.: 2, unused: 0#012#011#011 bucket size: 1096 - 1096 (first 1096) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 138 fragments no.: 1, unused: 0#012#011#011 bucket size: 1104 - 1104 (first 1104) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 139 fragments no.: 2, unused: 0#012#011#011 bucket size: 1112 - 1112 (first 1112) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 140 fragments no.: 3, unused: 0#012#011#011 bucket size: 1120 - 1120 (first 1120) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 141 fragments no.: 3, unused: 0#012#011#011 bucket size: 1128 - 1128 (first 1128) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 143 fragments no.: 1, unused: 0#012#011#011 bucket size: 1144 - 1144 (first 1144) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 144 fragments no.: 2, unused: 0#012#011#011 bucket size: 1152 - 1152 (first 1152) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 145 fragments no.: 4, unused: 0#012#011#011 bucket size: 1160 - 1160 (first 1160) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 146 fragments no.: 1, unused: 0#012#011#011 bucket size: 1168 - 1168 (first 1168) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 147 fragments no.: 2, unused: 0#012#011#011 bucket size: 1176 - 1176 (first 1176) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 148 fragments no.: 2, unused: 0#012#011#011 bucket size: 1184 - 1184 (first 1184) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 150 fragments no.: 2, unused: 0#012#011#011 bucket size: 1200 - 1200 (first 1200) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 151 fragments no.: 1, unused: 0#012#011#011 bucket size: 1208 - 1208 (first 1208) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 152 fragments no.: 1, unused: 0#012#011#011 bucket size: 1216 - 1216 (first 1216) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 153 fragments no.: 2, unused: 0#012#011#011 bucket size: 1224 - 1224 (first 1224) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 154 fragments no.: 2, unused: 0#012#011#011 bucket size: 1232 - 1232 (first 1232) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 155 fragments no.: 1, unused: 0#012#011#011 bucket size: 1240 - 1240 (first 1240) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 157 fragments no.: 2, unused: 0#012#011#011 bucket size: 1256 - 1256 (first 1256) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 159 fragments no.: 2, unused: 0#012#011#011 bucket size: 1272 - 1272 (first 1272) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 160 fragments no.: 2, unused: 0#012#011#011 bucket size: 1280 - 1280 (first 1280) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 161 fragments no.: 1, unused: 0#012#011#011 bucket size: 1288 - 1288 (first 1288) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 162 fragments no.: 2, unused: 0#012#011#011 bucket size: 1296 - 1296 (first 1296) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 163 fragments no.: 1, unused: 0#012#011#011 bucket size: 1304 - 1304 (first 1304) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 164 fragments no.: 1, unused: 0#012#011#011 bucket size: 1312 - 1312 (first 1312) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 165 fragments no.: 2, unused: 0#012#011#011 bucket size: 1320 - 1320 (first 1320) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 166 fragments no.: 1, unused: 0#012#011#011 bucket size: 1328 - 1328 (first 1328) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 168 fragments no.: 2, unused: 0#012#011#011 bucket size: 1344 - 1344 (first 1344) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 169 fragments no.: 2, unused: 0#012#011#011 bucket size: 1352 - 1352 (first 1352) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 170 fragments no.: 2, unused: 0#012#011#011 bucket size: 1360 - 1360 (first 1360) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 171 fragments no.: 1, unused: 0#012#011#011 bucket size: 1368 - 1368 (first 1368) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 172 fragments no.: 1, unused: 0#012#011#011 bucket size: 1376 - 1376 (first 1376) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 174 fragments no.: 1, unused: 0#012#011#011 bucket size: 1392 - 1392 (first 1392) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 175 fragments no.: 2, unused: 0#012#011#011 bucket size: 1400 - 1400 (first 1400) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 178 fragments no.: 1, unused: 0#012#011#011 bucket size: 1424 - 1424 (first 1424) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 182 fragments no.: 1, unused: 0#012#011#011 bucket size: 1456 - 1456 (first 1456) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 184 fragments no.: 1, unused: 0#012#011#011 bucket size: 1472 - 1472 (first 1472) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 185 fragments no.: 2, unused: 0#012#011#011 bucket size: 1480 - 1480 (first 1480) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 186 fragments no.: 2, unused: 0#012#011#011 bucket size: 1488 - 1488 (first 1488) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 187 fragments no.: 1, unused: 0#012#011#011 bucket size: 1496 - 1496 (first 1496) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 195 fragments no.: 1, unused: 0#012#011#011 bucket size: 1560 - 1560 (first 1560) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 203 fragments no.: 1, unused: 0#012#011#011 bucket size: 1624 - 1624 (first 1624) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 204 fragments no.: 1, unused: 0#012#011#011 bucket size: 1632 - 1632 (first 1632) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 216 fragments no.: 1, unused: 0#012#011#011 bucket size: 1728 - 1728 (first 1728) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 218 fragments no.: 1, unused: 0#012#011#011 bucket size: 1744 - 1744 (first 1744) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 221 fragments no.: 1, unused: 0#012#011#011 bucket size: 1768 - 1768 (first 1768) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 233 fragments no.: 2, unused: 0#012#011#011 bucket size: 1864 - 1864 (first 1864) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 235 fragments no.: 1, unused: 0#012#011#011 bucket size: 1880 - 1880 (first 1880) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 237 fragments no.: 1, unused: 0#012#011#011 bucket size: 1896 - 1896 (first 1896) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 241 fragments no.: 1, unused: 0#012#011#011 bucket size: 1928 - 1928 (first 1928) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 243 fragments no.: 1, unused: 0#012#011#011 bucket size: 1944 - 1944 (first 1944) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 246 fragments no.: 1, unused: 0#012#011#011 bucket size: 1968 - 1968 (first 1968) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 248 fragments no.: 1, unused: 0#012#011#011 bucket size: 1984 - 1984 (first 1984) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 255 fragments no.: 1, unused: 0#012#011#011 bucket size: 2040 - 2040 (first 2040) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 262 fragments no.: 1, unused: 0#012#011#011 bucket size: 2096 - 2096 (first 2096) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 265 fragments no.: 1, unused: 0#012#011#011 bucket size: 2120 - 2120 (first 2120) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 269 fragments no.: 1, unused: 0#012#011#011 bucket size: 2152 - 2152 (first 2152) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 270 fragments no.: 1, unused: 0#012#011#011 bucket size: 2160 - 2160 (first 2160) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 272 fragments no.: 1, unused: 0#012#011#011 bucket size: 2176 - 2176 (first 2176) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 277 fragments no.: 1, unused: 0#012#011#011 bucket size: 2216 - 2216 (first 2216) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 295 fragments no.: 1, unused: 0#012#011#011 bucket size: 2360 - 2360 (first 2360) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 300 fragments no.: 1, unused: 0#012#011#011 bucket size: 2400 - 2400 (first 2400) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 309 fragments no.: 1, unused: 0#012#011#011 bucket size: 2472 - 2472 (first 2472) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 326 fragments no.: 1, unused: 0#012#011#011 bucket size: 2608 - 2608 (first 2608) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 426 fragments no.: 1, unused: 0#012#011#011 bucket size: 3408 - 3408 (first 3408) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 450 fragments no.: 1, unused: 0#012#011#011 bucket size: 3600 - 3600 (first 3600) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1024 fragments no.: 2, unused: 0#012#011#011 bucket size: 8192 - 8192 (first 8192) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1313 fragments no.: 1, unused: 0#012#011#011 bucket size: 10504 - 10504 (first 10504) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1800 fragments no.: 1, unused: 0#012#011#011 bucket size: 14400 - 14400 (first 14400) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2048 fragments no.: 3, unused: 0#012#011#011 bucket size: 16384 - 16384 (first 16384) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2049 fragments no.: 15241, unused: 0#012#011#011 bucket size: 16384 - 32768 (first 16400) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2050 fragments no.: 3, unused: 0#012#011#011 bucket size: 32768 - 65536 (first 32768) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2052 fragments no.: 2, unused: 0#012#011#011 bucket size: 131072 - 262144 (first 131088) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2053 fragments no.: 2, unused: 0#012#011#011 bucket size: 262144 - 524288 (first 262144) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2054 fragments no.: 4, unused: 0#012#011#011 bucket size: 524288 - 1048576 (first 524288) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2056 fragments no.: 1, unused: 0#012#011#011 bucket size: 2097152 - 4194304 (first 2097152) Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: TOTAL: 15657 free fragments = 266687152 free bytes Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: ----------------------------- ```
### SECOND: catch it, syslog BEFORE kamailio free shm, it's hard to get the sweet spot
**below is time sequence where I stop kamailio** ``` root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 17630352 used: 249499264 real_used: 250805104 max_used: 250805496 fragments: 4630 } root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 12131352 used: 254980512 real_used: 256304104 max_used: 256304416 fragments: 3934 } root@SIP-B03:/var/log# kamcmd core.shmmem { total: 268435456 free: 7884928 used: 259213272 real_used: 260550528 max_used: 260550600 fragments: 3291 }
===== MANUALLY STOP KAMAILIO HERE ===== ===== BEFORE kamailio free shm ===== ```
`kamailio stop`
**below is `Memory status (shm)`, but no `Memory still-in-use summary (shm)` shows in syslog**
```
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : <core> [main.c:565]: cleanup(): Memory status (shm): Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: fm_status (0x7f41fbcd4000): Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: heap size= 268435456 Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: used= 263736, used+overhead=1510832, free=266924624 Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: max used (+overhead)= 268102440 Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: dumping free list: Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1 fragments no.: 30, unused: 0#012#011#011 bucket size: 8 - 8 (first 8) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2 fragments no.: 33, unused: 0#012#011#011 bucket size: 16 - 16 (first 16) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 3 fragments no.: 82, unused: 0#012#011#011 bucket size: 24 - 24 (first 24) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 4 fragments no.: 40, unused: 0#012#011#011 bucket size: 32 - 32 (first 32) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 5 fragments no.: 211, unused: 0#012#011#011 bucket size: 40 - 40 (first 40) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 6 fragments no.: 57, unused: 0#012#011#011 bucket size: 48 - 48 (first 48) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 7 fragments no.: 42, unused: 0#012#011#011 bucket size: 56 - 56 (first 56) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 8 fragments no.: 212, unused: 0#012#011#011 bucket size: 64 - 64 (first 64) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 9 fragments no.: 92, unused: 0#012#011#011 bucket size: 72 - 72 (first 72) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 10 fragments no.: 1, unused: 0#012#011#011 bucket size: 80 - 80 (first 80) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 11 fragments no.: 1, unused: 0#012#011#011 bucket size: 88 - 88 (first 88) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 12 fragments no.: 33, unused: 0#012#011#011 bucket size: 96 - 96 (first 96) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 13 fragments no.: 107, unused: 0#012#011#011 bucket size: 104 - 104 (first 104) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 14 fragments no.: 115, unused: 0#012#011#011 bucket size: 112 - 112 (first 112) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 15 fragments no.: 10, unused: 0#012#011#011 bucket size: 120 - 120 (first 120) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 16 fragments no.: 239, unused: 0#012#011#011 bucket size: 128 - 128 (first 128) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 17 fragments no.: 12, unused: 0#012#011#011 bucket size: 136 - 136 (first 136) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 18 fragments no.: 206, unused: 0#012#011#011 bucket size: 144 - 144 (first 144) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 19 fragments no.: 3, unused: 0#012#011#011 bucket size: 152 - 152 (first 152) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 20 fragments no.: 11, unused: 0#012#011#011 bucket size: 160 - 160 (first 160) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 21 fragments no.: 3, unused: 0#012#011#011 bucket size: 168 - 168 (first 168) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 22 fragments no.: 6, unused: 0#012#011#011 bucket size: 176 - 176 (first 176) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 23 fragments no.: 4, unused: 0#012#011#011 bucket size: 184 - 184 (first 184) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 24 fragments no.: 30, unused: 0#012#011#011 bucket size: 192 - 192 (first 192) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 25 fragments no.: 23, unused: 0#012#011#011 bucket size: 200 - 200 (first 200) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 26 fragments no.: 18, unused: 0#012#011#011 bucket size: 208 - 208 (first 208) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 27 fragments no.: 2, unused: 0#012#011#011 bucket size: 216 - 216 (first 216) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 28 fragments no.: 5, unused: 0#012#011#011 bucket size: 224 - 224 (first 224) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 29 fragments no.: 569, unused: 0#012#011#011 bucket size: 232 - 232 (first 232) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 30 fragments no.: 7, unused: 0#012#011#011 bucket size: 240 - 240 (first 240) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 31 fragments no.: 32, unused: 0#012#011#011 bucket size: 248 - 248 (first 248) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 32 fragments no.: 12, unused: 0#012#011#011 bucket size: 256 - 256 (first 256) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 33 fragments no.: 19, unused: 0#012#011#011 bucket size: 264 - 264 (first 264) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 34 fragments no.: 6, unused: 0#012#011#011 bucket size: 272 - 272 (first 272) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 35 fragments no.: 6, unused: 0#012#011#011 bucket size: 280 - 280 (first 280) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 36 fragments no.: 5, unused: 0#012#011#011 bucket size: 288 - 288 (first 288) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 37 fragments no.: 11, unused: 0#012#011#011 bucket size: 296 - 296 (first 296) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 38 fragments no.: 7, unused: 0#012#011#011 bucket size: 304 - 304 (first 304) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 39 fragments no.: 44, unused: 0#012#011#011 bucket size: 312 - 312 (first 312) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 40 fragments no.: 35, unused: 0#012#011#011 bucket size: 320 - 320 (first 320) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 41 fragments no.: 65, unused: 0#012#011#011 bucket size: 328 - 328 (first 328) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 42 fragments no.: 5, unused: 0#012#011#011 bucket size: 336 - 336 (first 336) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 43 fragments no.: 28, unused: 0#012#011#011 bucket size: 344 - 344 (first 344) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 44 fragments no.: 11, unused: 0#012#011#011 bucket size: 352 - 352 (first 352) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 45 fragments no.: 25, unused: 0#012#011#011 bucket size: 360 - 360 (first 360) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 46 fragments no.: 6, unused: 0#012#011#011 bucket size: 368 - 368 (first 368) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 47 fragments no.: 7, unused: 0#012#011#011 bucket size: 376 - 376 (first 376) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 48 fragments no.: 4, unused: 0#012#011#011 bucket size: 384 - 384 (first 384) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 49 fragments no.: 6, unused: 0#012#011#011 bucket size: 392 - 392 (first 392) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 50 fragments no.: 14, unused: 0#012#011#011 bucket size: 400 - 400 (first 400) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 51 fragments no.: 5, unused: 0#012#011#011 bucket size: 408 - 408 (first 408) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 52 fragments no.: 48, unused: 0#012#011#011 bucket size: 416 - 416 (first 416) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 53 fragments no.: 4, unused: 0#012#011#011 bucket size: 424 - 424 (first 424) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 54 fragments no.: 45, unused: 0#012#011#011 bucket size: 432 - 432 (first 432) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 55 fragments no.: 9, unused: 0#012#011#011 bucket size: 440 - 440 (first 440) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 56 fragments no.: 10, unused: 0#012#011#011 bucket size: 448 - 448 (first 448) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 57 fragments no.: 2, unused: 0#012#011#011 bucket size: 456 - 456 (first 456) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 58 fragments no.: 10, unused: 0#012#011#011 bucket size: 464 - 464 (first 464) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 59 fragments no.: 2, unused: 0#012#011#011 bucket size: 472 - 472 (first 472) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 60 fragments no.: 6, unused: 0#012#011#011 bucket size: 480 - 480 (first 480) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 61 fragments no.: 11, unused: 0#012#011#011 bucket size: 488 - 488 (first 488) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 62 fragments no.: 2, unused: 0#012#011#011 bucket size: 496 - 496 (first 496) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 63 fragments no.: 1, unused: 0#012#011#011 bucket size: 504 - 504 (first 504) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 64 fragments no.: 7, unused: 0#012#011#011 bucket size: 512 - 512 (first 512) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 65 fragments no.: 24, unused: 0#012#011#011 bucket size: 520 - 520 (first 520) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 66 fragments no.: 9, unused: 0#012#011#011 bucket size: 528 - 528 (first 528) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 67 fragments no.: 12, unused: 0#012#011#011 bucket size: 536 - 536 (first 536) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 68 fragments no.: 6, unused: 0#012#011#011 bucket size: 544 - 544 (first 544) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 69 fragments no.: 8, unused: 0#012#011#011 bucket size: 552 - 552 (first 552) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 70 fragments no.: 1, unused: 0#012#011#011 bucket size: 560 - 560 (first 560) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 71 fragments no.: 6, unused: 0#012#011#011 bucket size: 568 - 568 (first 568) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 72 fragments no.: 1, unused: 0#012#011#011 bucket size: 576 - 576 (first 576) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 73 fragments no.: 3, unused: 0#012#011#011 bucket size: 584 - 584 (first 584) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 74 fragments no.: 1, unused: 0#012#011#011 bucket size: 592 - 592 (first 592) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 75 fragments no.: 10, unused: 0#012#011#011 bucket size: 600 - 600 (first 600) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 76 fragments no.: 4, unused: 0#012#011#011 bucket size: 608 - 608 (first 608) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 77 fragments no.: 12, unused: 0#012#011#011 bucket size: 616 - 616 (first 616) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 78 fragments no.: 4, unused: 0#012#011#011 bucket size: 624 - 624 (first 624) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 79 fragments no.: 5, unused: 0#012#011#011 bucket size: 632 - 632 (first 632) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 81 fragments no.: 2, unused: 0#012#011#011 bucket size: 648 - 648 (first 648) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 82 fragments no.: 3, unused: 0#012#011#011 bucket size: 656 - 656 (first 656) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 83 fragments no.: 2, unused: 0#012#011#011 bucket size: 664 - 664 (first 664) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 84 fragments no.: 3, unused: 0#012#011#011 bucket size: 672 - 672 (first 672) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 85 fragments no.: 4, unused: 0#012#011#011 bucket size: 680 - 680 (first 680) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 86 fragments no.: 1, unused: 0#012#011#011 bucket size: 688 - 688 (first 688) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 87 fragments no.: 1, unused: 0#012#011#011 bucket size: 696 - 696 (first 696) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 88 fragments no.: 7, unused: 0#012#011#011 bucket size: 704 - 704 (first 704) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 89 fragments no.: 3, unused: 0#012#011#011 bucket size: 712 - 712 (first 712) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 90 fragments no.: 7, unused: 0#012#011#011 bucket size: 720 - 720 (first 720) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 91 fragments no.: 3, unused: 0#012#011#011 bucket size: 728 - 728 (first 728) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 92 fragments no.: 2, unused: 0#012#011#011 bucket size: 736 - 736 (first 736) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 94 fragments no.: 2, unused: 0#012#011#011 bucket size: 752 - 752 (first 752) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 95 fragments no.: 1, unused: 0#012#011#011 bucket size: 760 - 760 (first 760) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 97 fragments no.: 1, unused: 0#012#011#011 bucket size: 776 - 776 (first 776) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 98 fragments no.: 2, unused: 0#012#011#011 bucket size: 784 - 784 (first 784) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 99 fragments no.: 3, unused: 0#012#011#011 bucket size: 792 - 792 (first 792) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 100 fragments no.: 2, unused: 0#012#011#011 bucket size: 800 - 800 (first 800) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 101 fragments no.: 4, unused: 0#012#011#011 bucket size: 808 - 808 (first 808) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 102 fragments no.: 2, unused: 0#012#011#011 bucket size: 816 - 816 (first 816) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 103 fragments no.: 1, unused: 0#012#011#011 bucket size: 824 - 824 (first 824) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 104 fragments no.: 3, unused: 0#012#011#011 bucket size: 832 - 832 (first 832) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 105 fragments no.: 1, unused: 0#012#011#011 bucket size: 840 - 840 (first 840) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 106 fragments no.: 1, unused: 0#012#011#011 bucket size: 848 - 848 (first 848) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 107 fragments no.: 4, unused: 0#012#011#011 bucket size: 856 - 856 (first 856) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 108 fragments no.: 2, unused: 0#012#011#011 bucket size: 864 - 864 (first 864) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 109 fragments no.: 1, unused: 0#012#011#011 bucket size: 872 - 872 (first 872) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 110 fragments no.: 2, unused: 0#012#011#011 bucket size: 880 - 880 (first 880) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 111 fragments no.: 1, unused: 0#012#011#011 bucket size: 888 - 888 (first 888) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 112 fragments no.: 1, unused: 0#012#011#011 bucket size: 896 - 896 (first 896) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 113 fragments no.: 1, unused: 0#012#011#011 bucket size: 904 - 904 (first 904) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 116 fragments no.: 1, unused: 0#012#011#011 bucket size: 928 - 928 (first 928) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 117 fragments no.: 3, unused: 0#012#011#011 bucket size: 936 - 936 (first 936) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 119 fragments no.: 1, unused: 0#012#011#011 bucket size: 952 - 952 (first 952) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 123 fragments no.: 4, unused: 0#012#011#011 bucket size: 984 - 984 (first 984) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 124 fragments no.: 1, unused: 0#012#011#011 bucket size: 992 - 992 (first 992) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 126 fragments no.: 1, unused: 0#012#011#011 bucket size: 1008 - 1008 (first 1008) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 130 fragments no.: 1, unused: 0#012#011#011 bucket size: 1040 - 1040 (first 1040) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 131 fragments no.: 1, unused: 0#012#011#011 bucket size: 1048 - 1048 (first 1048) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 132 fragments no.: 1, unused: 0#012#011#011 bucket size: 1056 - 1056 (first 1056) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 134 fragments no.: 1, unused: 0#012#011#011 bucket size: 1072 - 1072 (first 1072) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 136 fragments no.: 2, unused: 0#012#011#011 bucket size: 1088 - 1088 (first 1088) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 137 fragments no.: 1, unused: 0#012#011#011 bucket size: 1096 - 1096 (first 1096) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 139 fragments no.: 1, unused: 0#012#011#011 bucket size: 1112 - 1112 (first 1112) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 145 fragments no.: 1, unused: 0#012#011#011 bucket size: 1160 - 1160 (first 1160) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 154 fragments no.: 1, unused: 0#012#011#011 bucket size: 1232 - 1232 (first 1232) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 158 fragments no.: 1, unused: 0#012#011#011 bucket size: 1264 - 1264 (first 1264) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 164 fragments no.: 1, unused: 0#012#011#011 bucket size: 1312 - 1312 (first 1312) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 166 fragments no.: 1, unused: 0#012#011#011 bucket size: 1328 - 1328 (first 1328) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 170 fragments no.: 1, unused: 0#012#011#011 bucket size: 1360 - 1360 (first 1360) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 171 fragments no.: 1, unused: 0#012#011#011 bucket size: 1368 - 1368 (first 1368) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 186 fragments no.: 1, unused: 0#012#011#011 bucket size: 1488 - 1488 (first 1488) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 187 fragments no.: 1, unused: 0#012#011#011 bucket size: 1496 - 1496 (first 1496) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 191 fragments no.: 1, unused: 0#012#011#011 bucket size: 1528 - 1528 (first 1528) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 201 fragments no.: 1, unused: 0#012#011#011 bucket size: 1608 - 1608 (first 1608) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 450 fragments no.: 1, unused: 0#012#011#011 bucket size: 3600 - 3600 (first 3600) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1024 fragments no.: 2, unused: 0#012#011#011 bucket size: 8192 - 8192 (first 8192) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1313 fragments no.: 1, unused: 0#012#011#011 bucket size: 10504 - 10504 (first 10504) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1800 fragments no.: 1, unused: 0#012#011#011 bucket size: 14400 - 14400 (first 14400) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2048 fragments no.: 3, unused: 0#012#011#011 bucket size: 16384 - 16384 (first 16384) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2049 fragments no.: 15234, unused: 0#012#011#011 bucket size: 16384 - 32768 (first 16400) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2050 fragments no.: 3, unused: 0#012#011#011 bucket size: 32768 - 65536 (first 32768) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2052 fragments no.: 2, unused: 0#012#011#011 bucket size: 131072 - 262144 (first 131088) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2053 fragments no.: 2, unused: 0#012#011#011 bucket size: 262144 - 524288 (first 262144) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2054 fragments no.: 4, unused: 0#012#011#011 bucket size: 524288 - 1048576 (first 524288) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2056 fragments no.: 1, unused: 0#012#011#011 bucket size: 2097152 - 4194304 (first 2097152) Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: TOTAL: 18177 free fragments = 266924624 free bytes Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: ----------------------------- ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-135972338
Back to this after some time caught with other stuff.
The status with the list of free chunks doesn't help at all.
To get the summary, try:
``` memlog=1 ```
in kamailio.cfg
Then, at runtime issue the command:
``` kamcmd corex.shm_summary ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-137688458
Looks like problems not in jsonrpc-c now?
#### points of the command `kamcmd corex.shm_summary` issued ``` root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem { total: 268435456 free: 7586408 used: 259640648 real_used: 260849048 max_used: 260849120 fragments: 2985 }
===FIRST LOG HERE (00:03:32) === root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd corex.shm_summary
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem { total: 268435456 free: 187520 used: 267043232 real_used: 268247936 max_used: 268248120 fragments: 771 }
===SECOND LOG HERE (00:03:35) === root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd corex.shm_summary
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem { total: 268435456 free: 1860208 used: 265306536 real_used: 266575248 max_used: 268430848 fragments: 120 } root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem { total: 268435456 free: 3415784 used: 263698488 real_used: 265019672 max_used: 268430848 fragments: 218 } ```
#### FIRST LOG ( at 00:03:32 )
``` Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: summarizing all alloc'ed. fragments: Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 464 bytes from usrloc: ucontact.c: new_ucontact(98) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 64 bytes from usrloc: urecord.c: new_urecord(58) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 24 bytes from usrloc: urecord.c: new_urecord(65) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 10 size= 336 bytes from usrloc: ../../ut.h: shm_str_dup(691) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 19 size= 1368 bytes from <core>: usr_avp.c: create_avp(175) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1216 size= 81216 bytes from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 24 size= 6144 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 24 size= 3280 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 45 size= 1136 bytes from dialog: dlg_var.c: new_dlg_var(106) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 45 size= 2160 bytes from dialog: dlg_var.c: new_dlg_var(97) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 45 size= 552 bytes from dialog: dlg_var.c: new_dlg_var(116) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 15112 size= 258959232 bytes from <core>: tcp_main.c: tcpconn_new(957) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 57600 bytes from <core>: counters.c: counters_prefork_init(207) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 24 bytes from pv: pv_shv.c: set_shvar_value(243) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 3600 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 14400 bytes from sl: sl_stats.c: init_sl_stats_child(125) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 19200 bytes from tm: t_stats.c: init_tm_stats_child(60) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 6 size= 88 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 784 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 163840 bytes from usrloc: udomain.c: new_udomain(93) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 10504 bytes from <core>: pt.c: init_pt(104) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(932) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(926) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(918) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2432 size= 194672 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 62 size= 6264 bytes from htable: ht_api.c: ht_cell_new(183) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 7 size= 2392320 bytes from htable: ht_api.c: ht_init_tables(336) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8192 bytes from nat_traversal: nat_traversal.c: HashTable_new(552) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from nat_traversal: nat_traversal.c: HashTable_new(543) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from dialog: dlg_cb.c: init_dlg_callback(55) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 80 bytes from dialog: dlg_cb.c: register_dlgcb(124) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16400 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 131088 bytes from dialog: dlg_hash.c: init_dlg_table(277) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_timer.c: init_dlg_timer(57) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32 bytes from dialog: dlg_timer.c: init_dlg_timer(48) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 320 bytes from dialog: dlg_profile.c: new_dlg_profile(187) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4634) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4628) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4620) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4613) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4606) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4600) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4588) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 22 size= 496 bytes from pv: pv_shv.c: add_shvar(177) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 22 size= 1408 bytes from pv: pv_shv.c: add_shvar(170) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 80 bytes from pv: ../../lock_alloc.h: lock_set_alloc(70) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 7 size= 672 bytes from htable: ht_api.c: ht_add_table(275) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: ----------------------------- ```
#### SECOND LOG ( at 00:03:35 )
```
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: summarizing all alloc'ed. fragments: Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230140]: ERROR: <core> [tcp_main.c:959]: tcpconn_new(): mem. allocation failure Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230140]: ERROR: <core> [tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 64 bytes from usrloc: urecord.c: new_urecord(58) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 24 bytes from usrloc: urecord.c: new_urecord(65) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 752 bytes from usrloc: ucontact.c: new_ucontact(98) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 10 size= 368 bytes from usrloc: ../../ut.h: shm_str_dup(691) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 17 size= 1392 bytes from <core>: usr_avp.c: create_avp(175) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 48 size= 2384 bytes from dialog: dlg_var.c: new_dlg_var(97) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 24 size= 6144 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 24 size= 3280 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 48 size= 1312 bytes from dialog: dlg_var.c: new_dlg_var(106) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 48 size= 576 bytes from dialog: dlg_var.c: new_dlg_var(116) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1813 size= 137808 bytes from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 15252 size= 261371984 bytes from <core>: tcp_main.c: tcpconn_new(957) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 57600 bytes from <core>: counters.c: counters_prefork_init(207) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 24 bytes from pv: pv_shv.c: set_shvar_value(243) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 3600 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 14400 bytes from sl: sl_stats.c: init_sl_stats_child(125) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 19200 bytes from tm: t_stats.c: init_tm_stats_child(60) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 6 size= 88 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 784 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 163840 bytes from usrloc: udomain.c: new_udomain(93) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(110) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: pt.c: init_pt(105) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 10504 bytes from <core>: pt.c: init_pt(104) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(932) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(926) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(918) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 3626 size= 344928 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 62 size= 6264 bytes from htable: ht_api.c: ht_cell_new(183) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 7 size= 2392320 bytes from htable: ht_api.c: ht_init_tables(336) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8192 bytes from nat_traversal: nat_traversal.c: HashTable_new(552) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from nat_traversal: nat_traversal.c: HashTable_new(543) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from dialog: dlg_cb.c: init_dlg_callback(55) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 80 bytes from dialog: dlg_cb.c: register_dlgcb(124) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16400 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 131088 bytes from dialog: dlg_hash.c: init_dlg_table(277) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from dialog: dlg_timer.c: init_dlg_timer(57) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32 bytes from dialog: dlg_timer.c: init_dlg_timer(48) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 320 bytes from dialog: dlg_profile.c: new_dlg_profile(187) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4634) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4628) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4620) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4613) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4606) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4600) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4588) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: dst_blacklist.c: init_dst_blacklist(437) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dst_blacklist.c: init_dst_blacklist(430) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(283) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(281) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(280) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(269) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(237) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(219) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: timer.c: init_timer(206) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c: cfg_child_cb_new(830) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 22 size= 496 bytes from pv: pv_shv.c: add_shvar(177) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 22 size= 1408 bytes from pv: pv_shv.c: add_shvar(170) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 80 bytes from pv: ../../lock_alloc.h: lock_set_alloc(70) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 7 size= 672 bytes from htable: ht_api.c: ht_add_table(275) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: count= 1 size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179) Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status: -----------------------------
```
#### Some of my kamailio.cfg settings
``` children=24 async_workers=12 dns=no rev_dns=no udp_mtu = 1472 udp_mtu_try_proto = tcp tcp_max_connections = 200000 tcp_connection_lifetime = 650 tcp_keepalive = yes tcp_keepcnt = 7 tcp_keepintvl = 3 tcp_keepidle = 60 tcp_crlf_ping = yes tcp_defer_accept = 15 tcp_rd_buf_size = 16384 # 16M tcp_conn_wq_max = 200000 # 200k tcp_poll_method = epoll_et open_files_limit = 500000 # Max open files ulimit
modparam("usrloc", "handle_lost_tcp", 1) # --- close_expired_tcp seemed may cause double free on a missed socket sometimes and leads segfault, especially in large amount of online users. disable it explicitly. modparam("usrloc", "close_expired_tcp", 0)
```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-137973153
It looks like a lot of tcp connections are open (over 15 000) -- for each of them is allocated the read/write buffer. Do you expect so many tcp connections? If yes, then you need to allocate more memory for kamailio, it runs out of it.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-138292095
Thanks, we're trying to serve as large amount as we can on a single kamailio server. Anyway, after some test, I found the problem may related to debug level.
If I switch on memdbg, it'll have a HUGE impact to kamailio and cause TCP connection drop. If I leave debug=-2, then it runs well.
Normally it should not switch on memlog on a running production server. So I think this issue can be closed now.
Again, thank you for all your great contributions, really appreciate your help.
### Further information if anyone interest in
Env: ``` start kamailio with pkg 2G, shm 2G sipp with 1000 RPS ```
#### TEST 1: with memdbg on ( lots of logs write to syslog )
It will drop tcp connection along with shm goes down. And sipp will stop and shows `Broken pipe on TCP connection, remote peer probably closed the socket.`
#### TEST 2: disable all logs, at log level -2, nearly no logs at all
It runs well, and did not keep swallowing shm at this level. free shm 208811808 more or less. ``` root@SIP-B03:~# kamcmd core.shmmem { total: 1073741824 free: 208811808 used: 862058976 real_used: 864930016 max_used: 864952488 fragments: 253 } ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#issuecomment-138469916
Closed #296.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/296#event-403343460
-
Daniel-Constantin Mierla -
Hugh Waite
-
zxcpoiu