24 Aug
2015
24 Aug
'15
11:11 a.m.
#### Use Case Description:
I'm trying to log "user last register status" using function
`jsonrpc_notification` from module `jsonrpc-c`
##### invoke in kamailio.cfg:
`jsonrpc_notification("last_reg", "{'user': '$au',
'data':['$Ts', '$au', '$ct', '$ua',
'$sel(received.proto_ip_port)']}");`
**according to
[doc](http://kamailio.org/docs/modules/4.3.x/modules/jsonrpc-c.html#idp24639…
jsonrpc_notification is *asynchronous delayed job* and *do not receive response*
notifications do not receive a response. Script
processing continues in the usual fashion as soon as the notification has been sent.
The rpc server on localhost is implemented with Node.js which never return responses and
never close connections ( it only support jsonrpc notification )
#### Observation and Test:
##### Crash occurs
I found kamailio crashed periodically about 2-3 days. ( back trace attached at the end)
**settings:**
1. the amount of total registered users are about 15k ~ 50k; register expiration is 1200
Secs.
2. kamailio start with parameter pkg 1G and shm 5G.
3. I had patched jsonrpc_io.c from #268
**json crash log:**
```
Aug 18 22:12:04 SIP-4 kernel: [50538943.565484] kamailio[84670]: segfault at 0 ip
00007f424a3ea3bf sp 00007fff8b3eab60 error 4 in libjson.so.0.1.0[7f424a3e6000+9000]
Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core>
[main.c:728]: handle_sigs(): child process 84670 exited by a signal 11
Aug 18 22:12:23 SIP-4 /home/pkg/kamailio/sbin/kamailio[84620]: ALERT: <core>
[main.c:731]: handle_sigs(): core was generated
```
sometimes it would not crash, but result in allocation failure. no new connections can be
accept.
```
Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core>
[tcp_main.c:959]: tcpconn_new(): mem. allocation failure
Aug 20 12:26:04 SIP-5 /home/pkg/kamailio/sbin/kamailio[181104]: ERROR: <core>
[tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket
```
##### Test
use `sip-p` to start a stress test with RPS 2000 register and observing results both from
`kamcmd core.shmmem` and `top` command
Kamailio died quickly along with depleting shm memory.
`top` shows kamailio SHR consumed 4.8G
`kamcmd core.shmmem` shows **free** is less than 1MB
##### the more `jsonrpc_notification` call we've made, the quicker kamailio out of
memory.
#### MEMDBG
switch on memdbg and shutdown kamailio to get mem status.
see Memory still-in-use summary (shm):
`shm_strdup` and `jsonrpc_notification` from jsonrpc_request.c
```
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status:
-----------------------------
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : <core>
[main.c:569]: cleanup(): Memory still-in-use summary (shm):
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: summarizing
all alloc'ed. fragments:
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96
size= 4608 bytes from dialog: dlg_var.c: new_dlg_var(97)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96
size= 2016 bytes from dialog: dlg_var.c: new_dlg_var(106)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 96
size= 1152 bytes from dialog: dlg_var.c: new_dlg_var(116)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40
size= 10568 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 40
size= 5520 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 993
size= 7960 bytes from dialog: dlg_handlers.c: dlg_get_iuid_shm_clone(279)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=997818
size= 113547928 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3
size= 64 bytes from usrloc: udomain.c: build_stat_name(51)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from <core>: pt.c: init_pt(110)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from <core>: pt.c: init_pt(105)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count=498909
size= 40377480 bytes from jsonrpc-c: jsonrpc_request.c: jsonrpc_notification(135)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 3
size= 216 bytes from <core>: timer.c: register_timer(1011)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from <core>: usr_avp.c: init_avps(90)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from <core>: usr_avp.c: init_avps(89)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: count= 1
size= 8 bytes from <core>: mem/shm_mem.c: shm_mem_init_mallocs(179)
Aug 24 14:39:42 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status:
-----------------------------
```
#### crash on jsonrpc back trace
```
#0 0x00007f424a3ea3bf in json_tokener_parse_ex () from
/lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#1 0x00007f424a3ebda6 in json_tokener_parse_verbose () from
/lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#2 0x00007f424a3ebdfe in json_tokener_parse () from /lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#3 0x00007f424a5f4e1a in cmd_pipe_cb (fd=6, event=2, arg=0x7fff8b3eae60) at
jsonrpc_io.c:152
cmd = 0x7f4304a93cf8
__FUNCTION__ = "cmd_pipe_cb"
params = 0x1bc0aa0
payload = 0x3985b4d6
req = 0x304a9c5
json = 0x7fff8b3ead00 "0\f\274\001"
ns = 0x7f4391ebdc60 "298:{ \"jsonrpc\": \"2.0\",
\"method\": \"last_reg\", \"params\": { \"user\":
\"597438910\", \"data\": [ \"1439907102\",
\"597438910\",
\"<sip:597438910@211.246.212.119:55430;transport=tcp>;+sip.instance=\\\"<urn:uuid:f1016"...
bytes = 140735529528624
g = 0x7f424a1ac2f8
sent = 0
#4 0x00007f424a1af254 in event_base_loop () from
/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#5 0x00007f424a5f45ef in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7f4391aba578
"10.1.3.9:5013,3 10.1.3.10:5013,2 10.1.3.11:5013,1")
at jsonrpc_io.c:92
__FUNCTION__ = "jsonrpc_io_child_process"
pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x1bb9dc0}, ev_next =
{tqe_next = 0x7f4391ebda20, tqe_prev = 0x1bc0c00},
ev_timeout_pos = {ev_next_with_common_timeout = {tqe_next = 0xffffffff, tqe_prev
= 0x0}, min_heap_idx = -1}, ev_fd = 6,
ev_base = 0x1bc0aa0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev =
0x1bc0fd0}, ev_timeout = {tv_sec = 0, tv_usec = 0}},
ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev = 0x1bc0fd0},
ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18,
ev_res = 2, ev_flags = 130, ev_pri = 0 '\000', ev_closure = 2
'\002', ev_timeout = {tv_sec = 140733193388081, tv_usec = 6631108320},
ev_callback = 0x7f424a5f4aa7 <cmd_pipe_cb>, ev_arg = 0x7fff8b3eae60}
#6 0x00007f424a5fd50b in child_init (rank=0) at jsonrpc_mod.c:135
pid = 0
#7 0x0000000000596a41 in init_mod_child (m=0x7f4391ab0db8, rank=0) at sr_module.c:898
__FUNCTION__ = "init_mod_child"
#8 0x0000000000596779 in init_mod_child (m=0x7f4391ab1110, rank=0) at sr_module.c:895
__FUNCTION__ = "init_mod_child"
#9 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924
No locals.
#10 0x00000000004aafd3 in main_loop () at main.c:1650
i = 24
pid = 84647
si = 0x0
si_desc = "udp receiver child=23
sock=38.106.58.4:5063\000n\000\000\000~/q\000\001\000\000\000\260\344\213OB\177\000\000\020\261>\213\377\177\000\000\032\264N\000\000\000\000\000`\261>\213\377\177\000\000\000\224\351\221C\177\000\000
\333\330OB\177\000\000\260\344\213OB\177\000\000$!\314\a\000\000\000\000\260\210A\000\001\000\000"
nrprocs = 24
__FUNCTION__ = "main_loop"
#11 0x00000000004b0b5b in main (argc=13, argv=0x7fff8b3eb3d8) at main.c:2533
cfg_stream = 0x1b1a010
c = -1
r = 0
tmp = 0x7fff8b3ecd59 ""
tmp_len = 32579
port = -767281296
proto = 0
options = 0x707db8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 189487301
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x76
p = 0x7fff8b3eb2de ""
__FUNCTION__ = "main"
quit
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296
25 Aug
25 Aug
4:57 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I pushed a patch to fix the leak -- it is in master branch
79459d3f87731309641794a2b881ee1cacc5e1f2
If you test it and report it solves the leak, I will backport.
For the crash, not sure it is related, can you get the output in gdb for the next
commands?
```
frame 4
p *cmd
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-134593966
26 Aug
26 Aug
9:54 a.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I'm afraid that core files has been deleted.
Anyway, after test your patch, it seems there are still have issues.
#### So, let's from the ground up again:
I will attach **two** back trace below,
one is crashed **BEFORE your patch** that I reproduced, the other one is crashed **AFTER
your patch**
both tested under `kamailio-4.3.1 stable` with only recompiled `jsonrpc-c.so`
if you need any further info, just let me know which corefiles and the gdb command.
I will keep corefiles this time : )
#### bt_BEFORE_patch.corefile
```
#0 0x00007fdc4c1e03bf in json_tokener_parse_ex () from
/lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#1 0x00007fdc4c1e1da6 in json_tokener_parse_verbose () from
/lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#2 0x00007fdc4c1e1dfe in json_tokener_parse () from /lib/x86_64-linux-gnu/libjson.so.0
No symbol table info available.
#3 0x00007fdc4c3eae1a in cmd_pipe_cb (fd=6, event=2, arg=0x7ffff51d9750) at
jsonrpc_io.c:152
cmd = 0x7fdc71690608
__FUNCTION__ = "cmd_pipe_cb"
params = 0x15d4bf0
payload = 0x2ea971e6
req = 0x1c3115d
json = 0x7ffff51d95f0 "\200M]\001"
ns = 0x7fdc73cce438 "201:{ \"jsonrpc\": \"2.0\",
\"method\": \"last_reg\", \"params\": { \"user\":
\"5978002\", \"data\": [ \"1440567940\",
\"5978002\", \"<sip:5978002@38.106.57.69:5060>;transport=TCP\",
\"SIPp\\/Linux\", \"tcp:38.106.57.68:5003\" "...
bytes = 140737305744928
g = 0x7fdc4bfa22f8
sent = 0
#4 0x00007fdc4bfa5254 in event_base_loop () from
/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#5 0x00007fdc4c3ea5ef in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7fdc738b16e8
"10.1.4.3:5013,3 10.1.4.1:5013,2 10.1.4.2:5013,1") at jsonrpc_io.c:93
__FUNCTION__ = "jsonrpc_io_child_process"
pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x15cdf10}, ev_next =
{tqe_next = 0x7fdc73cce1f8, tqe_prev = 0x15d4d50}, ev_timeout_pos =
{ev_next_with_common_timeout = {
tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6,
ev_base = 0x15d4bf0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x15d5120},
ev_timeout = {tv_sec = 0,
tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev =
0x15d5120}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 130,
ev_pri = 0 '\000',
ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec =
8407324624}, ev_callback = 0x7fdc4c3eaaa7 <cmd_pipe_cb+8>, ev_arg = 0x7ffff51d9750}
#6 0x00007fdc4c3f350b in child_init (rank=0) at jsonrpc_mod.c:130
pid = 0
#7 0x0000000000596a41 in init_mod_child (m=0x7fdc738a7db8, rank=0) at sr_module.c:898
__FUNCTION__ = "init_mod_child"
#8 0x0000000000596779 in init_mod_child (m=0x7fdc738a8110, rank=0) at sr_module.c:895
__FUNCTION__ = "init_mod_child"
#9 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924
No locals.
#10 0x00000000004aafd3 in main_loop () at main.c:1650
i = 24
pid = 106880
si = 0x0
si_desc = "udp receiver child=23
sock=38.106.57.68:5003\000\000\000\000~/q\000\001\000\000\000\260TkQ\334\177\000\000\000\232\035\365\377\177\000\000\032\264N\000\000\000\000\000P\232\035\365\377\177\000\000\270:\312s\334\177\000\000\310K\270Q\334\177\000\000\260TkQ\334\177\000\000N\354\070p\000\000\000\000\260\210A\000\001\000\000"
nrprocs = 24
__FUNCTION__ = "main_loop"
#11 0x00000000004b0b5b in main (argc=13, argv=0x7ffff51d9cc8) at main.c:2533
cfg_stream = 0x152e010
c = -1
r = 0
tmp = 0x7ffff51db7c1 ""
tmp_len = 32732
port = -1809602704
proto = 0
options = 0x707db8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1532920115
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x76
p = 0x7ffff51d9bce ""
__FUNCTION__ = "main"
quit
```
#### bt_AFTER_patch.corefile
```
#0 0x00007fe61e702165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1 0x00007fe61e7053e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2 0x0000000000620cc2 in fm_free (qm=0x7fe5dc50a000, p=0x112dca4002a, file=0x7fe5d7254d6e
"jsonrpc-c: jsonrpc_io.c", func=0x7fe5d725710c "free_pipe_cmd",
line=551) at mem/f_malloc.c:564
f = 0x7fe5dca4a198
__FUNCTION__ = "fm_free"
#3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551
__FUNCTION__ = "free_pipe_cmd"
#4 0x00007fe5d72499d1 in cmd_pipe_cb (fd=6, event=2, arg=0x7fff85af5370) at
jsonrpc_io.c:241
cmd = 0x7fe5dca4a1d0
__FUNCTION__ = "cmd_pipe_cb"
params = 0x2328a40
payload = 0x23285b0
req = 0x0
json = 0x2329170 ""
ns = 0x7fe5feb2b438 "201:{ \"jsonrpc\": \"2.0\",
\"method\": \"last_reg\", \"params\": { \"user\":
\"5978002\", \"data\": [ \"1440568494\",
\"5978002\", \"<sip:5978002@38.106.57.69:5060>;transport=TCP\",
\"SIPp\\/Linux\", \"tcp:38.106.57.68:5003\" "...
bytes = 206
g = 0x7fe5feac9a98
sent = 1
#5 0x00007fe5d6e02254 in event_base_loop () from
/usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#6 0x00007fe5d72475e7 in jsonrpc_io_child_process (cmd_pipe=6, _servers=0x7fe5fe70e6e8
"10.1.4.3:5013,3 10.1.4.1:5013,2 10.1.4.2:5013,1") at jsonrpc_io.c:91
__FUNCTION__ = "jsonrpc_io_child_process"
pipe_ev = {ev_active_next = {tqe_next = 0x0, tqe_prev = 0x2320f10}, ev_next =
{tqe_next = 0x7fe5feb2b1f8, tqe_prev = 0x2327d50}, ev_timeout_pos =
{ev_next_with_common_timeout = {
tqe_next = 0xffffffff, tqe_prev = 0x0}, min_heap_idx = -1}, ev_fd = 6,
ev_base = 0x2327bf0, _ev = {ev_io = {ev_io_next = {tqe_next = 0x0, tqe_prev = 0x2328120},
ev_timeout = {tv_sec = 0,
tv_usec = 0}}, ev_signal = {ev_signal_next = {tqe_next = 0x0, tqe_prev =
0x2328120}, ev_ncalls = 0, ev_pncalls = 0x0}}, ev_events = 18, ev_res = 2, ev_flags = 130,
ev_pri = 0 '\000',
ev_closure = 2 '\002', ev_timeout = {tv_sec = 140733193388081, tv_usec =
6537827312}, ev_callback = 0x7fe5d7247a9f <cmd_pipe_cb>, ev_arg = 0x7fff85af5370}
#7 0x00007fe5d7250547 in child_init (rank=0) at jsonrpc_mod.c:135
pid = 0
#8 0x0000000000596a41 in init_mod_child (m=0x7fe5fe704db8, rank=0) at sr_module.c:898
__FUNCTION__ = "init_mod_child"
#9 0x0000000000596779 in init_mod_child (m=0x7fe5fe705110, rank=0) at sr_module.c:895
__FUNCTION__ = "init_mod_child"
#10 0x0000000000596d55 in init_child (rank=0) at sr_module.c:924
No locals.
#11 0x00000000004aafd3 in main_loop () at main.c:1650
i = 24
pid = 108881
si = 0x0
si_desc = "udp receiver child=23
sock=38.106.57.68:5003\000\000\000\000~/q\000\001\000\000\000\260$Q\334\345\177\000\000
V\257\205\377\177\000\000\032\264N\000\000\000\000\000pV\257\205\377\177\000\000\270\n\260\376\345\177\000\000\310\033\236\334\345\177\000\000\260$Q\334\345\177\000\000ZI\301\026\000\000\000\000\260\210A\000\001\000\000"
nrprocs = 24
__FUNCTION__ = "main_loop"
#12 0x00000000004b0b5b in main (argc=13, argv=0x7fff85af58e8) at main.c:2533
cfg_stream = 0x2281010
c = -1
r = 0
tmp = 0x7fff85af77c1 ""
tmp_len = 32742
port = 520714096
proto = 0
options = 0x707db8
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"
ret = -1
seed = 1048170556
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x76
p = 0x7fff85af57ee ""
__FUNCTION__ = "main"
quit
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-134866858
9:59 a.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
#### bt_BEFORE_patch.corefile with frame3 p *cmd
**Addition gdb command**
```
frame 3
p *cmd
```
**Output**
```
#3 0x00007fdc4c3eae1a in cmd_pipe_cb (fd=6, event=2, arg=0x7ffff51d9750) at
jsonrpc_io.c:152
152 json_object *payload = NULL;
$1 = {method = 0x0, params = 0x0, cb_route = 0x0, err_route = 0x0, t_hash = 0, t_label =
0, notify_only = 1, cb_pv = 0x0, msg = 0x0}
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-134869099
10:10 a.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Attach **kamailio syslog** when crash:
#### BEFORE_patch.syslog
```
Aug 26 13:45:42 SIP-B03 kernel: [29523793.119992] kamailio[106903]: segfault at 0 ip
00007fdc4c1e03bf sp 00007ffff51d9450 error 4 in libjson.so.0.1.0[7fdc4c1dc000+9000]
Aug 26 13:45:42 SIP-B03 /home/pkg/kamailio/sbin/kamailio[106853]: ALERT: <core>
[main.c:728]: handle_sigs(): child process 106903 exited by a signal 11
```
#### AFTER_patch.syslog
```
Aug 26 13:53:49 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108633]: : <core>
[mem/f_malloc.c:562]: fm_free(): BUG: fm_free: bad pointer 0x7fc30000000b (out of memory
block!), called from jsonrpc-c: jsonrpc_io.c: free_pipe_cmd(553) - aborting
Aug 26 13:53:50 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108583]: ALERT: <core>
[main.c:728]: handle_sigs(): child process 108633 exited by a signal 6
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-134872654
2:19 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Can you get what is at modules/jsonrpc-c/jsonrpc_io.c line 551?
You applied the patch to your version and I cannot match in the source code.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-134951398
5:26 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
oh, sorry for the chaos.
here it is:
#### bt_AFTER_patch.corefile
```
(gdb) frame 3
#3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551
551 shm_free(cmd->method);
(gdb) l
546
547
548 void free_pipe_cmd(struct jsonrpc_pipe_cmd *cmd)
549 {
550 if (cmd->method)
551 shm_free(cmd->method);
552 if (cmd->params)
553 shm_free(cmd->params);
554 if (cmd->cb_route)
555 shm_free(cmd->cb_route);
```
#### below is my current diff between patched jsonrpc_io.c and 4.3.1 stable (
jsonrpc_io.c.orig )
```
--- jsonrpc_io.c.orig 2015-07-20 21:16:25.000000000 +0800
+++ jsonrpc_io.c 2015-08-26 13:25:24.628835993 +0800
@@ -85,8 +85,7 @@
if (!connect_servers(server_group))
{
- LM_ERR("failed to connect to any servers\n");
- return -1;
+ LM_WARN("failed to connect to any servers\n");
}
event_dispatch();
@@ -163,7 +162,7 @@
if (!payload) {
LM_ERR("Failed to build jsonrpc_request_t (method: %s, params: %s)\n",
cmd->method, cmd->params);
- return;
+ goto error;
}
char *json = (char*)json_object_get_string(payload);
@@ -201,7 +200,7 @@
if (timerfd == -1) {
LM_ERR("Could not create timerfd.");
- return;
+ goto error;
}
req->timerfd = timerfd;
@@ -215,7 +214,7 @@
if (timerfd_settime(timerfd, 0, itime, NULL) == -1)
{
LM_ERR("Could not set timer.");
- return;
+ goto error;
}
pkg_free(itime);
struct event *timer_ev = pkg_malloc(sizeof(struct event));
@@ -223,7 +222,7 @@
event_set(timer_ev, timerfd, EV_READ, timeout_cb, req);
if(event_add(timer_ev, NULL) == -1) {
LM_ERR("event_add failed while setting request timer (%s).",
strerror(errno));
- return;
+ goto error;
}
req->timer_ev = timer_ev;
} else if (!sent) {
@@ -237,6 +236,10 @@
pkg_free(ns);
json_object_put(payload);
+ if (cmd->notify_only) free_pipe_cmd(cmd);
+error:
+ if (cmd->notify_only) free_pipe_cmd(cmd);
+ return;
}
void socket_cb(int fd, short event, void *arg)
@@ -338,6 +341,7 @@
struct jsonrpc_server *server = pkg_malloc(sizeof(struct jsonrpc_server));
CHECK_MALLOC(server);
+ memset(server, 0, sizeof(struct jsonrpc_server));
char *h = pkg_malloc(strlen(host)+1);
CHECK_MALLOC(h);
@@ -365,6 +369,7 @@
selected_group = pkg_malloc(sizeof(struct jsonrpc_server_group));
CHECK_MALLOC(selected_group);
+ memset(selected_group, 0, sizeof(struct jsonrpc_server_group));
selected_group->priority = priority;
selected_group->next_server = server;
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135039373
27 Aug
27 Aug
4:49 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Hi,
I think your problem is at line 238
```
@@ -237,6 +236,10 @@
pkg_free(ns);
json_object_put(payload);
+ if (cmd->notify_only) free_pipe_cmd(cmd);
+error:
+ if (cmd->notify_only) free_pipe_cmd(cmd);
+ return;
}
```
You are calling free_pipe_cmd before and after the error: label
Hugh
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135437733
4:52 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Are the backtrace and the syslog messages from the same case? One is reporting a crash at
line 551 and the other at 553.
```
(gdb) frame 3
#3 0x00007fe5d724f9ea in free_pipe_cmd (cmd=0x7fe5dca4a1d0) at jsonrpc_io.c:551
```
```
Aug 26 13:53:49 SIP-B03 /home/pkg/kamailio/sbin/kamailio[108633]: : <core>
[mem/f_malloc.c:562]: fm_free(): BUG: fm_free: bad pointer 0x7fc30000000b (out of memory
block!), called from jsonrpc-c: jsonrpc_io.c: free_pipe_cmd(553) - aborting
```
Also, rather interesting that pointer fields of the cmd are null (0x0), but the free is
reporting a non-NULL pointer:
```
$1 = {method = 0x0, params = 0x0, cb_route = 0x0, err_route = 0x0, t_hash = 0, t_label =
0, notify_only = 1, cb_pv = 0x0, msg = 0x0}
```
Can you reproduce this crash easily?
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135439537
4:54 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
@que273 great hit, thanks! It is missing a return there ...
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135440046
5 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Pushed a commit for it, including some other cleanup operations.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135443860
5:20 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I will recompile and test it from the latest master branch
ec0ba37b9f2b6c60303227ea0527d69896f2930e
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135450282
6:05 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I compiled brand new kamailio, and use my kamailio.cfg
```
$> git branch -v
* master ec0ba37 jsonrpc-c: proper error handling for pipe cmd callback
$> kamcmd core.info
{
version: kamailio 4.4.0-dev2
id: ec0ba3
compiler: gcc 4.7.2
compiled: 22:33:06 Aug 27 2015
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
}
```
#### Strange Result
* **It NEVER crash**
* use `kamcmd core.shmmem` shows still consumed shmmem and did not free
* when shmmem free reaches 0 ( or near 0 ), it frees all memory suddenly, and free mem
restore max.
* when restored all mem, the sipp shows:
`Error on TCP connection, remote peer probably closed the socket: Connection reset by
peer.`
and the test finished.
* in the mean time, in kamailio syslog, it only shows:
```
ERROR: <core> [tcp_read.c:271]: tcp_read_data(): error reading: Connection reset by
peer (104)
ERROR: <core> [tcp_read.c:1296]: tcp_read_req(): ERROR: tcp_read_req: error reading
```
seems like all fd were gone.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135462739
28 Aug
28 Aug
12:12 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Can you get the shm summary again, when shm free is very low?
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135701765
4:27 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
#### This is initial state kamailio just restart
```
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 531530384
used: 5295024
real_used: 5340528
max_used: 5340528
fragments: 2
}
```
#### begin test:
issue command `kamcmd.shmmem` within every 0.5 secs
p.s. during the test, fragments sometimes up to 4~5000, but reduced quickly.
```
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 2352
used: 534157344
real_used: 536868560
max_used: 536870912
fragments: 26
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 1936
used: 534157760
real_used: 536868976
max_used: 536870912
fragments: 31
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 1648
used: 534158216
real_used: 536869264
max_used: 536870912
fragments: 18
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 696
used: 534159224
real_used: 536870216
max_used: 536870912
fragments: 13
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 336
used: 534159584
real_used: 536870576
max_used: 536870912
fragments: 6
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 424
used: 534159440
real_used: 536870488
max_used: 536870912
fragments: 11
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 1456
used: 534158464
real_used: 536869456
max_used: 536870912
fragments: 22
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 280
used: 534159640
real_used: 536870632
max_used: 536870912
fragments: 8
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 720
used: 534159200
real_used: 536870192
max_used: 536870912
fragments: 11
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 472
used: 534159560
real_used: 536870440
max_used: 536870912
fragments: 6
}
======APPROXIMATELY BREAK HERE======
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 1714168
used: 532426880
real_used: 535156744
max_used: 536870912
fragments: 111
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 3521856
used: 530602896
real_used: 533349056
max_used: 536870912
fragments: 213
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 5294736
used: 528814000
real_used: 531576176
max_used: 536870912
fragments: 320
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 7121304
used: 526971528
real_used: 529749608
max_used: 536870912
fragments: 436
}
root@SIP-B03:~# kamcmd core.shmmem
{
total: 536870912
free: 8955368
used: 525124920
real_used: 527915544
max_used: 536870912
fragments: 545
}
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135775378
4:34 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I was looking for the summary printed in syslog, like you attached in a comment before,
eg:
```
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status:
-----------------------------
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : <core>
[main.c:569]: cleanup(): Memory still-in-use summary (shm):
Aug 24 14:39:41 SIP-6 /home/pkg/kamailio/sbin/kamailio[263619]: : fm_status: summarizing
all alloc'ed. fragments:
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135776679
6:21 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
There are lots of
```
Aug 28 23:04:05 SIP-B03 /home/pkg/kamailio/sbin/kamailio[35260]: : <core>
[mem/f_malloc.c:431]: fm_malloc(): fm_malloc(0x7f3b170e0000, 64) called from jsonrpc-c:
jsonrpc_request.c: jsonrpc_notification(135)
```
but no statistics, since it will not shutdown or SIGUSR1.
I test under:
```
debug = 0
memdbg = -2
memlog = -2
```
is anything else I can help?
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135804167
29 Aug
29 Aug
2 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
I played some human reaction game with kamailio, try to issue `kamailio stop` when I saw
shm go down near 0 to get the shm status.
hope this helps.
I can play more games if you need : )
### FIRST: missed the point, syslog AFTER kamailio just free shm
**below is time sequence where I stop kamailio**
```
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 616576
used: 266686528
real_used: 267818880
max_used: 267920976
fragments: 1321
}
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 435008
used: 266839704
real_used: 268000448
max_used: 268001752
fragments: 554
}
===== kamailio free shm between this two messages. =====
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 1076152
used: 266186912
real_used: 267359304
max_used: 268031368
fragments: 487
}
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 3740208
used: 263494632
real_used: 264695248
max_used: 268031368
fragments: 484
}
===== MANUALLY STOP KAMAILIO HERE =====
===== AFTER kamailio free shm when shm go down near 0 =====
```
`kamailio stop`
**below is `Memory status (shm)`, but no `Memory still-in-use summary (shm)` shows in
syslog**
```
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : <core>
[main.c:565]: cleanup(): Memory status (shm):
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: fm_status
(0x7ff92bb1d000):
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: heap size=
268435456
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: used=
530664, used+overhead=1748304, free=266687152
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: max used
(+overhead)= 268031368
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: dumping
free list:
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1
fragments no.: 30, unused: 0#012#011#011 bucket size: 8 - 8 (first
8)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2
fragments no.: 14, unused: 0#012#011#011 bucket size: 16 - 16 (first
16)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 3
fragments no.: 21, unused: 0#012#011#011 bucket size: 24 - 24 (first
24)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 4
fragments no.: 7, unused: 0#012#011#011 bucket size: 32 - 32 (first
32)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 5
fragments no.: 7, unused: 0#012#011#011 bucket size: 40 - 40 (first
40)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 6
fragments no.: 6, unused: 0#012#011#011 bucket size: 48 - 48 (first
48)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 8
fragments no.: 24, unused: 0#012#011#011 bucket size: 64 - 64 (first
64)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 10
fragments no.: 1, unused: 0#012#011#011 bucket size: 80 - 80 (first
80)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 11
fragments no.: 1, unused: 0#012#011#011 bucket size: 88 - 88 (first
88)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 12
fragments no.: 33, unused: 0#012#011#011 bucket size: 96 - 96 (first
96)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 13
fragments no.: 31, unused: 0#012#011#011 bucket size: 104 - 104 (first
104)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 14
fragments no.: 5, unused: 0#012#011#011 bucket size: 112 - 112 (first
112)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 29
fragments no.: 2, unused: 0#012#011#011 bucket size: 232 - 232 (first
232)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 32
fragments no.: 1, unused: 0#012#011#011 bucket size: 256 - 256 (first
256)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 35
fragments no.: 1, unused: 0#012#011#011 bucket size: 280 - 280 (first
280)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 40
fragments no.: 1, unused: 0#012#011#011 bucket size: 320 - 320 (first
320)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 86
fragments no.: 1, unused: 0#012#011#011 bucket size: 688 - 688 (first
688)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 98
fragments no.: 1, unused: 0#012#011#011 bucket size: 784 - 784 (first
784)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 99
fragments no.: 4, unused: 0#012#011#011 bucket size: 792 - 792 (first
792)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 100
fragments no.: 4, unused: 0#012#011#011 bucket size: 800 - 800 (first
800)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 101
fragments no.: 11, unused: 0#012#011#011 bucket size: 808 - 808 (first
808)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 102
fragments no.: 5, unused: 0#012#011#011 bucket size: 816 - 816 (first
816)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 103
fragments no.: 4, unused: 0#012#011#011 bucket size: 824 - 824 (first
824)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 104
fragments no.: 7, unused: 0#012#011#011 bucket size: 832 - 832 (first
832)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 105
fragments no.: 2, unused: 0#012#011#011 bucket size: 840 - 840 (first
840)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 106
fragments no.: 5, unused: 0#012#011#011 bucket size: 848 - 848 (first
848)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 107
fragments no.: 3, unused: 0#012#011#011 bucket size: 856 - 856 (first
856)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 108
fragments no.: 2, unused: 0#012#011#011 bucket size: 864 - 864 (first
864)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 109
fragments no.: 3, unused: 0#012#011#011 bucket size: 872 - 872 (first
872)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 110
fragments no.: 3, unused: 0#012#011#011 bucket size: 880 - 880 (first
880)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 111
fragments no.: 5, unused: 0#012#011#011 bucket size: 888 - 888 (first
888)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 112
fragments no.: 4, unused: 0#012#011#011 bucket size: 896 - 896 (first
896)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 113
fragments no.: 4, unused: 0#012#011#011 bucket size: 904 - 904 (first
904)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 114
fragments no.: 3, unused: 0#012#011#011 bucket size: 912 - 912 (first
912)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 115
fragments no.: 3, unused: 0#012#011#011 bucket size: 920 - 920 (first
920)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 116
fragments no.: 1, unused: 0#012#011#011 bucket size: 928 - 928 (first
928)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 117
fragments no.: 4, unused: 0#012#011#011 bucket size: 936 - 936 (first
936)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 119
fragments no.: 2, unused: 0#012#011#011 bucket size: 952 - 952 (first
952)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 120
fragments no.: 3, unused: 0#012#011#011 bucket size: 960 - 960 (first
960)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 122
fragments no.: 5, unused: 0#012#011#011 bucket size: 976 - 976 (first
976)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 123
fragments no.: 1, unused: 0#012#011#011 bucket size: 984 - 984 (first
984)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 124
fragments no.: 2, unused: 0#012#011#011 bucket size: 992 - 992 (first
992)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 125
fragments no.: 1, unused: 0#012#011#011 bucket size: 1000 - 1000 (first
1000)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 126
fragments no.: 2, unused: 0#012#011#011 bucket size: 1008 - 1008 (first
1008)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 127
fragments no.: 4, unused: 0#012#011#011 bucket size: 1016 - 1016 (first
1016)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 128
fragments no.: 5, unused: 0#012#011#011 bucket size: 1024 - 1024 (first
1024)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 129
fragments no.: 3, unused: 0#012#011#011 bucket size: 1032 - 1032 (first
1032)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 130
fragments no.: 4, unused: 0#012#011#011 bucket size: 1040 - 1040 (first
1040)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 132
fragments no.: 2, unused: 0#012#011#011 bucket size: 1056 - 1056 (first
1056)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 134
fragments no.: 3, unused: 0#012#011#011 bucket size: 1072 - 1072 (first
1072)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 135
fragments no.: 1, unused: 0#012#011#011 bucket size: 1080 - 1080 (first
1080)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 136
fragments no.: 3, unused: 0#012#011#011 bucket size: 1088 - 1088 (first
1088)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 137
fragments no.: 2, unused: 0#012#011#011 bucket size: 1096 - 1096 (first
1096)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 138
fragments no.: 1, unused: 0#012#011#011 bucket size: 1104 - 1104 (first
1104)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 139
fragments no.: 2, unused: 0#012#011#011 bucket size: 1112 - 1112 (first
1112)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 140
fragments no.: 3, unused: 0#012#011#011 bucket size: 1120 - 1120 (first
1120)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 141
fragments no.: 3, unused: 0#012#011#011 bucket size: 1128 - 1128 (first
1128)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 143
fragments no.: 1, unused: 0#012#011#011 bucket size: 1144 - 1144 (first
1144)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 144
fragments no.: 2, unused: 0#012#011#011 bucket size: 1152 - 1152 (first
1152)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 145
fragments no.: 4, unused: 0#012#011#011 bucket size: 1160 - 1160 (first
1160)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 146
fragments no.: 1, unused: 0#012#011#011 bucket size: 1168 - 1168 (first
1168)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 147
fragments no.: 2, unused: 0#012#011#011 bucket size: 1176 - 1176 (first
1176)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 148
fragments no.: 2, unused: 0#012#011#011 bucket size: 1184 - 1184 (first
1184)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 150
fragments no.: 2, unused: 0#012#011#011 bucket size: 1200 - 1200 (first
1200)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 151
fragments no.: 1, unused: 0#012#011#011 bucket size: 1208 - 1208 (first
1208)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 152
fragments no.: 1, unused: 0#012#011#011 bucket size: 1216 - 1216 (first
1216)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 153
fragments no.: 2, unused: 0#012#011#011 bucket size: 1224 - 1224 (first
1224)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 154
fragments no.: 2, unused: 0#012#011#011 bucket size: 1232 - 1232 (first
1232)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 155
fragments no.: 1, unused: 0#012#011#011 bucket size: 1240 - 1240 (first
1240)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 157
fragments no.: 2, unused: 0#012#011#011 bucket size: 1256 - 1256 (first
1256)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 159
fragments no.: 2, unused: 0#012#011#011 bucket size: 1272 - 1272 (first
1272)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 160
fragments no.: 2, unused: 0#012#011#011 bucket size: 1280 - 1280 (first
1280)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 161
fragments no.: 1, unused: 0#012#011#011 bucket size: 1288 - 1288 (first
1288)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 162
fragments no.: 2, unused: 0#012#011#011 bucket size: 1296 - 1296 (first
1296)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 163
fragments no.: 1, unused: 0#012#011#011 bucket size: 1304 - 1304 (first
1304)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 164
fragments no.: 1, unused: 0#012#011#011 bucket size: 1312 - 1312 (first
1312)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 165
fragments no.: 2, unused: 0#012#011#011 bucket size: 1320 - 1320 (first
1320)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 166
fragments no.: 1, unused: 0#012#011#011 bucket size: 1328 - 1328 (first
1328)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 168
fragments no.: 2, unused: 0#012#011#011 bucket size: 1344 - 1344 (first
1344)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 169
fragments no.: 2, unused: 0#012#011#011 bucket size: 1352 - 1352 (first
1352)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 170
fragments no.: 2, unused: 0#012#011#011 bucket size: 1360 - 1360 (first
1360)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 171
fragments no.: 1, unused: 0#012#011#011 bucket size: 1368 - 1368 (first
1368)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 172
fragments no.: 1, unused: 0#012#011#011 bucket size: 1376 - 1376 (first
1376)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 174
fragments no.: 1, unused: 0#012#011#011 bucket size: 1392 - 1392 (first
1392)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 175
fragments no.: 2, unused: 0#012#011#011 bucket size: 1400 - 1400 (first
1400)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 178
fragments no.: 1, unused: 0#012#011#011 bucket size: 1424 - 1424 (first
1424)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 182
fragments no.: 1, unused: 0#012#011#011 bucket size: 1456 - 1456 (first
1456)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 184
fragments no.: 1, unused: 0#012#011#011 bucket size: 1472 - 1472 (first
1472)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 185
fragments no.: 2, unused: 0#012#011#011 bucket size: 1480 - 1480 (first
1480)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 186
fragments no.: 2, unused: 0#012#011#011 bucket size: 1488 - 1488 (first
1488)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 187
fragments no.: 1, unused: 0#012#011#011 bucket size: 1496 - 1496 (first
1496)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 195
fragments no.: 1, unused: 0#012#011#011 bucket size: 1560 - 1560 (first
1560)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 203
fragments no.: 1, unused: 0#012#011#011 bucket size: 1624 - 1624 (first
1624)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 204
fragments no.: 1, unused: 0#012#011#011 bucket size: 1632 - 1632 (first
1632)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 216
fragments no.: 1, unused: 0#012#011#011 bucket size: 1728 - 1728 (first
1728)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 218
fragments no.: 1, unused: 0#012#011#011 bucket size: 1744 - 1744 (first
1744)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 221
fragments no.: 1, unused: 0#012#011#011 bucket size: 1768 - 1768 (first
1768)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 233
fragments no.: 2, unused: 0#012#011#011 bucket size: 1864 - 1864 (first
1864)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 235
fragments no.: 1, unused: 0#012#011#011 bucket size: 1880 - 1880 (first
1880)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 237
fragments no.: 1, unused: 0#012#011#011 bucket size: 1896 - 1896 (first
1896)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 241
fragments no.: 1, unused: 0#012#011#011 bucket size: 1928 - 1928 (first
1928)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 243
fragments no.: 1, unused: 0#012#011#011 bucket size: 1944 - 1944 (first
1944)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 246
fragments no.: 1, unused: 0#012#011#011 bucket size: 1968 - 1968 (first
1968)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 248
fragments no.: 1, unused: 0#012#011#011 bucket size: 1984 - 1984 (first
1984)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 255
fragments no.: 1, unused: 0#012#011#011 bucket size: 2040 - 2040 (first
2040)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 262
fragments no.: 1, unused: 0#012#011#011 bucket size: 2096 - 2096 (first
2096)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 265
fragments no.: 1, unused: 0#012#011#011 bucket size: 2120 - 2120 (first
2120)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 269
fragments no.: 1, unused: 0#012#011#011 bucket size: 2152 - 2152 (first
2152)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 270
fragments no.: 1, unused: 0#012#011#011 bucket size: 2160 - 2160 (first
2160)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 272
fragments no.: 1, unused: 0#012#011#011 bucket size: 2176 - 2176 (first
2176)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 277
fragments no.: 1, unused: 0#012#011#011 bucket size: 2216 - 2216 (first
2216)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 295
fragments no.: 1, unused: 0#012#011#011 bucket size: 2360 - 2360 (first
2360)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 300
fragments no.: 1, unused: 0#012#011#011 bucket size: 2400 - 2400 (first
2400)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 309
fragments no.: 1, unused: 0#012#011#011 bucket size: 2472 - 2472 (first
2472)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 326
fragments no.: 1, unused: 0#012#011#011 bucket size: 2608 - 2608 (first
2608)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 426
fragments no.: 1, unused: 0#012#011#011 bucket size: 3408 - 3408 (first
3408)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 450
fragments no.: 1, unused: 0#012#011#011 bucket size: 3600 - 3600 (first
3600)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1024
fragments no.: 2, unused: 0#012#011#011 bucket size: 8192 - 8192 (first
8192)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1313
fragments no.: 1, unused: 0#012#011#011 bucket size: 10504 - 10504 (first
10504)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 1800
fragments no.: 1, unused: 0#012#011#011 bucket size: 14400 - 14400 (first
14400)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2048
fragments no.: 3, unused: 0#012#011#011 bucket size: 16384 - 16384 (first
16384)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2049
fragments no.: 15241, unused: 0#012#011#011 bucket size: 16384 - 32768 (first
16400)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2050
fragments no.: 3, unused: 0#012#011#011 bucket size: 32768 - 65536 (first
32768)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2052
fragments no.: 2, unused: 0#012#011#011 bucket size: 131072 - 262144 (first
131088)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2053
fragments no.: 2, unused: 0#012#011#011 bucket size: 262144 - 524288 (first
262144)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2054
fragments no.: 4, unused: 0#012#011#011 bucket size: 524288 - 1048576 (first
524288)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: hash = 2056
fragments no.: 1, unused: 0#012#011#011 bucket size: 2097152 - 4194304 (first
2097152)
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status: TOTAL:
15657 free fragments = 266687152 free bytes
Aug 29 18:23:51 SIP-B03 /home/pkg/kamailio/sbin/kamailio[204853]: : fm_status:
-----------------------------
```
### SECOND: catch it, syslog BEFORE kamailio free shm, it's hard to get the sweet
spot
**below is time sequence where I stop kamailio**
```
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 17630352
used: 249499264
real_used: 250805104
max_used: 250805496
fragments: 4630
}
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 12131352
used: 254980512
real_used: 256304104
max_used: 256304416
fragments: 3934
}
root@SIP-B03:/var/log# kamcmd core.shmmem
{
total: 268435456
free: 7884928
used: 259213272
real_used: 260550528
max_used: 260550600
fragments: 3291
}
===== MANUALLY STOP KAMAILIO HERE =====
===== BEFORE kamailio free shm =====
```
`kamailio stop`
**below is `Memory status (shm)`, but no `Memory still-in-use summary (shm)` shows in
syslog**
```
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : <core>
[main.c:565]: cleanup(): Memory status (shm):
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: fm_status
(0x7f41fbcd4000):
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: heap size=
268435456
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: used=
263736, used+overhead=1510832, free=266924624
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: max used
(+overhead)= 268102440
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: dumping
free list:
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1
fragments no.: 30, unused: 0#012#011#011 bucket size: 8 - 8 (first
8)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2
fragments no.: 33, unused: 0#012#011#011 bucket size: 16 - 16 (first
16)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 3
fragments no.: 82, unused: 0#012#011#011 bucket size: 24 - 24 (first
24)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 4
fragments no.: 40, unused: 0#012#011#011 bucket size: 32 - 32 (first
32)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 5
fragments no.: 211, unused: 0#012#011#011 bucket size: 40 - 40 (first
40)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 6
fragments no.: 57, unused: 0#012#011#011 bucket size: 48 - 48 (first
48)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 7
fragments no.: 42, unused: 0#012#011#011 bucket size: 56 - 56 (first
56)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 8
fragments no.: 212, unused: 0#012#011#011 bucket size: 64 - 64 (first
64)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 9
fragments no.: 92, unused: 0#012#011#011 bucket size: 72 - 72 (first
72)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 10
fragments no.: 1, unused: 0#012#011#011 bucket size: 80 - 80 (first
80)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 11
fragments no.: 1, unused: 0#012#011#011 bucket size: 88 - 88 (first
88)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 12
fragments no.: 33, unused: 0#012#011#011 bucket size: 96 - 96 (first
96)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 13
fragments no.: 107, unused: 0#012#011#011 bucket size: 104 - 104 (first
104)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 14
fragments no.: 115, unused: 0#012#011#011 bucket size: 112 - 112 (first
112)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 15
fragments no.: 10, unused: 0#012#011#011 bucket size: 120 - 120 (first
120)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 16
fragments no.: 239, unused: 0#012#011#011 bucket size: 128 - 128 (first
128)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 17
fragments no.: 12, unused: 0#012#011#011 bucket size: 136 - 136 (first
136)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 18
fragments no.: 206, unused: 0#012#011#011 bucket size: 144 - 144 (first
144)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 19
fragments no.: 3, unused: 0#012#011#011 bucket size: 152 - 152 (first
152)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 20
fragments no.: 11, unused: 0#012#011#011 bucket size: 160 - 160 (first
160)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 21
fragments no.: 3, unused: 0#012#011#011 bucket size: 168 - 168 (first
168)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 22
fragments no.: 6, unused: 0#012#011#011 bucket size: 176 - 176 (first
176)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 23
fragments no.: 4, unused: 0#012#011#011 bucket size: 184 - 184 (first
184)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 24
fragments no.: 30, unused: 0#012#011#011 bucket size: 192 - 192 (first
192)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 25
fragments no.: 23, unused: 0#012#011#011 bucket size: 200 - 200 (first
200)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 26
fragments no.: 18, unused: 0#012#011#011 bucket size: 208 - 208 (first
208)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 27
fragments no.: 2, unused: 0#012#011#011 bucket size: 216 - 216 (first
216)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 28
fragments no.: 5, unused: 0#012#011#011 bucket size: 224 - 224 (first
224)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 29
fragments no.: 569, unused: 0#012#011#011 bucket size: 232 - 232 (first
232)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 30
fragments no.: 7, unused: 0#012#011#011 bucket size: 240 - 240 (first
240)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 31
fragments no.: 32, unused: 0#012#011#011 bucket size: 248 - 248 (first
248)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 32
fragments no.: 12, unused: 0#012#011#011 bucket size: 256 - 256 (first
256)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 33
fragments no.: 19, unused: 0#012#011#011 bucket size: 264 - 264 (first
264)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 34
fragments no.: 6, unused: 0#012#011#011 bucket size: 272 - 272 (first
272)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 35
fragments no.: 6, unused: 0#012#011#011 bucket size: 280 - 280 (first
280)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 36
fragments no.: 5, unused: 0#012#011#011 bucket size: 288 - 288 (first
288)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 37
fragments no.: 11, unused: 0#012#011#011 bucket size: 296 - 296 (first
296)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 38
fragments no.: 7, unused: 0#012#011#011 bucket size: 304 - 304 (first
304)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 39
fragments no.: 44, unused: 0#012#011#011 bucket size: 312 - 312 (first
312)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 40
fragments no.: 35, unused: 0#012#011#011 bucket size: 320 - 320 (first
320)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 41
fragments no.: 65, unused: 0#012#011#011 bucket size: 328 - 328 (first
328)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 42
fragments no.: 5, unused: 0#012#011#011 bucket size: 336 - 336 (first
336)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 43
fragments no.: 28, unused: 0#012#011#011 bucket size: 344 - 344 (first
344)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 44
fragments no.: 11, unused: 0#012#011#011 bucket size: 352 - 352 (first
352)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 45
fragments no.: 25, unused: 0#012#011#011 bucket size: 360 - 360 (first
360)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 46
fragments no.: 6, unused: 0#012#011#011 bucket size: 368 - 368 (first
368)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 47
fragments no.: 7, unused: 0#012#011#011 bucket size: 376 - 376 (first
376)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 48
fragments no.: 4, unused: 0#012#011#011 bucket size: 384 - 384 (first
384)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 49
fragments no.: 6, unused: 0#012#011#011 bucket size: 392 - 392 (first
392)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 50
fragments no.: 14, unused: 0#012#011#011 bucket size: 400 - 400 (first
400)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 51
fragments no.: 5, unused: 0#012#011#011 bucket size: 408 - 408 (first
408)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 52
fragments no.: 48, unused: 0#012#011#011 bucket size: 416 - 416 (first
416)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 53
fragments no.: 4, unused: 0#012#011#011 bucket size: 424 - 424 (first
424)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 54
fragments no.: 45, unused: 0#012#011#011 bucket size: 432 - 432 (first
432)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 55
fragments no.: 9, unused: 0#012#011#011 bucket size: 440 - 440 (first
440)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 56
fragments no.: 10, unused: 0#012#011#011 bucket size: 448 - 448 (first
448)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 57
fragments no.: 2, unused: 0#012#011#011 bucket size: 456 - 456 (first
456)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 58
fragments no.: 10, unused: 0#012#011#011 bucket size: 464 - 464 (first
464)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 59
fragments no.: 2, unused: 0#012#011#011 bucket size: 472 - 472 (first
472)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 60
fragments no.: 6, unused: 0#012#011#011 bucket size: 480 - 480 (first
480)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 61
fragments no.: 11, unused: 0#012#011#011 bucket size: 488 - 488 (first
488)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 62
fragments no.: 2, unused: 0#012#011#011 bucket size: 496 - 496 (first
496)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 63
fragments no.: 1, unused: 0#012#011#011 bucket size: 504 - 504 (first
504)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 64
fragments no.: 7, unused: 0#012#011#011 bucket size: 512 - 512 (first
512)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 65
fragments no.: 24, unused: 0#012#011#011 bucket size: 520 - 520 (first
520)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 66
fragments no.: 9, unused: 0#012#011#011 bucket size: 528 - 528 (first
528)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 67
fragments no.: 12, unused: 0#012#011#011 bucket size: 536 - 536 (first
536)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 68
fragments no.: 6, unused: 0#012#011#011 bucket size: 544 - 544 (first
544)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 69
fragments no.: 8, unused: 0#012#011#011 bucket size: 552 - 552 (first
552)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 70
fragments no.: 1, unused: 0#012#011#011 bucket size: 560 - 560 (first
560)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 71
fragments no.: 6, unused: 0#012#011#011 bucket size: 568 - 568 (first
568)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 72
fragments no.: 1, unused: 0#012#011#011 bucket size: 576 - 576 (first
576)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 73
fragments no.: 3, unused: 0#012#011#011 bucket size: 584 - 584 (first
584)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 74
fragments no.: 1, unused: 0#012#011#011 bucket size: 592 - 592 (first
592)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 75
fragments no.: 10, unused: 0#012#011#011 bucket size: 600 - 600 (first
600)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 76
fragments no.: 4, unused: 0#012#011#011 bucket size: 608 - 608 (first
608)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 77
fragments no.: 12, unused: 0#012#011#011 bucket size: 616 - 616 (first
616)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 78
fragments no.: 4, unused: 0#012#011#011 bucket size: 624 - 624 (first
624)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 79
fragments no.: 5, unused: 0#012#011#011 bucket size: 632 - 632 (first
632)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 81
fragments no.: 2, unused: 0#012#011#011 bucket size: 648 - 648 (first
648)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 82
fragments no.: 3, unused: 0#012#011#011 bucket size: 656 - 656 (first
656)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 83
fragments no.: 2, unused: 0#012#011#011 bucket size: 664 - 664 (first
664)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 84
fragments no.: 3, unused: 0#012#011#011 bucket size: 672 - 672 (first
672)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 85
fragments no.: 4, unused: 0#012#011#011 bucket size: 680 - 680 (first
680)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 86
fragments no.: 1, unused: 0#012#011#011 bucket size: 688 - 688 (first
688)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 87
fragments no.: 1, unused: 0#012#011#011 bucket size: 696 - 696 (first
696)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 88
fragments no.: 7, unused: 0#012#011#011 bucket size: 704 - 704 (first
704)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 89
fragments no.: 3, unused: 0#012#011#011 bucket size: 712 - 712 (first
712)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 90
fragments no.: 7, unused: 0#012#011#011 bucket size: 720 - 720 (first
720)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 91
fragments no.: 3, unused: 0#012#011#011 bucket size: 728 - 728 (first
728)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 92
fragments no.: 2, unused: 0#012#011#011 bucket size: 736 - 736 (first
736)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 94
fragments no.: 2, unused: 0#012#011#011 bucket size: 752 - 752 (first
752)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 95
fragments no.: 1, unused: 0#012#011#011 bucket size: 760 - 760 (first
760)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 97
fragments no.: 1, unused: 0#012#011#011 bucket size: 776 - 776 (first
776)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 98
fragments no.: 2, unused: 0#012#011#011 bucket size: 784 - 784 (first
784)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 99
fragments no.: 3, unused: 0#012#011#011 bucket size: 792 - 792 (first
792)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 100
fragments no.: 2, unused: 0#012#011#011 bucket size: 800 - 800 (first
800)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 101
fragments no.: 4, unused: 0#012#011#011 bucket size: 808 - 808 (first
808)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 102
fragments no.: 2, unused: 0#012#011#011 bucket size: 816 - 816 (first
816)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 103
fragments no.: 1, unused: 0#012#011#011 bucket size: 824 - 824 (first
824)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 104
fragments no.: 3, unused: 0#012#011#011 bucket size: 832 - 832 (first
832)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 105
fragments no.: 1, unused: 0#012#011#011 bucket size: 840 - 840 (first
840)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 106
fragments no.: 1, unused: 0#012#011#011 bucket size: 848 - 848 (first
848)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 107
fragments no.: 4, unused: 0#012#011#011 bucket size: 856 - 856 (first
856)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 108
fragments no.: 2, unused: 0#012#011#011 bucket size: 864 - 864 (first
864)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 109
fragments no.: 1, unused: 0#012#011#011 bucket size: 872 - 872 (first
872)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 110
fragments no.: 2, unused: 0#012#011#011 bucket size: 880 - 880 (first
880)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 111
fragments no.: 1, unused: 0#012#011#011 bucket size: 888 - 888 (first
888)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 112
fragments no.: 1, unused: 0#012#011#011 bucket size: 896 - 896 (first
896)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 113
fragments no.: 1, unused: 0#012#011#011 bucket size: 904 - 904 (first
904)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 116
fragments no.: 1, unused: 0#012#011#011 bucket size: 928 - 928 (first
928)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 117
fragments no.: 3, unused: 0#012#011#011 bucket size: 936 - 936 (first
936)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 119
fragments no.: 1, unused: 0#012#011#011 bucket size: 952 - 952 (first
952)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 123
fragments no.: 4, unused: 0#012#011#011 bucket size: 984 - 984 (first
984)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 124
fragments no.: 1, unused: 0#012#011#011 bucket size: 992 - 992 (first
992)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 126
fragments no.: 1, unused: 0#012#011#011 bucket size: 1008 - 1008 (first
1008)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 130
fragments no.: 1, unused: 0#012#011#011 bucket size: 1040 - 1040 (first
1040)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 131
fragments no.: 1, unused: 0#012#011#011 bucket size: 1048 - 1048 (first
1048)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 132
fragments no.: 1, unused: 0#012#011#011 bucket size: 1056 - 1056 (first
1056)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 134
fragments no.: 1, unused: 0#012#011#011 bucket size: 1072 - 1072 (first
1072)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 136
fragments no.: 2, unused: 0#012#011#011 bucket size: 1088 - 1088 (first
1088)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 137
fragments no.: 1, unused: 0#012#011#011 bucket size: 1096 - 1096 (first
1096)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 139
fragments no.: 1, unused: 0#012#011#011 bucket size: 1112 - 1112 (first
1112)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 145
fragments no.: 1, unused: 0#012#011#011 bucket size: 1160 - 1160 (first
1160)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 154
fragments no.: 1, unused: 0#012#011#011 bucket size: 1232 - 1232 (first
1232)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 158
fragments no.: 1, unused: 0#012#011#011 bucket size: 1264 - 1264 (first
1264)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 164
fragments no.: 1, unused: 0#012#011#011 bucket size: 1312 - 1312 (first
1312)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 166
fragments no.: 1, unused: 0#012#011#011 bucket size: 1328 - 1328 (first
1328)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 170
fragments no.: 1, unused: 0#012#011#011 bucket size: 1360 - 1360 (first
1360)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 171
fragments no.: 1, unused: 0#012#011#011 bucket size: 1368 - 1368 (first
1368)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 186
fragments no.: 1, unused: 0#012#011#011 bucket size: 1488 - 1488 (first
1488)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 187
fragments no.: 1, unused: 0#012#011#011 bucket size: 1496 - 1496 (first
1496)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 191
fragments no.: 1, unused: 0#012#011#011 bucket size: 1528 - 1528 (first
1528)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 201
fragments no.: 1, unused: 0#012#011#011 bucket size: 1608 - 1608 (first
1608)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 450
fragments no.: 1, unused: 0#012#011#011 bucket size: 3600 - 3600 (first
3600)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1024
fragments no.: 2, unused: 0#012#011#011 bucket size: 8192 - 8192 (first
8192)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1313
fragments no.: 1, unused: 0#012#011#011 bucket size: 10504 - 10504 (first
10504)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 1800
fragments no.: 1, unused: 0#012#011#011 bucket size: 14400 - 14400 (first
14400)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2048
fragments no.: 3, unused: 0#012#011#011 bucket size: 16384 - 16384 (first
16384)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2049
fragments no.: 15234, unused: 0#012#011#011 bucket size: 16384 - 32768 (first
16400)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2050
fragments no.: 3, unused: 0#012#011#011 bucket size: 32768 - 65536 (first
32768)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2052
fragments no.: 2, unused: 0#012#011#011 bucket size: 131072 - 262144 (first
131088)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2053
fragments no.: 2, unused: 0#012#011#011 bucket size: 262144 - 524288 (first
262144)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2054
fragments no.: 4, unused: 0#012#011#011 bucket size: 524288 - 1048576 (first
524288)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: hash = 2056
fragments no.: 1, unused: 0#012#011#011 bucket size: 2097152 - 4194304 (first
2097152)
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status: TOTAL:
18177 free fragments = 266924624 free bytes
Aug 29 18:45:03 SIP-B03 /home/pkg/kamailio/sbin/kamailio[209062]: : fm_status:
-----------------------------
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-135972338
4 Sep
4 Sep
12:28 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Back to this after some time caught with other stuff.
The status with the list of free chunks doesn't help at all.
To get the summary, try:
```
memlog=1
```
in kamailio.cfg
Then, at runtime issue the command:
```
kamcmd corex.shm_summary
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-137688458
5 Sep
5 Sep
7:38 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Looks like problems not in jsonrpc-c now?
#### points of the command `kamcmd corex.shm_summary` issued
```
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem
{
total: 268435456
free: 7586408
used: 259640648
real_used: 260849048
max_used: 260849120
fragments: 2985
}
===FIRST LOG HERE (00:03:32) ===
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd corex.shm_summary
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem
{
total: 268435456
free: 187520
used: 267043232
real_used: 268247936
max_used: 268248120
fragments: 771
}
===SECOND LOG HERE (00:03:35) ===
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd corex.shm_summary
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem
{
total: 268435456
free: 1860208
used: 265306536
real_used: 266575248
max_used: 268430848
fragments: 120
}
root@SIP-B03:/home/pkg/kamailio/etc/kamailio# kamcmd core.shmmem
{
total: 268435456
free: 3415784
used: 263698488
real_used: 265019672
max_used: 268430848
fragments: 218
}
```
#### FIRST LOG ( at 00:03:32 )
```
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
summarizing all alloc'ed. fragments:
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 464 bytes from usrloc: ucontact.c: new_ucontact(98)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 64 bytes from usrloc: urecord.c: new_urecord(58)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 24 bytes from usrloc: urecord.c: new_urecord(65)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 10 size= 336 bytes from usrloc: ../../ut.h: shm_str_dup(691)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 19 size= 1368 bytes from <core>: usr_avp.c: create_avp(175)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1216 size= 81216 bytes from jsonrpc-c: jsonrpc_request.c:
jsonrpc_notification(135)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 24 size= 6144 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 24 size= 3280 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 45 size= 1136 bytes from dialog: dlg_var.c: new_dlg_var(106)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 45 size= 2160 bytes from dialog: dlg_var.c: new_dlg_var(97)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 45 size= 552 bytes from dialog: dlg_var.c: new_dlg_var(116)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 15112 size= 258959232 bytes from <core>: tcp_main.c: tcpconn_new(957)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 57600 bytes from <core>: counters.c:
counters_prefork_init(207)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 24 bytes from pv: pv_shv.c: set_shvar_value(243)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 3600 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 14400 bytes from sl: sl_stats.c: init_sl_stats_child(125)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 19200 bytes from tm: t_stats.c: init_tm_stats_child(60)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 6 size= 88 bytes from <core>: cfg/cfg_struct.c:
cfg_clone_str(130)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 784 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 163840 bytes from usrloc: udomain.c: new_udomain(93)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: pt.c: init_pt(110)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: pt.c: init_pt(105)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 10504 bytes from <core>: pt.c: init_pt(104)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(932)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(926)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(918)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2432 size= 194672 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 62 size= 6264 bytes from htable: ht_api.c: ht_cell_new(183)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 7 size= 2392320 bytes from htable: ht_api.c: ht_init_tables(336)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8192 bytes from nat_traversal: nat_traversal.c:
HashTable_new(552)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from nat_traversal: nat_traversal.c:
HashTable_new(543)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from dialog: dlg_cb.c: init_dlg_callback(55)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 80 bytes from dialog: dlg_cb.c: register_dlgcb(124)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16400 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 131088 bytes from dialog: dlg_hash.c: init_dlg_table(277)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_timer.c: init_dlg_timer(57)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32 bytes from dialog: dlg_timer.c: init_dlg_timer(48)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 320 bytes from dialog: dlg_profile.c: new_dlg_profile(187)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4634)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4628)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4620)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4613)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4606)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4600)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4588)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: dst_blacklist.c:
init_dst_blacklist(437)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dst_blacklist.c:
init_dst_blacklist(430)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(283)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(281)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(280)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(269)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(237)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(219)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(206)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c:
cfg_child_cb_new(830)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 22 size= 496 bytes from pv: pv_shv.c: add_shvar(177)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 22 size= 1408 bytes from pv: pv_shv.c: add_shvar(170)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 80 bytes from pv: ../../lock_alloc.h: lock_set_alloc(70)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 7 size= 672 bytes from htable: ht_api.c: ht_add_table(275)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: mem/shm_mem.c:
shm_mem_init_mallocs(179)
Sep 6 00:03:32 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
-----------------------------
```
#### SECOND LOG ( at 00:03:35 )
```
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
summarizing all alloc'ed. fragments:
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230140]: ERROR: <core>
[tcp_main.c:959]: tcpconn_new(): mem. allocation failure
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230140]: ERROR: <core>
[tcp_main.c:3977]: handle_new_connect(): tcpconn_new failed, closing socket
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 64 bytes from usrloc: urecord.c: new_urecord(58)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 24 bytes from usrloc: urecord.c: new_urecord(65)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 752 bytes from usrloc: ucontact.c: new_ucontact(98)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 10 size= 368 bytes from usrloc: ../../ut.h: shm_str_dup(691)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 17 size= 1392 bytes from <core>: usr_avp.c: create_avp(175)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 48 size= 2384 bytes from dialog: dlg_var.c: new_dlg_var(97)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 24 size= 6144 bytes from tmx: tmx_pretran.c: tmx_check_pretran(264)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 24 size= 3280 bytes from tmx: tmx_pretran.c: tmx_check_pretran(243)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 48 size= 1312 bytes from dialog: dlg_var.c: new_dlg_var(106)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 48 size= 576 bytes from dialog: dlg_var.c: new_dlg_var(116)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1813 size= 137808 bytes from jsonrpc-c: jsonrpc_request.c:
jsonrpc_notification(135)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 15252 size= 261371984 bytes from <core>: tcp_main.c: tcpconn_new(957)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 57600 bytes from <core>: counters.c:
counters_prefork_init(207)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 24 bytes from pv: pv_shv.c: set_shvar_value(243)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 3600 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 14400 bytes from sl: sl_stats.c: init_sl_stats_child(125)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 1024 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 19200 bytes from tm: t_stats.c: init_tm_stats_child(60)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 6 size= 88 bytes from <core>: cfg/cfg_struct.c:
cfg_clone_str(130)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 784 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 163840 bytes from usrloc: udomain.c: new_udomain(93)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: pt.c: init_pt(110)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: pt.c: init_pt(105)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 10504 bytes from <core>: pt.c: init_pt(104)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(932)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(926)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from cfgutils: cfgutils.c: mod_init(918)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 3626 size= 344928 bytes from jsonrpc-c: jsonrpc_request.c: shm_strdup(158)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 62 size= 6264 bytes from htable: ht_api.c: ht_cell_new(183)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 7 size= 2392320 bytes from htable: ht_api.c: ht_init_tables(336)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 64 bytes from <core>: cfg/cfg_ctx.c: cfg_register_ctx(47)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8192 bytes from nat_traversal: nat_traversal.c:
HashTable_new(552)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from nat_traversal: nat_traversal.c:
HashTable_new(543)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from dialog: dlg_cb.c: init_dlg_callback(55)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 80 bytes from dialog: dlg_cb.c: register_dlgcb(124)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16400 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 131088 bytes from dialog: dlg_hash.c: init_dlg_table(277)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(269)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(262)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_hash.c: init_dlg_table(257)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from dialog: dlg_timer.c: init_dlg_timer(57)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32 bytes from dialog: dlg_timer.c: init_dlg_timer(48)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 3 size= 216 bytes from <core>: timer.c: register_timer(1011)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 40 bytes from tm: t_hooks.c: insert_tmcb(137)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 320 bytes from dialog: dlg_profile.c: new_dlg_profile(187)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(450)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8192 bytes from <core>: tcp_main.c: init_tcp(4634)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 32768 bytes from <core>: tcp_main.c: init_tcp(4628)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4620)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4613)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4606)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4600)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: tcp_main.c: init_tcp(4588)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(90)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: usr_avp.c: init_avps(89)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: dst_blacklist.c:
init_dst_blacklist(437)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dst_blacklist.c:
init_dst_blacklist(430)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 2 size= 96 bytes from <core>: timer.c: timer_alloc(514)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(366)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: dns_cache.c: init_dns_cache(358)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16 bytes from <core>: dns_cache.c: init_dns_cache(351)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: dns_cache.c: init_dns_cache(345)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(283)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 16384 bytes from <core>: timer.c: init_timer(282)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(281)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(280)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(269)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(237)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 278544 bytes from <core>: timer.c: init_timer(220)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(219)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: timer.c: init_timer(206)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 64 bytes from <core>: cfg/cfg_struct.c:
cfg_child_cb_new(830)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(361)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(354)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(347)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(335)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: cfg/cfg_struct.c: sr_cfg_init(323)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 22 size= 496 bytes from pv: pv_shv.c: add_shvar(177)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 22 size= 1408 bytes from pv: pv_shv.c: add_shvar(170)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 80 bytes from pv: ../../lock_alloc.h: lock_set_alloc(70)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 7 size= 672 bytes from htable: ht_api.c: ht_add_table(275)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
count= 1 size= 8 bytes from <core>: mem/shm_mem.c:
shm_mem_init_mallocs(179)
Sep 6 00:03:35 SIP-B03 /home/pkg/kamailio/sbin/kamailio[230109]: NOTICE: fm_status:
-----------------------------
```
#### Some of my kamailio.cfg settings
```
children=24
async_workers=12
dns=no
rev_dns=no
udp_mtu = 1472
udp_mtu_try_proto = tcp
tcp_max_connections = 200000
tcp_connection_lifetime = 650
tcp_keepalive = yes
tcp_keepcnt = 7
tcp_keepintvl = 3
tcp_keepidle = 60
tcp_crlf_ping = yes
tcp_defer_accept = 15
tcp_rd_buf_size = 16384 # 16M
tcp_conn_wq_max = 200000 # 200k
tcp_poll_method = epoll_et
open_files_limit = 500000 # Max open files ulimit
modparam("usrloc", "handle_lost_tcp", 1)
# --- close_expired_tcp seemed may cause double free on a missed socket sometimes and
leads segfault, especially in large amount of online users. disable it explicitly.
modparam("usrloc", "close_expired_tcp", 0)
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-137973153
7 Sep
7 Sep
4:03 p.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
It looks like a lot of tcp connections are open (over 15 000) -- for each of them is
allocated the read/write buffer. Do you expect so many tcp connections? If yes, then you
need to allocate more memory for kamailio, it runs out of it.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-138292095
8 Sep
8 Sep
11:11 a.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Thanks, we're trying to serve as large amount as we can on a single kamailio server.
Anyway, after some test, I found the problem may related to debug level.
If I switch on memdbg, it'll have a HUGE impact to kamailio and cause TCP connection
drop.
If I leave debug=-2, then it runs well.
Normally it should not switch on memlog on a running production server.
So I think this issue can be closed now.
Again, thank you for all your great contributions, really appreciate your help.
### Further information if anyone interest in
Env:
```
start kamailio with pkg 2G, shm 2G
sipp with 1000 RPS
```
#### TEST 1: with memdbg on ( lots of logs write to syslog )
It will drop tcp connection along with shm goes down.
And sipp will stop and shows `Broken pipe on TCP connection, remote peer probably closed
the socket.`
#### TEST 2: disable all logs, at log level -2, nearly no logs at all
It runs well, and did not keep swallowing shm at this level.
free shm 208811808 more or less.
```
root@SIP-B03:~# kamcmd core.shmmem
{
total: 1073741824
free: 208811808
used: 862058976
real_used: 864930016
max_used: 864952488
fragments: 253
}
```
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#issuecomment-138469916
11:21 a.m.
New subject: [kamailio] jsonrpc-c notification does not free shm memory (#296)
Closed #296.
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/296#event-403343460
3363
days inactive
3378
days old
22 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Hugh Waite
-
zxcpoiu