The easier way would be to get the memory usage summary, it will give a
report ordered by the size of used memory based on code location,
allowing to assert if it is some startup (allocated once per lifetime of
kamailio) or runtime.
Some hints about how to do it:
*
https://www.kamailio.org/wiki/tutorials/troubleshooting/memory
Cheers,
Daniel
On 03.10.18 08:48, Igor Olhovskiy wrote:
Thanks Daniel
But how to find out what exact eats all memory?
pkg.stats
{
entry: 0
pid: 13
rank: 0
used: 3639400
free: 3323464
real_used: 5065144
total_size: 8388608
total_frags: 38
}
{
entry: 1
pid: 14
rank: 1
used: 3672088
free: 3253128
real_used: 5135480
total_size: 8388608
total_frags: 125
}
{
entry: 2
pid: 15
rank: 2
used: 3675992
free: 3246312
real_used: 5142296
total_size: 8388608
total_frags: 121
}
{
entry: 3
pid: 16
rank: 3
used: 3670912
free: 3254512
real_used: 5134096
total_size: 8388608
total_frags: 129
}
{
entry: 4
pid: 17
rank: 4
used: 3673712
free: 3250152
real_used: 5138456
total_size: 8388608
total_frags: 122
}
{
entry: 5
pid: 18
rank: 5
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 6
pid: 19
rank: 6
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 7
pid: 20
rank: 7
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 8
pid: 21
rank: 8
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 9
pid: 22
rank: 9
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 10
pid: 23
rank: 10
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 11
pid: 24
rank: 11
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 12
pid: 25
rank: 12
used: 3643776
free: 3315032
real_used: 5073576
total_size: 8388608
total_frags: 54
}
{
entry: 13
pid: 26
rank: 13
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 14
pid: 27
rank: 14
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 15
pid: 28
rank: 15
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 16
pid: 29
rank: 16
used: 3643776
free: 3318152
real_used: 5070456
total_size: 8388608
total_frags: 24
}
{
entry: 17
pid: 30
rank: -1
used: 3645488
free: 3308328
real_used: 5080280
total_size: 8388608
total_frags: 89
}
{
entry: 18
pid: 31
rank: -1
used: 3645920
free: 3303528
real_used: 5085080
total_size: 8388608
total_frags: 122
}
{
entry: 19
pid: 32
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 20
pid: 33
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 21
pid: 34
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 22
pid: 35
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 23
pid: 36
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 24
pid: 37
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 25
pid: 38
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 26
pid: 39
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 27
pid: 40
rank: -128
used: 0
free: 0
real_used: 0
total_size: 8388608
total_frags: 0
}
{
entry: 28
pid: 41
rank: -2
used: 3657016
free: 3304912
real_used: 5083696
total_size: 8388608
total_frags: 23
}
{
entry: 29
pid: 42
rank: -2
used: 5810992
free: 3816
real_used: 8384792
total_size: 8388608
total_frags: 93
}
{
entry: 30
pid: 43
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 31
pid: 44
rank: -2
used: 3643744
free: 3318288
real_used: 5070320
total_size: 8388608
total_frags: 24
}
{
entry: 32
pid: 45
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 33
pid: 46
rank: -1
used: 3643600
free: 3318536
real_used: 5070072
total_size: 8388608
total_frags: 24
}
{
entry: 34
pid: 47
rank: 17
used: 3725080
free: 3234768
real_used: 5153840
total_size: 8388608
total_frags: 39
}
{
entry: 35
pid: 48
rank: 18
used: 3725064
free: 3234992
real_used: 5153616
total_size: 8388608
total_frags: 38
}
{
entry: 36
pid: 49
rank: 19
used: 3725048
free: 3235112
real_used: 5153496
total_size: 8388608
total_frags: 37
}
{
entry: 37
pid: 50
rank: 20
used: 3725048
free: 3234800
real_used: 5153808
total_size: 8388608
total_frags: 40
}
{
entry: 38
pid: 51
rank: -4
used: 3723728
free: 3238824
real_used: 5149784
total_size: 8388608
total_frags: 37
}
Regards, Igor
On Oct 2, 2018, 10:37 PM +0300, Daniel-Constantin Mierla
<miconda(a)gmail.com>om>, wrote:
Hello,
pkg is out of free size, not shm. See the output for:
kamctl rpc pkg.stats
You may need to increase the pkg allocated pool with -M command line
parameter.
Cheers,
Daniel
On 02.10.18 20:02, Igor Olhovskiy wrote:
Hi!
What is the best way to debug messages like...
29(42) ERROR: [core/mem/q_malloc.c:290]: qm_find_free():
qm_find_free(0x7fad301cf010, 56); Free fragment not found!
29(42) ERROR: [core/mem/q_malloc.c:423]: qm_malloc():
qm_malloc(0x7fad301cf010, 56) called from core: core/data_lump.c:
dup_lump_list_r(537), module: core; Free fragment not found!
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: [core/data_lump.c:573]: dup_lump_list_r(): out of mem
29(42) ERROR: tm [t_fwd.c:1735]: t_forward_nonack(): failure to add
branches
Cause core.shmmem shows
total: 536870912
free: 522859104
used: 12224144
real_used: 14011808
max_used: 14479048
fragments: 10560
Also
mod.stats all pkg
Module: core
{
init_io_wait(475): 6960
init_io_wait(524): 6312
new_db_id(272): 512
dupl_string_name(72): 32
db_do_init2(298): 488
mk_switch_cond_table(43): 144
sr_wtimer_add(330): 48
as_asciiz(161): 24
init_modules(1026): 16
rpc_hash_add(146): 2048
db_allocate_columns(150): 32
db_new_result(114): 56
get_hdr_field(116): 1160
db_allocate_columns(160): 16
cnt_hash_add(383): 1024
register_select_table(458): 48
add_callback(59): 320
sr_wtimer_init(308): 136
cfg_new_group(79): 704
cfg_declare(50): 3360
fix_sock_str(420): 480
init_dst_set(83): 32208
grp_hash_add(234): 808
fix_hostname(1301): 152
add_alias(91): 464
add_alias(93): 256
dupl_string(48): 176
parse_headers(325): 432
subst_parser(291): 144
subst_parser(274): 64
exp_optimize_right(411): 24
fix_expr(541): 600
mk_case_stm(3775): 400
fixup_regexp_null(213): 72
mk_elem(90): 560
route_new_list(200): 1344
fix_param(1196): 1336
parse_select(204): 1488
fix_param(1257): 536
fix_match_rve(3028): 320
mk_rval_expr1(2616): 142656
tr_new(1646): 1464
fixup_pvar_all(279): 1624
fix_param(1162): 16144
mk_action(118): 350168
mk_rval_expr2(2674): 489552
mk_rval_expr_v(2537): 1475600
yyparse(2757): 8792
pv_parse_format(1150): 89064
pv_cache_add(347): 31456
parse_params2(585): 1400
set_mod_param_regex(110): 376
set_mod_param_regex(121): 832
tr_table_add(1875): 400
sr_cmd_exports_convert(235): 36288
register_module(280): 6624
load_module(496): 2640
new_sock_info(235): 224
fix_socket_list(1509): 48
new_sock_info(230): 3920
add_alias(93): 56
add_alias(91): 96
yyparse(1774): 64
yyparse(667): 152
yyparse(2306): 128
yyparse(648): 144
sr_push_yy_state(1666): 32
sr_push_yy_state(1656): 104
sr_push_yy_state(1596): 184
addstr(1413): 415752
rpc_hash_add(100): 25656
str_hash_alloc(59): 512
pv_table_add(236): 17864
pv_init_buffer(2055): 327680
pv_init_buffer(2045): 320
init_nonsip_hooks(43): 24
init_rlist(146): 8
mk_rval_expr_v(2548): 1264
route_add(124): 5872
str_hash_alloc(59): 768
rval_get_str(1256): 56632
pp_define(1788): 2248
init_counters(122): 128
cnt_hash_add(332): 20608
str_hash_alloc(59): 1280
Total: 3602152
}
Module: rtpengine
{
child_init(1812): 256
Total: 256
}
Module: db_mysql
{
db_mysql_new_connection(65): 96
db_mysql_new_connection(75): 3040
Total: 3136
}
Module: siptrace
{
mod_init(344): 384
Total: 384
}
Module: sl
{
sl_register_callback(460): 88
Total: 88
}
Module: acc
{
parse_acc_extra(116): 1512
Total: 1512
}
Module: dialog
{
dlg_bridge_init_hdrs(66): 80
Total: 80
}
Module: rr
{
register_rrcb(61): 80
Total: 80
}
Module: db_text
{
dbt_get_columns(67): 64
Total: 64
}
Module: permissions
{
get_pathname(242): 64
Total: 64
}
Module: auth
{
generate_random_secret(239): 32
generate_random_secret(238): 32
Total: 64
}
Module: xlog
{
mod_init(214): 4104
xdbg_fixup_helper(500): 6032
xlog_fixup_helper(535): 26992
Total: 37128
}
Module: dmq
{
make_socket_str_from_uri(149): 24
Total: 24
}
Module: sqlops
{
sqlops_tr_buffer_init(46): 2048
sql_init_con(83): 240
Total: 2288
}
Module: textops
{
tr_txt_parse_re(212): 32
hname_fixup(3217): 768
fixup_method(3287): 1384
Total: 2184
}
Module: pv
{
tr_parse_string(2343): 120
tr_parse_string(2332): 32
tr_parse_string(2378): 96
tr_parse_string(2368): 184
tr_parse_string(2544): 576
add_var(58): 3616
add_var(65): 1112
Total: 5736
}
Module: textopsx
{
fixup_hname_param(598): 352
Total: 352
}
Module: avpops
{
avpops_parse_pvar(57): 416
Total: 416
}
Module: userblacklist
{
check_blacklist_fixup(522): 16
Total: 16
}
Module: htable
{
fixup_ht_key(348): 176
pv_parse_ht_name(158): 520
Total: 696
}
Module: dialplan
{
dp_trans_fixup(396): 296
Total: 296
}
Module: dispatcher
{
Total: 0
}
Module: tmx
{
Total: 0
}
Module: tm
{
Total: 0
}
Module: kex
{
Total: 0
}
Module: usrloc
{
Total: 0
}
Module: topos
{
Total: 0
}
Module: utils
{
Total: 0
}
Module: app_python
{
Total: 0
}
Module: cfgutils
{
Total: 0
}
Module: nathelper
{
Total: 0
}
Where is the place to look at? Means what module can cause this?
Regards, Igor
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla --
www.asipto.com
www.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference --
www.kamailioworld.com
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin --
www.asipto.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla --
www.asipto.com
www.twitter.com/miconda --
www.linkedin.com/in/miconda
Kamailio World Conference --
www.kamailioworld.com
Kamailio Advanced Training, Nov 12-14, 2018, in Berlin --
www.asipto.com