Hello, I'm monitoring these testes with "corex.shm_summary" provided by kamcmd.
I'm noticing that the core module - xavp.c is consuming lot of memory and it is increasing.
result of command corex.shm_summary is:
fm_status: summarizing all alloc'ed. fragments: fm_status: count= 2 size= 1120 bytes from tm: t_reply.c: _reply_light(542) fm_status: count= 2 size= 896 bytes from tm: t_msgbuilder.c: build_uac_req(1543) fm_status: count= 2182 size= 231088 bytes from tm: t_fwd.c: prepare_new_uac(479) fm_status: count= 12488 size= 12050808 bytes from tm: t_reply.c: relay_reply(1884) fm_status: count= 12490 size= 14564656 bytes from core: msg_translator.c: build_req_buf_from_sip_req(2149) fm_status: count= 12492 size= 71366536 bytes from tm: h_table.c: build_cell(317) fm_status: count= 12490 size= 75086184 bytes from core: sip_msg_clone.c: sip_msg_shm_clone(494) fm_status: count= 2 size= 96 bytes from tm: t_hooks.c: insert_tmcb(137) fm_status: count= 2182 size= 17800 bytes from tm: t_fwd.c: prepare_new_uac(524) fm_status: count= 3 size= 512 bytes from htable: ht_api.c: ht_cell_new(183) fm_status: count= 12490 size= 10586712 bytes from core: sip_msg_clone.c: msg_lump_cloner(978) fm_status: count= 56508 size= 3457904 bytes from core: usr_avp.c: create_avp(175) fm_status: count= 4 size= 600 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250) fm_status: count= 4 size= 1008 bytes from usrloc: ucontact.c: new_ucontact(98) fm_status: count= 4 size= 1320 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271) fm_status: count= 4 size= 144 bytes from usrloc: urecord.c: new_urecord(65) fm_status: count= 4 size= 328 bytes from usrloc: urecord.c: new_urecord(58) fm_status: count= 24 size= 952 bytes from usrloc: ../../ut.h: shm_str_dup(723) fm_status: count= 2182 size= 53040 bytes from tm: t_fwd.c: prepare_new_uac(509) fm_status: count=12545409 size=1420184592 bytes from core: xavp.c: xavp_new_value(94) fm_status: count= 4 size= 128 bytes from dmq: worker.c: alloc_job_queue(229) fm_status: count= 1 size= 13440 bytes from core: counters.c: counters_prefork_init(207) fm_status: count= 1 size= 4032 bytes from sl: sl_stats.c: init_sl_stats_child(125) fm_status: count= 1 size= 256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90) fm_status: count= 1 size= 5376 bytes from tm: t_stats.c: init_tm_stats_child(60) fm_status: count= 1 size= 1008 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79) fm_status: count= 3 size= 88 bytes from core: cfg/cfg_struct.c: cfg_clone_str(130) fm_status: count= 1 size= 744 bytes from core: cfg/cfg_struct.c: cfg_shmize(217) fm_status: count= 3 size= 64 bytes from usrloc: udomain.c: build_stat_name(51) fm_status: count= 1 size= 40960 bytes from usrloc: udomain.c: new_udomain(93) fm_status: count= 1 size= 48 bytes from usrloc: udomain.c: new_udomain(86) fm_status: count= 1 size= 16 bytes from usrloc: dlist.c: new_dlist(573) fm_status: count= 1 size= 32 bytes from usrloc: dlist.c: new_dlist(565) fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(110) fm_status: count= 1 size= 8 bytes from core: pt.c: init_pt(105) fm_status: count= 1 size= 2944 bytes from core: pt.c: init_pt(104) fm_status: count= 1 size= 32 bytes from usrloc: ul_callback.c: register_ulcb(94) fm_status: count= 1 size= 24 bytes from dmq: ../../ut.h: shm_str_dup(723) fm_status: count= 1 size= 448 bytes from dmq: dmqnode.c: build_dmq_node(156) fm_status: count= 2 size= 168 bytes from dmq: peer.c: add_peer(67) fm_status: count= 1 size= 8 bytes from dmq: dmq.c: mod_init(243) fm_status: count= 1 size= 96 bytes from dmq: dmq.c: mod_init(237) fm_status: count= 1 size= 24 bytes from dmq: dmqnode.c: init_dmq_node_list(66) fm_status: count= 1 size= 24 bytes from dmq: peer.c: init_peer_list(33) fm_status: count= 1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(45) fm_status: count= 1 size= 4112 bytes from usrloc: ../../lock_alloc.h: lock_set_alloc(70) fm_status: count= 1 size= 8 bytes from carrierroute: cr_data.c: rule_fixup_recursor(584) fm_status: count= 5 size= 56 bytes from carrierroute: ../../ut.h: shm_str_dup(723) fm_status: count= 1 size= 152 bytes from carrierroute: cr_rule.c: add_route_rule(74) fm_status: count= 3 size= 240 bytes from core: dtrie.c: dtrie_insert(157) fm_status: count= 3 size= 48 bytes from core: dtrie.c: dtrie_insert(148) fm_status: count= 1 size= 48 bytes from carrierroute: cr_rule.c: add_route_flags(225) fm_status: count= 2 size= 160 bytes from core: dtrie.c: dtrie_init(60) fm_status: count= 2 size= 32 bytes from core: dtrie.c: dtrie_init(51) fm_status: count= 1 size= 32 bytes from carrierroute: cr_domain.c: create_domain_data(83) fm_status: count= 1 size= 8 bytes from carrierroute: cr_carrier.c: create_carrier_data(59) fm_status: count= 1 size= 64 bytes from carrierroute: cr_carrier.c: create_carrier_data(50) fm_status: count= 1 size= 8 bytes from carrierroute: cr_db.c: load_route_data_db(295) fm_status: count= 5 size= 1008 bytes from dispatcher: dispatch.c: reindex_dests(600) fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c: load_domain_map(182) fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c: load_domain_map(171) fm_status: count= 1 size= 48 bytes from carrierroute: cr_db.c: load_carrier_map(126) fm_status: count= 1 size= 24 bytes from carrierroute: cr_db.c: load_carrier_map(115) fm_status: count= 7 size= 168 bytes from dispatcher: dispatch.c: add_dest2list(350) fm_status: count= 1 size= 64 bytes from carrierroute: cr_data.c: reload_route_data(172) fm_status: count= 1 size= 8 bytes from carrierroute: cr_data.c: init_route_data(74) fm_status: count= 5 size= 4200 bytes from dispatcher: dispatch.c: add_dest2list(324) fm_status: count= 1 size= 16 bytes from dispatcher: dispatch.c: init_data(204) fm_status: count= 1 size= 16 bytes from dispatcher: dispatch.c: init_data(195) fm_status: count= 1 size= 8 bytes from dispatcher: dispatcher.c: mod_init(309) fm_status: count= 1 size= 8 bytes from dispatcher: dispatcher.c: mod_init(307) fm_status: count= 1 size= 8 bytes from dispatcher: dispatch.c: ds_ping_active_init(102) fm_status: count= 1 size= 256 bytes from db_text: dbt_lib.c: dbt_init_cache(81) fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c: dbt_init_cache(69) fm_status: count= 1 size= 8 bytes from db_text: dbt_lib.c: dbt_init_cache(54) fm_status: count= 3 size= 288 bytes from core: timer.c: register_timer(1011) fm_status: count= 4 size= 8388608 bytes from htable: ht_api.c: ht_init_tables(381) fm_status: count= 1 size= 8 bytes from sl: sl_funcs.c: sl_startup(83) fm_status: count= 1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110) fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(74) fm_status: count= 1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(72) fm_status: count= 1 size= 2097152 bytes from tm: h_table.c: init_hash_table(467) fm_status: count= 2 size= 64 bytes from core: cfg/cfg_ctx.c: cfg_register_ctx(47) fm_status: count= 1 size= 80 bytes from db_postgres: ../../lock_alloc.h: lock_set_alloc(70) fm_status: count= 1 size= 8192 bytes from core: tcp_main.c: init_tcp(4635) fm_status: count= 1 size= 32768 bytes from core: tcp_main.c: init_tcp(4629) fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4621) fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4614) fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4607) fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4601) fm_status: count= 1 size= 8 bytes from core: tcp_main.c: init_tcp(4589) fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(90) fm_status: count= 1 size= 8 bytes from core: usr_avp.c: init_avps(89) fm_status: count= 1 size= 16384 bytes from core: dst_blacklist.c: init_dst_blacklist(437) fm_status: count= 1 size= 8 bytes from core: dst_blacklist.c: init_dst_blacklist(430) fm_status: count= 2 size= 96 bytes from core: timer.c: timer_alloc(514) fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(366) fm_status: count= 1 size= 16384 bytes from core: dns_cache.c: init_dns_cache(358) fm_status: count= 1 size= 16 bytes from core: dns_cache.c: init_dns_cache(351) fm_status: count= 1 size= 8 bytes from core: dns_cache.c: init_dns_cache(345) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(283) fm_status: count= 1 size= 16384 bytes from core: timer.c: init_timer(282) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(281) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(280) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(269) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(237) fm_status: count= 1 size= 278544 bytes from core: timer.c: init_timer(220) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(219) fm_status: count= 1 size= 8 bytes from core: timer.c: init_timer(206) fm_status: count= 1 size= 64 bytes from core: cfg/cfg_struct.c: cfg_child_cb_new(830) fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(361) fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(354) fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(347) fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(335) fm_status: count= 1 size= 8 bytes from core: cfg/cfg_struct.c: sr_cfg_init(323) fm_status: count= 4 size= 928 bytes from htable: ht_api.c: ht_add_table(278) fm_status: count= 1 size= 8 bytes from core: mem/shm.c: shm_core_lock_init(153) fm_status: -----------------------------
If i stop these tests and after few minutes, run again the command "corex.shm_summary" the entry that contains the memory consume for xavp.c doesn't decrease.
It seems that there is a memory leak in this module xavp.c.
Thank you for your help. Regards José
2016-07-02 11:45 GMT+01:00 José Seabra joseseabra4@gmail.com:
I'm running burning tests against kamailio and after some time ~ 2 days I see kamailio out of mem:
ERROR: tm [t_lookup.c:1271]: new_t(): ERROR: new_t: out of mem: ERROR: tm [t_lookup.c:1411]: t_newtran(): ERROR: t_newtran: new_t failed
The result of command kamcmd mod.stats all is:
Module: core { sip_msg_shm_clone(494): 10653392 build_req_buf_from_sip_req(2149): 2565480 msg_lump_cloner(978): 2943584 create_avp(175): 969736 xavp_new_value(94): 858469320 counters_prefork_init(207): 16896 cfg_clone_str(130): 88 cfg_shmize(217): 744 init_pt(110): 8 init_pt(105): 8 init_pt(104): 3080 dtrie_insert(157): 240 dtrie_insert(148): 48 dtrie_init(60): 160 dtrie_init(51): 32 register_timer(1011): 360 cfg_register_ctx(47): 32 init_tcp(4635): 8192 init_tcp(4629): 32768 init_tcp(4621): 8 init_tcp(4614): 8 init_tcp(4607): 8 init_tcp(4601): 8 init_tcp(4589): 8 init_avps(90): 8 init_avps(89): 8 init_dst_blacklist(437): 16384 init_dst_blacklist(430): 8 timer_alloc(514): 96 init_dns_cache(366): 8 init_dns_cache(358): 16384 init_dns_cache(351): 16 init_dns_cache(345): 8 init_timer(283): 8 init_timer(282): 16384 init_timer(281): 8 init_timer(280): 8 init_timer(269): 8 init_timer(237): 8 init_timer(220): 278544 init_timer(219): 8 init_timer(206): 8 cfg_child_cb_new(830): 64 sr_cfg_init(361): 8 sr_cfg_init(354): 8 sr_cfg_init(347): 8 sr_cfg_init(335): 8 sr_cfg_init(323): 8 shm_core_lock_init(153): 8 Total: 875992224 }
Module: dmq { alloc_job_queue(229): 128 shm_str_dup(723): 24 build_dmq_node(156): 448 add_peer(67): 168 mod_init(243): 8 mod_init(237): 96 init_dmq_node_list(66): 24 init_peer_list(33): 24 Total: 920 }
Module: auth_db { Total: 0 }
Module: dialog { init_dlg_table(284): 131088 init_dlg_table(276): 8 init_dlg_table(269): 8 init_dlg_table(264): 8 init_dlg_timer(57): 8 init_dlg_timer(48): 32 new_dlg_profile(187): 624 Total: 131776 }
Module: rr { Total: 0 }
Module: xlog { Total: 0 }
Module: kex { pkg_proc_stats_init(79): 1056 Total: 1056 }
Module: db_postgres { lock_set_alloc(70): 80 Total: 80 }
Module: pv { Total: 0 }
Module: textops { Total: 0 }
Module: htable { ht_cell_new(183): 1104 ht_init_tables(381): 8388608 ht_add_table(278): 928 Total: 8390640 }
Module: sqlops { Total: 0 }
Module: tm { build_uac_req(1543): 848 prepare_new_uac(479): 28880 _reply_light(542): 751504 build_local_reparse(341): 1168568 relay_reply(1884): 643088 build_cell(317): 9826624 prepare_new_uac(524): 8112 prepare_new_uac(509): 10192 init_tm_stats_child(60): 5632 insert_tmcb(137): 136 init_tmcb_lists(74): 16 init_tmcb_lists(72): 16 init_hash_table(467): 2097152 Total: 14540768 }
Module: usrloc { new_urecord(65): 136 new_ucontact(98): 1096 new_urecord(58): 288 shm_str_dup(723): 1200 build_stat_name(51): 64 new_udomain(93): 40960 new_udomain(86): 48 new_dlist(573): 16 new_dlist(565): 32 register_ulcb(94): 32 init_ulcb_list(45): 16 lock_set_alloc(70): 4112 Total: 48000 }
Module: tmx { tmx_check_pretran(250): 864 tmx_check_pretran(271): 1216 tmx_init_pretran_table(90): 256 Total: 2336 }
Module: sl { init_sl_stats_child(125): 4224 sl_startup(83): 8 init_sl_stats(110): 8 Total: 4240 }
Module: carrierroute { rule_fixup_recursor(584): 8 shm_str_dup(723): 56 add_route_rule(74): 152 add_route_flags(225): 48 create_domain_data(83): 32 create_carrier_data(59): 8 create_carrier_data(50): 64 load_route_data_db(295): 8 load_domain_map(182): 48 load_domain_map(171): 24 load_carrier_map(126): 48 load_carrier_map(115): 24 reload_route_data(172): 64 init_route_data(74): 8 Total: 592 }
Module: dispatcher { reindex_dests(600): 1008 add_dest2list(350): 168 add_dest2list(324): 4200 init_data(204): 16 init_data(195): 16 mod_init(309): 8 mod_init(307): 8 ds_ping_active_init(102): 8 Total: 5432 }
Module: db_text { dbt_init_cache(81): 256 dbt_init_cache(69): 8 dbt_init_cache(54): 8 Total: 272 }
The result of command kamctl shmem is:
shmem:fragments = 6353 shmem:free_size = 406208 shmem:max_used_size = 8464105472 shmem:real_used_size = 8463699264 shmem:total_size = 8464105472 shmem:used_size = 5190698208
The server has 8G of shared memory allocated, and the issue happens more or less after 2 days running tests.
I'm using sipp to generate traffic, it is sending 500CAPS.
There is any possibility of memory leak on Kamailio 4.4.0?
Best Regards
-- José Seabra