25 Jul
2023
25 Jul
'23
7 p.m.
### Description
We have some Kamailio servers running as an outboundproxy. Requests go to different
domains. Over the time, there is a slow but steady shared memory increase. We don't
see the behavior on our Kamailio servers serving as endpoint proxies using the same code.
I did a memory dump, and on two systems I inspected, I saw the same result: A really high
memory usage for the `dns_cache_mk_rd_entry` function.
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
### Troubleshooting
#### Reproduction
When we restart our Kamailio, it starts off with just a few megabytes of shared mem used.
After running for a month or two, this has increased to some hundred megabytes used.
<!--
If the issue can be reproduced, describe how it can be done.
-->
#### Debugging Data
This is the output of the shm dump:
```
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): summarizing
all alloc'ed. fragments:
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 592 bytes from tm: t_reply.c: _reply_light(500)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
45 size= 43576 bytes from core: core/msg_translator.c:
build_req_buf_from_sip_req(2233)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
44 size= 33232 bytes from tm: t_reply.c: relay_reply(2061)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
45 size= 376840 bytes from tm: h_table.c: build_cell(334)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
4 size= 224 bytes from pike: ip_tree.c: new_ip_node(201)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
45 size= 275304 bytes from core: core/sip_msg_clone.c: sip_msg_shm_clone(495)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
16 size= 1528 bytes from core: core/usr_avp.c: create_avp(176)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
45 size= 16728 bytes from core: core/sip_msg_clone.c: msg_lump_cloner(973)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
12 size= 488 bytes from tls: tls_ct_q.h: tls_ct_q_add(58)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
12 size= 584 bytes from tls: tls_server.c: tls_complete_init(276)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
59 size= 3776 bytes from rtpengine: ../../core/parser/../ut.h: shm_str_dup(859)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
212 size= 15720 bytes from core: core/xavp.c: xavp_new_value(116)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
59 size= 3888 bytes from rtpengine: rtpengine.c: rtpp_function_call(2775)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
29 size= 2680 bytes from core: core/dns_cache.c: dns_cache_mk_bad_entry(770)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums():
count=856568 size= 311761008 bytes from core: core/dns_cache.c:
dns_cache_mk_rd_entry(1117)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
13941 size= 2741784 bytes from htable: ht_api.c: ht_cell_new(186)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
2 size= 104 bytes from xhttp_prom: prom_metric.c: prom_metric_lvalue_create(513)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
24 size= 3752 bytes from tmx: tmx_pretran.c: tmx_check_pretran(250)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
24 size= 6776 bytes from tmx: tmx_pretran.c: tmx_check_pretran(271)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
553 size= 10177808 bytes from core: core/tcp_main.c: tcpconn_new(1175)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 53760 bytes from core: core/counters.c: counters_prefork_init(212)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
2 size= 976 bytes from tls: tls_domain.c: ksr_tls_fix_domain(1057)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8064 bytes from sl: sl_stats.c: init_sl_stats_child(125)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 512 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 10752 bytes from tm: t_stats.c: init_tm_stats_child(56)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 2016 bytes from kex: pkg_stats.c: pkg_proc_stats_init(78)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
8 size= 176 bytes from core: core/cfg/cfg_struct.c: cfg_clone_str(130)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 992 bytes from core: core/cfg/cfg_struct.c: cfg_shmize(217)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: main.c: main_loop(1334)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/pt.c: init_pt(107)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/pt.c: init_pt(106)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 6216 bytes from core: core/pt.c: init_pt(105)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from app_python3: apy_kemi.c: apy_sr_init_mod(2095)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from app_python3: apy_kemi.c: apy_sr_init_mod(2088)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from xhttp_prom: prom_metric.c: prom_metric_init(219)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from pike: pike.c: pike_init(127)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 1040 bytes from pike: ../../core/lock_alloc.h: lock_set_alloc(66)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 4112 bytes from pike: ip_tree.c: init_ip_tree(115)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from pike: pike.c: pike_init(109)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
4 size= 120 bytes from core: core/cfg_parser.c: cfg_parse_str(907)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
2 size= 448 bytes from tls: tls_domain.c: tls_new_domain(187)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 48 bytes from tls: tls_domain.c: tls_new_cfg(1553)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from tls: tls_ct_wrq.c: tls_ct_wq_init(49)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from tls: tls_mod.c: mod_init(373)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from tls: tls_mod.c: mod_init(357)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from siptrace: siptrace.c: mod_init(338)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
256 size= 14336 bytes from rtpengine: rtpengine_hash.c:
rtpengine_hash_table_init(111)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
256 size= 2048 bytes from rtpengine: rtpengine_hash.c: rtpengine_hash_table_init(96)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 1024 bytes from rtpengine: rtpengine_hash.c: rtpengine_hash_table_init(85)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 2048 bytes from rtpengine: rtpengine_hash.c: rtpengine_hash_table_init(76)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 2048 bytes from rtpengine: rtpengine_hash.c: rtpengine_hash_table_init(67)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 32 bytes from rtpengine: rtpengine_hash.c: rtpengine_hash_table_init(58)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 24 bytes from rtpengine: rtpengine.c: add_rtpengine_socks(1021)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 64 bytes from rtpengine: rtpengine.c: add_rtpengine_socks(1002)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from rtpengine: rtpengine.c: get_rtpp_set(916)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 56 bytes from rtpengine: rtpengine.c: get_rtpp_set(907)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from rtpengine: rtpengine.c: mod_init(1683)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 24 bytes from rtpengine: rtpengine.c: mod_init(1676)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from rtpengine: rtpengine.c: mod_init(1650)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from rtpengine: rtpengine.c: mod_init(1643)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from rtpengine: rtpengine.c: mod_init(1629)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
5 size= 1581568 bytes from htable: ht_api.c: ht_init_tables(394)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from usrloc: ul_callback.c: init_ulcb_list(44)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
4 size= 288 bytes from core: core/timer.c: register_timer(995)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from sl: sl_funcs.c: sl_startup(90)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from sl: sl_stats.c: init_sl_stats(110)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(71)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from tm: t_hooks.c: init_tmcb_lists(69)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 2097152 bytes from tm: h_table.c: init_hash_table(482)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from tm: rpc_uac.c: tm_rpc_response_list_init(77)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from kex: mod_stats.c: mod_stats_init_rpc(294)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
4 size= 128 bytes from core: core/cfg/cfg_ctx.c: cfg_register_ctx(47)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1089 size= 82016 bytes from tls: tls_init.c: ser_realloc(299)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8192 bytes from core: core/tcp_main.c: init_tcp(5051)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 32768 bytes from core: core/tcp_main.c: init_tcp(5046)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/tcp_main.c: init_tcp(5037)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/tcp_main.c: init_tcp(5030)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/tcp_main.c: init_tcp(5023)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/tcp_main.c: init_tcp(5017)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/tcp_main.c: init_tcp(5005)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/usr_avp.c: init_avps(90)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/usr_avp.c: init_avps(89)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16384 bytes from core: core/dst_blocklist.c: init_dst_blocklist(437)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/dst_blocklist.c: init_dst_blocklist(430)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
2 size= 96 bytes from core: core/timer.c: timer_alloc(498)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/dns_cache.c: init_dns_cache(368)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16384 bytes from core: core/dns_cache.c: init_dns_cache(359)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16 bytes from core: core/dns_cache.c: init_dns_cache(351)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/dns_cache.c: init_dns_cache(343)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(267)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 16384 bytes from core: core/timer.c: init_timer(266)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(265)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(264)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(253)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(221)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 278544 bytes from core: core/timer.c: init_timer(210)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(209)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/timer.c: init_timer(197)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 64 bytes from core: core/cfg/cfg_struct.c: cfg_child_cb_new(829)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/cfg/cfg_struct.c: sr_cfg_init(361)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/cfg/cfg_struct.c: sr_cfg_init(354)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/cfg/cfg_struct.c: sr_cfg_init(347)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/cfg/cfg_struct.c: sr_cfg_init(335)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/cfg/cfg_struct.c: sr_cfg_init(323)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/sr_module.c: ksr_shutdown_phase_init(124)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 36864 bytes from core: core/kemi.c: sr_kemi_cbname_list_init(3605)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/kemi.c: sr_kemi_cbname_list_init(3596)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/kemi.c: sr_kemi_cbname_list_init(3586)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
3 size= 72 bytes from xhttp_prom: ../../core/ut.h: shm_str_dup(859)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
5 size= 96 bytes from xhttp_prom: prom_metric.c: prom_lb_node_add(813)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
5 size= 120 bytes from xhttp_prom: prom_metric.c: prom_lb_node_add(791)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
3 size= 48 bytes from xhttp_prom: prom_metric.c: prom_lb_create(873)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
3 size= 168 bytes from xhttp_prom: prom_metric.c: prom_counter_create(978)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
9 size= 128 bytes from pv: pv_shv.c: set_shvar_value(243)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
40 size= 472 bytes from pv: pv_shv.c: add_shvar(177)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
40 size= 2608 bytes from pv: pv_shv.c: add_shvar(170)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 80 bytes from pv: ../../core/lock_alloc.h: lock_set_alloc(66)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
5 size= 1600 bytes from htable: ht_api.c: ht_add_table(289)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
10464 size= 1453800 bytes from tls: tls_init.c: ser_malloc(293)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums(): count=
1 size= 8 bytes from core: core/mem/q_malloc.c: qm_shm_lock_init(1277)
Jul 25 15:08:52 sipproxy /usr/sbin/kamailio[122]: NOTICE: qm_sums: qm_sums():
-----------------------------
```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.6.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 9.4.0
```
* **DNS related kamailio.cfg Settings**
```
dns_retr_time=3
dns_retr_no=3
dns_use_search_list=no
dns_try_naptr=yes
use_dns_failover=yes
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04,
CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Podman container with Ubuntu Focal.
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523(a)github.com>
12 Sep
12 Sep
3:02 p.m.
This one aged a bit due to summer break ... are there log error messages?
There should be some RPC commands to get dns cache content and other reports, can you
monitor and see what records accumulate?
Has it happened again?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1715593038
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1715593038(a)github.com>
3:18 p.m.
There are no error messages. It happens all the time. I just checked the graphs of one of
the affected systems. If we don't restart them from time to time, shared mem will fill
up eventually.
<img width="1761" alt="kamailio_shared_mem_graph"
src="https://github.com/kamailio/kamailio/assets/9820790/a173ed58-d3f0…
`kamcmd dns.view` shows about 70 entries in the cache, not more.
`kamcmd dns.delete_all_force` doesn't free any mem.
Anything else I can check here?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1715616058
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1715616058(a)github.com>
28 Sep
28 Sep
10:07 a.m.
I went a few times over the code but I could not spot anything. Do you have a test system
where this can be reproduced and run git master branch?
Can you estimate with how many DNS records the system may have to deal with (i.e., how
many domains are involved in SIP routing that have to be resolved and stored in the
cache)?
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1738597182
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1738597182(a)github.com>
18 Oct
18 Oct
3:44 p.m.
We will first update two systems to the latest 5.7 branch release (and a new OS as well)
and watch if the behavior has changed. Will take another week or so.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1768374938
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1768374938(a)github.com>
20 Oct
20 Oct
5:41 p.m.
Okay, the problem is not gone in current 5.7 (5.7.2 from the official repo).
We restarted one affected system 2 days ago. Since then, I have dumped the shm summary a
few times.
Shortly after restart:
```
NOTICE: qm_sums: qm_sums(): count= 159 size= 42000 bytes from core:
core/dns_cache.c: dns_cache_mk_rd_entry(1117)
```
After running for 10 hours:
```
NOTICE: qm_sums: qm_sums(): count= 9822 size= 2857152 bytes from core:
core/dns_cache.c: dns_cache_mk_rd_entry(1117)
```
After running for 2 days:
```
NOTICE: qm_sums: qm_sums(): count= 39931 size= 12716680 bytes from core:
core/dns_cache.c: dns_cache_mk_rd_entry(1117)
```
Answering your question about the amount of different domains: We have a pretty static
load sharing over all proxies, so without an outage the same amount of customer trunks
always get routed through the same proxy. And there are only 10-20 carriers that actually
get routed through those proxies. The only special thing are Telekom CompanyFlex trunks,
where every customer has their own dns name, which gets set in `$du` when handling the
packet. All other trunks get routed simply via `$ru`.
In total there are just about 1k trunks registered through each system.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1772868040
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1772868040(a)github.com>
15 Nov
15 Nov
7:46 p.m.
Closed #3523 as completed.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#event-10969811645
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issue/3523/issue_event/10969811645(a)github.com>
7:46 p.m.
Since we are obviously the only ones having this problem and it's not possible for us
to set up a useful test system and to find anything without it, I'm going to close it.
Seems to be an edge case and we'll write a "restart every 4 weeks" wrapper
around.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3523#issuecomment-1812894698
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3523/1812894698(a)github.com>
376
days inactive
489
days old
7 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
Sebastian Damm