Differences

This shows you the differences between two versions of the page.

--- tutorials:troubleshooting:memory [2015/09/03 22:01] – [Memory Manager Debugging] miconda
+++ tutorials:troubleshooting:memory [2021/06/01 18:44] (current) – [Insufficient Memory] giavac
@@ Line 36: / Line 36: @@
   * too small PKG or SHM - insufficient size to accommodate all data needed to be stored in memory
-  * memory leak - some part of code allocates memory at runtine and does not free it
+  * memory leak - some part of code allocates memory at runtime and does not free it
 ===== Monitoring Memory =====
@@ Line 46: / Line 46: @@
 <code>
 kamctl stats shmem
+kamcmd mod.stats all shm
 </code>
@@ Line 52: / Line 53: @@
 <code>
 kamcmd pkg.stats
+kamcmd mod.stats all pkg
 </code>
 Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child).
+In order to merge the free memory fragments one should enable memory join. Default is disabled (mem_join=0).
+<code>
+mem_join=1
+</code>
 ===== Analysis of Memory Incidents =====
@@ Line 77: / Line 85: @@
 <code>
-MEMMNG=1 MEMDBG=1 make cfg ...
+MEMDBG=1 make cfg ...
 </code>
@@ Line 98: / Line 106: @@
 Then restart and wait a bit for getting some traffic processed.
+MEMMNG=0/1/2 to select from different memory allocation algorithms (fm, qm, tlsf) is deprecated. Use "-x" parameter when running kamailio instead; see kamailio -h for more details.
 To get the list of chunks from memory manager, there are two ways:
@@ Line 135: / Line 145: @@
 For PKG is similar format, just SHM replaced with PKG in messages.
+To generate summary report, do:
+<code>
+# first set memlog lower than debug
+kamcmd cfg.set_now_int core memlog 1
+kamcmd corex.shm_summary
+</code>
+The log for f_malloc with debug enabled should look like:
+<code>
+(4082) NOTICE: fm_status: summarizing all alloc'ed. fragments:
+(4082) NOTICE: fm_status:  count=     1 size=     16640 bytes from <core>: counters.c: counters_prefork_init(207)
+(4082) NOTICE: fm_status:  count=     1 size=     14560 bytes from debugger: debugger_api.c: dbg_init_pid_list(572)
+(4082) NOTICE: fm_status:  count=     1 size=      4992 bytes from sl: sl_stats.c: init_sl_stats_child(125)
+(4082) NOTICE: fm_status:  count=     1 size=       256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
+(4082) NOTICE: fm_status:  count=     1 size=      6656 bytes from tm: t_stats.c: init_tm_stats_child(60)
+(4082) NOTICE: fm_status:  count=     1 size=      1248 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
+(4082) NOTICE: fm_status:  count=     2 size=        64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130)
+(4082) NOTICE: fm_status:  count=     1 size=       704 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217)
+(4082) NOTICE: fm_status:  count=     3 size=        64 bytes from usrloc: udomain.c: build_stat_name(51)
+</code>
+If you dumped the status with qm_malloc, you can extract the logs from syslog and count the unique allocations with next commands:
+<code>
+grep qm_status /var/log/syslog >qm_status.txt
+# or:
+# grep qm_status /var/log/messages >qm_status.txt
+grep alloc qm_status.txt | awk '{ print substr( $0, 16, length($0) ) }' | sort | uniq -c | sort -k1n
+</code>
 ===== Using GDB =====
@@ Line 224: / Line 268: @@
 Here is the article that presents better the situation:
-   * http://blog.logicmonitor.com/2014/10/09/more-linux-memory-free-memory-that-is-not-free-nor-cache/
+   * https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/
 An relevant excerpt from the blog article: