User Tools

Site Tools


tutorials:troubleshooting:memory

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
tutorials:troubleshooting:memory [2015/05/14 13:46]
miconda [OS Memory Reports]
tutorials:troubleshooting:memory [2021/06/01 20:44] (current)
giavac [Insufficient Memory]
Line 36: Line 36:
  
   * too small PKG or SHM - insufficient size to accommodate all data needed to be stored in memory   * too small PKG or SHM - insufficient size to accommodate all data needed to be stored in memory
-  * memory leak - some part of code allocates memory at runtine and does not free it+  * memory leak - some part of code allocates memory at runtime and does not free it
  
 ===== Monitoring Memory ===== ===== Monitoring Memory =====
Line 46: Line 46:
 <code> <code>
 kamctl stats shmem kamctl stats shmem
 +kamcmd mod.stats all shm
 </code> </code>
  
Line 52: Line 53:
 <code> <code>
 kamcmd pkg.stats kamcmd pkg.stats
 +kamcmd mod.stats all pkg
 </code> </code>
  
 Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child). Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child).
 +
 +In order to merge the free memory fragments one should enable memory join. Default is disabled (mem_join=0).
 +<code>
 +mem_join=1
 +</code>
 +
  
 ===== Analysis of Memory Incidents ===== ===== Analysis of Memory Incidents =====
Line 77: Line 85:
  
 <code> <code>
-MEMMNG=1 MEMDBG=1 make cfg ...+MEMDBG=1 make cfg ...
 </code> </code>
  
Line 98: Line 106:
  
 Then restart and wait a bit for getting some traffic processed. Then restart and wait a bit for getting some traffic processed.
 +
 +MEMMNG=0/1/2 to select from different memory allocation algorithms (fm, qm, tlsf) is deprecated. Use "-x" parameter when running kamailio instead; see kamailio -h for more details.
  
 To get the list of chunks from memory manager, there are two ways: To get the list of chunks from memory manager, there are two ways:
Line 113: Line 123:
     * for SHM memory:     * for SHM memory:
 <code> <code>
-kamcmd cfg.set_now_int core mem_dump_shm+kamcmd cfg.set_now_int core mem_dump_shm 1
 </code> </code>
  
Line 135: Line 145:
 For PKG is similar format, just SHM replaced with PKG in messages. For PKG is similar format, just SHM replaced with PKG in messages.
  
 +To generate summary report, do:
 +
 +<code>
 +# first set memlog lower than debug
 +kamcmd cfg.set_now_int core memlog 1
 +
 +kamcmd corex.shm_summary
 +</code>
 +
 +The log for f_malloc with debug enabled should look like:
 +
 +<code>
 +20(4082) NOTICE: fm_status: summarizing all alloc'ed. fragments:
 +20(4082) NOTICE: fm_status:  count=     1 size=     16640 bytes from <core>: counters.c: counters_prefork_init(207)
 +20(4082) NOTICE: fm_status:  count=     1 size=     14560 bytes from debugger: debugger_api.c: dbg_init_pid_list(572)
 +20(4082) NOTICE: fm_status:  count=     1 size=      4992 bytes from sl: sl_stats.c: init_sl_stats_child(125)
 +20(4082) NOTICE: fm_status:  count=     1 size=       256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
 +20(4082) NOTICE: fm_status:  count=     1 size=      6656 bytes from tm: t_stats.c: init_tm_stats_child(60)
 +20(4082) NOTICE: fm_status:  count=     1 size=      1248 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
 +20(4082) NOTICE: fm_status:  count=     2 size=        64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130)
 +20(4082) NOTICE: fm_status:  count=     1 size=       704 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217)
 +20(4082) NOTICE: fm_status:  count=     3 size=        64 bytes from usrloc: udomain.c: build_stat_name(51)
 +</code>
 +
 +If you dumped the status with qm_malloc, you can extract the logs from syslog and count the unique allocations with next commands:
 +
 +<code>
 +grep qm_status /var/log/syslog >qm_status.txt
 +
 +# or:
 +# grep qm_status /var/log/messages >qm_status.txt
 +
 +grep alloc qm_status.txt | awk '{ print substr( $0, 16, length($0) ) }' | sort | uniq -c | sort -k1n
 +</code>
 ===== Using GDB ===== ===== Using GDB =====
  
Line 213: Line 257:
  
 After updating Makefile.defs recompile and reinstall. After updating Makefile.defs recompile and reinstall.
 +
 +Other tools available out there (e.g., valgrind) can be then used to track the PKG memory operations done by Kamailio.
  
 ===== OS Memory Reports ===== ===== OS Memory Reports =====
Line 222: Line 268:
 Here is the article that presents better the situation: Here is the article that presents better the situation:
  
-   http://blog.logicmonitor.com/2014/10/09/more-linux-memory-free-memory-that-is-not-free-nor-cache/+   https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/
  
 An relevant excerpt from the blog article: An relevant excerpt from the blog article:
tutorials/troubleshooting/memory.1431603997.txt.gz ยท Last modified: 2015/05/14 13:46 by miconda