Kamailio SIP Server Wiki

User Tools

Site Tools

Trace:
tutorials:troubleshooting:memory

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
tutorials:troubleshooting:memory [2015/01/16 16:06] – [OS Memory Reports] micondatutorials:troubleshooting:memory [2021/06/01 18:44] (current) – [Insufficient Memory] giavac
Line 36: Line 36:
  
   * too small PKG or SHM - insufficient size to accommodate all data needed to be stored in memory   * too small PKG or SHM - insufficient size to accommodate all data needed to be stored in memory
-  * memory leak - some part of code allocates memory at runtine and does not free it+  * memory leak - some part of code allocates memory at runtime and does not free it
  
 ===== Monitoring Memory ===== ===== Monitoring Memory =====
Line 46: Line 46:
 <code> <code>
 kamctl stats shmem kamctl stats shmem
 +kamcmd mod.stats all shm
 </code> </code>
  
Line 52: Line 53:
 <code> <code>
 kamcmd pkg.stats kamcmd pkg.stats
 +kamcmd mod.stats all pkg
 </code> </code>
  
 Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child). Notice that for SHM only one group of statistics is printed, being one zone of memory, while for PKG you get a list with many groups of statistics, each specific for a Kamailio process (child).
 +
 +In order to merge the free memory fragments one should enable memory join. Default is disabled (mem_join=0).
 +<code>
 +mem_join=1
 +</code>
 +
  
 ===== Analysis of Memory Incidents ===== ===== Analysis of Memory Incidents =====
Line 63: Line 71:
   * if the number of subscribers, traffic is constant, no larger data was reloaded (e.g., dispacher, lcr), then there is very likely a memory leak that has to be discovered and fixed   * if the number of subscribers, traffic is constant, no larger data was reloaded (e.g., dispacher, lcr), then there is very likely a memory leak that has to be discovered and fixed
  
-===== Troubleshooting ===== 
  
 ===== Memory Manager Debugging ===== ===== Memory Manager Debugging =====
Line 78: Line 85:
  
 <code> <code>
-MEMMNG=1 MEMDBG=1 make cfg ...+MEMDBG=1 make cfg ...
 </code> </code>
  
Line 99: Line 106:
  
 Then restart and wait a bit for getting some traffic processed. Then restart and wait a bit for getting some traffic processed.
 +
 +MEMMNG=0/1/2 to select from different memory allocation algorithms (fm, qm, tlsf) is deprecated. Use "-x" parameter when running kamailio instead; see kamailio -h for more details.
  
 To get the list of chunks from memory manager, there are two ways: To get the list of chunks from memory manager, there are two ways:
Line 114: Line 123:
     * for SHM memory:     * for SHM memory:
 <code> <code>
-kamcmd cfg.set_now_int core mem_dump_shm+kamcmd cfg.set_now_int core mem_dump_shm 1
 </code> </code>
  
Line 136: Line 145:
 For PKG is similar format, just SHM replaced with PKG in messages. For PKG is similar format, just SHM replaced with PKG in messages.
  
 +To generate summary report, do:
 +
 +<code>
 +# first set memlog lower than debug
 +kamcmd cfg.set_now_int core memlog 1
 +
 +kamcmd corex.shm_summary
 +</code>
 +
 +The log for f_malloc with debug enabled should look like:
 +
 +<code>
 +20(4082) NOTICE: fm_status: summarizing all alloc'ed. fragments:
 +20(4082) NOTICE: fm_status:  count=     1 size=     16640 bytes from <core>: counters.c: counters_prefork_init(207)
 +20(4082) NOTICE: fm_status:  count=     1 size=     14560 bytes from debugger: debugger_api.c: dbg_init_pid_list(572)
 +20(4082) NOTICE: fm_status:  count=     1 size=      4992 bytes from sl: sl_stats.c: init_sl_stats_child(125)
 +20(4082) NOTICE: fm_status:  count=     1 size=       256 bytes from tmx: tmx_pretran.c: tmx_init_pretran_table(90)
 +20(4082) NOTICE: fm_status:  count=     1 size=      6656 bytes from tm: t_stats.c: init_tm_stats_child(60)
 +20(4082) NOTICE: fm_status:  count=     1 size=      1248 bytes from kex: pkg_stats.c: pkg_proc_stats_init(79)
 +20(4082) NOTICE: fm_status:  count=     2 size=        64 bytes from <core>: cfg/cfg_struct.c: cfg_clone_str(130)
 +20(4082) NOTICE: fm_status:  count=     1 size=       704 bytes from <core>: cfg/cfg_struct.c: cfg_shmize(217)
 +20(4082) NOTICE: fm_status:  count=     3 size=        64 bytes from usrloc: udomain.c: build_stat_name(51)
 +</code>
 +
 +If you dumped the status with qm_malloc, you can extract the logs from syslog and count the unique allocations with next commands:
 +
 +<code>
 +grep qm_status /var/log/syslog >qm_status.txt
 +
 +# or:
 +# grep qm_status /var/log/messages >qm_status.txt
 +
 +grep alloc qm_status.txt | awk '{ print substr( $0, 16, length($0) ) }' | sort | uniq -c | sort -k1n
 +</code>
 ===== Using GDB ===== ===== Using GDB =====
  
Line 148: Line 191:
 if($i>2000) if($i>2000)
 if($a->u.is_free==0) if($a->u.is_free==0)
 +printf "=========== non-free fragment: %d\n", $i
 +p $a
 +p (void*)((char*)($a)+sizeof(struct qm_frag))
 +printf "----------- content\n"
 p *$a p *$a
 end end
 +end
 +set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end)))
 +set $i = $i + 1
 +end
 +</code>
 +
 +An alternative is to print all used chunks, but be aware that it may take some time:
 +
 +
 +<code c>
 +set $i=0
 +set $a = mem_block->first_frag
 +while($a < mem_block->last_frag_end)
 +if($a->u.is_free==0)
 +printf "=========== non-free fragment: %d\n", $i
 +p $a
 +p (void*)((char*)($a)+sizeof(struct qm_frag))
 +printf "----------- content\n"
 +p *$a
 end end
 set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end))) set $a = ((struct qm_frag*)((char*)($a)+sizeof(struct qm_frag)+((struct qm_frag*)$a)->size+sizeof(struct qm_frag_end)))
Line 167: Line 233:
 gdb --batch --command=/tmp/kamailio-dump-used-pkg.gdb /usr/sbin/kamailio 21907 gdb --batch --command=/tmp/kamailio-dump-used-pkg.gdb /usr/sbin/kamailio 21907
 </code> </code>
 +
 +===== PKG With System Malloc =====
 +
 +Kamailio can be compiled to use system malloc and free for PKG needs. You have to remove the PKG_MALLOC define from Makefile.defs and can add DBG_SYS_MALLOC to get more verbosity with memory operations in debug mode.
 +
 +Next is a diff showing the changes in Makefile.defs, but note that lines may vary on your specific Kamailio version.
 +
 +<code c>
 +diff --git a/Makefile.defs b/Makefile.defs
 +index 3890668..12ca37a 100644
 +--- a/Makefile.defs
 ++++ b/Makefile.defs
 +@@ -621,7 +621,7 @@ C_DEFS= $(extra_defs) \
 +         -DSER_VER=$(SER_VER) \
 +         -DCFG_DIR='"$(cfg_target)"'\
 +         -DRUN_DIR='"$(run_target)"'\
 +-        -DPKG_MALLOC \
 ++        -DDBG_SYS_MALLOC \
 +         -DSHM_MEM  -DSHM_MMAP \
 +         -DDNS_IP_HACK \
 +         -DUSE_MCAST \
 +</code>
 +
 +After updating Makefile.defs recompile and reinstall.
 +
 +Other tools available out there (e.g., valgrind) can be then used to track the PKG memory operations done by Kamailio.
  
 ===== OS Memory Reports ===== ===== OS Memory Reports =====
Line 176: Line 268:
 Here is the article that presents better the situation: Here is the article that presents better the situation:
  
-   * http://blog.logicmonitor.com/2014/10/09/more-linux-memory-free-memory-that-is-not-free-nor-cache/+   * https://www.logicmonitor.com/blog/more-linux-memory-free-memory-that-is-not-free-nor-cache/
  
 An relevant excerpt from the blog article: An relevant excerpt from the blog article:
tutorials/troubleshooting/memory.1421424408.txt.gz · Last modified: 2015/01/16 16:06 by miconda
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki