I took a look on mem/q_malloc.c and I saw:<div><br></div><div>        /*size must be a multiple of 8*/</div><div><div>        size=ROUNDUP(size);</div><div><br></div><div>that pretty much answered most of my questions regarding the size of the memory chunks, but unfortunately not the reason of the crashing after 40k calls. MEMDBG doesn&#39;t report leaks on my module after finalizing Kamailio.</div>
<div><br></div><div>I don&#39;t see why the size should be changed since the values that I&#39;m passing are always equal to 27 in length. Why is this?</div><div><br></div><div>Regards.</div><div><br></div><div>Carlos.</div>
<div><br></div><br><div class="gmail_quote">On Mon, Sep 10, 2012 at 8:24 PM, Carlos Ruiz Díaz <span dir="ltr">&lt;<a href="mailto:carlos.ruizdiaz@gmail.com" target="_blank">carlos.ruizdiaz@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks for your help Daniel.<div><br></div><div>After trying memory dumping with sercmd and killall, I ended up using pkg_sums() before and after every block of dynamic memory allocation. What I found out (and scared me) is that my module finishes with no leak according to the memory summary but, somehow, the memory consumption grows in, what appears to be, a linear basis. </div>

<div><br></div><div>Below the outputs per stage after the entry point function is called:</div><div><br></div><div><b><u>Stage 1:</u></b></div><div><div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=    10 size=       256 bytes from cnx3a: cnx3a_parser.c: parse_body(122)</div>

<div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        40 bytes from cnx3a: cnx3a_parser.c: parse_header(170)</div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        40 bytes from cnx3a: cnx3a_mod.c: cnx3a_invite(1032)</div>

<div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=    10 size=        88 bytes from cnx3a: cnx3a_parser.c: parse_body(88)</div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=      5000 bytes from cnx3a: cnx3a_mod.c: cnx3a_invite(972)</div>

<div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=      5000 bytes from cnx3a: cnx3a_mod.c: cnx3a_invite(971)</div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=    10 size=        92 bytes from cnx3a: cnx3a_parser.c: parse_header(182)</div>

</div><div><br></div><div><i>In the final stage, the module is supposed to store a SIPURI and return. That SIPURI will later be consulted by the script using a PV, reason why, <b>stage 2</b> ends with 28 bytes of memory usage.</i></div>

<div><br></div><div>pkg_str_dup() is called.</div><div><br></div><div><b><u>Stage2:</u></b></div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        28 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

<div><br></div><div><i>Afterwards, the module returns with 1.</i></div><div><i><br></i></div><div>Later when the script tries to write to a PV, the function in charge is called and if the value is assigned (str.s != NULL) it frees the internal variable first and then does another pkg_str_dup().</div>

<div><br></div><div><b>1st assignment. Before trying to free: </b></div><div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        28 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

</div><div><br></div><div><b>1st assignment. </b><b>After trying to free:</b></div><div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        28 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

</div><div><br></div><div><i>Same output since the var is null</i></div><div><br></div><div><b>1st assignment. </b><b>After pkg_str_dup():</b></div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: &lt;core&gt; [mem/q_malloc.c:369]: qm_malloc(0xb71fd008, 27) called from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

<div><div>Sep 10 19:58:29 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     2 size=        56 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div></div><div><br></div><div>-------------------------------------------------------------</div>

<div><br></div><div><b>2nd assignment. B</b><b>efore trying to free:</b></div><div><div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     2 size=        56 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

</div><div><br></div><div><b>2nd assignment. </b><b>After trying to free</b><b>: </b></div><div><div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: &lt;core&gt; [mem/q_malloc.c:437]: qm_free(0xb71fd008, 0xb7275dec), called from cnx3a: cnx3a_mod.c: pv_set_routes(2041)</div>

<div><br></div><div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: &lt;core&gt; [mem/q_malloc.c:461]: qm_free: freeing frag. 0xb7275dd4 alloc&#39;ed from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

<div><br></div><div><div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     1 size=        28 bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div></div><div style="font-weight:bold">

<br></div></div><div style="font-weight:bold"><span style="font-weight:normal"><i>free() worked since var != NULL. We ended up with 28 bytes, which is correct.</i></span></div><div style="font-weight:bold"><br></div><div style="font-weight:bold">

<b>2nd assignment. </b><b>After pkg_str_dup():</b></div><div style="font-weight:bold"><b><br></b></div><div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: &lt;core&gt; [mem/q_malloc.c:369]: qm_malloc(0xb71fd008, 27) called from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

<div>Sep 10 19:58:33 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  count=     2 size=       <b> 56</b> bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div><div><br></div><div>56 is correct.</div>

<div><div><br></div><div>Now, when the whole circuit is repeated for the second time, in <u style="font-weight:bold">Stage2</u> happens the following:</div></div><div><br></div><div>Sep 10 19:58:39 carlosrdcnx-laptop kamailio: NOTICE: &lt;core&gt; [mem/q_malloc.c:369]: qm_malloc(0xb71fd008, <b>27</b>) called from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

<div><div><br></div><div>Sep 10 19:58:39 carlosrdcnx-laptop kamailio: NOTICE: qm_sums:  <b>count=     1</b> size=        <b><font color="#ff0000">32</font> </b>bytes from cnx3a: /usr/include/kamailio/parser/../ut.h: pkg_str_dup(720)</div>

</div><div><br></div><div>-------------------------------------------------------------</div><div><br></div><div>Why pkg_malloc always gives 28 bytes instead of the 27 requested? Why the returned size is equal to 32 instead of just 27 in the last example? Can this be a memory fragmentation problem?</div>

<div><br></div><div>This happened after a few calls. Can this be the cause for a crash after running 40k calls?</div><div><br></div><div>I&#39;m using kamailio 3.2.3.</div><div><br></div><div>Regards and thanks in advance.</div>
<span class="HOEnZb"><font color="#888888">
<div><br></div><div>Carlos</div></font></span><div><div class="h5"><br><div class="gmail_quote">On Mon, Sep 10, 2012 at 12:18 PM, Daniel-Constantin Mierla <span dir="ltr">&lt;<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    Hello,<div><br>
    <br>
    <div>On 9/10/12 2:33 PM, Carlos Ruiz Díaz
      wrote:<br>
    </div>
    <blockquote type="cite">Hi Daniel,
      <div><br>
      </div>
      <div>no, it&#39;s just a plain make &amp;&amp; make install. </div>
      <div><br>
      </div>
      <div>I spent a few hours on debugging and I figured out that it
        was an off by one error on a memcpy() on certain exceptional
        conditions. I fixed that but then I found a memory leak that&#39;s
        causing the crash after 40k calls but I&#39;m having a hard time
        interpreting the dump provided after sending the SIGUSR1 to the
        process. if I&#39;m not mistaken, there are two pkg_str_dup() leaks
        per call but I&#39;m still searching where I&#39;m missing the
        pkg_free(). </div>
      <div><br>
      </div>
      <div>Is there a way to know where in the code the pkg_str_dup was
        called?</div>
      <div><br>
      </div>
      <div>I tried with valgrind too, but after the macro expansion of
        the memory management functions, is very hard to find where it
        was called.</div>
    </blockquote>
    <br></div>
    here are some hints about troubleshooting memory:<br>
    <a href="http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting" target="_blank">http://www.asipto.com/pub/kamailio-devel-guide/#c04troubleshooting</a><br>
    <br>
    if you set memdbg and memlog lower than debug global parameter, then
    you get all memory operations in the syslog (be sure you compile
    with MEMDBG=1). Also look at mem_summary:<br>
    <a href="http://www.kamailio.org/wiki/cookbooks/3.3.x/core#mem_summary" target="_blank">http://www.kamailio.org/wiki/cookbooks/3.3.x/core#mem_summary</a><br>
    <br>
    and cfgutils module readme for some functions that can be used in
    config to dump the memory usage.<br>
    <br>
    Cheers,<br>
    Daniel<div><div><br>
    <br>
    <br>
    <blockquote type="cite">
      <div><br>
      </div>
      <div>Regards.</div>
      <div><br>
      </div>
      <div>Carlos.</div>
      <div><br>
      </div>
      <div><br>
        <br>
        <div class="gmail_quote">On Mon, Sep 10, 2012 at 4:02 AM,
          Daniel-Constantin Mierla <span dir="ltr">&lt;<a href="mailto:miconda@gmail.com" target="_blank">miconda@gmail.com</a>&gt;</span> wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000"> Hello,<br>
              <br>
              yes, by default is with debug symbols on. Do you provide
              special flags or parameters to the make command?<br>
              <br>
              Cheers,<br>
              Daniel
              <div>
                <div><br>
                  <br>
                  <div>On 9/8/12 9:39 PM, Carlos Ruiz Díaz wrote:<br>
                  </div>
                </div>
              </div>
              <blockquote type="cite">
                <div>
                  <div>Hello list,
                    <div><br>
                    </div>
                    <div>I&#39;m trying to debug my module after it crashed
                      by SIGSEGV, but the problem is that I&#39;m unable to
                      see the relevant data in the core file that&#39;s
                      being dumped after the crash.</div>
                    <div> <br>
                    </div>
                    <div>For example, if I do <i>bt full, </i>I get:</div>
                    <div><br>
                    </div>
                    <div>
                      <div>0  0x0ef0f494 in append_string () from
                        /opt/install/kamailio-3.2.3/lib/kamailio/modules_k/cnx3a.so</div>
                      <div>No symbol table info available.</div>
                      <div>#1  0x0eef9658 in build_message () from
                        /opt/install/kamailio-3.2.3/lib/kamailio/modules_k/cnx3a.so</div>
                      <div>No symbol table info available.</div>
                      <div>Backtrace stopped: frame did not save the PC</div>
                    </div>
                    <div><br>
                    </div>
                    <div>Only the functions that were called but no
                      parameter values. </div>
                    <div><br>
                    </div>
                    <div>AFAIK, if I compile kamailio from sources, it
                      is by default compiled with the -g option, so, it
                      looks like that I&#39;m missing something else.</div>
                    <div><br>
                    </div>
                    <div>Regards.</div>
                    <div><br>
                    </div>
                    <div>Carlos.</div>
                    <br>
                    <fieldset></fieldset>
                    <br>
                  </div>
                </div>
                <pre>_______________________________________________
sr-dev mailing list
<a href="mailto:sr-dev@lists.sip-router.org" target="_blank">sr-dev@lists.sip-router.org</a>
<a href="http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev" target="_blank">http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev</a><span><font color="#888888">
</font></span></pre>
                <span><font color="#888888"> </font></span></blockquote>
              <span><font color="#888888"> <br>
                  <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#%21/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Nov 5-8, 2012 - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a>
Kamailio Advanced Training, Miami, USA, Nov 12-14, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a></pre>
                </font></span></div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
    <pre cols="72">-- 
Daniel-Constantin Mierla - <a href="http://www.asipto.com" target="_blank">http://www.asipto.com</a>
<a href="http://twitter.com/#!/miconda" target="_blank">http://twitter.com/#!/miconda</a> - <a href="http://www.linkedin.com/in/miconda" target="_blank">http://www.linkedin.com/in/miconda</a>
Kamailio Advanced Training, Berlin, Nov 5-8, 2012 - <a href="http://asipto.com/u/kat" target="_blank">http://asipto.com/u/kat</a>
Kamailio Advanced Training, Miami, USA, Nov 12-14, 2012 - <a href="http://asipto.com/u/katu" target="_blank">http://asipto.com/u/katu</a></pre>
  </div></div></div>

</blockquote></div><br></div></div></div>
</blockquote></div><br></div>