Re: [SR-Users] Memory leak in 3.3.4

Hello, revising the patch I noticed I was moving the initialization of the variable after pushing it to perl environment (from the perl docs, the variable should have been initialized after initializing the environment -- what I tried to do with previous patch). There is a new smaller patch to be added: http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai… Practically, the line XPUSHs(m); has to be moved after the line with SvREADONLY_on(SvRV(m)); Cheers, Daniel On 7/31/13 8:26 AM, David Cunningham wrote: Hi Daniel, We tried that patch, but Kamailio logged lots of errors like the following. The undefined value in question is $m, which should be the SIP message. Would you have any advice? Thanks. Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl [openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an undefined value at Foo.pm line 247.#012 On 25 July 2013 17:11, David Cunningham &lt;dcunningham(a)voisonics.com&gt;wrote;wrote: > Hi Daniel, > > I'll suggest that to the customer. Thank you! > > > > On 25 July 2013 15:45, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt;wrote;wrote: > >> Hello, >> >> can you try the attached patch? It's the same patch, just for two >> versions, one is for 3.3.x and the other for devel version >> >> It initializes the SIP message variable that is passed to perl after >> creating the temporary environment, so it is actually destroyed by the perl >> embedded interpreter. >> >> Cheers, >> Daniel >> >> >> On 7/25/13 1:29 AM, David Cunningham wrote: >> >> Hi Daniel, >> >> The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server >> release 5.4. If I remember right programs running under Valgrind can have >> issues, so I'm not sure if the customer will want to do that. Ideally we'd >> do it on a test system, but I'm not sure if we have any RHEL available. >> I'll see what we can do. Thanks again. >> >> >> On 25 July 2013 04:55, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt;wrote;wrote: >> >>> Hello, >>> >>> I would say that perl_exec() is the one with the highest chances to >>> be the reason for the leak. Next is line would be db_mysql module, if liked >>> with some custom mysql client library, although even in this case will be >>> unlikely. >>> >>> Back to perl, the module itself does not call any malloc, so it >>> might be the embedding Perl API that is not used properly in the module. >>> >>> Can you use some testbed, set children=1 and run kamailio under >>> valgrind, then do some calls and see if it detects the source of the leak? >>> >>> I'm not using the perl module, I will try to check it whenever I get >>> a chance in the next days. What version of perl do you have installed? >>> >>> Cheers, >>> Daniel >>> >>> >>> On 7/24/13 10:31 AM, David Cunningham wrote: >>> >>> Hello, >>> >>> We don't do any kamctl commands at all. We do have various modules >>> loaded, as follows. The primary functions we use Kamailio for are phone >>> registrations through usrloc, and routing calls to Asterisk through logic >>> contained in Perl via perl_exec(). >>> Thanks for all your advice so far! >>> >>> loadmodule "tm.so" >>> loadmodule "tmx.so" >>> loadmodule "usrloc.so" >>> loadmodule "auth.so" >>> loadmodule "auth_db.so" >>> loadmodule "ctl.so" >>> loadmodule "db_mysql.so" >>> loadmodule "kex.so" >>> loadmodule "maxfwd.so" >>> loadmodule "mi_fifo.so" >>> loadmodule "mi_rpc.so" >>> loadmodule "nathelper.so" >>> loadmodule "perl.so" >>> loadmodule "pv.so" >>> loadmodule "registrar.so" >>> loadmodule "rr.so" >>> loadmodule "sanity.so" >>> loadmodule "siputils.so" >>> loadmodule "sl.so" >>> loadmodule "textops.so" >>> loadmodule "xlog.so" >>> >>> >>> On 24 July 2013 16:33, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt;wrote;wrote: >>> >>>> Hello, >>>> >>>> >>>> On 7/24/13 4:24 AM, David Cunningham wrote: >>>> >>>> Hello, >>>> >>>> Thank you very much for the email. In reply: >>>> >>>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio. >>>> >>>> then all the instructions I gave are useless, they are for >>>> debugging kamailio's internal memory manager, which handles pkg and shm >>>> mallocs. >>>> >>>> The chances to be from kamailio itself are very low now. Do you do >>>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but >>>> the rest of code should use internal memory manager which does not go >>>> beyond the limits set with -m and -M, thus not causing an OS memory >>>> exhaustion. >>>> >>>> Can you list what modules are you loading? At some point it was a >>>> leak in libssl, in case you use tls a lot. But could be another external >>>> library... >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> >>>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If >>>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1. >>>> >>>> 3. We'll try setting mem_summary=12, thanks. >>>> >>>> 4. We'll try setting asynchronous syslog, thanks. >>>> >>>> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a >>>> lot? >>>> >>>> 6. We'll try setting mem_join=1, thanks. >>>> >>>> >>>> >>>> On 23 July 2013 16:53, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt;wrote;wrote: >>>> >>>>> Hello, >>>>> >>>>> first, to clarify, is the system memory or kamailio's pkg/shm >>>>> memory running out? If the operating system runs out of memory, then should >>>>> be a leak in a library, because kamailio modules uses only from a >>>>> pre-allocated chunk, not going over it. >>>>> >>>>> >>>>> On 7/23/13 7:33 AM, David Cunningham wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly >>>>>> using more and more memory. Over a couple of weeks it will run out of >>>>>> system memory. >>>>>> >>>>>> We tried to enable memory debugging doing the following, but it >>>>>> resulted in Kamailio not responding to any SIP packets. Would anyone have >>>>>> advice please on how to debug the situation? >>>>>> >>>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio. >>>>>> 2. In kamailio.cfg add the line: >>>>>> #!define DEBUG_MEMORY 1 >>>>>> >>>>> do you set something special in config when DEBUG_MEMORY is 1? It >>>>> is not by default there, so I assume you added some rules based on this >>>>> pre-processor directive. >>>>> >>>>> For memory troubleshooting, set memlog to a value lower than debug >>>>> parameter in config file and try with mem_summary=12 for a more compact >>>>> output. See more about these parameters in the wiki: >>>>> >>>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog >>>>> >>>>> Run kamailio for a while in normal conditions, then restart it to >>>>> get the memory usage summaries. There should be indication if there is some >>>>> leak, by seeing memory chunks allocated many times from a function used at >>>>> runtime. You can send the memory summary for a process here, we can look at >>>>> it. >>>>> >>>>> >>>>> >>>>>> While this was running and Kamailio didn't respond to packets, it >>>>>> logged lots of lines like this: >>>>>> >>>>> >>>>> Do you have syslog to be configured in asynchronous mode? See the >>>>> notes from: >>>>> >>>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog >>>>> >>>>> The memdbg is less than debug value, that means printing few log >>>>> messages for each memory operation. You can make memdbg higher and rely on >>>>> memlog for memory summaries, otherwise will be lot of log messages related >>>>> to memory. >>>>> >>>>> >>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 >>>>>> (size=128) on 1 -th hit >>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 >>>>>> (size=128) on 1 -th hit >>>>>> >>>>> addstr() is a function used only for parsing configuration file, >>>>> as long as you can still see them, the configuration file parsing was not >>>>> finish. addstr() is not a source of leaks because it is not used at runtime. >>>>> >>>>> If you have large config file, then you can get close to the >>>>> limits of the private memory, which is set to 4MB. You can increase its >>>>> value using -M parameter (e.g., start kamailio with -M 8 to set it to use >>>>> 8MB of memory). >>>>> >>>>> Over the time, the private memory can get used due to >>>>> fragmentation, you can set the mem_join parameter in config file to avoid >>>>> it (works when compiled with MEMDBG=1). >>>>> >>>>> To monitor usage of internal pkg memory, then you can use sercmd >>>>> with pkg.stats command: >>>>> >>>>> >>>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640 >>>>> >>>>> Shared memory stats are printed by 'kamctl fifo get_statistics >>>>> shmem:' >>>>> >>>>> When you see significant increase of the memory usage, then you >>>>> can restart to get the summaries. >>>>> >>>>> You should run these commands after start, just to see the initial >>>>> usage of memory. >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> -- >>>>> Daniel-Constantin Mierla - http://www.asipto.com >>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>>> >>>>> >>>>> _______________________________________________ >>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing >>>>> list >>>>> sr-users(a)lists.sip-router.org >>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>> >>>> >>>> >>>> -- >>>> David Cunningham, Voisonics >>>> http://voisonics.com/ >>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092> >>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> >>>> Australia: +61 (0) 2 8063 9019 >>>> >>>> >>>> -- >>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> >>>> >>> >>> >>> -- >>> David Cunningham, Voisonics >>> http://voisonics.com/ >>> USA: +1 213 221 1092 <%2B1%20213%20221%201092> >>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> >>> Australia: +61 (0) 2 8063 9019 >>> >>> >>> -- >>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> >>> >> >> >> -- >> David Cunningham, Voisonics >> http://voisonics.com/ >> USA: +1 213 221 1092 <%2B1%20213%20221%201092> >> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> >> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019> >> >> >> -- >> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> >> > > > -- > David Cunningham, Voisonics > http://voisonics.com/ > USA: +1 213 221 1092 <%2B1%20213%20221%201092> > UK: +44 (0) 20 3298 1642 > Australia: +61 (0) 2 8063 9019 > -- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 <%2B1%20213%20221%201092> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019> -- Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda