Hello,

oh, 4.0 is not compiled with memory debugging by default, so it is not printing the details about allocation place. Can you recompile with MEMDBG=1? You can change that in Makefile.defs or give it to the command line when compiling with make.

Cheers,
Daniel

On 13/05/14 14:44, Igor Potjevlesch wrote:

Hello Daniel,


Thank you for back porting the patches. I will schedule an update. Maybe I will go in the latest 4.1.x.

Indeed, I have still the coredump. I tried the script you mentioned.

Is this result make sense for you?:

 

$1 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}

$2 = {size = 32, u = {nxt_free = 0x0, is_free = 0}}

$3 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}

$4 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}

$5 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}

$6 = {size = 256, u = {nxt_free = 0x0, is_free = 0}}

$7 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}

$8 = {size = 224, u = {nxt_free = 0x0, is_free = 0}}

[…]

$1289 = {size = 48, u = {nxt_free = 0x0, is_free = 0}}

$1290 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}

$1291 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}

$1292 = {size = 1728, u = {nxt_free = 0x0, is_free = 0}}

$1293 = {size = 128, u = {nxt_free = 0x0, is_free = 0}}

$1294 = {size = 848, u = {nxt_free = 0x0, is_free = 0}}

 

There are thousand and thousand lines.

 

Regards,

 

Igor.

 

De : Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Envoyé : lundi 12 mai 2014 20:26
À : Igor Potjevlesch; 'Kamailio (SER) - Users Mailing List'
Objet : Re: [SR-Users] Memory issue

 

Hello,

On 12/05/14 17:29, Igor Potjevlesch wrote:

Hello Daniel,

 

Thank you for this feedback.

We are running 4.0.4 without custom module.

I do a sqlops request in a very small table (40 rows) but on each REGISTER and INVITE.

The bigger table is “aliases” with 154k rows.

 

Can you highlight me the particular bug concerned and corrected?

 

In the meantime, for the memory allocation, I have to grow the shared and/or private memory? What would be the next increment : -M 128?

it doesn't really matter the increment. I think the private memory is already too big, so if you don't load big amount of data, there could be a leak.

I backported related patches to branch 4.0 -- you should upgrade to latest git branch 4.0. In this way you are sure you have all the fixes, because 4.0.4 is old in the 4.0 branch.

You can use 'kamcmd pkg.stats' to monitor available pkg memory. When you see the available memory keep decreasing, you can issue:

kam cmd cfg.set_now_int core mem_dump_pkg _PID_

replace PID with an appropriate value you take from pkg.stats. You can send the output from syslog here for investigation, to see what can be the potential leak.

If you still have the core file, you can look inside it to see allocated memory chunks --  see the gdb script at:

- http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:memory#walking_through_pkg_with_gdb

Cheers,
Daniel



 

Regards,

 

Igor.

 

De : sr-users-bounces@lists.sip-router.org [mailto:sr-users-bounces@lists.sip-router.org] De la part de Daniel-Constantin Mierla
Envoyé : lundi 12 mai 2014 16:43
À : Kamailio (SER) - Users Mailing List
Objet : Re: [SR-Users] Memory issue

 

Hello,

what version are you using? Are you doing sql queries that return large number of records (e.g., via sqlops)? Are you using any custom module you developed?

The crash was fixed in master branch, but somehow forgotten to be backported -- it is now in branch 4.1.

Cheers,
Daniel

On 12/05/14 15:58, Igor Potjevlesch wrote:

Hello,

 

I had a memory issue this morning.

First, I could see the following logs in /var/log/message :

May 12 11:08:21 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [rvalue.c:2370]: rval_expr_eval(): rv eval int expression: out of memory

May 12 11:08:25 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [msg_translator.c:2164]: build_res_buf_from_sip_req(): ERROR: build_res_buf_from_sip_req: out of memory  ; needs 482

May 12 11:08:26 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [msg_translator.c:2012]: build_res_buf_from_sip_res(): ERROR: build_res_buf_from_sip_res: out of mem

May 12 11:08:27 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [msg_translator.c:1910]: build_req_buf_from_sip_req(): ERROR: build_req_buf_from_sip_req: out of memory

May 12 11:08:28 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [rvalue.c:2370]: rval_expr_eval(): rv eval int expression: out of memory

May 12 11:08:31 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [msg_translator.c:1910]: build_req_buf_from_sip_req(): ERROR: build_req_buf_from_sip_req: out of memory

May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]: ERROR: <core> [db_res.c:181]: db_allocate_rows(): no private memory left

May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]: ERROR: db_mysql [km_res.c:175]: db_mysql_convert_rows(): could not allocate rows

May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]: ERROR: db_mysql [km_res.c:217]: db_mysql_convert_result(): error while converting rows

May 12 11:08:32 localhost /usr/local/sbin/kamailio[30268]: ERROR: db_mysql [km_dbase.c:191]: db_mysql_store_result(): error while converting result

May 12 11:08:33 localhost kernel: kamailio[30268]: segfault at 30 ip 00007ff44dd93f93 sp 00007fffcb00a200 error 4 in db_mysql.so[7ff44dd8f000+19000]

May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]: ALERT: <core> [main.c:788]: handle_sigs(): child process 30268 exited by a signal 11

May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]: ALERT: <core> [main.c:791]: handle_sigs(): core was generated

May 12 11:08:37 localhost /usr/local/sbin/kamailio[30235]: INFO: <core> [main.c:803]: handle_sigs(): INFO: terminating due to SIGCHLD

 

This problem have already occurred and I have set –M 64 to avoid it. It looks that it’s not enough anymore.

What is the next increment recommended after 64M?

 

On the contrary of the first occurrence, Kamailio crash with coredump just after these errors. I’m not sure that the memory message errors are linked to the crash. I have tried a “bt full”:

(gdb) bt full

#0  db_mysql_store_result (_h=0x7ff44e0e70e0, _r=0x7fffcb00a408) at km_dbase.c:198

        code = <value optimized out>

        __FUNCTION__ = "db_mysql_store_result"

#1  0x00007ff44d971e72 in db_do_query_internal (_h=0x7ff44e0e70e0, _k=0x7fffcb00a3e0, _op=0x0, _v=0x7fffcb00a3a0, _c=<value optimized out>,

    _n=<value optimized out>, _nc=2, _o=0x0, _r=0x7fffcb00a408, val2str=0x7ff44dd97fa0 <db_mysql_val2str>,

    submit_query=0x7ff44dd92ed0 <db_mysql_submit_query>, store_result=0x7ff44dd93d80 <db_mysql_store_result>, _l=0) at db_query.c:137

        tmp = <value optimized out>

        off = <value optimized out>

        ret = <value optimized out>

        __FUNCTION__ = "db_do_query_internal"

#2  0x00007ff44d97269a in db_do_query (_h=<value optimized out>, _k=<value optimized out>, _op=<value optimized out>,

    _v=<value optimized out>, _c=<value optimized out>, _n=<value optimized out>, _nc=2, _o=0x0, _r=0x7fffcb00a408,

    val2str=0x7ff44dd97fa0 <db_mysql_val2str>, submit_query=0x7ff44dd92ed0 <db_mysql_submit_query>,

    store_result=0x7ff44dd93d80 <db_mysql_store_result>) at db_query.c:156

 

Thanks in advance for your help.

 

Regards,

 

Igor.





_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users




-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda



-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

-- 
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda