19 Sep
2016
19 Sep
'16
10:04 p.m.
Hello,
yes, the db related allocated chunks look to be quite high in number.
That can be possible if there are db results with lots of records, but
in this case there are more related to column structure than row structure.
Tracing back in the history of commits, I noticed a place where db
result is not freed if no row is returned by query, in the presence
module, patch done by you. It was not backported to 4.3, iirc, but if
you did it in your branch, try with the latest patch pushed few minutes
ago in master and 4.4.
Cheers,
Daniel
On 19/09/16 20:20, Phil Lavin wrote:
Nevermind – I found this in the docs.
kamcmd cfg.set_now_int core mem_dump_pkg 17645
kamcmd cfg.set_now_int core mem_dump_pkg 1
There were 27526 allocations. I have taken and parsed the output,
summing the memory usage, grouping it by file/function. Parsed output
is at https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad
Do any of those stick out to you as being far higher than they should
be? My feeling is that it’s related to the db_* stuff.
Cheers
*From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On
Behalf Of *Phil Lavin
*Sent:* 19 September 2016 18:15
*To:* miconda(a)gmail.com; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.sip-router.org>
*Subject:* Re: [SR-Users] OOM Errors
4.3
*From:*Daniel-Constantin Mierla [mailto:miconda@gmail.com]
*Sent:* 19 September 2016 17:47
*To:* Phil Lavin <phil.lavin(a)cloudcall.com
<mailto:phil.lavin@cloudcall.com>>; Kamailio (SER) - Users Mailing
List <sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>>
*Subject:* Re: [SR-Users] OOM Errors
The rpc command is in 4.4, what version are you using?
Cheers,
Daniel
On 19/09/16 18:11, Phil Lavin wrote:
Having adding some graphing, I see this memory issue on another
production Kamailio server. A few of the processes are using
significantly more pkg memory than others. What I assume is
happening is that memory is leaking and will eventually start
hitting OOM errors. See
https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9
for pkg memory stats output.
When was corex.pkg_summary added? I’m getting not found error:
error: 500 - command corex.pkg_summary not found
*From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On
Behalf Of *Phil Lavin
*Sent:* 19 September 2016 14:36
*To:* miconda(a)gmail.com <mailto:miconda@gmail.com>; Kamailio (SER)
- Users Mailing List <sr-users(a)lists.sip-router.org>
<mailto:sr-users@lists.sip-router.org>
*Subject:* Re: [SR-Users] OOM Errors
Thanks, Daniel.
Kamailio has been restarted now so there’s not much of any use in
the pkg memory output.
I’m going to add some monitoring/graphing on this and see if
there’s an upward trend in usage.
*From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On
Behalf Of *Daniel-Constantin Mierla
*Sent:* 19 September 2016 13:41
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>>
*Subject:* Re: [SR-Users] OOM Errors
Hello,
it seems that the pkg is out of free space, not shm. You can check
it with:
kamcmd pkg.stats
and look for the same pid that printed the log messages.
To troubleshoot, it is good to print the pkg summary, see:
-
https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.p…
Then get the log messages from syslog printed by the command and
send them over here.
Cheers,
Daniel
On 19/09/16 14:02, Phil Lavin wrote:
Hi all,
Kamailio stopped performing some functions earlier today,
citing OOM errors. An excerpt from the log is below.
SHM usage, at the time, looked fine. Also see below for that
output.
A restart of Kamailio resolved the issue. Is this SHM related
or OS memory manager related?
root@ua-proxy-01:~# kamctl stats shmem
shmem:fragments = 910
shmem:free_size = 104131224
shmem:max_used_size = 51590800
shmem:real_used_size = 30086504
shmem:total_size = 134217728
shmem:used_size = 19558800
Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:161]: insert_new_lump_before(): out of memory
Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [msg_translator.c:2390]: build_res_buf_from_sip_req():
out of memory; needs 423
Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [msg_translator.c:2390]: build_res_buf_from_sip_req():
out of memory; needs 453
Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:136]: insert_new_lump_after(): out of memory
Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [ut.c:163]: as_asciiz(): Out of memory
Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [ut.c:163]: as_asciiz(): Out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:361]: anchor_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:361]: anchor_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:306]: del_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:306]: del_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:136]: insert_new_lump_after(): out of memory
Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [ut.c:163]: as_asciiz(): Out of memory
Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:161]: insert_new_lump_before(): out of memory
Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [ut.c:163]: as_asciiz(): Out of memory
Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc
[acc_extra.c:267]: extra2strar(): extra2strar: out of memory.
Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:186]: insert_subst_lump_after(): out of memory
Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR:
<core> [data_lump.c:161]: insert_new_lump_before(): out of memory
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com