19 Sep
2016
19 Sep
'16
9:20 p.m.
Nevermind – I found this in the docs.
kamcmd cfg.set_now_int core mem_dump_pkg 17645
kamcmd cfg.set_now_int core mem_dump_pkg 1
There were 27526 allocations. I have taken and parsed the output, summing the memory
usage, grouping it by file/function. Parsed output is at
https://gist.github.com/phil-lavin/d2607a4eed36065e01ed42c43a0d27ad
Do any of those stick out to you as being far higher than they should be? My feeling is
that it’s related to the db_* stuff.
Cheers
From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Phil Lavin
Sent: 19 September 2016 18:15
To: miconda(a)gmail.com; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.sip-router.org>
Subject: Re: [SR-Users] OOM Errors
4.3
From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: 19 September 2016 17:47
To: Phil Lavin <phil.lavin@cloudcall.com<mailto:phil.lavin@cloudcall.com>>;
Kamailio (SER) - Users Mailing List
<sr-users@lists.sip-router.org<mailto:sr-users@lists.sip-router.org>>
Subject: Re: [SR-Users] OOM Errors
The rpc command is in 4.4, what version are you using?
Cheers,
Daniel
On 19/09/16 18:11, Phil Lavin wrote:
Having adding some graphing, I see this memory issue on another production Kamailio
server. A few of the processes are using significantly more pkg memory than others. What I
assume is happening is that memory is leaking and will eventually start hitting OOM
errors. See https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9 for pkg
memory stats output.
When was corex.pkg_summary added? I’m getting not found error:
error: 500 - command corex.pkg_summary not found
From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Phil Lavin
Sent: 19 September 2016 14:36
To: miconda@gmail.com<mailto:miconda@gmail.com>; Kamailio (SER) - Users Mailing List
<sr-users@lists.sip-router.org><mailto:sr-users@lists.sip-router.org>
Subject: Re: [SR-Users] OOM Errors
Thanks, Daniel.
Kamailio has been restarted now so there’s not much of any use in the pkg memory output.
I’m going to add some monitoring/graphing on this and see if there’s an upward trend in
usage.
From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of
Daniel-Constantin Mierla
Sent: 19 September 2016 13:41
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.sip-router.org<mailto:sr-users@lists.sip-router.org>>
Subject: Re: [SR-Users] OOM Errors
Hello,
it seems that the pkg is out of free space, not shm. You can check it with:
kamcmd pkg.stats
and look for the same pid that printed the log messages.
To troubleshoot, it is good to print the pkg summary, see:
-
https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.p…
Then get the log messages from syslog printed by the command and send them over here.
Cheers,
Daniel
On 19/09/16 14:02, Phil Lavin wrote:
Hi all,
Kamailio stopped performing some functions earlier today, citing OOM errors. An excerpt
from the log is below.
SHM usage, at the time, looked fine. Also see below for that output.
A restart of Kamailio resolved the issue. Is this SHM related or OS memory manager
related?
root@ua-proxy-01:~# kamctl stats shmem
shmem:fragments = 910
shmem:free_size = 104131224
shmem:max_used_size = 51590800
shmem:real_used_size = 30086504
shmem:total_size = 134217728
shmem:used_size = 19558800
Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]:
insert_new_lump_before(): out of memory
Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 423
Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 453
Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]:
insert_new_lump_after(): out of memory
Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory
Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]:
anchor_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]:
anchor_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]:
del_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]:
del_lump(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core>
[msg_translator.c:326]: received_builder(): out of memory
Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]:
insert_new_lump_after(): out of memory
Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory
Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]:
insert_new_lump_before(): out of memory
Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]:
as_asciiz(): Out of memory
Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]:
extra2strar(): extra2strar: out of memory.
Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:186]:
insert_subst_lump_after(): out of memory
Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]:
insert_new_lump_before(): out of memory
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com