Having adding some graphing, I see this memory issue on another production Kamailio server. A few of the processes are using significantly more pkg memory than others. What I assume is happening is that memory is leaking and will eventually start hitting OOM errors. See https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9 for pkg memory stats output.

When was corex.pkg_summary added? I’m getting not found error:

error: 500 - command corex.pkg_summary not found

From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Phil Lavin
Sent: 19 September 2016 14:36
To: miconda@gmail.com; Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org>
Subject: Re: [SR-Users] OOM Errors

Thanks, Daniel.

Kamailio has been restarted now so there’s not much of any use in the pkg memory output.

I’m going to add some monitoring/graphing on this and see if there’s an upward trend in usage.

From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla
Sent: 19 September 2016 13:41
To: Kamailio (SER) - Users Mailing List <sr-users@lists.sip-router.org>
Subject: Re: [SR-Users] OOM Errors

Hello,

it seems that the pkg is out of free space, not shm. You can check it with:

kamcmd pkg.stats

and look for the same pid that printed the log messages.

To troubleshoot, it is good to print the pkg summary, see:

- https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.pkg_summary

Then get the log messages from syslog printed by the command and send them over here.

Cheers,
Daniel

On 19/09/16 14:02, Phil Lavin wrote:

Hi all,

Kamailio stopped performing some functions earlier today, citing OOM errors. An excerpt from the log is below.

SHM usage, at the time, looked fine. Also see below for that output.

A restart of Kamailio resolved the issue. Is this SHM related or OS memory manager related?

root@ua-proxy-01:~# kamctl stats shmem

shmem:fragments = 910

shmem:free_size = 104131224

shmem:max_used_size = 51590800

shmem:real_used_size = 30086504

shmem:total_size = 134217728

shmem:used_size = 19558800

Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory

Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 423

Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 453

Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory

Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory

Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory

Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory

Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory

Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory

Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory

Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory

Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]: extra2strar(): extra2strar: out of memory.

Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:186]: insert_subst_lump_after(): out of memory

Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

--

Daniel-Constantin Mierla

http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda

Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com