Re: [SR-Users] OOM Errors

Having adding some graphing, I see this memory issue on another production Kamailio server. A few of the processes are using significantly more pkg memory than others. What I assume is happening is that memory is leaking and will eventually start hitting OOM errors. See https://gist.github.com/phil-lavin/27f63b472ace53dc956125a47d210bc9 for pkg memory stats output. When was corex.pkg_summary added? I’m getting not found error: error: 500 - command corex.pkg_summary not found *From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Phil Lavin *Sent:* 19 September 2016 14:36 *To:* miconda(a)gmail.com; Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.sip-router.org&gt; *Subject:* Re: [SR-Users] OOM Errors Thanks, Daniel. Kamailio has been restarted now so there’s not much of any use in the pkg memory output. I’m going to add some monitoring/graphing on this and see if there’s an upward trend in usage. *From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Daniel-Constantin Mierla *Sent:* 19 September 2016 13:41 *To:* Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>> *Subject:* Re: [SR-Users] OOM Errors Hello, it seems that the pkg is out of free space, not shm. You can check it with: kamcmd pkg.stats and look for the same pid that printed the log messages. To troubleshoot, it is good to print the pkg summary, see: - https://www.kamailio.org/docs/modules/stable/modules/corex.html#corex.rpc.p… Then get the log messages from syslog printed by the command and send them over here. Cheers, Daniel On 19/09/16 14:02, Phil Lavin wrote: Hi all, Kamailio stopped performing some functions earlier today, citing OOM errors. An excerpt from the log is below. SHM usage, at the time, looked fine. Also see below for that output. A restart of Kamailio resolved the issue. Is this SHM related or OS memory manager related? root@ua-proxy-01:~# kamctl stats shmem shmem:fragments = 910 shmem:free_size = 104131224 shmem:max_used_size = 51590800 shmem:real_used_size = 30086504 shmem:total_size = 134217728 shmem:used_size = 19558800 Sep 19 11:35:56 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory Sep 19 11:40:10 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 423 Sep 19 11:40:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:2390]: build_res_buf_from_sip_req(): out of memory; needs 453 Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory Sep 19 11:42:46 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory Sep 19 11:42:57 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:361]: anchor_lump(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:306]: del_lump(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory Sep 19 11:43:12 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [msg_translator.c:326]: received_builder(): out of memory Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:136]: insert_new_lump_after(): out of memory Sep 19 11:43:16 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory Sep 19 11:44:18 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory Sep 19 11:45:07 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [ut.c:163]: as_asciiz(): Out of memory Sep 19 11:45:30 ua-proxy-01 /sbin/kamailio[20218]: ERROR: acc [acc_extra.c:267]: extra2strar(): extra2strar: out of memory. Sep 19 11:45:43 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:186]: insert_subst_lump_after(): out of memory Sep 19 11:45:52 ua-proxy-01 /sbin/kamailio[20218]: ERROR: <core> [data_lump.c:161]: insert_new_lump_before(): out of memory _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Kamailio Advanced Training, Berlin, Oct 24-26, 2016 - http://www.asipto.com