This particular system is a test system, so there’s not a TON of activity.  Though the system itself has maybe 8 GB of RAM.  So we’re talking a handful of calls throughout a given day.  The service appears to crash maybe once every couple of days.  I bumped the SHM_MEMORY up to 512 over the weekend and restarted to see what impact that has. 

Which specific command are you looking for the output of, I’m blanking on which to run.

If you have a lot of traffic and registered devices, you can run out of it quickly. What is the number of registered devices and call setups per second/active calls at peak?

When you do a restart, can you grab the stats and see what are the values for shared memory?

Cheers,
Daniel

# Amount of shared memory to allocate for the running Kamailio server (in Mb)

SHM_MEMORY=64

 

# Amount of private memory to allocate for the running Kamailio server (in Mb)

PKG_MEMORY=8

 

 

 

version: kamailio 5.0.2 (x86_64/linux)

flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES

ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB

poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

id: unknown

compiled on 11:34:41 Jun 28 2017 with gcc 4.8.5

 

From: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Sent: Friday, September 15, 2017 10:53 AM
To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>; Joe Baran <jbaran@whitelabelcomm.com>
Subject: Re: [SR-Users] Memory configuration

 

What is the value of shared memory you set for kamailio (-m parameter)? Maybe it is not sufficient for what you need to do.

Then what version of kamailio do you run (kamailio -v)?

The process that crashed (29438) has a different pid than the one (29468) printing the memory errors, so may or may not be related.

 

On 14.09.17 20:36, Joe Baran wrote:

I feel fairly confident I’m running into a memory tuning issue, but I’m struggling to determine the next step.  Can someone give me some guidance on how to adjust settings to allow the system to stop crashing kamailio?

 

Sep 14 18:25:15 server1 kamailio[29468]: ERROR: <core> [core/mem/q_malloc.c:292]: qm_find_free(): qm_find_free(0x7fa9eeeea000, 144); Free fragment not found!

Sep 14 18:25:15 server1 kamailio[29468]: ERROR: <core> [core/mem/q_malloc.c:425]: qm_malloc(): qm_malloc(0x7fa9eeeea000, 144) called from kazoo: kz_amqp.c: kz_amqp_async_query(1357), module: kazoo; Free fragment not found!

Sep 14 18:25:15 server1 kamailio[29468]: ERROR: kazoo [kz_amqp.c:1359]: kz_amqp_async_query(): failed to allocate kz_amqp_cmd in process 29468

Sep 14 18:25:15 server1 kamailio[29468]: INFO: <script>: kGddp3V0bA6nitnk25dB4w..|log|failed to send Kazoo query for authentication xxxxxxxxxxxxxxxxxx

Sep 14 18:25:15 server1 kamailio[29468]: ERROR: tm [t_reply.c:494]: _reply_light(): cannot allocate shmem buffer

Sep 14 18:25:15 server1 kamailio[29468]: ERROR: sl [sl.c:271]: send_reply(): failed to reply stateful (tm)

Sep 14 18:25:16 server1 kamailio[29483]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 108

Sep 14 18:25:16 server1 kamailio[29360]: ALERT: <core> [main.c:743]: handle_sigs(): child process 29438 exited by a signal 11

Sep 14 18:25:16 server1 kamailio[29360]: ALERT: <core> [main.c:746]: handle_sigs(): core was not generated

 

Thanks,

 

Joe Baran

_______________________________________________

Kamailio (SER) - Users Mailing List

sr-users@lists.kamailio.org

https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

 

-- 

Daniel-Constantin Mierla

www.twitter.com/miconda -- www.linkedin.com/in/miconda

Kamailio Advanced Training - www.asipto.com

Kamailio World Conference - www.kamailioworld.com

-- 

Daniel-Constantin Mierla

www.twitter.com/miconda -- www.linkedin.com/in/miconda

Kamailio Advanced Training - www.asipto.com

Kamailio World Conference - www.kamailioworld.com