Hello,

 

it looks like that you are observing a memory leak. You can look into it e.g. with memory debugging and try to debug it:

https://www.kamailio.org/wiki/tutorials/troubleshooting/memory

 

Generally, I would suggest decreasing the PGK memory, 256 MB is way too much, e.g, to 16-32 MB and increase the shared memory e.g. to 1GB.

If there is a large memory leak, the memory will be used anyway so it makes no sense to increase it too much.

 

Cheers,

 

Henning

 

--

Henning Westerholt – https://skalatan.de/blog/

Kamailio services – https://gilawa.com

 

From: paresh-panda via sr-dev <sr-dev@lists.kamailio.org>
Sent: Montag, 29. April 2024 14:06
To: Kamailio Devel List <sr-dev@lists.kamailio.org>
Cc: Subscribed <subscribed@noreply.github.com>; paresh-panda <notifications@github.com>
Subject: [sr-dev] [kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833)

 

Hi,
I am using Kamailio 5.8.1 ( ubuntu 22.04 Jammy 64 Bit),. Below are the observations.

I am running a load of 500 3GPP IMS SMS which kamailio decodes and sends to another application over Rabbit MQ (Kazoo Module).
-m 512 -M 256 , using 512 MB Shared memory and 256MB Package Memory.

When the load for 20 mins the available Memory drops from 80% to 25%

And I see below error logs

Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/pvapi.c:1136]: pv_parse_format(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1318]: fix_param(): bad PVE format: "PRD-IR92/11 term-Motorola/XT2205-1-T2STS33.105-51-11-1-2-2 device-type/smart-phone"
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:539
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:540
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:541
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:542
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:543
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request!
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:555
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:556
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1171]: do_action(): failed to convert RVE to string
Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401@192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:559

I am attaching the configuration for reference.

Help on this is much appreciated
configfiles.tar.gz
, I am looking into this issue for quite a long time.

Thank you!
Regards Paresh

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Image removed by sender.Message ID: <kamailio/kamailio/issues/3833@github.com>