[sr-dev] Re: [kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833)

Hello, it looks like that you are observing a memory leak. You can look into it e.g. with memory debugging and try to debug it: https://www.kamailio.org/wiki/tutorials/troubleshooting/memory Generally, I would suggest decreasing the PGK memory, 256 MB is way too much, e.g, to 16-32 MB and increase the shared memory e.g. to 1GB. If there is a large memory leak, the memory will be used anyway so it makes no sense to increase it too much. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com<https://gilawa.com/> From: paresh-panda via sr-dev &lt;sr-dev(a)lists.kamailio.org&gt; Sent: Montag, 29. April 2024 14:06 To: Kamailio Devel List &lt;sr-dev(a)lists.kamailio.org&gt; Cc: Subscribed &lt;subscribed(a)noreply.github.com&gt;om>; paresh-panda &lt;notifications(a)github.com&gt; Subject: [sr-dev] [kamailio/kamailio] Mem leak in smsops which leads to the udp receiver process leak memory continuously (Issue #3833) Hi, I am using Kamailio 5.8.1 ( ubuntu 22.04 Jammy 64 Bit),. Below are the observations. I am running a load of 500 3GPP IMS SMS which kamailio decodes and sends to another application over Rabbit MQ (Kazoo Module). -m 512 -M 256 , using 512 MB Shared memory and 256MB Package Memory. When the load for 20 mins the available Memory drops from 80% to 25% And I see below error logs Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mem/q_malloc.c:324]: qm_find_free(): qm_find_free(0x7ff912e21010, 192); Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mem/q_malloc.c:463]: qm_malloc(): qm_malloc(0x7ff912e21010, 192) called from smsops: smsops_impl.c: decode_3gpp_sms(813), module: smsops; Free fragment not found! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/pvapi.c:1136]: pv_parse_format(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/sr_module.c:1318]: fix_param(): bad PVE format: "PRD-IR92/11 term-Motorola/XT2205-1-T2STS33.105-51-11-1-2-2 device-type/smart-phone" Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:539 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:540 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/sr_module.c:1208]: fix_param(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/mod_fix.c:601]: fixup_spve_null(): Cannot convert function parameter 1 to spve Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): runtime fixup failed for jansson_set param 3 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:541 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:542 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:543 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:815]: decode_3gpp_sms(): Error allocating 192 bytes! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} smsops [smsops_impl.c:1320]: pv_get_sms(): Error getting/decoding RP-Data from request! Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:555 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1178]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:556 Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/rvalue.c:1321]: rval_get_str(): could not allocate private memory from pkg pool Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1171]: do_action(): failed to convert RVE to string Apr 29 13:28:53 stgn-haud-vm-ub-kamailio01 /usr/sbin/kamailio[5520]: ERROR: {1 1 MESSAGE 739708-3401(a)192.168.136.152} [core/action.c:1566]: do_action(): run action error at: /etc/kamailio/kamailio.cfg:559 I am attaching the configuration for reference. Help on this is much appreciated configfiles.tar.gz<https://github.com/kamailio/kamailio/files/15148792/c… , I am looking into this issue for quite a long time. Thank you! Regards Paresh — Reply to this email directly, view it on GitHub<https://github.com/kamailio/kamailio/issues/3833>33>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABO7UZJMTN…MA>. You are receiving this because you are subscribed to this thread.Message ID: &lt;kamailio/kamailio/issues/3833(a)github.com&gt;