[sr-dev] [kamailio/kamailio] Kamailio 5.4.2 - CentOS 7 - qm_free(): BUG: freeing already freed pointer, called from core: core/usr_avp.c: destroy_avp_list_unsafe (#2620)

Hi, Same issue with @farnk05 on https://github.com/kamailio/kamailio/issues/2224 Wanted to open a fresh issue to not reopen an old one. This is on kamailio 5.4.2, which appears to have these fixes from https://github.com/kamailio/kamailio/issues/2224#issuecomment-602730307 in them when I checked the src files. ``` $ sudo rpm -qi kamailio |grep Date Install Date: Fri 20 Nov 2020 12:03:26 AM PST Build Date : Tue 27 Oct 2020 05:37:31 AM PDT ``` ``` $ sudo rpm -qa | grep kamailio kamailio-mysql-5.4.2-0.el7.x86_64 kamailio-tls-5.4.2-0.el7.x86_64 kamailio-5.4.2-0.el7.x86_64 kamailio-websocket-5.4.2-0.el7.x86_64 kamailio-tcpops-5.4.2-0.el7.x86_64 kamailio-statsd-5.4.2-0.el7.x86_64 ``` Package Source: https://rpm.kamailio.org/centos/7/5.4/5.4.2/x86_64/ OS: RHEL 7.6.1810 Mem mgr: default/qm SHM is 4096, PKG is 1024 (system has 16gb ram, 4 core Intel Skylake CPU on a KVM. There are a few variations we see with the qm_free errors, here are the most common we see when kamailio segfaults, and we have to let systemd restart it, or Monit as we now have to have Monit check for CRITICAL messages since kamailio segfaults so often. ``` /usr/sbin/kamailio[32734]: CRITICAL: <core> [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer (0x7f0da5012fc0), called from core: core/usr_avp.c: destroy_avp_list_unsafe(626), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring /usr/sbin/kamailio[32734]: CRITICAL: <core> [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer (0x7f0da5012fc0), called from core: core/usr_avp.c: destroy_avp_list_unsafe(626), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring ... ``` ``` /usr/sbin/kamailio[32733]: CRITICAL: dialog [dlg_profile.c:574]: set_dlg_profile(): BUG - dialog not found in a non REQUEST route (1) /usr/sbin/kamailio[32733]: CRITICAL: dialog [dlg_profile.c:574]: set_dlg_profile(): BUG - dialog not found in a non REQUEST route (1) /usr/sbin/kamailio[6160]: CRITICAL: <core> [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer (0x7ff00842e320), called from tm: h_table.c: free_cell_helper(189), first free core: core/usr_avp.c: destroy_avp_list_unsafe(626) - ignoring ``` GDB as requested in the other Issue (note gdb was run on another VM, not the main system, hopefully that is not an issue: ``` $ gdb /usr/sbin/kamailio /core-kamailio-11-995-992-11912-1612143069 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/usr/sbin/kamailio.debug...done. done. [New LWP 11912] warning: .dynamic section for "/lib64/libc.so.6" is not at the expected address (wrong library or version mismatch?) [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by '/usr/sbin/kamailio -DD -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamai'. Program terminated with signal 11, Segmentation fault. #0 0x00000000006024be in destroy_avp_list_unsafe (list=0x7fdc559d4fd8) at core/usr_avp.c:625 625 avp = avp->next; Missing separate debuginfos, use: debuginfo-install glibc-2.17-260.el7_6.6.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_6.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-36.el7_6.2.x86_64 libmaxminddb-1.2.0-1.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libstdc++-4.8.5-36.el7_6.2.x86_64 libunistring-0.9.3-9.el7.x86_64 mariadb-libs-5.5.60-1.el7_5.x86_64 openssl-libs-1.0.2k-16.el7_6.1.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) frame 0 #0 0x00000000006024be in destroy_avp_list_unsafe (list=0x7fdc559d4fd8) at core/usr_avp.c:625 625 avp = avp->next; (gdb) list 620 avp_t *avp, *foo; 621 622 avp = *list; 623 while( avp ) { 624 foo = avp; 625 avp = avp->next; 626 shm_free_unsafe( foo ); 627 } 628 *list = 0; 629 } (gdb) p *p_entry No symbol "p_entry" in current context. (gdb) p *l No symbol "l" in current context. (gdb) No symbol "l" in current context. (gdb) p *lh No symbol "lh" in current context. (gdb) p *lh No symbol "lh" in current context. (gdb) frame 1 #1 0x00007fdd56fb8f92 in free_cell_helper (dead_cell=0x7fdc559d4dd8, silent=0, fname=0x7fdd570d1363 "timer.c", fline=643) at h_table.c:255 255 destroy_avp_list_unsafe(&dead_cell->uri_avps_from); (gdb) #1 0x00007fdd56fb8f92 in free_cell_helper (dead_cell=0x7fdc559d4dd8, silent=0, fname=0x7fdd570d1363 "timer.c", fline=643) at h_table.c:255 255 destroy_avp_list_unsafe(&dead_cell->uri_avps_from); (gdb) list 250 if(dead_cell->user_avps_from) 251 destroy_avp_list_unsafe(&dead_cell->user_avps_from); 252 if(dead_cell->user_avps_to) 253 destroy_avp_list_unsafe(&dead_cell->user_avps_to); 254 if(dead_cell->uri_avps_from) 255 destroy_avp_list_unsafe(&dead_cell->uri_avps_from); 256 if(dead_cell->uri_avps_to) 257 destroy_avp_list_unsafe(&dead_cell->uri_avps_to); 258 if(dead_cell->xavps_list) 259 xavp_destroy_list_unsafe(&dead_cell->xavps_list); (gdb) p *dlg No symbol "dlg" in current context. (gdb) p *msg No symbol "msg" in current context. (gdb) ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/2620