**Description**
facing crashes the server, raised by qm_debug_check_frag().
**Troubleshooting**
The error message:
Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdc31f360, fd 31166 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbcbf232a8, fd 31326 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbd990b9e8, fd 31327 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbd89cd1d8, fd 31360 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbcf0bab28, fd 31434 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbc6e031b0, fd 31471 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdb7fd598, fd 31472 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdeb334c8, fd 31494 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbddee6de8, fd 31524 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbdb91ef70, fd 31576 Jan 23 12:27:26 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: ERROR: <core> [core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7fdbc76a1da8, fd 31690 Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31327, not in [0, 2054) Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31360, not in [0, 2054) Jan 23 12:27:31 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2893]: CRITICAL: <core> [core/io_wait.h:596]: io_watch_del(): invalid fd 31434, not in [0, 2054) Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:777]: handle_sigs(): child process 2858 exited by a signal 6 Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:780]: handle_sigs(): core was generated Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: CRITICAL: <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdbd7e8a0e8 (address 0x7fdbd7e8a120) end overwritten (5fd4cd2e, abcdefed)! Memory allocator was called from core: core/usr_avp.c:626. Fragment marked by core: core/usr_avp.c:175. Exec from core/mem/q_malloc.c:511.
The output from GDB:
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/fep-kamailio/sbin/kamailio...done. [New LWP 2809] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/fep-kamailio/sbin/kamailio -f /usr/local/fep-kamailio/etc/kamailio/k'. Program terminated with signal 6, Aborted. #0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-21.el7.x86_64 glibc-2.17-222.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-19.el7.x86_64 libcom_err-1.42.9-12.el7_5.x86_64 libcurl-7.29.0-54.el7.x86_64 libgcc-4.8.5-36.el7_6.1.x86_64 libidn-1.28-4.el7.x86_64 libselinux-2.5-12.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-36.el7_6.1.x86_64 libunistring-0.9.3-9.el7.x86_64 nspr-4.21.0-1.el7.x86_64 nss-3.44.0-4.el7.x86_64 nss-softokn-freebl-3.44.0-5.el7.x86_64 nss-util-3.44.0-3.el7.x86_64 openldap-2.4.44-5.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6 #1 0x00007fde7acc0968 in abort () from /lib64/libc.so.6 #2 0x00000000006d7c95 in qm_debug_check_frag (qm=0x7fdbc25ae000, f=0x7fdbd7e8a0e8, file=0x817155 "core: core/usr_avp.c", line=626, efile=0x83f7c5 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:140 #3 0x00000000006db99e in qm_free (qmp=0x7fdbc25ae000, p=0x7fdbd7e8a120, file=0x817155 "core: core/usr_avp.c", func=0x8189a0 <__FUNCTION__.8526> "destroy_avp_list_unsafe", line=626, mname=0x817150 "core") at core/mem/q_malloc.c:511 #4 0x00000000005e7ede in destroy_avp_list_unsafe (list=0x7fdbd7e8a950) at core/usr_avp.c:626 #5 0x00007fde38d4bd8f in free_cell_helper (dead_cell=0x7fdbd7e8a750, silent=1, fname=0x7fde38e4267f "h_table.c", fline=466) at h_table.c:255 #6 0x00007fde38d4c9e1 in free_hash_table () at h_table.c:466 #7 0x00007fde38df9cd0 in tm_shutdown () at t_funcs.c:88 #8 0x0000000000581960 in destroy_modules () at core/sr_module.c:746 #9 0x000000000041cda7 in cleanup (show_status=1) at main.c:563 #10 0x000000000041e682 in shutdown_children (sig=15, show_status=1) at main.c:706 #11 0x0000000000421715 in handle_sigs () at main.c:806 #12 0x000000000042b9a7 in main_loop () at main.c:1817 #13 0x0000000000433b96 in main (argc=9, argv=0x7ffea05cd2c8) at main.c:2856 (gdb) bt full #0 0x00007fde7acbf277 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007fde7acc0968 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00000000006d7c95 in qm_debug_check_frag (qm=0x7fdbc25ae000, f=0x7fdbd7e8a0e8, file=0x817155 "core: core/usr_avp.c", line=626, efile=0x83f7c5 "core/mem/q_malloc.c", eline=511) at core/mem/q_malloc.c:140 p = 0x7ffea05cc3b0 __FUNCTION__ = "qm_debug_check_frag" #3 0x00000000006db99e in qm_free (qmp=0x7fdbc25ae000, p=0x7fdbd7e8a120, file=0x817155 "core: core/usr_avp.c", func=0x8189a0 <__FUNCTION__.8526> "destroy_avp_list_unsafe", line=626, mname=0x817150 "core") at core/mem/q_malloc.c:511 qm = 0x7fdbc25ae000 f = 0x7fdbd7e8a0e8 size = 408 next = 0x7fdbd7e8a5c8 prev = 0x7fdbd7e8a1d8 __FUNCTION__ = "qm_free" #4 0x00000000005e7ede in destroy_avp_list_unsafe (list=0x7fdbd7e8a950) at core/usr_avp.c:626 avp = 0x7fdbd7e8a080 foo = 0x7fdbd7e8a120 __FUNCTION__ = "destroy_avp_list_unsafe" #5 0x00007fde38d4bd8f in free_cell_helper (dead_cell=0x7fdbd7e8a750, silent=1, fname=0x7fde38e4267f "h_table.c", fline=466) at h_table.c:255 b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x7fde34a5df80 <__FUNCTION__.7259> cbs = 0x0 cbs_tmp = 0x7fde388fe570 __FUNCTION__ = "free_cell_helper" #6 0x00007fde38d4c9e1 in free_hash_table () at h_table.c:466 p_cell = 0x7fdbd7e8a750 tmp_cell = 0x7fdbc27e3060 i = 40845 __FUNCTION__ = "free_hash_table" #7 0x00007fde38df9cd0 in tm_shutdown () at t_funcs.c:88 __FUNCTION__ = "tm_shutdown" #8 0x0000000000581960 in destroy_modules () at core/sr_module.c:746 t = 0x7fde3c51e4c0 foo = 0x7fde3c51e020 __FUNCTION__ = "destroy_modules" #9 0x000000000041cda7 in cleanup (show_status=1) at main.c:563 memlog = 0 __FUNCTION__ = "cleanup" #10 0x000000000041e682 in shutdown_children (sig=15, show_status=1) at main.c:706 __FUNCTION__ = "shutdown_children" #11 0x0000000000421715 in handle_sigs () at main.c:806 chld = 0 chld_status = 134 any_chld_stopped = 1 memlog = -1027733624 __FUNCTION__ = "handle_sigs" #12 0x000000000042b9a7 in main_loop () at main.c:1817 i = 10 pid = 2893 si = 0x0 si_desc = "udp receiver child=9 sock=10.50.7.18:5060\000:0:0:1]:5060\000:5060)\000\000\000\004\000\000\000\000\000\000\000\000\340Z\302\333\177\000\000\000\000\000\000\000\000\000\000 \006\276\302\333\177\000\000`\315\\240\376\177\000\000\220\006\276\302\333\177\000\000r\fb3\336\177\000\000P\246\222<\336\177\000" nrprocs = 10 woneinit = 1 __FUNCTION__ = "main_loop" #13 0x0000000000433b96 in main (argc=9, argv=0x7ffea05cd2c8) at main.c:2856
**Additional Information**
Kamailio Version - output of Kamailio -v
version: kamailio 5.4.2 (x86_64/linux) f8885c flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: f8885c compiled on 04:30:53 Dec 16 2020 with gcc 4.8.5
**Operating System:**
NAME="CentOS Linux" VERSION="7 (Core)" ID="centos"
**Note:** It looks similar to the following issue
https://github.com/kamailio/kamailio/issues/2503
The backtrace seems to be from the core file generated on shutdown. Likely the core file generated for the runtime issue was overwritten.
Look in the logs and see if you can spot logs printed by PID `2858`, that is the process that crashed.
Also, enable core file per process/pid, see:
* https://www.kamailio.org/wiki/tutorials/troubleshooting/coredumpfile#how_do_...
Then try to reproduce and you should eventually get more than one core file. Paste here the gdb `bt full` for each of the core files.
here is the log for PID 2858 Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:777]: handle_sigs(): child process 2858 exited by a signal 6 Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: ALERT: <core> [main.c:780]: handle_sigs(): core was generated Jan 23 12:28:09 fep-1 /usr/local/fep-kamailio/sbin/kamailio[2809]: CRITICAL: <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fdbd7e8a0e8 (address 0x7fdbd7e8a120) end overwritten (5fd4cd2e, abcdefed)! Memory allocator was called from core: core/usr_avp.c:626. Fragment marked by core: core/usr_avp.c:175. Exec from core/mem/q_malloc.c:511.
The pid is after the application path/name in log messages, so those logs are printed by pid 2809 (which should be the kamailio main process). The logs printed by pid 2858 should contain:
/usr/local/fep-kamailio/sbin/kamailio[2858]
No further troubleshooting data was provided with core files per pid for very long time. Reopen when able to attach data as requested above.
Closed #2607.
-
aatif218 -
Daniel-Constantin Mierla