I have 3 kamailio 434 registrars replicating between each other using dmq usrloc module
I have recently noticed that all of the registrars crash at random intervals over the course of the day, they all dont crash at the same time, but randomly, independantly of each other
I cannot see any unusual traffic arriving at the systems that may be a cause for the crash
I'd appreciate assistance trying to get to the bottom of the issue
Kamailio Version:
```
# /usr/sbin/kamailio -V
version: kamailio 434 (x86_64/linux) 0ec860
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select
id: 0ec860
compiled on 17:42:12 Nov 25 2015 with gcc 447
```
Message in the log file when the kamailio application crashes:
```
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:728]: handle_sigs(): child process 16234 exited by a signal 11
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:731]: handle_sigs(): core was generated
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [mainc:743]: handle_sigs(): terminating due to SIGCHLD
Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a084f8 is still linked in hash table (h_tablec:440)
Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a12390 is still linked in hash table (h_tablec:440)
```
Core file synopsis is:
```
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailiopid -m 1024 -M 64 -u root -g root'
Program terminated with signal 11, Segmentation fault
#0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140, _ci=0x7f5a81418500) at urecordc:152
152 if_update_stat( _r->slot, _r->slot->d->contacts, 1);
```
I still have the core file, however, it contains some sensitive info, so I can send backtraces directly to you if required
Please let me know if you require any additional info
Thanks
---
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/440
Module: kamailio
Branch: master
Commit: a372f861f04b04f08d68f28a95f12a13776be891
URL: https://github.com/kamailio/kamailio/commit/a372f861f04b04f08d68f28a95f12a1…
Author: Stefan Mititelu <stefan.mititelu(a)1and1.ro>
Committer: Stefan Mititelu <stefan.mititelu(a)1and1.ro>
Date: 2016-01-07T11:11:35+02:00
debugger: doku dbg.get_mod_*
Overlooked due to copy/paste.
---
Modified: modules/debugger/doc/debugger_admin.xml
---
Diff: https://github.com/kamailio/kamailio/commit/a372f861f04b04f08d68f28a95f12a1…
Patch: https://github.com/kamailio/kamailio/commit/a372f861f04b04f08d68f28a95f12a1…
---
diff --git a/modules/debugger/doc/debugger_admin.xml b/modules/debugger/doc/debugger_admin.xml
index 04ba4ee..d0f3511 100644
--- a/modules/debugger/doc/debugger_admin.xml
+++ b/modules/debugger/doc/debugger_admin.xml
@@ -918,7 +918,7 @@ $ &ctltool; fifo get_dbg_mod_facility debugger
</title>
<para>
Get the module log level.
- If module does not exist in kamailio, the entry in the level hashtable is still added for the bogus module.
+ If mod_name does not exist in the level hashtable, returns the config file value.
</para>
<para>
Name: <emphasis>dbg.get_mod_level</emphasis>
@@ -937,7 +937,7 @@ $ &ctltool; fifo get_dbg_mod_facility debugger
</title>
<para>
Get the module log facility.
- If module does not exist in kamailio, the entry in the facility hashtable is still added for the bogus module.
+ If mod_name does not exist in the facility hashtable, returns the config file value.
</para>
<para>
Name: <emphasis>dbg.get_mod_facility</emphasis>