I have 3 kamailio 434 registrars replicating between each other using dmq usrloc module
I have recently noticed that all of the registrars crash at random intervals over the course of the day, they all dont crash at the same time, but randomly, independantly of each other
I cannot see any unusual traffic arriving at the systems that may be a cause for the crash
I'd appreciate assistance trying to get to the bottom of the issue
Kamailio Version: ``` # /usr/sbin/kamailio -V version: kamailio 434 (x86_64/linux) 0ec860 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select id: 0ec860 compiled on 17:42:12 Nov 25 2015 with gcc 447 ```
Message in the log file when the kamailio application crashes: ``` Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:728]: handle_sigs(): child process 16234 exited by a signal 11 Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [mainc:731]: handle_sigs(): core was generated Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [mainc:743]: handle_sigs(): terminating due to SIGCHLD Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [mainc:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a084f8 is still linked in hash table (h_tablec:440) Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_tablec:130]: free_cell_helper(): removed cell 0x7f5a40a12390 is still linked in hash table (h_tablec:440) ```
Core file synopsis is: ``` Core was generated by `/usr/sbin/kamailio -P /var/run/kamailiopid -m 1024 -M 64 -u root -g root' Program terminated with signal 11, Segmentation fault #0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140, _ci=0x7f5a81418500) at urecordc:152 152 if_update_stat( _r->slot, _r->slot->d->contacts, 1); ```
I still have the core file, however, it contains some sensitive info, so I can send backtraces directly to you if required
Please let me know if you require any additional info
Thanks
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440
Can you give the output of:
``` p *_r p *_r->slot p *_r->slot->d p *_r->slot->d->contacts ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-164469254
``` (gdb) p *_r $1 = {domain = 0x746d616e3a706973, aor = {s = 0x3931403130747365 <Address 0x3931403130747365 out of bounds>, len = 909192754}, aorhash = 893007672, contacts = 0x39383d7361696c61, slot = 0x3133312e3130312e, prev = 0x3336327e3032312e, next = 0x317e38} (gdb) p *_r->slot Cannot access memory at address 0x3133312e3130312e (gdb) p *_r->slot->d Cannot access memory at address 0x3133312e31303146 (gdb) p *_r->slot->d->contacts Cannot access memory at address 0x3133312e31303146 ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-164470877
It looks like the structure has invalid content. Do you see any error messages in syslog before the crash happened?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165020997
all I see in the kamailio log is what is posted in the 1st post, I re-post it below again, the previous log above these is for the 13 Dec at 20:57, with a warning, the next log after these below is the first line of Kamailio starting up.
``` Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:728]: handle_sigs(): child process 16234 exited by a signal 11 Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated Dec 14 01:46:54 /usr/sbin/kamailio[16205]: INFO: <core> [main.c:743]: handle_sigs(): terminating due to SIGCHLD Dec 14 01:46:54 /usr/sbin/kamailio[16246]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16244]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16232]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16243]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16242]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16228]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16245]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16241]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16235]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16240]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16225]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16236]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16233]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16220]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16221]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16230]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16237]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16231]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16217]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16229]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16218]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16219]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16226]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16215]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16216]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16239]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16212]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16223]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16214]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16213]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16208]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16207]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16211]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16238]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16222]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16210]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16227]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16224]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16206]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16209]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f5a40a084f8 is still linked in hash table (h_table.c:440) Dec 14 01:46:54 /usr/sbin/kamailio[16205]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f5a40a12390 is still linked in hash table (h_table.c:440) ```
I have multiple core files, I could try another one if we cannot get any info out of this one.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165025696
To be more specific now: any log messages before the crash from the process with pid 16234 ?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165028337
no, not around that time, but an entire log search for that pid results in the following:
the log file date starts from 30th June, so I presume it would have caught anything if it was logged.
``` # fgrep '[16234]' kamailio.log # fgrep '16234' kamailio.log Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 14 01:46:54 /usr/sbin/kamailio[16205]: ALERT: <core> [main.c:728]: handle_sigs(): child process 16234 exited by a signal 11 ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165029061
actually looking at the log for the mesages around 15:23 on Dec 13, it looks like it crashed and restarted at that time too, those dmq messages appear to be just after the restart, when, I presume, dmq is replacting the contacts upon the restart, full log at that time is as follows:
``` Dec 13 15:22:57 /usr/sbin/kamailio[15553]: ALERT: <core> [main.c:728]: handle_sigs(): child process 15582 exited by a signal 11 Dec 13 15:22:57 /usr/sbin/kamailio[15553]: ALERT: <core> [main.c:731]: handle_sigs(): core was generated Dec 13 15:22:57 /usr/sbin/kamailio[15553]: INFO: <core> [main.c:743]: handle_sigs(): terminating due to SIGCHLD Dec 13 15:22:57 /usr/sbin/kamailio[15590]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15592]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15585]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15594]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15584]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15589]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15581]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15579]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15583]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15593]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15580]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15571]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15575]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15578]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15554]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15556]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15567]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15569]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15566]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15561]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15568]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15565]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15588]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15570]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15591]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15559]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15563]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15560]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15562]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15557]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15564]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15555]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15572]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15587]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15558]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15576]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15586]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15574]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15573]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15577]: INFO: <core> [main.c:798]: sig_usr(): signal 15 received Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f380d9f91c8 is still linked in hash table (h_table.c:440) Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f380d9fdac0 is still linked in hash table (h_table.c:440) Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f380da1a8d0 is still linked in hash table (h_table.c:440) Dec 13 15:22:57 /usr/sbin/kamailio[15553]: WARNING: tm [h_table.c:130]: free_cell_helper(): removed cell 0x7f380da19228 is still linked in hash table (h_table.c:440) Dec 13 15:22:57 /usr/sbin/kamailio[15553]: INFO: <core> [sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Dec 13 15:23:08 kamailio: INFO: <core> [sctp_core.c:75]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module Dec 13 15:23:08 kamailio: INFO: <core> [tcp_main.c:4657]: init_tcp(): using epoll_lt as the io watch method (auto detected) Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: rr [../outbound/api.h:54]: ob_load_api(): Failed to import bind_ob Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: rr [rr_mod.c:174]: mod_init(): outbound module not available Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: usrloc [hslot.c:51]: ul_init_locks(): locks array size 1024 Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: path [../outbound/api.h:54]: ob_load_api(): Failed to import bind_ob Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: path [path_mod.c:147]: mod_init(): outbound module not available Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: dmq_usrloc [dmq_usrloc.c:67]: mod_init(): dmq usrloc replication mode = 1 Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: permissions [permissions.c:608]: mod_init(): default allow file (//etc/kamailio/permissions.allow) not found => empty rule set Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: permissions [permissions.c:617]: mod_init(): default deny file (//etc/kamailio/permissions.deny) not found => empty rule set Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: db_text [dbt_base.c:92]: dbt_init(): using database at: /etc/kamailio/db_text Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: db_text [dbt_base.c:92]: dbt_init(): using database at: /etc/kamailio/db_text Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:150]: probe_max_receive_buffer(): SO_RCVBUF is initially 124928 Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:200]: probe_max_receive_buffer(): SO_RCVBUF is finally 249856 Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:150]: probe_max_receive_buffer(): SO_RCVBUF is initially 124928 Dec 13 15:23:08 /usr/sbin/kamailio[16205]: INFO: <core> [udp_server.c:200]: probe_max_receive_buffer(): SO_RCVBUF is finally 249856 Dec 13 15:23:08 /usr/sbin/kamailio[16224]: INFO: db_text [dbt_base.c:92]: dbt_init(): using database at: /etc/kamailio/db_text Dec 13 15:23:08 /usr/sbin/kamailio[16224]: INFO: db_text [dbt_base.c:92]: dbt_init(): using database at: /etc/kamailio/db_text Dec 13 15:23:08 /usr/sbin/kamailio[16233]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop: using epoll_lt io watch method (config) Dec 13 15:23:08 /usr/sbin/kamailio[16236]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16235]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq Dec 13 15:23:08 /usr/sbin/kamailio[16234]: ERROR: dmq_usrloc [usrloc_sync.c:67]: add_contact(): Invalid cseq ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165029746
Can you attach the backtrace (just 'bt', not 'bt full' for the moment) from gdb here? Replace any IP/sensitive info there, but I would like to see the backtrace here to correlate easier with the logs above.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165247807
sure, here it is:
``` # gdb /usr/sbin/kamailio /core.16234
<snip loading symbols output>
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 1024 -M 64 -u root -g root'. Program terminated with signal 11, Segmentation fault. #0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140, _ci=0x7f5a81418500) at urecord.c:152 152 if_update_stat( _r->slot, _r->slot->d->contacts, 1); Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.3.x86_64 libxml2-2.7.6-20.el6_7.1.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f5a82f68985 in mem_insert_ucontact (_r=0x7f5a409ec948, _c=0x7ffcc26f3140, _ci=0x7f5a81418500) at urecord.c:152 #1 0x00007f5a82f6bb9d in insert_ucontact (_r=0x7f5a409ec948, _contact=0x7ffcc26f3140, _ci=0x7f5a81418500, _c=0x7ffcc26f3150) at urecord.c:568 #2 0x00007f5a8120b613 in add_contact (aor=..., ci=0x7f5a81418500) at usrloc_sync.c:73 #3 0x00007f5a8121178b in usrloc_dmq_handle_msg (msg=0x7f5a409f4000, resp=0x7ffcc26f33b0, node=0x7f5a409e88a8) at usrloc_sync.c:390 #4 0x00007f5a8143d22b in worker_loop (id=0) at worker.c:105 #5 0x00007f5a8142128c in child_init (rank=0) at dmq.c:293 #6 0x0000000000595308 in init_mod_child (m=0x7f5a84f32088, rank=0) at sr_module.c:915 #7 0x0000000000595043 in init_mod_child (m=0x7f5a84f32570, rank=0) at sr_module.c:912 #8 0x0000000000595043 in init_mod_child (m=0x7f5a84f32838, rank=0) at sr_module.c:912 #9 0x0000000000595043 in init_mod_child (m=0x7f5a84f33440, rank=0) at sr_module.c:912 #10 0x0000000000595043 in init_mod_child (m=0x7f5a84f33928, rank=0) at sr_module.c:912 #11 0x0000000000595043 in init_mod_child (m=0x7f5a84f33d68, rank=0) at sr_module.c:912 #12 0x0000000000595043 in init_mod_child (m=0x7f5a84f34640, rank=0) at sr_module.c:912 #13 0x0000000000595612 in init_child (rank=0) at sr_module.c:941 #14 0x00000000004a8c7d in main_loop () at main.c:1654 #15 0x00000000004aecd2 in main (argc=11, argv=0x7ffcc26f3c88) at main.c:2553 ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-165250172
I pushed a patch for it - commit 5937b04 - as I could spot, there was no locking done before the insert, but only unlocking. Would be good if you can test and report the results.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-166310109
Thanks, I am away this week, I will test it out early next week and update you here. Thanks for looking into it.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-166362706
I have updated the registars with v4.3.4 including the patch above. I will need to wait for a couple days to see if we can get the system to generate any core dumps with the new patch applied.
I'll update the issue if/when there is a dump generated.
Thanks for looking into this.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-167581996
So far no core files have been generated on the registrar's since the patch. I think we can safely close this issue now.
Thanks for taking the time to look into it, much appreciated.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#issuecomment-169617890
Closed #440.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/440#event-508074090
-
00Asgaroth00 -
Daniel-Constantin Mierla