dmq/dmq_userloc crash on startup (kamailio 4.3.0) - sr-users

1 Jul 2015


      Hi All,
Another thing I noticed while playing with kamailio and dmq/dmq_usrloc 
is that 7 out of 10 times the kamailio process crashes when I initially 
start it up.
Any tips/requests for further debugging would be greatly appreciated.
Kamailio version:
version: kamailio 4.3.0 (x86_64/linux) c6aa95
flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, 
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, 
F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, 
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, 
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, 
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: c6aa95
compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7
I start kamailio as follows, from commandline:
/usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f 
/shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173, 
10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80 
-u root -g root -ddd
DMQ related settings are as follows:
# grep -i dmq module_definitions.cfg
loadmodule "dmq.so"
loadmodule "dmq_usrloc.so"
# ----- dmq params -----
modparam("dmq", "server_address", DMQ_ADDRESS)
modparam("dmq", "notification_address", 
"sip:sip-registrars.internal_domain.com:5060")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
# ----- dmq_usrloc params -----
modparam("dmq_usrloc", "enable", 1)
DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case.
sip-registrars.internal_domain.com resolves to the following address 
list 10.6.0.173, 10.6.0.174, 10.6.0.175
Here is the debug log of the crash:
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: 
tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header 
reached, state=29
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[notification_peer.c:454]: dmq_notification_callback(): dmq triggered 
from dmq_notification_callback
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[dmqnode.c:153]: build_dmq_node(): build_dmq_node 
sip:10.6.0.173:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[dmqnode.c:153]: build_dmq_node(): build_dmq_node 
sip:10.6.0.175:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[dmqnode.c:153]: build_dmq_node(): build_dmq_node 
sip:10.6.0.174:5060;status=active with private memory
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[notification_peer.c:468]: dmq_notification_callback(): received 0 new 
or changed nodes
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[notification_peer.c:528]: build_notification_body(): body_len = 1024 - 
clen = 0
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[notification_peer.c:528]: build_notification_body(): body_len = 1024 - 
clen = 35
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[notification_peer.c:528]: build_notification_body(): body_len = 1024 - 
clen = 70
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc 
[usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting sync from dmq 
peers
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc 
[usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending serialized data 
{"action":3}
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc 
[usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq broadcast...
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && 
[10.6.0.173] == [10.6.0.173]
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 
0) matches port 5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm 
[uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: 
next_hop=sip:usrloc@10.6.0.174:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm 
[uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && 
[10.6.0.173] == [10.6.0.173]
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 
0) matches port 5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm 
[uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: 
next_hop=sip:usrloc@10.6.0.175:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm 
[uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[dmq_funcs.c:163]: bcast_dmq_message(): skipping node sip:10.6.0.173:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl 
[sl.c:280]: send_reply(): reply in stateless mode (sl)
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[msg_translator.c:158]: check_via_address(): (10.6.0.174, 10.6.0.174, 0)
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq 
[worker.c:134]: worker_loop(): sent reply
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: 
tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> 
[parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header 
reached, state=29
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc 
[usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message received from 
sip:usrloc@10.6.0.174:5060
Jul  1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc 
[usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received DMQ_SYNC. Sending 
all contacts...
And here is the backtrace from the core file:
# gdb /usr/sbin/kamailio /core.27445
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/sbin/kamailio...(no debugging symbols 
found)...done.
[New Thread 27445]
Reading symbols from /lib64/libdl.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so
Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libodbc.so.2
Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1
Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libkcore.so.1
Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libltdl.so.7
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols 
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so
Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libkmi.so.1
Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/tm.so
Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/sl.so
Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/pv.so
Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/kex.so
Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so
Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/registrar.so
Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/xlog.so
Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/auth.so
Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so
Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/rr.so
Reading symbols from /usr/lib64/kamailio/modules/path.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/path.so
Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so
Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/siputils.so
Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/textops.so
Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so
Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libxml2.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/ctl.so
Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so
Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq.so
Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no 
debugging symbols found)...done.
Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging 
symbols found)...done.
Loaded symbols for /usr/lib64/gconv/ISO8859-1.so
Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/psqlodbc.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libpq.so.5
Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libodbcinst.so.2
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Core was generated by `/usr/sbin/kamailio -P 
/var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
kamailio-4.3.0-0.el6.bf.x86_64
(gdb) bt
#0  0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6
#1  0x00007fdacfeff03d in usrloc_get_all_ucontact () from 
/usr/lib64/kamailio/modules/dmq_usrloc.so
#2  0x00007fdacff02dfd in usrloc_dmq_handle_msg () from 
/usr/lib64/kamailio/modules/dmq_usrloc.so
#3  0x00007fdad012d4a7 in worker_loop () from 
/usr/lib64/kamailio/modules/dmq.so
#4  0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so
#5  0x0000000000592675 in ?? ()
#6  0x00000000005923b0 in ?? ()
#7  0x000000000059297f in init_child ()
#8  0x00000000004a7680 in main_loop ()
#9  0x00000000004acd36 in main ()
(gdb) ^C(gdb) Quit