[SR-Users] dmq/dmq_userloc crash on startup (kamailio 4.3.0)

Hi All, Another thing I noticed while playing with kamailio and dmq/dmq_usrloc is that 7 out of 10 times the kamailio process crashes when I initially start it up. Any tips/requests for further debugging would be greatly appreciated. Kamailio version: version: kamailio 4.3.0 (x86_64/linux) c6aa95 flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: c6aa95 compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7 I start kamailio as follows, from commandline: /usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f /shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173, 10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80 -u root -g root -ddd DMQ related settings are as follows: # grep -i dmq module_definitions.cfg loadmodule "dmq.so" loadmodule "dmq_usrloc.so" # ----- dmq params ----- modparam("dmq", "server_address", DMQ_ADDRESS) modparam("dmq", "notification_address", "sip:sip-registrars.internal_domain.com:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) # ----- dmq_usrloc params ----- modparam("dmq_usrloc", "enable", 1) DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case. sip-registrars.internal_domain.com resolves to the following address list 10.6.0.173, 10.6.0.174, 10.6.0.175 Here is the debug log of the crash: Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header reached, state=29 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:454]: dmq_notification_callback(): dmq triggered from dmq_notification_callback Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.173:5060;status=active with private memory Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.175:5060;status=active with private memory Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmqnode.c:153]: build_dmq_node(): build_dmq_node sip:10.6.0.174:5060;status=active with private memory Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:468]: dmq_notification_callback(): received 0 new or changed nodes Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 0 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 35 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [notification_peer.c:528]: build_notification_body(): body_len = 1024 - clen = 70 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting sync from dmq peers Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending serialized data {"action":3} Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq broadcast... Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && [10.6.0.173] == [10.6.0.173] Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 0) matches port 5060 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: next_hop=<sip:usrloc@10.6.0.174:5060> Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && [10.6.0.173] == [10.6.0.173] Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [socket_info.c:566]: grep_sock_info(): checking if port 5060 (advertise 0) matches port 5060 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: next_hop=<sip:usrloc@10.6.0.175:5060> Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [dmq_funcs.c:163]: bcast_dmq_message(): skipping node sip:10.6.0.173:5060 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl [sl.c:280]: send_reply(): reply in stateless mode (sl) Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [msg_translator.c:158]: check_via_address(): (10.6.0.174, 10.6.0.174, 0) Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq [worker.c:134]: worker_loop(): sent reply Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header reached, state=29 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message received from sip:usrloc@10.6.0.174:5060 Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq_usrloc [usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received DMQ_SYNC. Sending all contacts... And here is the backtrace from the core file: # gdb /usr/sbin/kamailio /core.27445 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/kamailio...(no debugging symbols found)...done. [New Thread 27445] Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libodbc.so.2 Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1 Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/libkcore.so.1 Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libltdl.so.7 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/libkmi.so.1 Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/tm.so Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/sl.so Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/pv.so Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/kex.so Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/registrar.so Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/xlog.so Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/auth.so Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/rr.so Reading symbols from /usr/lib64/kamailio/modules/path.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/path.so Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/siputils.so Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/textops.so Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/ctl.so Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/dmq.so Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/gconv/ISO8859-1.so Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/psqlodbc.so Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libpq.so.5 Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libodbcinst.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libldap_r-2.4.so.2 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libfreebl3.so Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl3.so Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsmime3.so Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnss3.so Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnssutil3.so Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplds4.so Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplc4.so Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libnspr4.so Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_dns.so.2 Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'. Program terminated with signal 11, Segmentation fault. #0 0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install kamailio-4.3.0-0.el6.bf.x86_64 (gdb) bt #0 0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6 #1 0x00007fdacfeff03d in usrloc_get_all_ucontact () from /usr/lib64/kamailio/modules/dmq_usrloc.so #2 0x00007fdacff02dfd in usrloc_dmq_handle_msg () from /usr/lib64/kamailio/modules/dmq_usrloc.so #3 0x00007fdad012d4a7 in worker_loop () from /usr/lib64/kamailio/modules/dmq.so #4 0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so #5 0x0000000000592675 in ?? () #6 0x00000000005923b0 in ?? () #7 0x000000000059297f in init_child () #8 0x00000000004a7680 in main_loop () #9 0x00000000004acd36 in main () (gdb) ^C(gdb) Quit