Re: [SR-Users] dmq/dmq_userloc crash on startup (kamailio 4.3.0)

Hello, at the quick look it seems that dmq_usrloc was not updated with the latest format of the serialized structure returned by usrloc when fetching the contacts. It seems to use the same function as nathelper (which use it for sending keepalives) and that is returning both contact and received addresses, but dmq_usrloc seem to take only one of them (like in the older versions of the function). I am traveling for a while, but I guess some other devs that use dmq can look at it. Cheers, Daniel On 01/07/15 23:33, Asgaroth wrote: > Hi All, > > Another thing I noticed while playing with kamailio and dmq/dmq_usrloc > is that 7 out of 10 times the kamailio process crashes when I > initially start it up. > > Any tips/requests for further debugging would be greatly appreciated. > > Kamailio version: > > version: kamailio 4.3.0 (x86_64/linux) c6aa95 > flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, > DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, > F_MALLOC, DBG_F_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, > USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, > HAVE_RESOLV_RES > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, > MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > id: c6aa95 > compiled on 16:14:27 Jun 23 2015 with gcc 4.4.7 > > I start kamailio as follows, from commandline: > > /usr/sbin/kamailio -P /var/run/kamailio_registrar01/kamailio.pid -f > /shared/kamailio/etc/kamailio_registrar.cfg -l "udp:(10.6.0.173, > 10.7.0.173)" -m 1024 -M64 -A RESOURCE_REGISTRAR01 -l tcp:10.6.0.173:80 > -u root -g root -ddd > > DMQ related settings are as follows: > > # grep -i dmq module_definitions.cfg > loadmodule "dmq.so" > loadmodule "dmq_usrloc.so" > # ----- dmq params ----- > modparam("dmq", "server_address", DMQ_ADDRESS) > modparam("dmq", "notification_address", > "sip:sip-registrars.internal_domain.com:5060") > modparam("dmq", "multi_notify", 1) > modparam("dmq", "num_workers", 4) > # ----- dmq_usrloc params ----- > modparam("dmq_usrloc", "enable", 1) > > DMQ_ADDRESS is defined as "sip:10.6.0.173:5060" in this servers case. > sip-registrars.internal_domain.com resolves to the following address > list 10.6.0.173, 10.6.0.174, 10.6.0.175 > > Here is the debug log of the crash: > > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [worker.c:82]: worker_loop(): dmq_worker [0 27445] getting lock > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [worker.c:84]: worker_loop(): dmq_worker [0 27445] lock acquired > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: > tag=66fabe782d71b7a9842d4a6a5eb93c4d-1de2 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header > reached, state=29 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [notification_peer.c:454]: dmq_notification_callback(): dmq triggered > from dmq_notification_callback > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [dmqnode.c:153]: build_dmq_node(): build_dmq_node > sip:10.6.0.173:5060;status=active with private memory > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [dmqnode.c:153]: build_dmq_node(): build_dmq_node > sip:10.6.0.175:5060;status=active with private memory > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [dmqnode.c:153]: build_dmq_node(): build_dmq_node > sip:10.6.0.174:5060;status=active with private memory > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [notification_peer.c:468]: dmq_notification_callback(): received 0 new > or changed nodes > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [notification_peer.c:528]: build_notification_body(): body_len = 1024 > - clen = 0 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [notification_peer.c:528]: build_notification_body(): body_len = 1024 > - clen = 35 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [notification_peer.c:528]: build_notification_body(): body_len = 1024 > - clen = 70 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: > dmq_usrloc [usrloc_sync.c:388]: usrloc_dmq_request_sync(): requesting > sync from dmq peers > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: > dmq_usrloc [usrloc_sync.c:404]: usrloc_dmq_request_sync(): sending > serialized data {"action":3} > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: > dmq_usrloc [usrloc_sync.c:226]: usrloc_dmq_send(): sending dmq > broadcast... > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && > [10.6.0.173] == [10.6.0.173] > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [socket_info.c:566]: grep_sock_info(): checking if port 5060 > (advertise 0) matches port 5060 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm > [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: > next_hop=<sip:usrloc@10.6.0.174:5060> > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm > [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24685 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [socket_info.c:563]: grep_sock_info(): checking if host==us: 10==10 && > [10.6.0.173] == [10.6.0.173] > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [socket_info.c:566]: grep_sock_info(): checking if port 5060 > (advertise 0) matches port 5060 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm > [uac.c:249]: t_uac_prepare(): DEBUG:tm:t_uac: > next_hop=<sip:usrloc@10.6.0.175:5060> > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: tm > [uac.c:150]: dlg2hash(): DEBUG: dlg2hash: 24686 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [dmq_funcs.c:163]: bcast_dmq_message(): skipping node > sip:10.6.0.173:5060 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: sl > [sl.c:280]: send_reply(): reply in stateless mode (sl) > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [msg_translator.c:158]: check_via_address(): (10.6.0.174, > 10.6.0.174, 0) > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: dmq > [worker.c:134]: worker_loop(): sent reply > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [parser/parse_addr_spec.c:172]: parse_to_param(): DEBUG: add_param: > tag=66fabe782d71b7a9842d4a6a5eb93c4d-1ac2 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: <core> > [parser/parse_addr_spec.c:894]: parse_addr_spec(): end of header > reached, state=29 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: > dmq_usrloc [usrloc_sync.c:250]: usrloc_dmq_handle_msg(): dmq message > received from sip:usrloc@10.6.0.174:5060 > Jul 1 21:04:48 bfievkrl01 /usr/sbin/kamailio[27445]: DEBUG: > dmq_usrloc [usrloc_sync.c:360]: usrloc_dmq_handle_msg(): Received > DMQ_SYNC. Sending all contacts... > > And here is the backtrace from the core file: > > # gdb /usr/sbin/kamailio /core.27445 > GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6) > Copyright (C) 2010 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later > <http://gnu.org/licenses/gpl.html> > This is free software: you are free to change and redistribute it. > There is NO WARRANTY, to the extent permitted by law. Type "show > copying" > and "show warranty" for details. > This GDB was configured as "x86_64-redhat-linux-gnu". > For bug reporting instructions, please see: > <http://www.gnu.org/software/gdb/bugs/>... > Reading symbols from /usr/sbin/kamailio...(no debugging symbols > found)...done. > [New Thread 27445] > Reading symbols from /lib64/libdl.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libdl.so.2 > Reading symbols from /lib64/libresolv.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libresolv.so.2 > Reading symbols from /lib64/libc.so.6...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libc.so.6 > Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging > symbols found)...done. > Loaded symbols for /lib64/ld-linux-x86-64.so.2 > Reading symbols from /usr/lib64/kamailio/modules/db_unixodbc.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/db_unixodbc.so > Reading symbols from /usr/lib64/libodbc.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libodbc.so.2 > Reading symbols from /usr/lib64/kamailio/libsrdb1.so.1...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/libsrdb1.so.1 > Reading symbols from /usr/lib64/kamailio/libkcore.so.1...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/libkcore.so.1 > Reading symbols from /usr/lib64/libltdl.so.7...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libltdl.so.7 > Reading symbols from /lib64/libpthread.so.0...(no debugging symbols > found)...done. > [Thread debugging using libthread_db enabled] > Loaded symbols for /lib64/libpthread.so.0 > Reading symbols from /usr/lib64/kamailio/modules/mi_fifo.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/mi_fifo.so > Reading symbols from /usr/lib64/kamailio/libkmi.so.1...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/libkmi.so.1 > Reading symbols from /usr/lib64/kamailio/modules/tm.so...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/tm.so > Reading symbols from /usr/lib64/kamailio/modules/sl.so...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/sl.so > Reading symbols from /usr/lib64/kamailio/modules/pv.so...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/pv.so > Reading symbols from /usr/lib64/kamailio/libsrutils.so.1...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/libsrutils.so.1 > Reading symbols from /lib64/libm.so.6...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libm.so.6 > Reading symbols from /usr/lib64/kamailio/modules/kex.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/kex.so > Reading symbols from /usr/lib64/kamailio/modules/usrloc.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/usrloc.so > Reading symbols from /usr/lib64/kamailio/modules/registrar.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/registrar.so > Reading symbols from /usr/lib64/kamailio/modules/xlog.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/xlog.so > Reading symbols from /usr/lib64/kamailio/modules/auth.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/auth.so > Reading symbols from /usr/lib64/kamailio/modules/auth_db.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/auth_db.so > Reading symbols from /usr/lib64/kamailio/modules/rr.so...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/rr.so > Reading symbols from /usr/lib64/kamailio/modules/path.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/path.so > Reading symbols from /usr/lib64/kamailio/modules/nathelper.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/nathelper.so > Reading symbols from /usr/lib64/kamailio/modules/siputils.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/siputils.so > Reading symbols from /usr/lib64/kamailio/modules/textops.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/textops.so > Reading symbols from /usr/lib64/kamailio/modules/xmlrpc.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/xmlrpc.so > Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libxml2.so.2 > Reading symbols from /lib64/libz.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libz.so.1 > Reading symbols from /usr/lib64/kamailio/modules/ctl.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/ctl.so > Reading symbols from /usr/lib64/kamailio/modules/cfg_rpc.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/cfg_rpc.so > Reading symbols from /usr/lib64/kamailio/modules/dmq.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/dmq.so > Reading symbols from /usr/lib64/kamailio/modules/dmq_usrloc.so...(no > debugging symbols found)...done. > Loaded symbols for /usr/lib64/kamailio/modules/dmq_usrloc.so > Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libnss_files.so.2 > Reading symbols from /usr/lib64/gconv/ISO8859-1.so...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/gconv/ISO8859-1.so > Reading symbols from /usr/lib64/psqlodbc.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/psqlodbc.so > Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libssl.so.10 > Reading symbols from /usr/lib64/libpq.so.5...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libpq.so.5 > Reading symbols from /usr/lib64/libodbcinst.so.2...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/libodbcinst.so.2 > Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging > symbols found)...done. > Loaded symbols for /lib64/libgssapi_krb5.so.2 > Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libkrb5.so.3 > Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libcom_err.so.2 > Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libk5crypto.so.3 > Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging > symbols found)...done. > Loaded symbols for /usr/lib64/libcrypto.so.10 > Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libcrypt.so.1 > Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libldap_r-2.4.so.2 > Reading symbols from /lib64/libkrb5support.so.0...(no debugging > symbols found)...done. > Loaded symbols for /lib64/libkrb5support.so.0 > Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libkeyutils.so.1 > Reading symbols from /lib64/libfreebl3.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libfreebl3.so > Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/liblber-2.4.so.2 > Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libsasl2.so.2 > Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libssl3.so > Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libsmime3.so > Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libnss3.so > Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols > found)...done. > Loaded symbols for /usr/lib64/libnssutil3.so > Reading symbols from /lib64/libplds4.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libplds4.so > Reading symbols from /lib64/libplc4.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libplc4.so > Reading symbols from /lib64/libnspr4.so...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libnspr4.so > Reading symbols from /lib64/libselinux.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libselinux.so.1 > Reading symbols from /lib64/librt.so.1...(no debugging symbols > found)...done. > Loaded symbols for /lib64/librt.so.1 > Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols > found)...done. > Loaded symbols for /lib64/libnss_dns.so.2 > Core was generated by `/usr/sbin/kamailio -P > /var/run/kamailio_registrar01/kamailio.pid -f /shared/kam'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6 > Missing separate debuginfos, use: debuginfo-install > kamailio-4.3.0-0.el6.bf.x86_64 > (gdb) bt > #0 0x00007fdad84d76a5 in memcpy () from /lib64/libc.so.6 > #1 0x00007fdacfeff03d in usrloc_get_all_ucontact () from > /usr/lib64/kamailio/modules/dmq_usrloc.so > #2 0x00007fdacff02dfd in usrloc_dmq_handle_msg () from > /usr/lib64/kamailio/modules/dmq_usrloc.so > #3 0x00007fdad012d4a7 in worker_loop () from > /usr/lib64/kamailio/modules/dmq.so > #4 0x00007fdad011232c in ?? () from /usr/lib64/kamailio/modules/dmq.so > #5 0x0000000000592675 in ?? () > #6 0x00000000005923b0 in ?? () > #7 0x000000000059297f in init_child () > #8 0x00000000004a7680 in main_loop () > #9 0x00000000004acd36 in main () > (gdb) ^C(gdb) Quit > > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list > sr-users(a)lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users