I'm observing a segmentation fault when mongodb & tls enabled. It doesn't happen when one of the module is disabled. OS: centos 6.7 kamailio 4.3.2 mongo-c-driver version 1.1.10.
It happens immediately after i register a subscriber. The location table is updated with the new data and in parallel core is generated as well. I'm running with the offical kamailio script by just enabling mongo modules. It happens for NDB as well.
How did you build the mongo driver when you implemented it? May be i can try the same.
Please find the scripts used for testing in the email.
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio.pid -m 64 -M 8'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
1017 lock_get(_d->table[sl].lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-19.el6_6.x86_64 glibc-2.12-1.166.el6_7.1.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007f85a21da075 in lock_udomain (_d=0x7f859c7b5f08, _aor=0x7fff6de7ef70) at udomain.c:1017
#1 0x00007f85a1d7069b in add_contacts (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _a=0x7fff6de7ef70, _mode=0, _use_regid=1) at save.c:831
#2 0x00007f85a1d72602 in save (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08, _cflags=0, _uri=0x0) at save.c:986
#3 0x00007f85a1d5975a in w_save2 (_m=0x7f85a3f054b8, _d=0x7f859c7b5f08 "h^{\234\205\177", _cflags=0x0) at reg_mod.c:414
#4 0x000000000041decb in do_action (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1059
#5 0x000000000042a553 in run_actions (h=0x7fff6de7f720, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1548
#6 0x000000000042abb8 in run_actions_safe (h=0x7fff6de80a10, a=0x7f85a3edd830, msg=0x7f85a3f054b8) at action.c:1613
#7 0x0000000000543d50 in rval_get_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, i=0x7fff6de7fbf8, rv=0x7f85a3edf478, cache=0x0) at rvalue.c:912
#8 0x0000000000547f88 in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de7fbf8, rve=0x7f85a3edf470) at rvalue.c:1906
#9 0x000000000054837e in rval_expr_eval_int (h=0x7fff6de80a10, msg=0x7f85a3f054b8, res=0x7fff6de80080, rve=0x7f85a3edfb70) at rvalue.c:1914
#10 0x000000000041d927 in do_action (h=0x7fff6de80a10, a=0x7f85a3ee0270, msg=0x7f85a3f054b8) at action.c:1029
#11 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3edc990, msg=0x7f85a3f054b8) at action.c:1548
#12 0x000000000041a8c3 in do_action (h=0x7fff6de80a10, a=0x7f85a3ebdd98, msg=0x7f85a3f054b8) at action.c:677
#13 0x000000000042a553 in run_actions (h=0x7fff6de80a10, a=0x7f85a3ebdb08, msg=0x7f85a3f054b8) at action.c:1548
#14 0x000000000042ac80 in run_top_route (a=0x7f85a3ebdb08, msg=0x7f85a3f054b8, c=0x0) at action.c:1634
#15 0x000000000050a9f4 in receive_msg (
buf=0xa70b00 "REGISTER sip:192.168.2.142 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.2.119:60887;branch=z9hG4bK-524287-1---d670bd2004732b4a;rport\r\nMax-Forwards: 70\r\nContact: <sip:usera@192.168.2.119:60887;rinstance=d9f6274d7"..., len=534, rcv_info=0x7fff6de80d00) at receive.c:196
#16 0x000000000060a4a6 in udp_rcv_loop () at udp_server.c:495
#17 0x00000000004a7fb3 in main_loop () at main.c:1573
#18 0x00000000004ae38e in main (argc=7, argv=0x7fff6de81138) at main.c:2533
(gdb) quit
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360
Took me a while to get back to this one after traveling ... do you still have the core files? If yes, can you give the output of next commands inside gdb:
``` frame 0 info locals list ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153022834
I'm getting a different core now, ( no Core if i disable TLS in kamailio or mongo-c-driver)
Core was generated by `/usr/local/kamailio_proxy/sbin/kamailio -f /usr/local/kamailio_proxy/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f142bd7897e in db_mongodb_query (_h=0x7f1430510100, _k=0x0, _op=0x0, _v=0x0, _c=0x7ffcb93494b0, _n=0, _nc=21, _o=0x0, _r=0x7ffcb93494a8) at mongodb_dbase.c:846 846 if(CON_TABLE(_h)->s[CON_TABLE(_h)->len]!='\0') { Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f142bd7897e in db_mongodb_query (_h=0x7f1430510100, _k=0x0, _op=0x0, _v=0x0, _c=0x7ffcb93494b0, _n=0, _nc=21, _o=0x0, _r=0x7ffcb93494a8) at mongodb_dbase.c:846 #1 0x00007f1428dce63c in preload_udomain (_c=0x7f1430510100, _d=0x7f142bfd2f20) at udomain.c:432 #2 0x00007f1428ddff87 in child_init (_rank=1) at ul_mod.c:437 #3 0x00000000005940ed in init_mod_child (m=0x7f143045a5f8, rank=1) at sr_module.c:898 #4 0x0000000000593e28 in init_mod_child (m=0x7f143045a8e0, rank=1) at sr_module.c:895 #5 0x0000000000593e28 in init_mod_child (m=0x7f143045b038, rank=1) at sr_module.c:895 #6 0x0000000000593e28 in init_mod_child (m=0x7f143045bd30, rank=1) at sr_module.c:895 #7 0x0000000000593e28 in init_mod_child (m=0x7f143045c670, rank=1) at sr_module.c:895 #8 0x0000000000593e28 in init_mod_child (m=0x7f143045cba8, rank=1) at sr_module.c:895 #9 0x0000000000593e28 in init_mod_child (m=0x7f143045cf38, rank=1) at sr_module.c:895 #10 0x0000000000593e28 in init_mod_child (m=0x7f143045d428, rank=1) at sr_module.c:895 #11 0x0000000000593e28 in init_mod_child (m=0x7f143045e9b8, rank=1) at sr_module.c:895 #12 0x0000000000593e28 in init_mod_child (m=0x7f143045eed0, rank=1) at sr_module.c:895 #13 0x0000000000593e28 in init_mod_child (m=0x7f143045f280, rank=1) at sr_module.c:895 #14 0x0000000000593e28 in init_mod_child (m=0x7f143045f630, rank=1) at sr_module.c:895 #15 0x0000000000593e28 in init_mod_child (m=0x7f143045f9c0, rank=1) at sr_module.c:895 #16 0x0000000000593e28 in init_mod_child (m=0x7f1430460030, rank=1) at sr_module.c:895 #17 0x0000000000593e28 in init_mod_child (m=0x7f1430460a20, rank=1) at sr_module.c:895 #18 0x00000000005943f7 in init_child (rank=1) at sr_module.c:924 #19 0x00000000004ebdf5 in fork_tcp_process (child_id=1, desc=0x7ffcb934a330 "tcp receiver (generic)", r=0, reader_fd_1=0x7ffcb934a3b0) at pt.c:465 #20 0x00000000005e20bf in tcp_init_children () at tcp_main.c:4777 #21 0x00000000004a8ba1 in main_loop () at main.c:1658 #22 0x00000000004ae38e in main (argc=15, argv=0x7ffcb934a758) at main.c:2533 (gdb) info locals i = 4 mgcon = 0x7f1430510288 client = 0x2a12500 seldoc = 0x0 cname = 0x0 b1 = 0 '\000' jstr = 0x7ffcb9349500 "\320|\377(\024\177" __FUNCTION__ = "db_mongodb_query" (gdb) list 841 } 842 843 if(_r) *_r = NULL; 844 845 b1 = '\0'; 846 if(CON_TABLE(_h)->s[CON_TABLE(_h)->len]!='\0') { 847 b1 = CON_TABLE(_h)->s[CON_TABLE(_h)->len]; 848 CON_TABLE(_h)->s[CON_TABLE(_h)->len] = '\0'; 849 } 850 cname = CON_TABLE(_h)->s; (gdb)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153386433
Get the output of next commands in gdb:
``` p _h p *_h p _h->table p *_h->table ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153404062
Program terminated with signal 11, Segmentation fault. #0 0x00007f7963c9e97e in db_mongodb_query (_h=0x7f7968437cd8, _k=0x0, _op=0x0, _v=0x0, _c=0x7ffc7373ba40, _n=0, _nc=21, _o=0x0, _r=0x7ffc7373ba38) at mongodb_dbase.c:846 846 if(CON_TABLE(_h)->s[CON_TABLE(_h)->len]!='\0') { Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-46.el6.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6_6.1.x86_64 libunistring-0.9.3-5.el6.x86_64 libxml2-2.7.6-20.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nspr-4.10.8-1.el6_6.x86_64 nss-3.18.0-5.3.el6_6.x86_64 nss-softokn-freebl-3.14.3-22.el6_6.x86_64 nss-util-3.18.0-1.el6_6.x86_64 openldap-2.4.40-5.el6.x86_64 openssl-1.0.1e-42.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x00007f7963c9e97e in db_mongodb_query (_h=0x7f7968437cd8, _k=0x0, _op=0x0, _v=0x0, _c=0x7ffc7373ba40, _n=0, _nc=21, _o=0x0, _r=0x7ffc7373ba38) at mongodb_dbase.c:846 #1 0x00007f7960ae663c in preload_udomain (_c=0x7f7968437cd8, _d=0x7f7963ef8f20) at udomain.c:432 #2 0x00007f7960af7f87 in child_init (_rank=1) at ul_mod.c:437 #3 0x00000000005940ed in init_mod_child (m=0x7f7968380b18, rank=1) at sr_module.c:898 #4 0x0000000000593e28 in init_mod_child (m=0x7f7968380e00, rank=1) at sr_module.c:895 #5 0x0000000000593e28 in init_mod_child (m=0x7f7968381558, rank=1) at sr_module.c:895 #6 0x0000000000593e28 in init_mod_child (m=0x7f7968382250, rank=1) at sr_module.c:895 #7 0x0000000000593e28 in init_mod_child (m=0x7f7968382b90, rank=1) at sr_module.c:895 #8 0x0000000000593e28 in init_mod_child (m=0x7f79683830c8, rank=1) at sr_module.c:895 #9 0x0000000000593e28 in init_mod_child (m=0x7f7968383458, rank=1) at sr_module.c:895 #10 0x0000000000593e28 in init_mod_child (m=0x7f7968383948, rank=1) at sr_module.c:895 #11 0x0000000000593e28 in init_mod_child (m=0x7f7968384ed8, rank=1) at sr_module.c:895 #12 0x0000000000593e28 in init_mod_child (m=0x7f79683853f0, rank=1) at sr_module.c:895 #13 0x0000000000593e28 in init_mod_child (m=0x7f79683857a0, rank=1) at sr_module.c:895 #14 0x0000000000593e28 in init_mod_child (m=0x7f7968385b50, rank=1) at sr_module.c:895 #15 0x0000000000593e28 in init_mod_child (m=0x7f7968385ee0, rank=1) at sr_module.c:895 #16 0x0000000000593e28 in init_mod_child (m=0x7f7968386550, rank=1) at sr_module.c:895 #17 0x0000000000593e28 in init_mod_child (m=0x7f7968386f40, rank=1) at sr_module.c:895 #18 0x00000000005943f7 in init_child (rank=1) at sr_module.c:924 #19 0x00000000004ebdf5 in fork_tcp_process (child_id=1, desc=0x7ffc7373c8c0 "tcp receiver (generic)", r=0, reader_fd_1=0x7ffc7373c940) at pt.c:465 #20 0x00000000005e20bf in tcp_init_children () at tcp_main.c:4777 #21 0x00000000004a8ba1 in main_loop () at main.c:1658 #22 0x00000000004ae38e in main (argc=15, argv=0x7ffc7373cce8) at main.c:2533 (gdb) info locals i = 0 mgcon = 0x7f7968437e60 client = 0x1988fb0 seldoc = 0x0 cname = 0x7ffc7373ba80 "\260\374\320`y\177" b1 = 0 '\000' jstr = 0xd0e12daad083a4fd <Address 0xd0e12daad083a4fd out of bounds> __FUNCTION__ = "db_mongodb_query" (gdb) list 841 } 842 843 if(_r) *_r = NULL; 844 845 b1 = '\0'; 846 if(CON_TABLE(_h)->s[CON_TABLE(_h)->len]!='\0') { 847 b1 = CON_TABLE(_h)->s[CON_TABLE(_h)->len]; 848 CON_TABLE(_h)->s[CON_TABLE(_h)->len] = '\0'; 849 } 850 cname = CON_TABLE(_h)->s; (gdb) p _h $1 = (const db1_con_t *) 0x7f7968437cd8 (gdb) p *_h $2 = {table = 0x7f7963ef81c0, tquote = 0x0, tail = 140159417024096} (gdb) p _h->table $3 = (const str *) 0x7f7963ef81c0 (gdb) p *_h->table $4 = {s = 0x7f79ffffffff <Address 0x7f79ffffffff out of bounds>, len = 8} (gdb)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153410984
Few more:
``` p *_h->table->s p _h->table->s ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153485859
(gdb) p *_h->table->s Cannot access memory at address 0x7f79ffffffff (gdb) p _h->table->s $1 = 0x7f79ffffffff <Address 0x7f79ffffffff out of bounds> (gdb)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153489740
So table name seems inaccessible.
Let's see what domain structure has:
``` frame 1 p *_d p *_d->name p *_d->name->s ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153490977
(gdb) f 1 #1 0x00007f7960ae663c in preload_udomain (_c=0x7f7968437cd8, _d=0x7f7963ef8f20) at udomain.c:432 432 if (ul_dbf.query(_c, 0, 0, 0, columns, 0, (use_domain)?(21):(20), 0, (gdb) p *_d $2 = {name = 0x7f7963ef81c0, size = 1024, table = 0x7f7964259d90, users = 0x64, contacts = 0x65, expires = 0x66} (gdb) p *_d->name $3 = {s = 0x7f79ffffffff <Address 0x7f79ffffffff out of bounds>, len = 8} (gdb) p *_d->name->s Cannot access memory at address 0x7f79ffffffff (gdb)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153501454
Quite strange...
Can you load tls module before mongo modules?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153503996
Yes that's what i did.
/* add local domain aliases */ #alias="sip.mydomain.com"
#listen=SIP_LISTEN_SOCKET listen=SIP_LISTEN_TCP_SOCKET #listen=WS_LISTEN_SOCKET #!ifdef WITH_TLS #listen=WSS_LISTEN_SOCKET listen=SIP_LISTEN_TLS_SOCKET #!endif
/* port to listen to * - can be specified more than once if needed to listen on many ports */ #port=SIP_PORT
# life time of TCP connection when there is no traffic # - a bit higher than registration expires to cope with UA behind NAT tcp_connection_lifetime=3605 tcp_accept_no_cl=yes tcp_rd_buf_size=16384
####### Modules Section ########
# set paths to location of modules mpath="/usr/local/kamailio_proxy/lib64/kamailio/modules/"
#!ifdef WITH_TLS loadmodule "tls.so" #!endif
#!ifdef WITH_MONGODB loadmodule "db_mongodb.so" #!endif #!ifdef WITH_NDB_MONGODB loadmodule "ndb_mongodb.so" #!endif loadmodule "db_mysql.so" loadmodule "mi_fifo.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "rr.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "textops.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" loadmodule "mi_rpc.so" loadmodule "acc.so"
#!ifdef WITH_AUTH loadmodule "auth.so" loadmodule "auth_db.so" #!endif
# loadmodule "alias_db.so"
# loadmodule "domain.so"
#!ifdef WITH_PIKE
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153509470
I think there is a memory corruption as kamailio and mongo db module is initializing the openssl context from the same process. This might have some side effects.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153509907
I can't reproduce it with kamailio master and mongo-c-driver 1.2.1 (configured with --enable-ssl=yes). Can you try same combination of versions?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153515692
Yes. I will try that and let you know.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-153516198
@jijo2010 please use ``` Markdown in order not to mess with references of issues like in #1
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-159609667
Closing this one as couldn't be reproduced with the latest version of mongo-c-driver and no follow up for few weeks. Reopen if the issue is discovered to be still there and have new details.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#issuecomment-160118506
Closed #360.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/360#event-476259342
-
Daniel-Constantin Mierla -
jijo2010
-
Victor Seva