Re: [SR-Users] Kamailio fail to start with db_mongodb and ndb_mongodb modules

Hi Daniel & K-Team, I setup kamailio 5.0.5 from git on vanilla Debian GNU/Linux 8.3 (jessie) # kamailio -v version: kamailio 5.0.5 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 16:53:42 Jan 5 2018 with gcc 4.9.2 Installed mongodb from debian repo # dpkg -l | grep mongodb ii mongodb 1:3.2.17-1 amd64 object/document-oriented database (metapackage) ii mongodb-clients 1:3.2.17-1 amd64 object/document-oriented database (client apps) ii mongodb-dev 1:2.4.10-5+deb8u1 amd64 object/document-oriented database (development) ii mongodb-server 1:3.2.17-1 amd64 object/document-oriented database (server package) # mongo --version MongoDB shell version: 3.2.17 Will try sometime with mongodb latest version 3.6 as well as I will get time. Installed libmongoc driver through sid repo "deb http://ftp.de.debian.org/debian sid main" # dpkg -l | grep libmongoc ii libmongoc-1.0-0 1.9.0+dfsg-1 amd64 MongoDB C client library - runtime files ii libmongoc-dev 1.9.0+dfsg-1 amd64 MongoDB C client library - dev files Kamailio compiled with # make cfg include_modules="utils app_lua json evapi db_mongodb ndb_mongodb" Created version collection in mongodb # cat /usr/local/src/kamailio/utils/kamctl/mongodb/kamailio/version-create.mongo | mongo Setup kamailio.cfg as per https://telerain.com/kamailio This time kamailio started successfully without any crash :) --------- Now its time to use mongodb for kamailio backend database. Used default kamailio configuration and defined mongodb connection as #!define DBURL "mongodb://localhost/kamailio" I enabled WITH_USRLOCDB [#!define WITH_USRLOCDB], kamailio refuse to start with errors. Jan 5 19:16:05 kmaongo kamailio: CRITICAL: <core> [core/cfg.y:3411]: yyerror_at(): parse error in config file /usr/local/etc/kamailio/kamailio.cfg, line 365, column 30-34: syntax error Jan 5 19:16:05 kmaongo kamailio: CRITICAL: <core> [core/cfg.y:3411]: yyerror_at(): parse error in config file /usr/local/etc/kamailio/kamailio.cfg, line 365, column 30-34: Invalid arguments Jan 5 19:16:05 kmaongo kamailio: CRITICAL: <core> [core/cfg.y:3414]: yyerror_at(): parse error in config file /usr/local/etc/kamailio/kamailio.cfg, line 365, column 35: Jan 5 19:16:05 kmaongo kamailio[28067]: ERROR: bad config file (3 errors) Jan 5 19:16:05 kmaongo kamailio: WARNING: <core> [core/ppcfg.c:221]: pp_ifdef_level_check(): different number of preprocessor directives: N(#!IF[N]DEF) - N(#!ENDIF) = 1 Jan 5 19:16:05 kmaongo kamailio: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Jan 5 19:16:05 kmaongo systemd[1]: kamailio.service: control process exited, code=exited status=255 [FAILJan 5 19:16:05 kmaongo systemd[1]: Failed to start Kamailio (OpenSER) - the Open Source SIP Server. Jan 5 19:16:05 kmaongo systemd[1]: Unit kamailio.service entered failed state. failed! Default userloc params in kamailio.cfg states line 365 as # ----- usrloc params ----- /* enable DB persistency for location entries */ #!ifdef WITH_USRLOCDB modparam("usrloc", "db_url", DBURL) [ << line 365 ] modparam("usrloc", "db_mode", 2) modparam("usrloc", "use_domain", MULTIDOMAIN) modparam("usrloc", "db_insert_null", 1) #!endif I created location collection as well but it didn't help. I experienced same error when I enable WITH_ACCDB or any other section that use DBURL. Whats the syntax error here? Is there any special configuration required to work with mongodb? How can we create collections using 'kamdbctl create' ? What else should I share for resolving the issue? -- regards, abdul basit On 4 January 2018 at 12:50, Abdul Basit &lt;basit.engg(a)gmail.com&gt; wrote: > > Hi Daniel, > > Thank you for your reply. > > I tested libmongoc with debian package available with repo. Then I moved > to compile from source. result was same. > > But i will try it again with fresh installation and will post here. > > -- > regards, > > abdul basit > > On 2 January 2018 at 16:38, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; > wrote: >> >> Hello, >> >> the issue happens inside mongoc library. Have you tried with the default >> one from Debian, without compiling it from sources? >> >> Cheers, >> Daniel >> >> >> On 01.01.18 16:51, Abdul Basit wrote: >> >> Hi, >> >> My last email was help for admin review. >> >> Sharing logs through pastebin https://pastebin.com/6HhdYLJK >> >> Interesting part will start from line 983 where it start initializing >> ndb_mongodb. >> >> >> >> On 1 January 2018 at 20:32, Abdul Basit &lt;basit.engg(a)gmail.com&gt; wrote: >>> >>> Hi, >>> >>> I enabled but kamailio didn't generate any dump file >>> To enable code dump followed >>> https://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles >>> >>> I also un-commented lines in /etc/init.d/kamailio script >>> >>> if test "$DUMP_CORE" = "yes" ; then >>> # set proper ulimit >>> ulimit -c unlimited >>> >>> # directory for the core dump files >>> COREDIR=/tmp/corefiles >>> [ -d $COREDIR ] || mkdir -p $COREDIR >>> chmod 777 $COREDIR >>> echo "$COREDIR/core.%e.sig%s.%p" > /proc/sys/kernel/core_pattern >>> fi >>> >>> >>> I think reason is that kamailio process didn't exit after crash and >>> keep on restarting. each time with new PID. >>> >>> However, debug file is attached. I hope this will help. >>> -- >>> regards, >>> >>> abdul basit >>> >>> On 29 December 2017 at 20:11, M S &lt;shaheryarkh(a)gmail.com&gt; wrote: >>>> >>>> Hi, >>>> >>>> Enable core dump and provide backtrace of sigfault. See this for more >>>> info, >>>> >>>> https://sourceware.org/gdb/onlinedocs/gdb/Backtrace.html >>>> >>>> Thank you. >>>> >>>> On Fri, Dec 29, 2017 at 12:25 PM, Abdul Basit &lt;basit.engg(a)gmail.com&gt; >>>> wrote: >>>>> >>>>> Adding further from >>>>> https://stackoverflow.com/questions/2549214/interpreting-segfault-messages >>>>> >>>>> error 6 means, fault is >>>>> >>>>> * bit 2 == 1 means kernel, 1 means user-mode >>>>> * bit 4 == 1 means fault was an instruction fetch >>>>> >>>>> That looks like kamailio is trying to invoke for db connection where >>>>> libmongo-c don't like to be. >>>>> -- >>>>> regards, >>>>> >>>>> abdul basit >>>>> >>>>> On 29 December 2017 at 15:39, Abdul Basit &lt;basit.engg(a)gmail.com&gt; >>>>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am trying to setup mongo db as database engine for kamailio as >>>>>> described at >>>>>> http://www.kamailio.org/wiki/tutorials/kamailio-and-mongodb >>>>>> and >>>>>> https://telerain.com/kamailio >>>>>> >>>>>> OS: Debian GNU/Linux 8.10 (jessie) >>>>>> MongoDB server version: 3.4.10 >>>>>> mongo-c-driver-1.9.0.tar.gz compiled with >>>>>> ./configure --disable-static --disable-silent-rules >>>>>> --enable-debug-symbols --enable-ssl=no --enable-sasl=yes >>>>>> --with-libbson=bundled --enable-optimizations >>>>>> --disable-automatic-init-and-cleanup >>>>>> >>>>>> version: kamailio 5.0.4 (x86_64/linux) >>>>>> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, >>>>>> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, >>>>>> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, >>>>>> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >>>>>> USE_DST_BLACKLIST, HAVE_RESOLV_RES >>>>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN >>>>>> 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >>>>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>>>> id: unknown >>>>>> compiled on 20:45:44 Dec 27 2017 with gcc 4.9.2 >>>>>> >>>>>> include_modules= utils app_lua db_postgres json evapi db_mongodb >>>>>> ndb_mongodb >>>>>> >>>>>> Kamailio is crashing with segfault as soon as it try to load >>>>>> libmongo-c >>>>>> >>>>>> Dec 28 04:44:14 sipserver systemd[1]: Starting Kamailio (OpenSER) - >>>>>> the Open Source SIP Server... >>>>>> Dec 28 04:44:14 sipserver kamailio: INFO: <core> >>>>>> [core/sctp_core.c:75]: sctp_core_check_support(): SCTP API not enabled - if >>>>>> you want to use it, load sctp module >>>>>> Dec 28 04:44:14 sipserver kamailio: INFO: <core> >>>>>> [core/tcp_main.c:4671]: init_tcp(): using epoll_lt as the io watch method >>>>>> (auto detected) >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: Listening on >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: udp: 127.0.0.1:5060 >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: udp: 192.168.10.20:5060 >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: tcp: 127.0.0.1:5060 >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: tcp: 192.168.10.20:5060 >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: Aliases: >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: tcp: localhost:5060 >>>>>> Dec 28 04:44:14 sipserver kamailio[11685]: udp: localhost:5060 >>>>>> Dec 28 04:44:14 sipserver /usr/local/sbin/kamailio[11687]: INFO: rr >>>>>> [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe >>>>>> module is not loaded >>>>>> ..... >>>>>> Dec 28 04:44:14 sipserver /usr/local/sbin/kamailio[11687]: INFO: >>>>>> <core> [core/udp_server.c:203]: probe_max_receive_buffer(): SO_RCVBUF is >>>>>> finally 425984 >>>>>> Dec 28 04:44:14 sipserver systemd[1]: Started Kamailio (OpenSER) - >>>>>> the Open Source SIP Server. >>>>>> Dec 28 04:44:14 sipserver kernel: [53109.390171] show_signal_msg: 20 >>>>>> callbacks suppressed >>>>>> Dec 28 04:44:14 sipserver kernel: [53109.390176] kamailio[11696] >>>>>> segfault at 78 ip 00007f690c0e6e98 sp 00007ffcde1a4770 error 6 in >>>>>> libmongoc-1.0.so.0.0.0[7f690c0ca000+6e000] >>>>>> Dec 28 04:44:14 sipserver kernel: [53109.390753] kamailio[11694] >>>>>> segfault at 78 ip 00007f690c0e6e98 sp 00007ffcde1a4770 error 6 in >>>>>> libmongoc-1.0.so.0.0.0[7f690c0ca000+6e000] >>>>>> Dec 28 04:44:14 sipserver kernel: [53109.393147] kamailio[11699] >>>>>> segfault at 38 ip 00007f690c0e6e98 sp 00007ffcde1a4770 error 6 in >>>>>> libmongoc-1.0.so.0.0.0[7f690c0ca000+6e000] >>>>>> Dec 28 04:44:14 sipserver /usr/local/sbin/kamailio[11687]: ALERT: >>>>>> <core> [main.c:743]: handle_sigs(): child process 11689 exited by a signal >>>>>> 11 >>>>>> Dec 28 04:44:14 sipserver /usr/local/sbin/kamailio[11687]: ALERT: >>>>>> <core> [main.c:746]: handle_sigs(): core was not generated >>>>>> Dec 28 04:44:14 sipserver /usr/local/sbin/kamailio[11687]: ALERT: >>>>>> <core> [main.c:743]: handle_sigs(): child process 11688 exited by a signal >>>>>> 11 >>>>>> ....... >>>>>> Dec 28 04:44:14 sipserver systemd[1]: kamailio.service: main process >>>>>> exited, code=exited, status=1/FAILURE >>>>>> Dec 28 04:44:14 sipserver systemd[1]: Unit kamailio.service entered >>>>>> failed state. >>>>>> >>>>>> >>>>>> As I tried googling the issue and found few issues with tls >>>>>> connection. >>>>>> I set in default kamailio.cfg but still facing the saem issue. >>>>>> >>>>>> enable_tls=no >>>>>> ... >>>>>> loadmodule "ndb_mongodb.so" >>>>>> modparam("ndb_mongodb", "server", >>>>>> "name=cgr91rc8;uri='mongodb://localhost/kamailio'") >>>>>> >>>>>> Its mongodb default installation and its accessible without any auth >>>>>> yet. >>>>>> >>>>>> Please suggest the way farward. >>>>>> >>>>>> -- >>>>>> regards, >>>>>> >>>>>> abdul basit