Hi im currently using freetds via unixodbc to connect to a sql azure db which is ok but suffers from various connection busy errors. I would like to use the ms odbc driver as it supports MARS (multiple active result sets) which should stop the connection busy errors, and numerous other sql server specific enhancements However despite the odbc driver working fine with the unixodbc cli tools kamailio crashes when I try and start the service.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501
Do you get a coredump file? If yes, paste here the backtrace taken with gdb.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-181085754
**Backtrace** ``` gdb kamailio /core GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...done. [New LWP 892] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -f /etc/kamailio/kamailio.cfg -DDDddEe'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000000000 in ?? () (gdb) backtrace #0 0x0000000000000000 in ?? () #1 0x00007fab4634134d in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #2 0x00007fab463419ca in lh_retrieve () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #3 0x00007fab46343ef1 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fab46344830 in ERR_get_state () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fab46344a6f in ERR_clear_error () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fab4665e1d0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 #7 0x00007fab46667b30 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 #8 0x00007fab3e081aea in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #9 0x00007fab3e08140c in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #10 0x00007fab3e077791 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #11 0x00007fab3e07864d in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #12 0x00007fab3e06251a in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #13 0x00007fab3e05d15e in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #14 0x00007fab3e05d9b2 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #15 0x00007fab3dfe253a in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #16 0x00007fab3e011246 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #17 0x00007fab3dfe4e1b in SQLDriverConnectW () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-11.0.so.2270.0 #18 0x00007fab45527978 in SQLDriverConnect () from /usr/lib/x86_64-linux-gnu/libodbc.so.2 #19 0x00007fab4577fff6 in db_unixodbc_new_connection (id=0x7fab4e2dccd8) at connection.c:155 #20 0x00007fab4850c456 in db_do_init2 (url=0x7fab482a2bd0 <db_url>, new_connection=0x7fab4577e922 <db_unixodbc_new_connection>, pooling=DB_POOLING_PERMITTED) at db.c:316 #21 0x00007fab4850b4c9 in db_do_init (url=0x7fab482a2bd0 <db_url>, new_connection=0x7fab4577e922 <db_unixodbc_new_connection>) at db.c:269 #22 0x00007fab45784aa0 in db_unixodbc_init (_url=0x7fab482a2bd0 <db_url>) at dbase.c:175 #23 0x00007fab48065f08 in acc_db_init_child (db_url=0x7fab482a2bd0 <db_url>) at acc.c:422 #24 0x00007fab4806f61a in child_init (rank=11) at acc_mod.c:711 #25 0x000000000051093d in init_mod_child (m=0x7fab4e0ba2d0, rank=11) at sr_module.c:915 #26 0x0000000000510678 in init_mod_child (m=0x7fab4e0bcb88, rank=11) at sr_module.c:912 #27 0x0000000000510678 in init_mod_child (m=0x7fab4e0be0a0, rank=11) at sr_module.c:912 #28 0x0000000000510678 in init_mod_child (m=0x7fab4e0be988, rank=11) at sr_module.c:912 #29 0x0000000000510678 in init_mod_child (m=0x7fab4e0bf4b8, rank=11) at sr_module.c:912 #30 0x0000000000510678 in init_mod_child (m=0x7fab4e0bffa0, rank=11) at sr_module.c:912 #31 0x0000000000510678 in init_mod_child (m=0x7fab4e0c09a8, rank=11) at sr_module.c:912 #32 0x0000000000510678 in init_mod_child (m=0x7fab4e0c1ec0, rank=11) at sr_module.c:912 #33 0x0000000000510678 in init_mod_child (m=0x7fab4e0c4a18, rank=11) at sr_module.c:912 #34 0x0000000000510678 in init_mod_child (m=0x7fab4e0c52f8, rank=11) at sr_module.c:912 #35 0x0000000000510678 in init_mod_child (m=0x7fab4e0c6348, rank=11) at sr_module.c:912 #36 0x0000000000510678 in init_mod_child (m=0x7fab4e0c6958, rank=11) at sr_module.c:912 #37 0x0000000000510678 in init_mod_child (m=0x7fab4e0c92e0, rank=11) at sr_module.c:912 #38 0x0000000000510678 in init_mod_child (m=0x7fab4e0c9bc8, rank=11) at sr_module.c:912 #39 0x0000000000510678 in init_mod_child (m=0x7fab4e0ca468, rank=11) at sr_module.c:912 #40 0x0000000000510678 in init_mod_child (m=0x7fab4e0caeb8, rank=11) at sr_module.c:912 #41 0x0000000000510678 in init_mod_child (m=0x7fab4e0cbf30, rank=11) at sr_module.c:912 #42 0x0000000000510678 in init_mod_child (m=0x7fab4e0cc828, rank=11) at sr_module.c:912 #43 0x0000000000510678 in init_mod_child (m=0x7fab4e0d2ce8, rank=11) at sr_module.c:912 #44 0x0000000000510678 in init_mod_child (m=0x7fab4e0d3260, rank=11) at sr_module.c:912 #45 0x0000000000510678 in init_mod_child (m=0x7fab4e0d4010, rank=11) at sr_module.c:912 #46 0x0000000000510678 in init_mod_child (m=0x7fab4e0d4a08, rank=11) at sr_module.c:912 #47 0x0000000000510678 in init_mod_child (m=0x7fab4e0d5020, rank=11) at sr_module.c:912 #48 0x0000000000510678 in init_mod_child (m=0x7fab4e0d5468, rank=11) at sr_module.c:912 #49 0x0000000000510678 in init_mod_child (m=0x7fab4e0d5d90, rank=11) at sr_module.c:912 #50 0x0000000000510678 in init_mod_child (m=0x7fab4e0d7658, rank=11) at sr_module.c:912 #51 0x0000000000510678 in init_mod_child (m=0x7fab4e0d7ca0, rank=11) at sr_module.c:912 #52 0x0000000000510678 in init_mod_child (m=0x7fab4e0d8040, rank=11) at sr_module.c:912 #53 0x0000000000510678 in init_mod_child (m=0x7fab4e0d8488, rank=11) at sr_module.c:912 #54 0x0000000000510678 in init_mod_child (m=0x7fab4e0d8ac8, rank=11) at sr_module.c:912 #55 0x0000000000510678 in init_mod_child (m=0x7fab4e0d9168, rank=11) at sr_module.c:912 #56 0x0000000000510678 in init_mod_child (m=0x7fab4e0d9600, rank=11) at sr_module.c:912 #57 0x0000000000510678 in init_mod_child (m=0x7fab4e0d9a50, rank=11) at sr_module.c:912 #58 0x0000000000510678 in init_mod_child (m=0x7fab4e0da110, rank=11) at sr_module.c:912 #59 0x0000000000510c44 in init_child (rank=11) at sr_module.c:941 #60 0x00000000004907f1 in fork_process (child_id=11, desc=0x7ffd3d895c60 "udp receiver child=2 sock=127.0.0.1:5050", make_sock=1) at pt.c:327 #61 0x0000000000501672 in main_loop () at main.c:1567 #62 0x00000000005083c3 in main (argc=4, argv=0x7ffd3d896008) at main.c:2553 (gdb) ```
**Local OS Info** ``` lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.2 (jessie) Release: 8.2 Codename: jessie
uname -a Linux kam01 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u2 (2016-01-02) x86_64 GNU/Linux ```
**Reproduction Steps** wget https://download.microsoft.com/download/B/C/D/BCDD264C-7517-4B7D-8159-C99FC5... extract the drivers clone https://github.com/Andrewpk/Microsoft--SQL-Server--ODBC-Driver-1.0-for-Linux... Copy the install scripts you just cloned to the unpacked msodbc driver folder Run `build_dm.sh` Run `install.sh install` Configure unixodbc dsn for a sql server using MS ODBC Driver i.e. you have an entry like this in /etc/odbc.ini [sqlserver.domain.name] Driver = ODBC Driver 11 for SQL Server ServerName = sqlserver.domain.name Database = kamdb UID = kamailio_user PWD = password TDS_Version = 7.3 MARS_Connection = Yes Configure kamailio to use that unixodbc dsn to connect to
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-181387891
Can you do in gdb:
``` frame 19 p *id p ptr p *ptr p con_str info locals ```
Is tls module of kamailio loaded?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-181417562
TLS module is loaded, and when i issue the commands you asked for above it displays the connection string, which i cant really share publically. I have also rebuilt the build box in case it was an issue of conflicting versions. Behavious is slightly different now rather than an outright crash the process now hangs instead, and can only be stopped by manually killing the process i.e. pkill -3 -e kamailio. I have had to disable certain modules now such as corex and sdpops to get this far other wise it hangs slightly earlier in the boot process., with the backtrace suggesting something is up with the daemonization of the process. If i switch to freetds driver in the odbc.ini file then everything works as expected.
I have increased the debug log level to 8 in the hope of spotting additional detail, and can confirm that some db activity takes place with the msodbc driver i.e. the dispatcher table is read etc.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-181845883
You can always replace the sensitive information with generic tokens such as USER, HOST, PASSWORD, IP1, etc... I am more interested where is invalid data rather than valid values.
Can you try without tls module?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-182054713
Just disabled the TLS module and its working, unfortunatly TLS is a requirement i cannot live without... I wonder if this could be caused by a symlink that has to be created to get this version of the driver to load, as its compiled for redhat (further details here: http://onefinepub.com/2013/03/ms-sql-odbc-ubuntu/ step 6,) `sudo ln -s /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 /usr/lib/x86_64-linux-gnu/libcrypto.so.10; sudo ln -s /lib/x86_64-linux-gnu/libssl.so.1.0.0 /usr/lib/x86_64-linux-gnu/libssl.so.10;`
There is a preview of v13 of the driver compiled for ubuntu, shall give that a try and see if it resolves the issue.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-182303807
Same issue with the ubuntu compiled v13 driver, everything works fine with the tls module disabled, hangs when its enabled.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-182319382
Latest backtrace, made using the ubuntu compiled version of the driver, as frame 19 is the same as previous backtrace included the requested info below.
``` gdb kamailio /core GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...done. [New LWP 23380] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -f /etc/kamailio/kamailio.cfg -DDddEe'. Program terminated with signal SIGQUIT, Quit. #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 38 ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory. (gdb) bt #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 #1 0x00007fc3b4f5e423 in futex_get (lock=0x7fc3b79e05a4) at ../../futexlock.h:121 #2 0x00007fc3b4f5f98c in locking_f (mode=9, n=1, file=0x7fc3b4a69c9b "err.c", line=446) at tls_locking.c:107 #3 0x00007fc3b49e39c3 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #4 0x00007fc3b49e3ecd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #5 0x00007fc3b49e4840 in ERR_get_state () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #6 0x00007fc3b49e4a7f in ERR_clear_error () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 #7 0x00007fc3b4cfe270 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 #8 0x00007fc3ac2e3282 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #9 0x00007fc3ac2e44cc in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #10 0x00007fc3ac2d95b8 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #11 0x00007fc3ac2d9c9f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #12 0x00007fc3ac2b711f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #13 0x00007fc3ac2b502f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #14 0x00007fc3ac2b59a2 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #15 0x00007fc3ac22fea1 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #16 0x00007fc3ac263de6 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #17 0x00007fc3ac232c66 in SQLDriverConnectW () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 #18 0x00007fc3b3bc7abd in SQLDriverConnect (hdbc=0x22df030, hwnd=0x0, conn_str_in=0x23006e0 "D", len_conn_str_in=-1, conn_str_out=0x7fffeeef1930 "`\031\357\356\377\177", conn_str_out_max=0, ptr_conn_str_out=0x7fffeeef192e, driver_completion=1) at SQLDriverConnect.c:1490 #19 0x00007fc3b3e1f016 in db_unixodbc_new_connection (id=0x7fc3bca2ca08) at connection.c:155 #20 0x00007fc3b6bac456 in db_do_init2 (url=0x7fc3b7011e10 <db_url>, new_connection=0x7fc3b3e1d942 <db_unixodbc_new_connection>, pooling=DB_POOLING_PERMITTED) at db.c:316 #21 0x00007fc3b6bab4c9 in db_do_init (url=0x7fc3b7011e10 <db_url>, new_connection=0x7fc3b3e1d942 <db_unixodbc_new_connection>) at db.c:269 #22 0x00007fc3b3e24749 in db_unixodbc_init (_url=0x7fc3b7011e10 <db_url>) at dbase.c:175 #23 0x00007fc3b6dea471 in mi_child_init () at ul_mod.c:459 #24 0x00007fc3bc2e78a0 in init_mi_child (rank=-128, mode=1) at mi.c:118 #25 0x00007fc3bb9b0035 in fifo_process (rank=1) at mi_fifo.c:255 #26 0x00007fc3bb9afa0d in mi_child_init (rank=0) at mi_fifo.c:234 #27 0x000000000051093d in init_mod_child (m=0x7fc3bc755a20, rank=0) at sr_module.c:915 #28 0x0000000000510678 in init_mod_child (m=0x7fc3bc756008, rank=0) at sr_module.c:912 #29 0x0000000000510678 in init_mod_child (m=0x7fc3bc758990, rank=0) at sr_module.c:912 #30 0x0000000000510678 in init_mod_child (m=0x7fc3bc759690, rank=0) at sr_module.c:912 #31 0x0000000000510678 in init_mod_child (m=0x7fc3bc759c98, rank=0) at sr_module.c:912 #32 0x0000000000510678 in init_mod_child (m=0x7fc3bc75aae8, rank=0) at sr_module.c:912 #33 0x0000000000510678 in init_mod_child (m=0x7fc3bc75b5c8, rank=0) at sr_module.c:912 #34 0x0000000000510678 in init_mod_child (m=0x7fc3bc75c4a0, rank=0) at sr_module.c:912 #35 0x0000000000510678 in init_mod_child (m=0x7fc3bc75ed58, rank=0) at sr_module.c:912 #36 0x0000000000510678 in init_mod_child (m=0x7fc3bc760270, rank=0) at sr_module.c:912 #37 0x0000000000510678 in init_mod_child (m=0x7fc3bc760b58, rank=0) at sr_module.c:912 #38 0x0000000000510678 in init_mod_child (m=0x7fc3bc761688, rank=0) at sr_module.c:912 #39 0x0000000000510678 in init_mod_child (m=0x7fc3bc762170, rank=0) at sr_module.c:912 #40 0x0000000000510678 in init_mod_child (m=0x7fc3bc762b78, rank=0) at sr_module.c:912 #41 0x0000000000510678 in init_mod_child (m=0x7fc3bc764090, rank=0) at sr_module.c:912 #42 0x0000000000510678 in init_mod_child (m=0x7fc3bc766be8, rank=0) at sr_module.c:912 #43 0x0000000000510678 in init_mod_child (m=0x7fc3bc7674c8, rank=0) at sr_module.c:912 #44 0x0000000000510678 in init_mod_child (m=0x7fc3bc768518, rank=0) at sr_module.c:912 #45 0x0000000000510678 in init_mod_child (m=0x7fc3bc768b28, rank=0) at sr_module.c:912 #46 0x0000000000510678 in init_mod_child (m=0x7fc3bc76b7c0, rank=0) at sr_module.c:912 #47 0x0000000000510678 in init_mod_child (m=0x7fc3bc76c0a8, rank=0) at sr_module.c:912 #48 0x0000000000510678 in init_mod_child (m=0x7fc3bc76c948, rank=0) at sr_module.c:912 #49 0x0000000000510678 in init_mod_child (m=0x7fc3bc76d398, rank=0) at sr_module.c:912 #50 0x0000000000510678 in init_mod_child (m=0x7fc3bc76e410, rank=0) at sr_module.c:912 #51 0x0000000000510678 in init_mod_child (m=0x7fc3bc76ed08, rank=0) at sr_module.c:912 #52 0x0000000000510678 in init_mod_child (m=0x7fc3bc7751c8, rank=0) at sr_module.c:912 #53 0x0000000000510678 in init_mod_child (m=0x7fc3bc775740, rank=0) at sr_module.c:912 #54 0x0000000000510678 in init_mod_child (m=0x7fc3bc7764f0, rank=0) at sr_module.c:912 #55 0x0000000000510678 in init_mod_child (m=0x7fc3bc776ee8, rank=0) at sr_module.c:912 #56 0x0000000000510678 in init_mod_child (m=0x7fc3bc777500, rank=0) at sr_module.c:912 #57 0x0000000000510678 in init_mod_child (m=0x7fc3bc777948, rank=0) at sr_module.c:912 #58 0x0000000000510678 in init_mod_child (m=0x7fc3bc778270, rank=0) at sr_module.c:912 #59 0x0000000000510678 in init_mod_child (m=0x7fc3bc779b38, rank=0) at sr_module.c:912 #60 0x0000000000510678 in init_mod_child (m=0x7fc3bc77a180, rank=0) at sr_module.c:912 #61 0x0000000000510678 in init_mod_child (m=0x7fc3bc77a520, rank=0) at sr_module.c:912 #62 0x0000000000510678 in init_mod_child (m=0x7fc3bc77a968, rank=0) at sr_module.c:912 #63 0x0000000000510678 in init_mod_child (m=0x7fc3bc77afa8, rank=0) at sr_module.c:912 #64 0x0000000000510678 in init_mod_child (m=0x7fc3bc77b648, rank=0) at sr_module.c:912 #65 0x0000000000510678 in init_mod_child (m=0x7fc3bc77bae0, rank=0) at sr_module.c:912 #66 0x0000000000510678 in init_mod_child (m=0x7fc3bc77bf30, rank=0) at sr_module.c:912 #67 0x0000000000510678 in init_mod_child (m=0x7fc3bc77c5f0, rank=0) at sr_module.c:912 #68 0x0000000000510678 in init_mod_child (m=0x7fc3bc77ccd8, rank=0) at sr_module.c:912 #69 0x0000000000510c44 in init_child (rank=0) at sr_module.c:941 #70 0x0000000000502258 in main_loop () at main.c:1654 #71 0x00000000005083c3 in main (argc=4, argv=0x7fffeeef3088) at main.c:2553 (gdb) frame 19 #19 0x00007fc3b3e1f016 in db_unixodbc_new_connection (id=0x7fc3bca2ca08) at connection.c:155 155 ret = SQLDriverConnect(ptr->dbc, NULL, (SQLCHAR*)conn_str, SQL_NTS, (gdb) p *id $1 = {url = {s = 0x7fc3bca2ca50 "unixodbc://username:password@localhost/dsn", len = 72}, scheme = 0x7fc3bc905be8 "unixodbc", username = 0x7fc3bc905b78 "username", password = 0x7fc3bc77d950 "password", host = 0x7fc3bc77df90 "localhost", port = 0, database = 0x7fc3bc7ea410 "dsn", pid = 23380, poolid = 0} (gdb) p ptr $2 = (struct my_con *) 0x7fc3bc77e380 (gdb) p *ptr $3 = {id = 0x0, ref = 1, next = 0x0, env = 0x22dea40, stmt_handle = 0x0, dbc = 0x22df030, row = 0x0, timestamp = 0} (gdb) p con_str No symbol "con_str" in current context. (gdb) info locals outstr = "`\031\357\356\377\177\000\000\\204\365\264\303\177\000\000\210\000\000\000\000\000\000\000(\005\236\267\303\177\000\000\021", '\000' <repeats 11 times>, "\001\000\000\000\240\031\357\356\377\177\000\000\a\205\365\264\303\177\000\000\212\306\371\264\303\177\000\000\212\306\371\264\303\177\000\000\360\230\371\264\303\177\000\000\022\001\000\000\210\000\000\000\320\031\357\356\001\000\000\000 mF\270\303\177\000\000\300\031\357\356\377\177\000\000\253\205\365\264\303\177\000\000\030\000\000\000\000\000\000\000\210\000\000\000\000\000\000\000\344\300\245\264\303\177\000\000\370\201\225\264\303\177\000\000\020\032\357\356\377\177\000\000\250lF\270\303\177\000\000\250lF\270\303\177\000\000\200\000\000\000\000\000\000\000\021\000\000\000\000\000\000\000"... outstrlen = 0 ret = 0 ptr = 0x7fc3bc77e380 conn_str = "DSN=dsn;UID=username;PWD=password;\000_A\000\000\000\000\000\200\060\357\356\377\177", '\000' <repeats 19 times>, "\022\357\356\377\177\000\000\276\236A\000\000\000\000\000@_A\000\000\000\000\000H\022\357\356\377\177\000\000L\022\357\356\377\177\000\000\000\000\000\000\000\000\000\000\310Û\274\001", '\000' <repeats 19 times>, "\001\000\000\000\000\000\000\000\320͓\274\303\177\000\000h舼\303\177\000\000`\022\357\356\377\177\000\000z\235A", '\000' <repeats 13 times>... __FUNCTION__ = "db_unixodbc_new_connection" ```
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-182322646
It appears to be a deadlock -- somewhere, a component does two times lock on a mutex set for libcrypto.
I added some debug messages in master branch, can you test with it? First set debug=3 in config file, then start and make the log message from syslog available somewhere (you can mask sensitive values: ip, passwords, etc...) -- here or some website/pastebin.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-183064112
Sure no worries will post back the results, i have also tested changing the threading model used by UNIXODBC which has had no impact on the issue, so will see what the latest master shows :)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-183253915
Log file is too big for pastebin (8.5Mb) so i have attached it to this post [kam-tls-debugging.log.txt](https://github.com/kamailio/kamailio/files/127835/kam-tls-debugging.log.txt)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-183267539
You need to recreate the active watcher table and update its version number in 'version' table -- there is an error:
``` Feb 12 10:26:01 vs-kam-uat01 /usr/local/sbin/kamailio[35513]: ERROR: <core> [db.c:439]: db_check_table_version(): invalid version 11 for table active_watchers found, expected 12 (check table structure and table "version") ```
Look at utils/kamctl/mysql/presence-create.sql
They try again, right now the problem above is the reason for not starting.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-183633160
Doh, silly mistake to make, never mind, have rerun with updated table definitions, same behavior, redacted log is attached in a zip as it was over 15Mb uncompressed.
[kam-tls-debugging.log.13-02-16.zip](https://github.com/kamailio/kamailio/files/129144/kam-tls-debugging.log.13-0...)
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-183662367
Is there any additional analysis i can do on the log files to try and narrow down where the problem lies?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-187668002
Keen to resolve this issue, is there anything else i can do my end to try and trackdown where the deadlock is occurring?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-196551712
Got distracted by traveling and other tasks...
The last logs point to a crash in a kamailio process:
``` Feb 13 12:57:46 vs-kam-uat01 /usr/local/sbin/kamailio[51225]: ALERT: <core> [main.c:739]: handle_sigs(): child process 51256 exited by a signal 11 ```
Can you do 'ulimit -c unlimited' before starting Kamailio as root? It should generate a coredump file from where you have to get the bracktrace using gdb command. That will reveal the stack of execution till the crash happens.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-196765061
Ok, i have pulled the latest commits from master and rebuilt. Here is the backtrace, it appears that latest master no longer emits lock tracing information when debug is set to level 3. No errors or alerts are logged either.
Im starting kamailio as root with the following command: `kamailio -f /etc/kamailio/kamailio.cfg -DDddEe` Then waiting 20 seconds after it freezes during the TLS info part of boot and atempting to stop kamailio by pressing `Ctrl + C` which emits a bunch of signal 2 received info messages and hangs until i run `pkill -3 -e kamailio` in annother terminal. 2 Core files are being created, one with the time stamp of when the process crashes, with a PID of the process marked as Defunct when i issues an `ps aux`, with the second created with the PID of main kamailio process and timestamp of when i pkill kamailio.
This is BT of the process marked as Defunct ``` GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...done. [New LWP 14214] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -f /etc/kamailio/kamailio.cfg -DDddEe -w /root/kamcore/16032016/'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x0000000000000040 in ?? () (gdb) set pagination off (gdb) bt full #0 0x0000000000000040 in ?? () No symbol table info available. #1 0x00007fbc5741135d in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #2 0x00007fbc574119da in lh_retrieve () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #3 0x00007fbc57413f01 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #4 0x00007fbc57414840 in ERR_get_state () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #5 0x00007fbc574150e9 in ERR_set_mark () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #6 0x00007fbc5740354c in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #7 0x00007fbc573f3685 in RSA_new_method () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #8 0x00007fbc573f673d in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #9 0x00007fbc574327ca in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #10 0x00007fbc574353d1 in ASN1_item_ex_d2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #11 0x00007fbc57435a0b in ASN1_item_d2i () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #12 0x00007fbc573f7bcf in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #13 0x00007fbc5742d4e7 in X509_PUBKEY_get () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #14 0x00007fbc57449dba in X509_get_pubkey_parameters () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #15 0x00007fbc57449fde in X509_verify_cert () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #16 0x00007fbc5774d368 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #17 0x00007fbc5772a423 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #18 0x00007fbc5772eba4 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #19 0x00007fbc57737bd0 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #20 0x00007fbc4ecc8282 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #21 0x00007fbc4ecc94cc in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #22 0x00007fbc4ecbe5b8 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #23 0x00007fbc4ecbec9f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #24 0x00007fbc4ec9c11f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #25 0x00007fbc4ec9a02f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #26 0x00007fbc4ec9a9a2 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #27 0x00007fbc4ec14ea1 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #28 0x00007fbc4ec48de6 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #29 0x00007fbc4ec17c66 in SQLDriverConnectW () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #30 0x00007fbc565deabd in SQLDriverConnect (hdbc=0x1121920, hwnd=0x7ffd06240460, conn_str_in=0x1142cb0 "D", len_conn_str_in=25892, conn_str_out=0x7ffd0624bf60 "\001\200\255\373\274\177", conn_str_out_max=0, ptr_conn_str_out=0x7ffd0624bf5e, driver_completion=1) at SQLDriverConnect.c:1490 uc_conn_str_in = 0x1142cb0 s2 = '\000' <repeats 127 times> con_struct = {count = 0, list = 0x111f110} driver = 0x1132cc0 "\001" dsn = 0x1132cc0 "\001" filedsn = 0xfffffffd <error: Cannot access memory at address 0xfffffffd> savefile = "\000S\021\001\000\000\000\000\320\357\016\001\000\000\000\000\260S$\006\375\177\000\000\254S$\006\375\177\000\000\003\000\000\000\000\000\000\000\376\334`V\274\177\000\000\000\004\000\000\000\000\000\000\254S$\006\375\177\000\000\300S$\006\375\177\000\000@\247\024\001\000\000\000\000\250S$\006\375\177\000\000\300W$\006\375\177\000\000\001\000\000\000\000\000\000\000x\000\000\000\000\000\000\000\200\351\026\001\000\000\000\000HZ\021\001\000\000\000\000PZ\021\001\000\000\000\000P[\021\001", '\000' <repeats 12 times>, "G\026\000\000\000\000\000\000\060\000\061\000\060\000\060\000\060\000\000\000\000\000\000\000[\000M\000i\000c\000r\000o\000s\000o\000f\000t\000]\000[\000"... lib_name = "/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0\000\000\000\000\000\000\000\000\000\314\351\016\001\000\000\000\000\320\357\016\001\000\000\000\000\360\357\016\001", '\000' <repeats 20 times>, " \240\025\001\000\000\000\000\300\351\016\001\000\000\000\000\021\273`V\274\177\000\000\020\232\025\001\000\000\000\000\300\351\016\001\000\000\000\000\310Y\021\001\000\000\000\000y\274`V\274\177\000\000\306\$\006\375\177\000\000\310Y\021\001\000\000\000\000X\241\025\001", '\000' <repeats 12 times>, "[\000M\000i\000c\000"... driver_name = "ODBC Driver 13 for SQL Server", '\000' <repeats 11 times>, "PY\366N\274\177\000\000 \$\006\375\177\000\000v\216\025\001\000\000\000\000\260\215\025\001\000\000\000\000\310[$\006\375\177\000\000\300\307\021\001\000\000\000\000\212\216\231_\274\177", '\000' <repeats 26 times>, "\320\307\021\001\000\000\000\000(\$\006\375\177\000\000\001", '\000' <repeats 15 times>, "(\$\006\375\177\000\000 \$\006\375\177\000\000\020\$\006\375\177\000\000\260\215\025\001\000\000\000\000\030\$\006\375\177\000\000@\311$\006\375\177\000\000"... s1 = '\000' <repeats 128 times>, "/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0", '\000' <repeats 329 times>... local_conn_str_in = '\000' <repeats 2047 times> local_out_conection = '\000' <repeats 2047 times> warnings = 0 #31 0x00007fbc56836060 in db_unixodbc_new_connection (id=0x7fbc5f457488) at connection.c:155 outstr = "\001\200\255\373\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\023C\027_\274\177\000\000\037C\027_\274\177\000\000\005C\027_\274\177\000\000\037C\027_\274\177", '\000' <repeats 42 times>, "\300\202\350Z\000\000\000\000\240\336nW\274\177\000\000\000\000nW\274\177\000\000\000\000\000\000\000\000\000\000\300\202\350Z\274\177\000\000\376\335BW\274\177\000\000\377\377\377\061\064\062\061\064P\300$\006\375\177\000\000P\333AZ\274\177\000\000P\300$\006\375\177\000\000\226\201\037Z\274\177\000\000"... outstrlen = 0 ret = 0 ptr = 0x7fbc5f1c12a8 conn_str = "DSN=db.domain.tld;UID=username;PWD=password;\000\000\000\000\000\000\000\000v\237\370E\000\000\000\000\063\nv\000\000\000\000\000 \332$\006\375\177", '\000' <repeats 18 times>, "\340\270$\006\375\177\000\000B\004C\000\000\000\000\000p\270$\006\375\177\000\000H\271$\006\375\177\000\000L\271$\006\375\177\000\000P\271$\006\375\177\000\000h\247A_\001\000\000\000ЮA_\274\177\000\000\314Cq\000\000\000\000\000\000\000\000\000\375\177\000\000ܸ$\006\375\177\000\000v\237"... __FUNCTION__ = "db_unixodbc_new_connection" #32 0x00007fbc5960f2ef in db_do_init2 (url=0x7fbc59a77e10 <db_url>, new_connection=0x7fbc56834881 <db_unixodbc_new_connection>, pooling=DB_POOLING_PERMITTED) at db.c:316 id = 0x7fbc5f457488 con = 0x0 res = 0x7fbc5f45fdb0 con_size = 32 __FUNCTION__ = "db_do_init2" #33 0x00007fbc5960e298 in db_do_init (url=0x7fbc59a77e10 <db_url>, new_connection=0x7fbc56834881 <db_unixodbc_new_connection>) at db.c:269 No locals. #34 0x00007fbc5683aef6 in db_unixodbc_init (_url=0x7fbc59a77e10 <db_url>) at dbase.c:175 No locals. #35 0x00007fbc5984f387 in child_init (_rank=-1) at ul_mod.c:441 ptr = 0x7ffd0624c560 i = 32765 __FUNCTION__ = "child_init" #36 0x000000000051da2d in init_mod_child (m=0x7fbc5f1a0dc0, rank=-1) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #37 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a1750, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #38 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a2510, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #39 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a4948, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #40 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a5c08, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #41 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a63e8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #42 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a6e98, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #43 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a7848, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #44 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a8148, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #45 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a9420, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #46 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ab9d8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #47 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ac1b0, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #48 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ad020, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #49 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ad630, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #50 0x000000000051d74b in init_mod_child (m=0x7fbc5f1afde8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #51 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b05c8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #52 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b0d60, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #53 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b16a8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #54 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b25b8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #55 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b2db0, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #56 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b8a00, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #57 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b8ee8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #58 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b9c90, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #59 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ba6e8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #60 0x000000000051d74b in init_mod_child (m=0x7fbc5f1badf8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #61 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bb1c8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #62 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bb9d0, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #63 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd0a0, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #64 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd628, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #65 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd950, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #66 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bdd20, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #67 0x000000000051d74b in init_mod_child (m=0x7fbc5f1be2e8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #68 0x000000000051d74b in init_mod_child (m=0x7fbc5f1be8f8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #69 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bed00, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #70 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bf0d8, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #71 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bf778, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #72 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bfe20, rank=-1) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #73 0x000000000051dd5b in init_child (rank=-1) at sr_module.c:947 No locals. #74 0x0000000000495abd in fork_process (child_id=-1, desc=0x74ca8d "slow timer", make_sock=1) at pt.c:327 pid = 0 child_process_no = 29 ret = -1 new_seed1 = 1228121956 new_seed2 = 471266841 sockfd = {54, 55} __FUNCTION__ = "fork_process" #75 0x000000000050dd99 in main_loop () at main.c:1647 i = 4 pid = 14213 si = 0x0 si_desc = "udp receiver child=3 sock=192.168.1.150:5060\000(185.29.208.21:5060)\000AZ\274\177\000\000\232\246d\000\001\000\000\000\270\332tZ\001\000\000\000\300dBZ\274\177\000\000\300\326$\006\375\177\000\000()uZ\274\177\000\000P\t\235W\274\177\000\000P\257,_\274\177\000" nrprocs = 4 woneinit = 1 __FUNCTION__ = "main_loop" #76 0x0000000000514e9d in main (argc=6, argv=0x7ffd0624da28) at main.c:2616 cfg_stream = 0x1047010 c = -1 r = 0 tmp = 0x7fbc6014573d <_dl_lookup_symbol_x+349> "\203", <incomplete sequence \370> tmp_len = 0 port = 1 proto = 32700 options = 0x74d640 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 2268810091 rfd = 4 debug_save = 2 debug_flag = 1 dont_fork_cnt = 2 n_lst = 0x7ffd0624d8e0 p = 0x7ffd0624da60 "q\356$\006\375\177" st = {st_dev = 15, st_ino = 12804, st_nlink = 2, st_mode = 16895, st_uid = 109, st_gid = 114, __pad0 = 0, st_rdev = 0, st_size = 100, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1458128144, tv_nsec = 692000000}, st_mtim = {tv_sec = 1458138912, tv_nsec = 180254641}, st_ctim = {tv_sec = 1458138912, tv_nsec = 180254641}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" ``` This is the BT from main kamailio process created by pkilling it ``` root@vs-kam-uat01:/# gdb kamailio core.kamailio.14214.vs-kam-uat01.1458139004 GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from kamailio...done. [New LWP 14185] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -f /etc/kamailio/kamailio.cfg -DDddEe -w /root/kamcore/16032016/'. Program terminated with signal SIGQUIT, Quit. #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 38 ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory. (gdb) set pagination off (gdb) bt full #0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 No locals. #1 0x00007fbc5798f5e9 in futex_get (lock=0x7fbc5a426524) at ../../futexlock.h:108 v = 2 i = 1024 #2 0x00007fbc57990c7e in locking_f (mode=9, n=1, file=0x7fbc57499c9b "err.c", line=446) at tls_locking.c:110 __FUNCTION__ = "locking_f" #3 0x00007fbc574139c3 in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #4 0x00007fbc57413ecd in ?? () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #5 0x00007fbc57414840 in ERR_get_state () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #6 0x00007fbc57414a7f in ERR_clear_error () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 No symbol table info available. #7 0x00007fbc577374b5 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 No symbol table info available. #8 0x00007fbc4ecc8282 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #9 0x00007fbc4ecc94cc in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #10 0x00007fbc4ecbe5b8 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #11 0x00007fbc4ecbec9f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #12 0x00007fbc4ec9c11f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #13 0x00007fbc4ec9a02f in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #14 0x00007fbc4ec9a9a2 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #15 0x00007fbc4ec14ea1 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #16 0x00007fbc4ec48de6 in ?? () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #17 0x00007fbc4ec17c66 in SQLDriverConnectW () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #18 0x00007fbc565deabd in SQLDriverConnect (hdbc=0x1121920, hwnd=0x0, conn_str_in=0x1142cb0 "D", len_conn_str_in=-1, conn_str_out=0x7ffd0624c000 "\001\200\255\373\274\177", conn_str_out_max=0, ptr_conn_str_out=0x7ffd0624bffe, driver_completion=1) at SQLDriverConnect.c:1490 uc_conn_str_in = 0x1142cb0 s2 = '\000' <repeats 127 times> con_struct = {count = 0, list = 0x111f110} driver = 0x1132cc0 "\001" dsn = 0x1132cc0 "\001" filedsn = 0xfffffffd <error: Cannot access memory at address 0xfffffffd> savefile = "\000\000\061\000\060\000\060\000\060\000\000\000\000\000\000\000[\000M\000i\000c\000r\000o\000s\000o\000f\000t\000]\000[\000O\000D\000B\000C\000 \000D\000r\000i\000v\000e\000r\000 \000\061\000\063\000 \000f\000o\000r\000 \000S\000Q\000L\000 \000S\000e\000r\000v\000e\000r\000]\000[\000S\000Q\000L\000 \000S\000e\000r\000v\000e\000r\000]\000C\000h\000a\000n\000g\000e\000d\000 \000l\000a\000n\000g\000u\000a\000g\000e\000 \000s\000e\000t\000t\000i\000n\000g\000 \000t\000o\000 \000u\000s\000_\000e\000n\000g\000l\000i\000s\000h\000"... lib_name = "/opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0\000O\000D\000B\000C\000 \000D\000r\000i\000v\000e\000r\000 \000\061\000\063\000 \000f\000o\000r\000 \000S\000Q\000L\000 \000S\000e\000r\000v\000e\000r\000]\000[\000S\000Q\000L\000 \000S\000e\000r\000v\000e\000r\000]\000C\000h\000a\000n\000g\000e\000d\000 \000l\000a\000n\000g\000u\000a\000g\000e\000 \000s\000e\000t\000t\000i\000n\000g\000 \000t\000o\000 \000u\000s\000"... driver_name = "ODBC Driver 13 for SQL Server\000\000\000@\311$\006\375\177\000\000)\204\231_\274\177", '\000' <repeats 26 times>, "@\311$\006\375\177\000\000c\000\000\000\000\000\000\000\260\215\025\001\000\000\000\000\000\000\000\000\001\000\000\000\360S\021\001\000\000\000\000\260\215\025\001\000\000\000\000\001\000\000\000\000\000\000\000\360S\021\001\000\000\000\000\002\000\000\000\000\000\000\000@\311$\006\375\177\000\000\300\200$\006\375\177\000\000\260\215\025\001\000\000\000\000\261\353`V\274\177\000\000\360S\021\001\000\000\000\000\314\$\006\375\177\000\000\320\$\006\375\177\000\000"... s1 = "ibmsodbcsql-13.0.so.0.0", '\000' <repeats 329 times>... local_conn_str_in = '\000' <repeats 2047 times> local_out_conection = '\000' <repeats 2047 times> warnings = 0 #19 0x00007fbc56836060 in db_unixodbc_new_connection (id=0x7fbc5f457488) at connection.c:155 outstr = "\001\200\255\373\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\005C\027_\274\177\000\000\023C\027_\274\177\000\000\037C\027_\274\177\000\000\005C\027_\274\177\000\000\037C\027_\274\177", '\000' <repeats 42 times>, "\300\202\350Z\000\000\000\000\022JCW\274\177\000\000\000\000$\006\375\177\000\000\000\000\000\000\000\000\000\000h\200\350Z\274\177\000\000\000\340AZ\274\177\000\000\377\377\377\061\064\061\070\065\360\300$\006\375\177\000\000P\333AZ\274\177\000\000\360\300$\006\375\177\000\000\226\201\037Z\274\177\000\000"... outstrlen = 0 ret = 0 ptr = 0x7fbc5f1c12a8 conn_str = "DSN=db.domain.tld;UID=username;PWD=password;", '\000' <repeats 16 times>, "\340\270$\006\375\177\000\000R\231A\000\000\000\000\000\063\nv\000\000\000\000\000 \332$\006\375\177\000\000\000\000\000\000\000\000\000\000\b\023>_\274\177\000\000p\300$\006\375\177\000\000H\271$\006\375\177\000\000L\271$\006\031\000\000\000\220\222B_\274\177", '\000' <repeats 18 times>, "\220\237A_\274\177\000\000h\247A_\274\177\000\000\340\270$\006\375\177\000\000"... __FUNCTION__ = "db_unixodbc_new_connection" #20 0x00007fbc5960f2ef in db_do_init2 (url=0x7fbc59a77e10 <db_url>, new_connection=0x7fbc56834881 <db_unixodbc_new_connection>, pooling=DB_POOLING_PERMITTED) at db.c:316 id = 0x7fbc5f457488 con = 0x0 res = 0x7fbc5f45fdb0 con_size = 32 __FUNCTION__ = "db_do_init2" #21 0x00007fbc5960e298 in db_do_init (url=0x7fbc59a77e10 <db_url>, new_connection=0x7fbc56834881 <db_unixodbc_new_connection>) at db.c:269 No locals. #22 0x00007fbc5683aef6 in db_unixodbc_init (_url=0x7fbc59a77e10 <db_url>) at dbase.c:175 No locals. #23 0x00007fbc5984f387 in child_init (_rank=0) at ul_mod.c:441 ptr = 0x7fbc57435ca2 i = 32765 __FUNCTION__ = "child_init" #24 0x000000000051da2d in init_mod_child (m=0x7fbc5f1a0dc0, rank=0) at sr_module.c:921 __FUNCTION__ = "init_mod_child" #25 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a1750, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #26 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a2510, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #27 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a4948, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #28 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a5c08, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #29 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a63e8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #30 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a6e98, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #31 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a7848, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #32 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a8148, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #33 0x000000000051d74b in init_mod_child (m=0x7fbc5f1a9420, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #34 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ab9d8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #35 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ac1b0, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #36 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ad020, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #37 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ad630, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #38 0x000000000051d74b in init_mod_child (m=0x7fbc5f1afde8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #39 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b05c8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #40 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b0d60, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #41 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b16a8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #42 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b25b8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #43 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b2db0, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #44 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b8a00, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #45 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b8ee8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #46 0x000000000051d74b in init_mod_child (m=0x7fbc5f1b9c90, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #47 0x000000000051d74b in init_mod_child (m=0x7fbc5f1ba6e8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #48 0x000000000051d74b in init_mod_child (m=0x7fbc5f1badf8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #49 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bb1c8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #50 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bb9d0, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #51 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd0a0, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #52 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd628, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #53 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bd950, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #54 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bdd20, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #55 0x000000000051d74b in init_mod_child (m=0x7fbc5f1be2e8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #56 0x000000000051d74b in init_mod_child (m=0x7fbc5f1be8f8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #57 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bed00, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #58 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bf0d8, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #59 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bf778, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #60 0x000000000051d74b in init_mod_child (m=0x7fbc5f1bfe20, rank=0) at sr_module.c:918 __FUNCTION__ = "init_mod_child" #61 0x000000000051dd5b in init_child (rank=0) at sr_module.c:947 No locals. #62 0x000000000050e60f in main_loop () at main.c:1682 i = 4 pid = 14215 si = 0x0 si_desc = "udp receiver child=3 sock=192.168.1.150:5060\000(185.29.208.21:5060)\000AZ\274\177\000\000\232\246d\000\001\000\000\000\270\332tZ\001\000\000\000\300dBZ\274\177\000\000\300\326$\006\375\177\000\000()uZ\274\177\000\000P\t\235W\274\177\000\000P\257,_\274\177\000" nrprocs = 4 woneinit = 1 __FUNCTION__ = "main_loop" #63 0x0000000000514e9d in main (argc=6, argv=0x7ffd0624da28) at main.c:2616 cfg_stream = 0x1047010 c = -1 r = 0 tmp = 0x7fbc6014573d <_dl_lookup_symbol_x+349> "\203", <incomplete sequence \370> tmp_len = 0 port = 1 proto = 32700 options = 0x74d640 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 2268810091 rfd = 4 debug_save = 2 debug_flag = 1 dont_fork_cnt = 2 n_lst = 0x7ffd0624d8e0 p = 0x7ffd0624da60 "q\356$\006\375\177" st = {st_dev = 15, st_ino = 12804, st_nlink = 2, st_mode = 16895, st_uid = 109, st_gid = 114, __pad0 = 0, st_rdev = 0, st_size = 100, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1458128144, tv_nsec = 692000000}, st_mtim = {tv_sec = 1458138912, tv_nsec = 180254641}, st_ctim = {tv_sec = 1458138912, tv_nsec = 180254641}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" ```
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-197362180
The first core is relevant, the other one is because kamailio tries to shut down due to other process crash, but ends in a deadlock, likely because the process that crashed didn't release a lock.
In the fist core, the problem seems to be in the odbc driver - the backtrace leaves the code of kamailio with the next frames:
``` #29 0x00007fbc4ec17c66 in SQLDriverConnectW () from /opt/microsoft/msodbcsql/lib64/libmsodbcsql-13.0.so.0.0 No symbol table info available. #30 0x00007fbc565deabd in SQLDriverConnect (hdbc=0x1121920, hwnd=0x7ffd06240460, conn_str_in=0x1142cb0 "D", len_conn_str_in=25892, conn_str_out=0x7ffd0624bf60 "\001\200\255\373\274\177", conn_str_out_max=0, ptr_conn_str_out=0x7ffd0624bf5e, driver_completion=1) at SQLDriverConnect.c:1490 uc_conn_str_in = 0x1142cb0 ```
Not sure I can really help much more here.
Can you test with other versions of odbc and ms odbc drivers?
Does libmsodbcsql has a package for debug symbols? Or, if libmsodbcsql is open source, maybe you can install it from sources with debug symbols on. Then the backtrace will show where in the source code of libmsodbcsql the crash happens and from there further investigation can be done in order to spot the problem.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-198356246
Its weird it works absolutely fine when i have TLS disabled in kamailio, and is a lot faster than freetds which works alright apart from an occasional hung connection which requires a kamailio restart to recover from (no kamailio processes hang when this happens). It also works fine with any other application we use unixodbc with, which does lead me back to thinking that root cause lies in kamailio...
I will contact MS support, and linux odbc driver team and see if they can shed any light on it, as far as i know they havnt made the debugging symbols publically available, although i imagine with the recently announced sql server linux varient they would be keen to fix any issues in the driver.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-198376134
What triggers the issue could be in Kamailio, but it is hard to figure out with no clue of what the driver doesn't like. The parameters should be ok, as freetds goes on fine.
Kamailio initiates libssl with its own shared memory manager, which then is reused by other libs linked to libssl, as libssl was designed with a context per process.
Also, the mutex locking systems is set to the one for kamailio, that's why the second core leads back to kamailio to locking_f() function from tls module. But for first core, the backtrace stack doesn't go back to kamailio, so the crash happens in external code.
Could be some options/params that the drivers doesn't like, but cannot be guessed without sources or someone from their dev team saying. Once that is learn, kamailio should be easy to adjust, all sources are out there...
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-198386863
Wondering if you had time to investigate further in the odbc driver side, to see where the issue is.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-214179642
Reopen if you get new details to investigate here.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#issuecomment-224085479
Closed #501.
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/501#event-683447158
-
Daniel-Constantin Mierla -
Tim Chubb