Hello,

I think the relevant here are the client library. I asked about mysql vs mariadb because some distro had (meta-)packages named mysql but installing mariadb. Debian at some point switched to something like default-mysql-client...

Based on backtrace, the crashes would prevent kamailio to start, as carrierroute_db_open() is used in mod_init/child_init callbacks.

A variant you could try is the use of mysql-proxy to bridge from a local unix socket to remote mysql, so kamailio connects to mysql server via  local unix socket and by that won't use libssl.

First you can check with ldd if mysql client libraries are compiled against libssl1.1 and the libssl1.0 is also used for other packages.

Cheers, Daniel

On 02.06.21 21:56, Andrew Chen wrote:

Hi Daniel, So our DB is in rds and this is what I can gather:

+-------------------------+------------------------------+

| Variable_name           | Value                        |

+-------------------------+------------------------------+

| innodb_version          | 5.6.34                       |

| protocol_version        | 10                           |

| slave_type_conversions  |                              |

| version                 | 5.6.34-log                   |

| version_comment         | MySQL Community Server (GPL) |

| version_compile_machine | x86_64                       |

| version_compile_os      | Linux                        |

+-------------------------+------------------------------+

The mysql server/client library we received from my apt-cache madison output earlier.

I don't know if upgrading to 20.04 is an option.

Given this said, what are the symptoms of these crashes?  I'm guessing one of them is a brief disconnect from the DB for dialplan/dispatcher/carrierroute lookup?

Thanks.

On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

did you upgrade the Kamailio only or also mysql server/client
library? Is it Oracle's MySQL or MariaDB?

A while ago, there were a few reports of problems with
mysql-client (Oracle) library on Ubuntu 16.04 and 18.04, which
typically happened after upgrades of the library, as I understood,
the upgrade brought in a dependency on libssl 1.1, while libssl1.0
was used by most of the other packages, resulting in mixed
deployments. Upgrading to 20.04 seemed to solve it, as it came
with libssl1.1 only.

Cheers,
Daniel

On 02.06.21 20:51, Andrew Chen wrote:

...

Hi all,

Today I upgraded Kamailio to 5.4.5 and a random crash was just
reported in our crash monitoring script.  Below is the output of
the crash:

#3  0x00007fa149ebe58d in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so

#4  0x00007fa14989e277 in ERR_clear_error () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

#5  0x00007fa149933d81 in X509_STORE_set_default_paths () from
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

#6  0x00007fa10310d374 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

#7  0x00007fa10310d9f3 in ?? () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

#8  0x00007fa1030e0427 in mysql_real_connect () from
/usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

#9  0x00007fa10368428c in db_mysql_new_connection () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

#10 0x00007fa1501f96dc in db_do_init2 () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

#11 0x00007fa1501f83f9 in db_do_init () from
/usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

#12 0x00007fa103689710 in db_mysql_init () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

#13 0x00007fa15062d35f in carrierroute_db_open () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so

#14 0x00007fa1506719c6 in ?? () from
/usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so


Is this crash due to libcrypto or mysql client?
Currently I have the following mysql client installed on the
kamailio instance:

ii  lib*mysql*client20:amd64              
5.7.34-0ubuntu0.18.04.1             amd64        MySQL database
client library

ii  *mysql*-client                        
5.7.34-0ubuntu0.18.04.1             all          MySQL database
client (metapackage depending on the latest version)

ii  *mysql*-client-5.7                    
5.7.34-0ubuntu0.18.04.1             amd64        MySQL database
client binaries

ii  *mysql*-client-core-5.7               
5.7.34-0ubuntu0.18.04.1             amd64        MySQL database
core client binaries

ii  *mysql*-common                         5.8+1.0.4            
              all          MySQL database common files, e.g.
/etc/*mysql*/my.cnf

root@ashintgtpsg51:/var/lib/cores # apt-cache madison mysql-client

mysql-client | 5.7.34-0ubuntu0.18.04.1 |
http://us-east-1.ec2.archive.ubuntu.com/ubuntu
<http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
bionic-updates/main amd64 Packages

mysql-client | 5.7.34-0ubuntu0.18.04.1 |
http://security.ubuntu.com/ubuntu
<http://security.ubuntu.com/ubuntu> bionic-security/main amd64
Packages

mysql-client | 5.7.21-1ubuntu1 |
http://us-east-1.ec2.archive.ubuntu.com/ubuntu
<http://us-east-1.ec2.archive.ubuntu.com/ubuntu> bionic/main
amd64 Packages

root@ashintgtpsg51:/var/lib/cores # 


Any suggestions would be greatly appreciated.
Thanks in advance.

-- 
Andy Chen
Sr. Telephony Lead Engineer
achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>



*Confidentiality Notice: The information contained in this e-mail
and any
attachments may be confidential. If you are not an intended
recipient, you
are hereby notified that any dissemination, distribution or
copying of this
e-mail is strictly prohibited. If you have received this e-mail
in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this
e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*

__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
  * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
  * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

-- 
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
  * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*

Hello,

it is not related to kamailio tls being compiled with libssl1.1. The issues were observed on the systems having both libssl1.0 (which was somehow the default choice) and libssl1.1. I got the chance to test a bit and it was still crashing even I compiled manually with libssl1.1. I didn't dig much in the external libraries given that upgrading OS seemed to fix it. There is a mod param for db_mysql that you can try to play with: opt_ssl_mode - I don't recall solving much for mysql servers requiring tls.

Otherwise, on a different direction: if you can get mariadb client library, you can try to compile kamailio against it (the db_mysql module) -- if the MySQL server is on a remote system, it should work.

I can't confirm it is same issue, but if you upgraded in kamailio 5.4.x series, there were no changes to carrierroute or db_mysql to

As a side note, for newer operating systems and newer libssl, it may also require to set --atexit cli parameter for kamailio.

Cheers, Daniel

On 03.06.21 14:42, Andrew Chen wrote:

Hi Daniel,

I just confirmed mysql is compiled using libssl.so.1.1.

As for the other kamailio packages, I only see two that are installed where both are depends on libssl1.1

kamailio-tls-modules:amd64

*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>= 7.16.2), libssl1.1 (>= 1.1.0)

kamailio-websocket-modules:amd64

*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>= 1.1.0), libunistring2 (>= 0.9.7)

So in my case here there shouldn't be any conflict with mysql client libssl version?

Thanks.

On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

I think the relevant here are the client library. I asked about
mysql vs mariadb because some distro had (meta-)packages named
mysql but installing mariadb. Debian at some point switched to
something like default-mysql-client...

Based on backtrace, the crashes would prevent kamailio to start,
as carrierroute_db_open() is used in mod_init/child_init callbacks.

A variant you could try is the use of mysql-proxy to bridge from a
local unix socket to remote mysql, so kamailio connects to mysql
server via  local unix socket and by that won't use libssl.

First you can check with ldd if mysql client libraries are
compiled against libssl1.1 and the libssl1.0 is also used for
other packages.

Cheers,
Daniel

On 02.06.21 21:56, Andrew Chen wrote:

...

Hi Daniel,
So our DB is in rds and this is what I can gather:

+-------------------------+------------------------------+

| Variable_name           | Value                        |

+-------------------------+------------------------------+

| innodb_version          | 5.6.34                       |

| protocol_version        | 10                           |

| slave_type_conversions  |                              |

| version                 | 5.6.34-log                   |

| version_comment         | MySQL Community Server (GPL) |

| version_compile_machine | x86_64                       |

| version_compile_os      | Linux                        |

+-------------------------+------------------------------+


The mysql server/client library we received from my apt-cache
madison output earlier.


I don't know if upgrading to 20.04 is an option.


Given this said, what are the symptoms of these crashes?  I'm
guessing one of them is a brief disconnect from the DB for
dialplan/dispatcher/carrierroute lookup?


Thanks.


On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    did you upgrade the Kamailio only or also mysql server/client
    library? Is it Oracle's MySQL or MariaDB?

    A while ago, there were a few reports of problems with
    mysql-client (Oracle) library on Ubuntu 16.04 and 18.04,
    which typically happened after upgrades of the library, as I
    understood, the upgrade brought in a dependency on libssl
    1.1, while libssl1.0 was used by most of the other packages,
    resulting in mixed deployments. Upgrading to 20.04 seemed to
    solve it, as it came with libssl1.1 only.

    Cheers,
    Daniel

    On 02.06.21 20:51, Andrew Chen wrote:

...

    Hi all,

    Today I upgraded Kamailio to 5.4.5 and a random crash was
    just reported in our crash monitoring script.  Below is the
    output of the crash:

    #3  0x00007fa149ebe58d in ?? () from
    /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so

    #4  0x00007fa14989e277 in ERR_clear_error () from
    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

    #5  0x00007fa149933d81 in X509_STORE_set_default_paths ()
    from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

    #6  0x00007fa10310d374 in ?? () from
    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

    #7  0x00007fa10310d9f3 in ?? () from
    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

    #8  0x00007fa1030e0427 in mysql_real_connect () from
    /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

    #9  0x00007fa10368428c in db_mysql_new_connection () from
    /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

    #10 0x00007fa1501f96dc in db_do_init2 () from
    /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

    #11 0x00007fa1501f83f9 in db_do_init () from
    /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

    #12 0x00007fa103689710 in db_mysql_init () from
    /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

    #13 0x00007fa15062d35f in carrierroute_db_open () from
    /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so

    #14 0x00007fa1506719c6 in ?? () from
    /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so


    Is this crash due to libcrypto or mysql client?
    Currently I have the following mysql client installed on the
    kamailio instance:

    ii  lib*mysql*client20:amd64              
    5.7.34-0ubuntu0.18.04.1             amd64        MySQL
    database client library

    ii  *mysql*-client                        
    5.7.34-0ubuntu0.18.04.1             all          MySQL
    database client (metapackage depending on the latest version)

    ii  *mysql*-client-5.7                    
    5.7.34-0ubuntu0.18.04.1             amd64        MySQL
    database client binaries

    ii  *mysql*-client-core-5.7               
    5.7.34-0ubuntu0.18.04.1             amd64        MySQL
    database core client binaries

    ii  *mysql*-common                         5.8+1.0.4        
                      all          MySQL database common files,
    e.g. /etc/*mysql*/my.cnf

    root@ashintgtpsg51:/var/lib/cores # apt-cache madison
    mysql-client

    mysql-client | 5.7.34-0ubuntu0.18.04.1 |
    http://us-east-1.ec2.archive.ubuntu.com/ubuntu
    <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
    bionic-updates/main amd64 Packages

    mysql-client | 5.7.34-0ubuntu0.18.04.1 |
    http://security.ubuntu.com/ubuntu
    <http://security.ubuntu.com/ubuntu> bionic-security/main
    amd64 Packages

    mysql-client | 5.7.21-1ubuntu1 |
    http://us-east-1.ec2.archive.ubuntu.com/ubuntu
    <http://us-east-1.ec2.archive.ubuntu.com/ubuntu> bionic/main
    amd64 Packages

    root@ashintgtpsg51:/var/lib/cores # 


    Any suggestions would be greatly appreciated.
    Thanks in advance.

    -- 
    Andy Chen
    Sr. Telephony Lead Engineer
    achen@ <mailto:achen@thinkingphones.com>fuze.com
    <http://fuze.com>



    *Confidentiality Notice: The information contained in this
    e-mail and any
    attachments may be confidential. If you are not an intended
    recipient, you
    are hereby notified that any dissemination, distribution or
    copying of this
    e-mail is strictly prohibited. If you have received this
    e-mail in error,
    please notify the sender and permanently delete the e-mail
    and any
    attachments immediately. You should not retain, copy or use
    this e-mail or
    any attachment for any purpose, nor disclose all or any part
    of the
    contents to any other person. Thank you.*

    __________________________________________________________
    Kamailio - Users Mailing List - Non Commercial Discussions
      * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
    Important: keep the mailing list in the recipients, do not reply only to the sender!
    Edit mailing list options or unsubscribe:
      * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

    -- 
    Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
    www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
    Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
      * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



-- 
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>


*Confidentiality Notice: The information contained in this e-mail
and any
attachments may be confidential. If you are not an intended
recipient, you
are hereby notified that any dissemination, distribution or
copying of this
e-mail is strictly prohibited. If you have received this e-mail
in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this
e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.* 

-- 
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
  * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*

Again, not to be blamed later: I am not sure you face the same kind of issue like reported in the past. I was looking at it from the perspective that db_mysql and carrierroute modules were not touched that much in the 5.4.x series recently.

The libssl did a lot of internal/api changes between 1.0 and 1.1, so for a few past years many libs and apps (including kamailio) faced some "strange" crashes/behaviours. Now seems stabilized and, at least on debian 10, there were no related open issues recently. In the future we may need to make --atexit=no the default (maybe needs to be set even on latest ubuntu 20.04), it's the last I remember that surfaced and it appeared after a minor version upgrade of libssl1.1, iirc.

Cheers, Daniel

On 03.06.21 15:31, Andrew Chen wrote:

Thanks Daniel. It sounds like upgrading to a newer Ubuntu seems like the cleanest way of resolving this issue.

On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Hello,

it is not related to kamailio tls being compiled with libssl1.1.
The issues were observed on the systems having both libssl1.0
(which was somehow the default choice) and libssl1.1. I got the
chance to test a bit and it was still crashing even I compiled
manually with libssl1.1. I didn't dig much in the external
libraries given that upgrading OS seemed to fix it. There is a mod
param for db_mysql that you can try to play with: opt_ssl_mode - I
don't recall solving much for mysql servers requiring tls.

Otherwise, on a different direction: if you can get mariadb client
library, you can try to compile kamailio against it (the db_mysql
module) -- if the MySQL server is on a remote system, it should work.

I can't confirm it is same issue, but if you upgraded in kamailio
5.4.x series, there were no changes to carrierroute or db_mysql to

As a side note, for newer operating systems and newer libssl, it
may also require to set --atexit cli parameter for kamailio.

Cheers,
Daniel

On 03.06.21 14:42, Andrew Chen wrote:

...

Hi Daniel,

I just confirmed mysql is compiled using libssl.so.1.1.

As for the other kamailio packages, I only see two that are
installed where both are depends on libssl1.1

kamailio-tls-modules:amd64

*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>=
7.16.2), libssl1.1 (>= 1.1.0)



kamailio-websocket-modules:amd64

*Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1 (>=
1.1.0), libunistring2 (>= 0.9.7)


So in my case here there shouldn't be any conflict with mysql
client libssl version?

Thanks.


On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    I think the relevant here are the client library. I asked
    about mysql vs mariadb because some distro had
    (meta-)packages named mysql but installing mariadb. Debian at
    some point switched to something like default-mysql-client...

    Based on backtrace, the crashes would prevent kamailio to
    start, as carrierroute_db_open() is used in
    mod_init/child_init callbacks.

    A variant you could try is the use of mysql-proxy to bridge
    from a local unix socket to remote mysql, so kamailio
    connects to mysql server via  local unix socket and by that
    won't use libssl.

    First you can check with ldd if mysql client libraries are
    compiled against libssl1.1 and the libssl1.0 is also used for
    other packages.

    Cheers,
    Daniel

    On 02.06.21 21:56, Andrew Chen wrote:

...

    Hi Daniel,
    So our DB is in rds and this is what I can gather:

    +-------------------------+------------------------------+

    | Variable_name           | Value                        |

    +-------------------------+------------------------------+

    | innodb_version          | 5.6.34                       |

    | protocol_version        | 10                           |

    | slave_type_conversions  |                              |

    | version                 | 5.6.34-log                   |

    | version_comment         | MySQL Community Server (GPL) |

    | version_compile_machine | x86_64                       |

    | version_compile_os      | Linux                        |

    +-------------------------+------------------------------+


    The mysql server/client library we received from my
    apt-cache madison output earlier.


    I don't know if upgrading to 20.04 is an option.


    Given this said, what are the symptoms of these crashes? 
    I'm guessing one of them is a brief disconnect from the DB
    for dialplan/dispatcher/carrierroute lookup?


    Thanks.


    On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        did you upgrade the Kamailio only or also mysql
        server/client library? Is it Oracle's MySQL or MariaDB?

        A while ago, there were a few reports of problems with
        mysql-client (Oracle) library on Ubuntu 16.04 and 18.04,
        which typically happened after upgrades of the library,
        as I understood, the upgrade brought in a dependency on
        libssl 1.1, while libssl1.0 was used by most of the
        other packages, resulting in mixed deployments.
        Upgrading to 20.04 seemed to solve it, as it came with
        libssl1.1 only.

        Cheers,
        Daniel

        On 02.06.21 20:51, Andrew Chen wrote:

...

        Hi all,

        Today I upgraded Kamailio to 5.4.5 and a random crash
        was just reported in our crash monitoring script. 
        Below is the output of the crash:

        #3  0x00007fa149ebe58d in ?? () from
        /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so

        #4  0x00007fa14989e277 in ERR_clear_error () from
        /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

        #5  0x00007fa149933d81 in X509_STORE_set_default_paths
        () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

        #6  0x00007fa10310d374 in ?? () from
        /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

        #7  0x00007fa10310d9f3 in ?? () from
        /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

        #8  0x00007fa1030e0427 in mysql_real_connect () from
        /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

        #9  0x00007fa10368428c in db_mysql_new_connection ()
        from /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

        #10 0x00007fa1501f96dc in db_do_init2 () from
        /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

        #11 0x00007fa1501f83f9 in db_do_init () from
        /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

        #12 0x00007fa103689710 in db_mysql_init () from
        /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

        #13 0x00007fa15062d35f in carrierroute_db_open () from
        /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so

        #14 0x00007fa1506719c6 in ?? () from
        /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so


        Is this crash due to libcrypto or mysql client?
        Currently I have the following mysql client installed
        on the kamailio instance:

        ii  lib*mysql*client20:amd64              
        5.7.34-0ubuntu0.18.04.1             amd64        MySQL
        database client library

        ii  *mysql*-client                        
        5.7.34-0ubuntu0.18.04.1             all          MySQL
        database client (metapackage depending on the latest
        version)

        ii  *mysql*-client-5.7                    
        5.7.34-0ubuntu0.18.04.1             amd64        MySQL
        database client binaries

        ii  *mysql*-client-core-5.7               
        5.7.34-0ubuntu0.18.04.1             amd64        MySQL
        database core client binaries

        ii  *mysql*-common                         5.8+1.0.4  
                                all          MySQL database
        common files, e.g. /etc/*mysql*/my.cnf

        root@ashintgtpsg51:/var/lib/cores # apt-cache madison
        mysql-client

        mysql-client | 5.7.34-0ubuntu0.18.04.1 |
        http://us-east-1.ec2.archive.ubuntu.com/ubuntu
        <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
        bionic-updates/main amd64 Packages

        mysql-client | 5.7.34-0ubuntu0.18.04.1 |
        http://security.ubuntu.com/ubuntu
        <http://security.ubuntu.com/ubuntu>
        bionic-security/main amd64 Packages

        mysql-client | 5.7.21-1ubuntu1 |
        http://us-east-1.ec2.archive.ubuntu.com/ubuntu
        <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
        bionic/main amd64 Packages

        root@ashintgtpsg51:/var/lib/cores # 


        Any suggestions would be greatly appreciated.
        Thanks in advance.

        -- 
        Andy Chen
        Sr. Telephony Lead Engineer
        achen@ <mailto:achen@thinkingphones.com>fuze.com
        <http://fuze.com>



        *Confidentiality Notice: The information contained in
        this e-mail and any
        attachments may be confidential. If you are not an
        intended recipient, you
        are hereby notified that any dissemination,
        distribution or copying of this
        e-mail is strictly prohibited. If you have received
        this e-mail in error,
        please notify the sender and permanently delete the
        e-mail and any
        attachments immediately. You should not retain, copy or
        use this e-mail or
        any attachment for any purpose, nor disclose all or any
        part of the
        contents to any other person. Thank you.*

        __________________________________________________________
        Kamailio - Users Mailing List - Non Commercial Discussions
          * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
        Important: keep the mailing list in the recipients, do not reply only to the sender!
        Edit mailing list options or unsubscribe:
          * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

        -- 
        Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
        www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
        Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
          * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



    -- 
    Andy Chen
    Sr. Telephony Lead Engineer
    415 516 5535 (M)
    achen@ <mailto:achen@thinkingphones.com>fuze.com
    <http://fuze.com>


    *Confidentiality Notice: The information contained in this
    e-mail and any
    attachments may be confidential. If you are not an intended
    recipient, you
    are hereby notified that any dissemination, distribution or
    copying of this
    e-mail is strictly prohibited. If you have received this
    e-mail in error,
    please notify the sender and permanently delete the e-mail
    and any
    attachments immediately. You should not retain, copy or use
    this e-mail or
    any attachment for any purpose, nor disclose all or any part
    of the
    contents to any other person. Thank you.* 

    -- 
    Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
    www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
    Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
      * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



-- 
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>


*Confidentiality Notice: The information contained in this e-mail
and any
attachments may be confidential. If you are not an intended
recipient, you
are hereby notified that any dissemination, distribution or
copying of this
e-mail is strictly prohibited. If you have received this e-mail
in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this
e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.* 

-- 
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
  * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*

Well, probably a misunderstanding: the issue was observed on OSes having the two libssl 1.0 & 1.1 versions, but it does not mean (it was not proved) that the problem was created by having the two version were installed at the same time. I would rather think it was more that libssl1.1 was not very mature at that moment when OSes shipped both of them.

If I would make a suggestion, that would be to use an OS version that comes only with one of them by default. Think like: when libssl1.0 was completely removed from an OS, the libssl1.1 was more tested/stable.

Cheers, Daniel

On 03.06.21 15:58, Andrew Chen wrote:

No blaming here.  :). Just looking for answers.

After chatting with my peers on options around this, we are going to remove libssl1.0 and just have libssl1.1 running.

Thanks again for you input.

On Thu, Jun 3, 2021 at 9:48 AM Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:

Again, not to be blamed later: I am not sure you face the same
kind of issue like reported in the past. I was looking at it from
the perspective that db_mysql and carrierroute modules were not
touched that much in the 5.4.x series recently.

The libssl did a lot of internal/api changes between 1.0 and 1.1,
so for a few past years many libs and apps (including kamailio)
faced some "strange" crashes/behaviours. Now seems stabilized and,
at least on debian 10, there were no related open issues recently.
In the future we may need to make --atexit=no the default (maybe
needs to be set even on latest ubuntu 20.04), it's the last I
remember that surfaced and it appeared after a minor version
upgrade of libssl1.1, iirc.

Cheers,
Daniel

On 03.06.21 15:31, Andrew Chen wrote:

...

Thanks Daniel.
It sounds like upgrading to a newer Ubuntu seems like the
cleanest way of resolving this issue.

On Thu, Jun 3, 2021 at 8:58 AM Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    it is not related to kamailio tls being compiled with
    libssl1.1. The issues were observed on the systems having
    both libssl1.0 (which was somehow the default choice) and
    libssl1.1. I got the chance to test a bit and it was still
    crashing even I compiled manually with libssl1.1. I didn't
    dig much in the external libraries given that upgrading OS
    seemed to fix it. There is a mod param for db_mysql that you
    can try to play with: opt_ssl_mode - I don't recall solving
    much for mysql servers requiring tls.

    Otherwise, on a different direction: if you can get mariadb
    client library, you can try to compile kamailio against it
    (the db_mysql module) -- if the MySQL server is on a remote
    system, it should work.

    I can't confirm it is same issue, but if you upgraded in
    kamailio 5.4.x series, there were no changes to carrierroute
    or db_mysql to

    As a side note, for newer operating systems and newer libssl,
    it may also require to set --atexit cli parameter for kamailio.

    Cheers,
    Daniel

    On 03.06.21 14:42, Andrew Chen wrote:

...

    Hi Daniel,

    I just confirmed mysql is compiled using libssl.so.1.1.

    As for the other kamailio packages, I only see two that are
    installed where both are depends on libssl1.1

    kamailio-tls-modules:amd64

    *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libcurl4 (>=
    7.16.2), libssl1.1 (>= 1.1.0)



    kamailio-websocket-modules:amd64

    *Depends*: kamailio (= 5.4.5), libc6 (>= 2.14), libssl1.1
    (>= 1.1.0), libunistring2 (>= 0.9.7)


    So in my case here there shouldn't be any conflict with
    mysql client libssl version?

    Thanks.


    On Thu, Jun 3, 2021 at 3:10 AM Daniel-Constantin Mierla
    <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

        Hello,

        I think the relevant here are the client library. I
        asked about mysql vs mariadb because some distro had
        (meta-)packages named mysql but installing mariadb.
        Debian at some point switched to something like
        default-mysql-client...

        Based on backtrace, the crashes would prevent kamailio
        to start, as carrierroute_db_open() is used in
        mod_init/child_init callbacks.

        A variant you could try is the use of mysql-proxy to
        bridge from a local unix socket to remote mysql, so
        kamailio connects to mysql server via  local unix socket
        and by that won't use libssl.

        First you can check with ldd if mysql client libraries
        are compiled against libssl1.1 and the libssl1.0 is also
        used for other packages.

        Cheers,
        Daniel

        On 02.06.21 21:56, Andrew Chen wrote:

...

        Hi Daniel,
        So our DB is in rds and this is what I can gather:

        +-------------------------+------------------------------+

        | Variable_name           | Value                        |

        +-------------------------+------------------------------+

        | innodb_version          | 5.6.34                       |

        | protocol_version        | 10                           |

        | slave_type_conversions  |                              |

        | version                 | 5.6.34-log                   |

        | version_comment         | MySQL Community Server (GPL) |

        | version_compile_machine | x86_64                       |

        | version_compile_os      | Linux                        |

        +-------------------------+------------------------------+


        The mysql server/client library we received from my
        apt-cache madison output earlier.


        I don't know if upgrading to 20.04 is an option.


        Given this said, what are the symptoms of these
        crashes?  I'm guessing one of them is a brief
        disconnect from the DB for
        dialplan/dispatcher/carrierroute lookup?


        Thanks.


        On Wed, Jun 2, 2021 at 3:38 PM Daniel-Constantin Mierla
        <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

            Hello,

            did you upgrade the Kamailio only or also mysql
            server/client library? Is it Oracle's MySQL or MariaDB?

            A while ago, there were a few reports of problems
            with mysql-client (Oracle) library on Ubuntu 16.04
            and 18.04, which typically happened after upgrades
            of the library, as I understood, the upgrade
            brought in a dependency on libssl 1.1, while
            libssl1.0 was used by most of the other packages,
            resulting in mixed deployments. Upgrading to 20.04
            seemed to solve it, as it came with libssl1.1 only.

            Cheers,
            Daniel

            On 02.06.21 20:51, Andrew Chen wrote:

...

            Hi all,

            Today I upgraded Kamailio to 5.4.5 and a random
            crash was just reported in our crash monitoring
            script.  Below is the output of the crash:

            #3  0x00007fa149ebe58d in ?? () from
            /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so

            #4  0x00007fa14989e277 in ERR_clear_error () from
            /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

            #5  0x00007fa149933d81 in
            X509_STORE_set_default_paths () from
            /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

            #6  0x00007fa10310d374 in ?? () from
            /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

            #7  0x00007fa10310d9f3 in ?? () from
            /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

            #8  0x00007fa1030e0427 in mysql_real_connect ()
            from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20

            #9  0x00007fa10368428c in db_mysql_new_connection
            () from
            /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

            #10 0x00007fa1501f96dc in db_do_init2 () from
            /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

            #11 0x00007fa1501f83f9 in db_do_init () from
            /usr/lib/x86_64-linux-gnu/kamailio/libsrdb1.so.1

            #12 0x00007fa103689710 in db_mysql_init () from
            /usr/lib/x86_64-linux-gnu/kamailio/modules/db_mysql.so

            #13 0x00007fa15062d35f in carrierroute_db_open ()
            from
            /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so

            #14 0x00007fa1506719c6 in ?? () from
            /usr/lib/x86_64-linux-gnu/kamailio/modules/carrierroute.so


            Is this crash due to libcrypto or mysql client?
            Currently I have the following mysql client
            installed on the kamailio instance:

            ii  lib*mysql*client20:amd64              
            5.7.34-0ubuntu0.18.04.1             amd64       
            MySQL database client library

            ii  *mysql*-client                        
            5.7.34-0ubuntu0.18.04.1             all         
            MySQL database client (metapackage depending on
            the latest version)

            ii  *mysql*-client-5.7                    
            5.7.34-0ubuntu0.18.04.1             amd64       
            MySQL database client binaries

            ii  *mysql*-client-core-5.7               
            5.7.34-0ubuntu0.18.04.1             amd64       
            MySQL database core client binaries

            ii  *mysql*-common                        
            5.8+1.0.4                           all         
            MySQL database common files, e.g. /etc/*mysql*/my.cnf

            root@ashintgtpsg51:/var/lib/cores # apt-cache
            madison mysql-client

            mysql-client | 5.7.34-0ubuntu0.18.04.1 |
            http://us-east-1.ec2.archive.ubuntu.com/ubuntu
            <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
            bionic-updates/main amd64 Packages

            mysql-client | 5.7.34-0ubuntu0.18.04.1 |
            http://security.ubuntu.com/ubuntu
            <http://security.ubuntu.com/ubuntu>
            bionic-security/main amd64 Packages

            mysql-client | 5.7.21-1ubuntu1 |
            http://us-east-1.ec2.archive.ubuntu.com/ubuntu
            <http://us-east-1.ec2.archive.ubuntu.com/ubuntu>
            bionic/main amd64 Packages

            root@ashintgtpsg51:/var/lib/cores # 


            Any suggestions would be greatly appreciated.
            Thanks in advance.

            -- 
            Andy Chen
            Sr. Telephony Lead Engineer
            achen@ <mailto:achen@thinkingphones.com>fuze.com
            <http://fuze.com>



            *Confidentiality Notice: The information contained
            in this e-mail and any
            attachments may be confidential. If you are not an
            intended recipient, you
            are hereby notified that any dissemination,
            distribution or copying of this
            e-mail is strictly prohibited. If you have
            received this e-mail in error,
            please notify the sender and permanently delete
            the e-mail and any
            attachments immediately. You should not retain,
            copy or use this e-mail or
            any attachment for any purpose, nor disclose all
            or any part of the
            contents to any other person. Thank you.*

            __________________________________________________________
            Kamailio - Users Mailing List - Non Commercial Discussions
              * sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
            Important: keep the mailing list in the recipients, do not reply only to the sender!
            Edit mailing list options or unsubscribe:
              * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>

            -- 
            Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
            www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
            Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
              * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



        -- 
        Andy Chen
        Sr. Telephony Lead Engineer
        415 516 5535 (M)
        achen@ <mailto:achen@thinkingphones.com>fuze.com
        <http://fuze.com>


        *Confidentiality Notice: The information contained in
        this e-mail and any
        attachments may be confidential. If you are not an
        intended recipient, you
        are hereby notified that any dissemination,
        distribution or copying of this
        e-mail is strictly prohibited. If you have received
        this e-mail in error,
        please notify the sender and permanently delete the
        e-mail and any
        attachments immediately. You should not retain, copy or
        use this e-mail or
        any attachment for any purpose, nor disclose all or any
        part of the
        contents to any other person. Thank you.* 

        -- 
        Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
        www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
        Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
          * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



    -- 
    Andy Chen
    Sr. Telephony Lead Engineer
    415 516 5535 (M)
    achen@ <mailto:achen@thinkingphones.com>fuze.com
    <http://fuze.com>


    *Confidentiality Notice: The information contained in this
    e-mail and any
    attachments may be confidential. If you are not an intended
    recipient, you
    are hereby notified that any dissemination, distribution or
    copying of this
    e-mail is strictly prohibited. If you have received this
    e-mail in error,
    please notify the sender and permanently delete the e-mail
    and any
    attachments immediately. You should not retain, copy or use
    this e-mail or
    any attachment for any purpose, nor disclose all or any part
    of the
    contents to any other person. Thank you.* 

    -- 
    Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
    www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
    Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
      * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>



-- 
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>


*Confidentiality Notice: The information contained in this e-mail
and any
attachments may be confidential. If you are not an intended
recipient, you
are hereby notified that any dissemination, distribution or
copying of this
e-mail is strictly prohibited. If you have received this e-mail
in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this
e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.* 

-- 
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online - June 7-10, 2021 (America Timezone)
  * https://www.asipto.com/sw/kamailio-advanced-training-online/ <https://www.asipto.com/sw/kamailio-advanced-training-online/>

-- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ mailto:achen@thinkingphones.comfuze.com http://fuze.com

*Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.*