Re: [sr-dev] [SR-Users] Info: initialization of libssl 1.1.+ with PTHREAD_PROCESS_SHARED attribute for pthread mutex and rwlock

Hello Andrew, strange - it should be there, as it is available on deb.kamailio.org. Here e.g. the one for 64bit stretch: https://deb.kamailio.org/kamailiodev-nightly/pool/main/k/kamailio/kamailio-… Cheers, Henning Am 18.09.19 um 18:52 schrieb Andrew Chen: I don't see one for debug: root@ashintgkama51:~ # apt-cache search kamailio resource-agents - Cluster Resource Agents kamailio - very fast, dynamic and configurable SIP server kamailio-autheph-modules - authentication using ephemeral credentials for the Kamailio SIP server kamailio-berkeley-bin - Berkeley database module for Kamailio - helper program kamailio-berkeley-modules - Berkeley database module for the Kamailio SIP server kamailio-carrierroute-modules - carrierroute module for the Kamailio SIP server kamailio-cnxcc-modules - cnxcc modules for the Kamailio SIP server kamailio-cpl-modules - CPL module (CPL interpreter engine) for the Kamailio SIP server kamailio-erlang-modules - Erlang modules for the Kamailio SIP server kamailio-geoip-modules - geoip module for the Kamailio SIP Server kamailio-geoip2-modules - geoip2 module for the Kamailio SIP Server kamailio-ims-modules - IMS module for the Kamailio SIP server kamailio-json-modules - Json parser and Json-RPC modules for the Kamailio SIP server On Wed, Sep 18, 2019 at 11:26 AM Daniel-Constantin Mierla < miconda(a)gmail.com&gt; wrote: > You have to install the debug symbols package, otherwise the backtrace > doesn't show any useful information. It just require to install that > package and then get again the backtraces. The package name should be > kamailio-dbg. > > Cheers, > Daniel > On 18.09.19 16:37, Andrew Chen wrote: > > Sorry..forgot bt full output > > (gdb) bt full > > #0 0x0000562fcc3544c0 in ?? () > > No symbol table info available. > > #1 0x0000562fcc354632 in ?? () > > No symbol table info available. > > #2 0x0000562fcc354974 in local_timer_run () > > No symbol table info available. > > #3 0x0000562fcc3fb256 in ?? () > > No symbol table info available. > > #4 0x0000562fcc3fb64c in tcp_receive_loop () > > No symbol table info available. > > #5 0x0000562fcc2acf52 in tcp_init_children () > > No symbol table info available. > > #6 0x0000562fcc1ad94a in main_loop () > > No symbol table info available. > > #7 0x0000562fcc1b4a7b in main () > > No symbol table info available. > > (gdb) > > On Wed, Sep 18, 2019 at 10:35 AM Andrew Chen &lt;achen(a)fuze.com&gt; wrote: > >> backtrace output: >> >> >> (gdb) bt >> >> #0 0x0000562fcc3544c0 in ?? () >> >> #1 0x0000562fcc354632 in ?? () >> >> #2 0x0000562fcc354974 in local_timer_run () >> >> #3 0x0000562fcc3fb256 in ?? () >> >> #4 0x0000562fcc3fb64c in tcp_receive_loop () >> >> #5 0x0000562fcc2acf52 in tcp_init_children () >> >> #6 0x0000562fcc1ad94a in main_loop () >> >> #7 0x0000562fcc1b4a7b in main () >> >> (gdb) >> >> >> >> >> On Wed, Sep 18, 2019 at 10:01 AM Andrew Chen &lt;achen(a)fuze.com&gt; wrote: >> >>> I'll look for the core. >>> >>> As part of our release process, I don't think we can release master >>> branch of Kamailio. >>> >>> On Wed, Sep 18, 2019 at 9:54 AM Daniel-Constantin Mierla < >>> miconda(a)gmail.com&gt; wrote: >>> >>>> Hello, >>>> >>>> did you get core files? If yes, it would be good to see the gdb >>>> bracktraces. >>>> >>>> Have you used the pre-loaded library workaround? >>>> >>>> The code was not backported to branch 5.2 yet, I wanted to wait a >>>> bit and see if there are reports with good/bad results. If you want to use >>>> it, then you have to install nightly builds from master branch (development >>>> version, should be 5.3.0-pre1 at this moment). >>>> >>>> Cheers, >>>> Daniel >>>> On 18.09.19 15:26, Andrew Chen wrote: >>>> >>>> Hi Daniel, >>>> Which apt repository can we get this workaround? Today I noticed >>>> our Kamailio 5.2.4 cored during one of our load test sessions: >>>> >>>> Wed Sep 18 09:31:14 2019] kamailio[26997]: segfault at >>>> 7f8501366000 ip 00007f8501e58585 sp 00007fffd5c84248 error 4 in >>>> libcrypto.so.1.1[7f8501dd4000+29b000] >>>> >>>> [Wed Sep 18 10:12:29 2019] kamailio[27110]: segfault at >>>> 7f1a8eea1000 ip 00007f1a8f993585 sp 00007ffddc8848c8 error 4 in >>>> libcrypto.so.1.1[7f1a8f90f000+29b000] >>>> >>>> [Wed Sep 18 10:12:43 2019] kamailio[11060]: segfault at 20 ip >>>> 00007f0a85dac128 sp 00007fff6847aad0 error 4 in >>>> libcrypto.so.1.1[7f0a85c40000+29b000] >>>> >>>> [Wed Sep 18 10:12:44 2019] kamailio[11059]: segfault at f0 ip >>>> 00007f0a85cc43ea sp 00007fff6847a9e8 error 4 in >>>> libcrypto.so.1.1[7f0a85c40000+29b000] >>>> >>>> [Wed Sep 18 10:13:02 2019] kamailio[11187]: segfault at >>>> 7f75e06c8000 ip 00007f75e11ba585 sp 00007ffe29b47c88 error 4 in >>>> libcrypto.so.1.1[7f75e1136000+29b000] >>>> >>>> [Wed Sep 18 10:13:17 2019] kamailio[11445]: segfault at >>>> 7f0f3743f000 ip 00007f0f37f31585 sp 00007ffde54daa38 error 4 in >>>> libcrypto.so.1.1[7f0f37ead000+29b000] >>>> >>>> [Wed Sep 18 10:13:36 2019] kamailio[11644]: segfault at >>>> 7f804309c000 ip 00007f8043b8e585 sp 00007ffe89fe8228 error 4 in >>>> libcrypto.so.1.1[7f8043b0a000+29b000] >>>> >>>> [Wed Sep 18 10:13:52 2019] kamailio[11799]: segfault at >>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in >>>> libcrypto.so.1.1[7ff59d250000+29b000] >>>> >>>> [Wed Sep 18 10:13:52 2019] kamailio[11792]: segfault at >>>> 7ff59c7e2000 ip 00007ff59d2d4585 sp 00007ffd69d76bb8 error 4 in >>>> libcrypto.so.1.1[7ff59d250000+29b000] >>>> >>>> [Wed Sep 18 12:48:01 2019] kamailio[12066]: segfault at >>>> 7f22dca6d000 ip 00007f22dd55f585 sp 00007fffbc766ab8 error 4 in >>>> libcrypto.so.1.1[7f22dd4db000+29b000] >>>> >>>> [Wed Sep 18 12:48:14 2019] kamailio[16344]: segfault at >>>> 7f7cca42d000 ip 00007f7ccaf1f585 sp 00007ffe99f16108 error 4 in >>>> libcrypto.so.1.1[7f7ccae9b000+29b000] >>>> >>>> [Wed Sep 18 12:48:32 2019] kamailio[16708]: segfault at >>>> 7fb309141000 ip 00007fb309c33585 sp 00007ffd07a5f3c8 error 4 in >>>> libcrypto.so.1.1[7fb309baf000+29b000] >>>> >>>> [Wed Sep 18 12:48:49 2019] kamailio[16824]: segfault at f0 ip >>>> 00007fc59095b3ea sp 00007ffe58a490b8 error 4 in >>>> libcrypto.so.1.1[7fc5908d7000+29b000] >>>> >>>> [Wed Sep 18 12:49:04 2019] kamailio[16956]: segfault at >>>> 7fcc22e58000 ip 00007fcc2394a585 sp 00007fffa9c52258 error 4 in >>>> libcrypto.so.1.1[7fcc238c6000+29b000] >>>> >>>> [Wed Sep 18 12:49:04 2019] kamailio[16945]: segfault at f0 ip >>>> 00007fcc2394a3ea sp 00007fffa9c52128 error 4 in >>>> libcrypto.so.1.1[7fcc238c6000+29b000] >>>> >>>> [Wed Sep 18 12:49:16 2019] kamailio[17244]: segfault at >>>> 7f586ed20000 ip 00007f586f812585 sp 00007ffeae1adcd8 error 4 in >>>> libcrypto.so.1.1[7f586f78e000+29b000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17431]: segfault at 8 ip >>>> 0000562fcc44c5d5 sp 00007ffe825c9dd0 error 4 in >>>> kamailio[562fcc17c000+4a2000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17386]: segfault at 10 ip >>>> 00007f96c3735c3e sp 00007ffe825ca270 error 4 in >>>> dialog.so[7f96c36d6000+be000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17423]: segfault at 0 ip >>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>> kamailio[562fcc17c000+4a2000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17378]: segfault at 0 ip >>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17417]: segfault at 0 ip >>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>> kamailio[562fcc17c000+4a2000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17416]: segfault at 0 ip >>>> 0000562fcc3544c0 sp 00007ffe825ca1f0 error 6 in >>>> kamailio[562fcc17c000+4a2000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17380]: segfault at 0 ip >>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] >>>> >>>> [Wed Sep 18 12:49:33 2019] kamailio[17381]: segfault at 0 ip >>>> 00007f96c5dd099f sp 00007ffe825c98e0 error 4 in tm.so[7f96c5da5000+13f000] >>>> >>>> [Wed Sep 18 12:49:34 2019] kamailio[17387]: segfault at 8 ip >>>> 0000562fcc25d1f3 sp 00007ffe825ca370 error 6 in >>>> kamailio[562fcc17c000+4a2000] >>>> >>>> >>>> and I was wondering if this core is related. >>>> >>>> Thanks >>>> >>>> >>>> On Fri, Sep 13, 2019 at 5:18 PM Alex Balashov < >>>> abalashov(a)evaristesys.com&gt; wrote: >>>> >>>>> Hi Daniel, >>>>> >>>>> Thanks a lot for this work; it's definitely important that Kamailio >>>>> build against OpenSSL v1.1, and I'll be happy to contribute some >>>>> testing >>>>> in a live environment in the coming weeks. Much appreciated. >>>>> >>>>> Have a good weekend, >>>>> >>>>> -- Alex >>>>> >>>>> On Fri, Sep 13, 2019 at 01:34:12PM +0200, Daniel-Constantin Mierla >>>>> wrote: >>>>> >>>>> > Hello, >>>>> > after couple of months waiting for feedback about the workaround >>>>> with the >>>>> > preloaded shared library to set the PTHREAD_PROCESS_SHARED >>>>> attribute for >>>>> > pthread mutex/rwlock in order to avoid deadlocks and as it seems >>>>> to be the >>>>> > solution, I just pushed a few commits to move that code in the >>>>> core of >>>>> > kamailio. >>>>> > >>>>> > It this way the initialization with the PTHREAD_PROCESS_SHARED >>>>> attribute is >>>>> > done by default when libssl v1.1 is detected at compile time, >>>>> without the >>>>> > need of using the approach with a preloaded library. >>>>> > >>>>> > The shared library is still kept in the tls module: >>>>> > * >>>>> > >>>>> https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/open… >>>>> > >>>>> > Probably is going to be removed in the near future. >>>>> > >>>>> > I am asking the community members that use Kamailio on systems >>>>> with libssl >>>>> > v1.1+ to test the master branch and give feedback. The master >>>>> branch is >>>>> > anyhow in testing phase for preparing the next major release >>>>> v5.3.0, so it >>>>> > will help for that as well. >>>>> > >>>>> > If no issue is going to be reported soon, the commits will be >>>>> backported to >>>>> > stable branches, to re-enable smooth use of Kamailio with libssl >>>>> v1.1+. >>>>> > >>>>> > When libssl v1.1+ is detected, the core links with pthread, but >>>>> that was >>>>> > already present in the Makefiles.defs for different compile >>>>> options. The >>>>> > pthread more or less a default library on linuxes/unixes, so >>>>> this should >>>>> > not introduce any major dependency to the Kamailio core. >>>>> > >>>>> > Cheers, >>>>> > Daniel >>>>> > >>>>> > -- >>>>> > Daniel-Constantin Mierla - http://www.asipto.com >>>>> > http://twitter.com/#!/miconda - >>>>> http://www.linkedin.com/in/miconda >>>>> >>>>> > _______________________________________________ >>>>> > Kamailio (SER) - Users Mailing List >>>>> > sr-users(a)lists.kamailio.org >>>>> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>>> >>>>> -- >>>>> Alex Balashov | Principal | Evariste Systems LLC >>>>> >>>>> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) >>>>> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> sr-users(a)lists.kamailio.org >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>> >>>> >>>> >>>> -- >>>> Andy Chen >>>> Sr. Telephony Lead Engineer >>>> 415 516 5535 (M) >>>> achen@ &lt;achen(a)thinkingphones.com&gt;fuze.com >>>> >>>> >>>> *Confidentiality Notice: The information contained in this e-mail >>>> and any >>>> attachments may be confidential. If you are not an intended >>>> recipient, you >>>> are hereby notified that any dissemination, distribution or copying >>>> of this >>>> e-mail is strictly prohibited. If you have received this e-mail in >>>> error, >>>> please notify the sender and permanently delete the e-mail and any >>>> attachments immediately. You should not retain, copy or use this >>>> e-mail or >>>> any attachment for any purpose, nor disclose all or any part of the >>>> contents to any other person. Thank you.* >>>> >>>> _______________________________________________ >>>> Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> -- >>>> Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda >>>> Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat >>>> >>>> >>> >>> -- >>> Andy Chen >>> Sr. Telephony Lead Engineer >>> 415 516 5535 (M) >>> achen@ &lt;achen(a)thinkingphones.com&gt;fuze.com >>> >>> >> >> -- >> Andy Chen >> Sr. Telephony Lead Engineer >> 415 516 5535 (M) >> achen@ &lt;achen(a)thinkingphones.com&gt;fuze.com >> >> > > -- > Andy Chen > Sr. Telephony Lead Engineer > 415 516 5535 (M) > achen@ &lt;achen(a)thinkingphones.com&gt;fuze.com > > > *Confidentiality Notice: The information contained in this e-mail and > any > attachments may be confidential. If you are not an intended recipient, > you > are hereby notified that any dissemination, distribution or copying of > this > e-mail is strictly prohibited. If you have received this e-mail in > error, > please notify the sender and permanently delete the e-mail and any > attachments immediately. You should not retain, copy or use this > e-mail or > any attachment for any purpose, nor disclose all or any part of the > contents to any other person. Thank you.* > > -- > Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda > Kamailio Advanced Training, Oct 21-23, 2019, Berlin, Germany -- https://asipto.com/u/kat > > -- Andy Chen Sr. Telephony Lead Engineer 415 516 5535 (M) achen@ &lt;achen(a)thinkingphones.com&gt;fuze.com *Confidentiality Notice: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you.* _______________________________________________ Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Kamailio Merchandising - https://skalatan.de/merchandising/ Kamailio services - https://skalatan.de/services Henning Westerholt - https://skalatan.de/blog/