20 May
2019
20 May
'19
10:49 a.m.
Hello,
this kind of behaviour, with long time blocking and then moving on, is a
symptom of the same issue. One of the observed behaviours was that
attaching with gdb and detaching make code running further, that's what
kamctl trap does. I haven't looked deeper, but my guess is that some
signals are sent during the gdb operations.
It would be good if you can test with the workaround and see the
results. There was already a report that the issue was not seen after a
rather long running time.
Cheers,
Daniel
On 17.05.19 16:03, Aymeric Moizard wrote:
Hi!
I haven't used the workaround yet: I'm focusing on trying to make sure
I have the same issue
or trying to figure out how to force it to happen.
I have started to check again the server today and I started by this
command:
$> sudo kamcmd tls.list
In my previous description, the above was a dead lock. Today, It
finally completed, but
after 5 minutes. (I suspect 5 minutes is abnormal)
During the long running command:
-> UDP was working
-> TCP was not:
-> The TCP connection is being ESTABLISHED, but the SIP message was
not replied.
(this was the behavior I had before)
At the same time, I took a trap "sudo kamctl trap". (during the dead lock)
-> one thread is on "tls_list" (tls_rpc.c:154)
-> one thread is on tcpconn_get (core/tcp_main.c:1449) called
from tcp_send (core/tcp_main.c:1716)
and seems to be sending a 484 Address Incomplete on a TLS connection
-> 2 threads are on CRYPTO_THREAD_write_lock on a backtrace showing
"SSL_do_handshake/tls_accept"
Suddenly, "sudo kamcmd tls.list" completed, and then, my TCP Agent
received
4 answers from kamailio for the last 4 REGISTER sent.
I have a network capture for my TCP agent.
I have a trap showing 2 thread waiting on "CRYPTO_THREAD_write_lock"
Conclusion:
The use-case showed that the lock was VERY long.
The use-case showed that the lock was TEMPORARY...
Side-note: From my understanding of the multi-fork/openssl issue, I
would expect
to see dead lock happening very fast after a kamailio restart?
Do you expect the preload workaround to work in such behavior?
Or do you consider that my issue is different?
Because there is no "real" dead-lock, I don't understand why
"my"
issue would be related to libssl1.1...
My gdb trap, network capture are available in private exchange if you
need! (please ask me by direct email)
Tks
Aymeric
Le lun. 13 mai 2019 à 12:48, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> a écrit :
Hello,
thanks for the feedback! It is good to know that it works well so
far for you. I don't see any reason not to make the library to
preload as part of the next release.
Just to let everyone know, for now, the built packages are pinned
to link against libssl 1.0.x.
Soon, I will approach the openssl project in order to find a
proper solution for long term.
Cheers,
Daniel
On 13.05.19 10:48, Floimair Florian wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Hi all!
We have used the work-around with the pre-loaded library and so
far this seems to have fixed our problem (that my colleague
Kristijan Vrban reported).
At least we did not have a single failure within the last week,
whereas before the issue happened about once every 2 days.
Would be nice if this would be part of the next Kamailio version.
With best regards
*Florian Floimair
*Innovation - Software-Development
*COMMEND INTERNATIONAL GMBH
*A-5020 Salzburg, Saalachstraße 51
http://www.commend.com <http://www.commend.com/>
*Security and Communication by Commend
*FN 178618z | LG Salzburg
*Von: *sr-users <sr-users-bounces(a)lists.kamailio.org>
<mailto:sr-users-bounces@lists.kamailio.org> im Auftrag von
Daniel-Constantin Mierla <miconda(a)gmail.com>
<mailto:miconda@gmail.com>
*Antworten an: *"miconda(a)gmail.com" <mailto:miconda@gmail.com>
<miconda(a)gmail.com> <mailto:miconda@gmail.com>, "Kamailio (SER) -
Users Mailing List" <sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>
*Datum: *Montag, 15. April 2019 um 09:07
*An: *Aymeric Moizard <amoizard(a)gmail.com>
<mailto:amoizard@gmail.com>, "Kamailio (SER) - Users Mailing
List" <sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>
*Betreff: *Re: [SR-Users] Kamailio stop to process incoming SIP
traffic via TCP.
Hello Aymeric,
would you be able to test with tls module compiled against libssl
1.1 and using the pre-loaded shared object workaround?
*
https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/open…
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkamailio%2Fkamailio%2Ftree%2Fmaster%2Fsrc%2Fmodules%2Ftls%2Futils%2Fopenssl_mutex_shared&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526480174&sdata=d9E%2Fy4cvdLkGCPUexoCJ7tws3QL4rFqz5ebnMGnsESQ%3D&reserved=0>
You should be able to use it with any version, no need to test
with kamailio master branch.
Just clone the master branch, then:
cd src/modules/tls/utils/openssl_mutex_shared
make
Either from there or copy openssl_mutex_shared.so to a location
you want, then pre-load it before starting your version of Kamailio.
The README.md in the folder has some more details.
I would like to have some validation that it works fine before
approaching this topic with libssl project to allow to init the
locks with shared process option.
Thanks,
Daniel
On 26.03.19 16:18, Daniel-Constantin Mierla wrote:
Hello,
yep, locking there is expected, as listing the tls
connections wait for no other processes to change the content
of internal tls connection structures. So it is a side effect
of libssl/libcrypto getting stuck and the other processing
waiting for it to move one. I have the Kamailio training in
USA these days, so the trip and schedule of the day didn't
allow me to look more at the libsll/libcrypto code in order
to find a solution here. It is a high priority in my list, as
I get time during the next days.
Cheers,
Daniel
On 26.03.19 15:55, Aymeric Moizard wrote:
Hi All,
I was debugging a TCP issue (most probably, I may start a
thread for this question).
I was trying to get some info for TCP and TLS.
I typed:
$> sudo kamctl rpc tls.list
And waited for a while.... until... I realized that my
User-Agent, connected with TCP was not able to register
any more. I think the rpc command has introduced
something wrong.
The device can successfully "connect", send the REGISTER
over the established TCP connection. The REGISTER do not
appear in the logs any more, I don't see any traffic for
TCP any more. So the behavior is the same as I had
before: TCP and TLS are both not working and UDP is still
working fine.
kamctl do not work any more... so kamctl trap do not work...
I have been able to type.. manually... for (all?)
kamailio threads:
gdb /usr/sbin/kamailio 16500 -batch --eval-command="bt
full" >> kamailio-trap-tcp-down.txt
I'm temporarly puting the backtrace I have here:
https://sip.antisip.com/kamailio-trap-tcp-down.txt
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsip.antisip.com%2Fkamailio-trap-tcp-down.txt&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=1lfFxvR0m4PVcfnYsrrIO%2FM2nbGK6zfpl2C01O2c7M0%3D&reserved=0>
You can see a thread stuck on the json command line:
"tls_list"
And many other waiting on CRYPTO_THREAD_write_lock
? might be related to:
https://github.com/openssl/openssl/issues/5376
<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenssl%2Fopenssl%2Fissues%2F5376&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526490178&sdata=D5Fb4U3trdbRUY7ifMLSc5KE4mAxjK2%2BzOy8nSD1Rks%3D&reserved=0>
SIDE NOTE:
Right before I was typing the last gdb command for the
last thread, kamailio
has crashed: This was around 5 minutes after the dead
lock started.
Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core>
[core/tcp_main.c:2561]: tcpconn_do_send(): failed to send
on 0x7ff8dfc2fdc8 (91.121.30.149:5061->62.210.97.21:49351
<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2F62.210.97.21%3A49351&data=02%7C01%7Cf.floimair%40commend.com%7C4008d49af1b347abe20308d6c1710532%7C13b1ddb756454e7fbe663171548559da%7C0%7C0%7C636909088526500195&sdata=9XqEUKoMwNEvCPFtKfvB0c43yk1GcSzYOiPdY9Pj1uo%3D&reserved=0>):
Broken pipe (32)
Mar 26 14:47:11 sip kamailio[16493]: ERROR: <core>
[core/tcp_read.c:1505]: tcp_read_req(): ERROR:
tcp_read_req: error reading - c: 0x7ff8dfc2fdc8 r:
0x7ff8dfc2fe48 (-1)
Mar 26 14:47:11 sip kamailio[16493]: WARNING: <core>
[core/tcp_read.c:1848]: handle_io(): F_TCPCONN connection
marked as bad: 0x7ff8dfa6a408 id 846 refcnt 3
Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core>
[main.c:755]: handle_sigs(): child process 16374 exited
by a signal 11
Mar 26 14:47:11 sip kamailio[16371]: ALERT: <core>
[main.c:758]: handle_sigs(): core was not generated
Mar 26 14:47:11 sip kamailio[16371]: INFO: <core>
[main.c:781]: handle_sigs(): terminating due to SIGCHLD
Mar 26 14:47:11 sip kamailio[16493]: INFO: <core>
[main.c:836]: sig_usr(): signal 15 received
Mar 26 14:47:11 sip kamailio[16500]: INFO: <core>
[main.c:836]: sig_usr(): signal 15 received
Mar 26 14:47:11 sip kamailio[16479]: INFO: <core>
[main.c:836]: sig_usr(): signal 15 received
Unfortunalty, even if I did my best to setup my service
to generate a core on crash, I still have "core was not
generated".... (debian stretch)
Tks for reading!
Regards
Aymeric
--
Antisip - http://www.antisip.com