6 Jan
2020
6 Jan
'20
10:52 p.m.
You can download the code from git repository and build the
openssl_mutex_shared.so locally.
Or install from the nightly builts, there should be the version with the
fix embedded -- after installation check kamailio -I and see if it lists
TLS_PTHREAD_MUTEX_SHARED.
Cheers,
Daniel
On 06.01.20 16:48, Andrew Chen wrote:
I have seen similar issues with pike and running 5.2.5
installed using
deb repo. I don't see this openssl_mutex_shared directory:
root@ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio # ls -lart
total 400
-rw-r--r-- 1 root root 22528 Oct 10 12:29 libtrie.so.1.0
lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so.1 -> libtrie.so.1.0
lrwxrwxrwx 1 root root 14 Oct 10 12:29 libtrie.so -> libtrie.so.1.0
-rw-r--r-- 1 root root 63864 Oct 10 12:29 libsrutils.so.1.0
lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so.1 ->
libsrutils.so.1.0
lrwxrwxrwx 1 root root 17 Oct 10 12:29 libsrutils.so ->
libsrutils.so.1.0
-rw-r--r-- 1 root root 51472 Oct 10 12:29 libsrdb2.so.1.0
lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so.1 ->
libsrdb2.so.1.0
lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb2.so -> libsrdb2.so.1.0
-rw-r--r-- 1 root root 211264 Oct 10 12:29 libsrdb1.so.1.0
lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so.1 ->
libsrdb1.so.1.0
lrwxrwxrwx 1 root root 15 Oct 10 12:29 libsrdb1.so -> libsrdb1.so.1.0
drwxr-xr-x 4 root root 4096 Dec 3 21:32 .
drwxr-xr-x 3 root root 4096 Dec 3 21:33 kamctl
drwxr-xr-x 2 root root 4096 Dec 3 21:33 modules
drwxr-xr-x 47 root root 36864 Dec 10 20:13 ..
root@ashmainkama51:/usr/lib/x86_64-linux-gnu/kamailio #
and here are my command outputs:
root@ashmainkama51:~ # ldd
/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
linux-vdso.so.1 (0x00007ffd28de0000)
libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
(0x00007f925de6d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f925da7c000)
libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
(0x00007f925d5b1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f925d392000)
/lib64/ld-linux-x86-64.so.2 (0x00007f925e39d000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007f925d18e000)
root@ashmainkama51:~ # kamailio -I
Print out of kamailio internals
Version: kamailio 5.2.5 (x86_64/linux)
Default config: /etc/kamailio/kamailio.cfg
Default paths to modules: /usr/lib/x86_64-linux-gnu/kamailio/modules
Compile flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
MAX_RECV_BUFFER_SIZE=262144
MAX_URI_SIZE=1024
BUF_SIZE=65535
DEFAULT PKG_SIZE=8MB
DEFAULT SHM_SIZE=64MB
ADAPTIVE_WAIT_LOOPS=1024
TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt, select
Source code revision ID: unknown
Compiled with: gcc 7.4.0
Compiled on:
Thank you for flying kamailio!
On Mon, Dec 16, 2019 at 12:51 PM Aymeric Moizard <amoizard(a)gmail.com
<mailto:amoizard@gmail.com>> wrote:
Hi Daniel,
The file openssl_mutex_shared.so is there and same installation
time than the other files!
I have added
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
To my kamailio.service and I have restarted!
I'm sure it will behave better now ;)
Tks a lot for the help.
Regards,
Aymeric
Le lun. 16 déc. 2019 à 17:23, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> a écrit :
I pinged Victor to see if he can figure out what happens
within the deb building process that makes the libssl mutex
fix not enabled.
The extra .so preload object should be still installed, try to
see if it is at:
/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so
Cheers,
Daniel
On 16.12.19 12:09, Aymeric Moizard wrote:
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
Good catch!
As I said in my first mail, I also add the issue with latest
5.2.X so I suppose the deb package has the same issue for 52X.
Is the extra binary to load still there? I will check that as
soon as I'm online...
Tks a lot!
Aymeric
Le lun. 16 déc. 2019 à 11:16, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> a écrit :
Hello,
for some reason the binary doesn't seem to have the
libssl mutex fix, in my system with the libssl 1.1 gives:
# kamailio -I
Print out of kamailio internals
Version: kamailio 5.3.1 (x86_64/linux) f36ac2
Default config: /tmp/kamailio-5.3/etc/kamailio/kamailio.cfg
Default paths to modules:
/tmp/kamailio-5.3/lib64/kamailio/modules
Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
MAX_RECV_BUFFER_SIZE=262144
MAX_URI_SIZE=1024
BUF_SIZE=65535
DEFAULT PKG_SIZE=8MB
DEFAULT SHM_SIZE=64MB
ADAPTIVE_WAIT_LOOPS=1024
TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt,
select
Source code revision ID: f36ac2
Compiled with: gcc 9.2.1
Compiled architecture: x86_64
Compiled on: 11:11:20 Dec 16 2019
Thank you for flying kamailio!
The important part above is the presence of
TLS_PTHREAD_MUTEX_SHARED compile time flag in the output.
Needs to be investigated why the dep packages have the
kamailio binary without the libssl mutex fix enabled.
Cheers,
Daniel
On 16.12.19 09:22, Aymeric Moizard wrote:
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin -- www.kamailioworld.com
<http://www.kamailioworld.com>
--
Antisip - http://www.antisip.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Andy Chen
Sr. Telephony Lead Engineer
415 516 5535 (M)
achen@ <mailto:achen@thinkingphones.com>fuze.com <http://fuze.com>
*Confidentiality Notice: The information contained in this e-mail and any
attachments may be confidential. If you are not an intended recipient, you
are hereby notified that any dissemination, distribution or copying of
this
e-mail is strictly prohibited. If you have received this e-mail in error,
please notify the sender and permanently delete the e-mail and any
attachments immediately. You should not retain, copy or use this e-mail or
any attachment for any purpose, nor disclose all or any part of the
contents to any other person. Thank you.*
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Hi Daniel,
Tks a lot for lookint at it.
$ ldd /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so
linux-vdso.so.1 (0x00007fff997dd000)
libssl.so.1.1 =>
/usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fe40b53c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
(0x00007fe40b19d000)
libcrypto.so.1.1 =>
/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
(0x00007fe40ad03000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007fe40aaff000)
libpthread.so.0 =>
/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe40a8e2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe40ba4a000)
$ /usr/sbin/kamailio -I
Print out of kamailio internals
Version: kamailio 5.3.1 (x86_64/linux)
Default config: /etc/kamailio/kamailio.cfg
Default paths to modules:
/usr/lib/x86_64-linux-gnu/kamailio/modules
Compile flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS,
USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
MAX_RECV_BUFFER_SIZE=262144
MAX_URI_SIZE=1024
BUF_SIZE=65535
DEFAULT PKG_SIZE=8MB
DEFAULT SHM_SIZE=64MB
ADAPTIVE_WAIT_LOOPS=1024
TCP poll methods: poll, epoll_lt, epoll_et, sigio_rt,
select
Source code revision ID: unknown
Compiled with: gcc 6.3.0
Compiled architecture: x86_64
Compiled on:
Thank you for flying kamailio!
Additional note:
I have tried to better understand the pike module and
after reading the "end" of the module documentation,
I do better understand the "Tree of IP" and settings.
The pike documentation, for each settins and
description, should refer to the section "Chapter 3.
Developer Guide",
otherwise, the parameters cannot be understood. Also,
it's not possible to understand, according to me, the
real time
for removing an IP from the tree (removing it 100% or
only last node of IP)
Looking again at my statistics, I feel the first graph
is definitly showing an issue. This graph is showing
"$stat(location-users)" and "$stat(location-contacts)".
During the 10 hours, many users are banned, unregistred,
etc..
so it is really not expected that the number of
registred users is maintained. From what I understand,
the fact
that the stats went down when deadlock dissapeared
obviouly means kamailio threads was in a bad state for the
last 10 hours...
https://www.antisip.com/sip-antisip-com-register/status2.htm
If you need more information, let me know...
Regards
Aymeric
Le lun. 16 déc. 2019 à 08:22, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> a écrit :
Hello,
can you provide output of ldd for tls.so and output
of "kamailio -I" (that's an uppercase i)?
Cheers,
Daniel
On 13.12.19 16:39, Aymeric Moizard wrote:
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin --
www.kamailioworld.com <http://www.kamailioworld.com>
Hi List,
History:
* In the past, I had deadlock which was, most
probably, related to ssl1.1.
We have discussed this issue, and a fix is
supposed to workaround the issue that was detected.
* With latest 5.2.X, I have experienced ONCE a
similar behavior with TCP and TLS being mostly
stuck. I have not been using this version much, but
the fix was supposed to be in the core of kamailio.
The status of the server this night:
* I'm today running version: kamailio 5.3.1
(x86_64/linux),
* Installed on stretch using
http://deb.kamailio.org/kamailio53 repository.
* This versions use libssl1.1
* A user reported that he can't connect with TCP
* An average of 5000 IPs per 10 minutes are being
banned by the pike module
(could be twice the same)
Yesterday/Today:
* at the end of the outage, I had 2479 IP in my
ipban htable. (which is equivalent to my statistics
showing 2 bans/IP every 10 minutes = 5000)
* looking at my logs, it appears that most (ALL?)
ip being banned... are my regular users.
* looking at my logs, I can't understand why pike
would block them.
This is a graph for statistics on my service for
the last 24 hours:
https://www.antisip.com/sip-antisip-com-register/status2.html
Yesterday, at 22:18:39, kamailio started to BAN
some IPs. 52 IPs were banned in a period of 10
minutes. I can confirm this from my logs.
My pike configuration is this one:
modparam("pike", "sampling_time_unit", 2)
modparam("pike", "reqs_density_per_unit", 64)
modparam("pike", "remove_latency", 4)
When detecting the issue, this morning, I typed:
$> sudo kamctl stats
$> sudo kamcmd htable.dump ipban
//FAILURE (answer too large...)
$> sudo kamctl trap
Then, I started an agent with TCP and it worked...???
Then, a few seconds, may be a minute after:
$> sudo kamcmd htable.dump ipban
//SUCCESS and shows 2479 banned ip.
and... everything is back to normal in a few minutes.
I haven't restarted kamailio, and all statistics
are as expected, as usual.
Thus, it looks that " sudo kamctl trap" has
triggered something. I already
experienced a similar behavior -when testing my
ssl1.1 deadlock last year-.
2 questions:
1/ I beleive my "pike" configuration should not ban
users. Is my pike configuration wrong?
As an example, pike has banned an IP sending one
message/second. I believe my configuration should
accept that?
2/ Could there still be a TLS issue with libssl1.1?
This is the result of the "kamctl trap":
https://sip.antisip.com/kamailio-pike-or-tls-issue-13-12-2019.kamctl-trap
Sorry for the long story & hoping to find a long
term solution or at least a workaround!
Regards
Aymeric
--
Antisip - http://www.antisip.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com>
www.twitter.com/miconda <http://www.twitter.com/miconda> --
www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda>
Kamailio World Conference - April 27-29, 2020, in Berlin --
www.kamailioworld.com <http://www.kamailioworld.com>
--
Antisip - http://www.antisip.com