22 Feb
2010
22 Feb
'10
8:16 p.m.
Hi!
With kamailio 3.0 I encounter lots of crashes. I am using SNOM 320 and
eyebeam clients. I think crashes are mostly correlated with SNOM 320.
Server is CentOS5.4. OpenSSL is openssl-0.9.8e-12.el5_4.1.
First crashed Kamailio always with the same backtrace:
Program terminated with signal 11, Segmentation fault.
[New process 1580]
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
(gdb) bt
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
#1 0x00156d01 in ssl3_choose_cipher () from /lib/libssl.so.6
#2 0x00151b45 in ssl3_get_client_hello () from /lib/libssl.so.6
#3 0x00152525 in ssl3_accept () from /lib/libssl.so.6
#4 0x00167eda in SSL_accept () from /lib/libssl.so.6
#5 0x004a6cff in tls_accept (c=0xb60ff4a0, error=0x0) at tls_server.c:327
#6 0x004ac15b in tls_h_fix_read_conn (c=0xb60ff4a0) at tls_server.c:1005
#7 0x0813d146 in tcp_read_req (con=0xb60ff4a0, bytes_read=0xbfe248b4,
read_flags=0xbfe248b0) at tcp_read.c:654
#8 0x0813d8ef in handle_io (fm=0x82aa9d0, events=1, idx=-1) at
tcp_read.c:930
#9 0x08141513 in tcp_receive_loop (unix_sock=27) at io_wait.h:1057
#10 0x0810fc1b in tcp_init_children () at tcp_main.c:4253
#11 0x0809ae69 in main_loop () at main.c:1525
#12 0x0809bc02 in main (argc=1, argv=0xbfe24cc4) at main.c:2251
(gdb) quit
Then I configured RSA ciphers (as suggested by others) and now I get
different backtraces. Here is an example where Kamailio wrote 2 core files:
1. Program terminated with signal 11, Segmentation fault.
[New process 1735]
#0 0x004d01d3 in free_hash_table () at h_table.c:423
423
clist_foreach_safe(&_tm_table->entries[i], p_cell, tmp_cell,
(gdb) bt
#0 0x004d01d3 in free_hash_table () at h_table.c:423
#1 0x004de1a0 in tm_shutdown () at t_funcs.c:126
#2 0x08104ef2 in destroy_modules () at sr_module.c:635
#3 0x08098400 in cleanup (show_status=1) at main.c:509
#4 0x0809912e in shutdown_children (sig=<value optimized out>,
show_status=1) at main.c:649
#5 0x08099aa4 in handle_sigs () at main.c:740
#6 0x0809aab9 in main_loop () at main.c:1562
#7 0x0809bc02 in main (argc=1, argv=0xbfc70b74) at main.c:2251
2. Program terminated with signal 11, Segmentation fault.
[New process 1759]
#0 0x004f24ce in t_reply_matching (p_msg=0x8298cf0,
p_branch=0xbfc70424) at t_lookup.c:983
983 if (p_cell->label != entry_label)
(gdb) bt
#0 0x004f24ce in t_reply_matching (p_msg=0x8298cf0,
p_branch=0xbfc70424) at t_lookup.c:983
#1 0x004f5559 in t_check_msg (p_msg=0x8298cf0, param_branch=0xbfc70424)
at t_lookup.c:1138
#2 0x004f5e94 in t_check (p_msg=0x8298cf0, param_branch=0xbfc70424) at
t_lookup.c:1180
#3 0x005140d9 in reply_received (p_msg=0x8298cf0) at t_reply.c:1897
#4 0x0808c764 in forward_reply (msg=0x8298cf0) at forward.c:689
#5 0x080c401e in receive_msg (
buf=0xb60fe088 "SIP/2.0 200 Ok\r\nVia: SIP/2.0/TLS
83.136.32.167:5061;branch=z9hG4bKcc38.0a7d6be7.0;i=2\r\nVia: SIP/2.0/TLS
10.10.0.51:40487;received=83.136.33.3;branch=z9hG4bK-d8754z-7b0f1727475bda32-1---d8754z-;rport=2"...,
len=1150, rcv_info=0xb60fded4) at receive.c:257
#6 0x0813cf41 in tcp_read_req (con=0xb60fdec0, bytes_read=0xbfc70768,
read_flags=0xbfc70760) at tcp_read.c:761
#7 0x0813da0b in handle_io (fm=0x82aa708, events=1, idx=-1) at
tcp_read.c:980
#8 0x08141513 in tcp_receive_loop (unix_sock=25) at io_wait.h:1057
#9 0x0810fc1b in tcp_init_children () at tcp_main.c:4253
#10 0x0809ae69 in main_loop () at main.c:1525
#11 0x0809bc02 in main (argc=1, argv=0xbfc70b74) at main.c:2251
Attached is the log file.
I can provided intensive logs (memdbg) too.
thanks
klaus
22 Feb
22 Feb
9 p.m.
On Feb 22, 2010 at 19:16, Klaus Darilion <klaus.mailinglists(a)pernau.at> wrote:
Hi!
With kamailio 3.0 I encounter lots of crashes. I am using SNOM 320 and
eyebeam clients. I think crashes are mostly correlated with SNOM 320.
Server is CentOS5.4. OpenSSL is openssl-0.9.8e-12.el5_4.1.
First crashed Kamailio always with the same backtrace:
Program terminated with signal 11, Segmentation fault.
[New process 1580]
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
(gdb) bt
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
This one looks like some openssl bug. What's strange is that it's in
some kerberos code, which is supposed to be disabled.
Could you try with another openssl version, or even better compile
openssl with debug info and link with it?
What exactly did you need to do for reproducing it with a snom (any
settings you think there might be relevant both on the phone and in the
tls handling in the script)?
[...]
Then I configured RSA ciphers (as suggested by others) and now I get
different backtraces. Here is an example where Kamailio wrote 2 core files:
1. Program terminated with signal 11, Segmentation fault.
[New process 1735]
#0 0x004d01d3 in free_hash_table () at h_table.c:423
This crashed on shutdown for some reason (corrupted list?).
[...]
2. Program terminated with signal 11, Segmentation fault.
[New process 1759]
#0 0x004f24ce in t_reply_matching (p_msg=0x8298cf0,
p_branch=0xbfc70424) at t_lookup.c:983
983 if (p_cell->label != entry_label)
Hmm, this looks like a corrupted transaction list. It's hard to say if
it's caused by tls or is something unrelated.
Could you print p_cell and p_cell->label in gdb?
(gdb) bt
#0 0x004f24ce in t_reply_matching (p_msg=0x8298cf0,
p_branch=0xbfc70424) at t_lookup.c:983
#1 0x004f5559 in t_check_msg (p_msg=0x8298cf0, param_branch=0xbfc70424)
at t_lookup.c:1138
#2 0x004f5e94 in t_check (p_msg=0x8298cf0, param_branch=0xbfc70424) at
t_lookup.c:1180
#3 0x005140d9 in reply_received (p_msg=0x8298cf0) at t_reply.c:1897
#4 0x0808c764 in forward_reply (msg=0x8298cf0) at forward.c:689
#5 0x080c401e in receive_msg (
buf=0xb60fe088 "SIP/2.0 200 Ok\r\nVia: SIP/2.0/TLS
83.136.32.167:5061;branch=z9hG4bKcc38.0a7d6be7.0;i=2\r\nVia: SIP/2.0/TLS
10.10.0.51:40487;received=83.136.33.3;branch=z9hG4bK-d8754z-7b0f1727475bda32-1---d8754z-;rport=2"...,
len=1150, rcv_info=0xb60fded4) at receive.c:257
#6 0x0813cf41 in tcp_read_req (con=0xb60fdec0, bytes_read=0xbfc70768,
read_flags=0xbfc70760) at tcp_read.c:761
#7 0x0813da0b in handle_io (fm=0x82aa708, events=1, idx=-1) at
tcp_read.c:980
#8 0x08141513 in tcp_receive_loop (unix_sock=25) at io_wait.h:1057
#9 0x0810fc1b in tcp_init_children () at tcp_main.c:4253
#10 0x0809ae69 in main_loop () at main.c:1525
#11 0x0809bc02 in main (argc=1, argv=0xbfc70b74) at main.c:2251
Andrei
11:21 p.m.
On Feb 22, 2010 at 20:00, Andrei Pelinescu-Onciul <andrei(a)iptel.org> wrote:
On Feb 22, 2010 at 19:16, Klaus Darilion
<klaus.mailinglists(a)pernau.at> wrote:
It seems there is a code path inside openssl, for which the old disable
kerberos trick does not work and causes crashes instead. Probably we
haven't seen it before because we didn't tried a client that advertised
KRB5 key exchange and an openssl compiled with kerberos support.
The proper fix is to upgrade to a newer openssl (real 0.9.8e or above).
For people stuck with older openssl versions, I will try to find a way to
disable kerberos in a better way.
Could you try in the same configuration (no RSA workaround on the
phone), but with the following tls module options:
modparam("tls", "cipher_list", "DEFAULT:!KRB5")
If this works, please try also the RSA stuff on the phone and see if it
still crashes in that case (although that case might be unrelated).
If everything is ok I'll add it to the code, so that it will be done
automatically for openssl < 0.9.8e release.
Note that another very strange thing is the version number.
Hi!
With kamailio 3.0 I encounter lots of crashes. I am using SNOM 320 and
eyebeam clients. I think crashes are mostly correlated with SNOM 320.
Server is CentOS5.4. OpenSSL is openssl-0.9.8e-12.el5_4.1.
First crashed Kamailio always with the same backtrace:
Program terminated with signal 11, Segmentation fault.
[New process 1580]
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
(gdb) bt
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
This one looks like some openssl bug. What's strange is that it's in
some kerberos code, which is supposed to be disabled. From your log:
INFO: tls [tls_init.c:486]: tls:
_init_tls_h: compiled with openssl
version "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" (0x0090802f), kerberos
^^^^^^^ ^^^^^^^^^^
0x0090802f means openssl 0.9.8b (release) and not 0.9.8e.
It looks like OPENSSL_VERSION_NUMBER is different from
OPENSSL_VERSION_TEXT in you include/openssl/opensslv.h. Out o curiosity
could you verify that that's the case?
Andrei
23 Feb
23 Feb
10:15 a.m.
Some analysis ...
Am 22.02.2010 22:21, schrieb Andrei Pelinescu-Onciul:
Note that another very strange thing is the version
number.
From your log:
INFO: tls [tls_init.c:486]: tls: _init_tls_h: compiled with openssl
version "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" (0x0090802f), kerberos
^^^^^^^ ^^^^^^^^^^
0x0090802f means openssl 0.9.8b (release) and not 0.9.8e.
It looks like OPENSSL_VERSION_NUMBER is different from
OPENSSL_VERSION_TEXT in you include/openssl/opensslv.h. Out o curiosity
could you verify that that's the case?
# cat /etc/redhat-release
CentOS release 5.4 (Final)
# rpm -qa|grep ssl
openssl-devel-0.9.8e-12.el5_4.1
openssl-0.9.8e-12.el5_4.1
extract from /usr/include/openssl/opensslv.h:
#define OPENSSL_VERSION_NUMBER 0x0090802fL
#ifdef OPENSSL_FIPS
#define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008"
#else
#define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-rhel5 01 Jul 2008"
#endif
#define OPENSSL_VERSION_PTEXT " part of " OPENSSL_VERSION_TEXT
really strange. I downloaded and extracted RH
openssl-0.9.8e-12.el5_4.1.src.rpm and greped for "OPENSSL_VERSION":
openssl-fips-0.9.8e-abi.patch:-#define OPENSSL_VERSION_NUMBER 0x00908060L
openssl-fips-0.9.8e-abi.patch:+#define OPENSSL_VERSION_NUMBER 0x0090802fL
openssl-fips-0.9.8e-abi.patch:-#define OPENSSL_VERSION_TEXT "OpenSSL
0.9.8f-fips-dev xx XXXX xxxx"
openssl-fips-0.9.8e-abi.patch:+#define OPENSSL_VERSION_TEXT "OpenSSL
0.9.8e-fips-rhel5 01 Jul 2008"
Further the SRPM contains openssl-fips-0.9.8e-usa.tar.bz2 which contains
a opensslv.h with:
#define OPENSSL_VERSION_NUMBER 0x00908060L
#ifdef OPENSSL_FIPS
#define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8f-fips-dev xx XXXX xxxx"
#else
#define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8f-dev xx XXXX xxxx"
#endif
#define OPENSSL_VERSION_PTEXT " part of " OPENSSL_VERSION_TEXT
Looks like RH is using a 0.9.8f-dev version, which is named 0.9.8e, and
numbered like 0.9.8b - maybe for compatibility reasons with earlier
RHEL5 releases.
regards
klaus
12:22 a.m.
Andrei Pelinescu-Onciul wrote:
On Feb 22, 2010 at 19:16, Klaus Darilion
<klaus.mailinglists(a)pernau.at> wrote:
I also guess this is something RH/CentOS related. Rebuilding openssl
myself could help, but the thing is that this is current CentOS/Redhat
which is used rather often.
klaus
Hi!
With kamailio 3.0 I encounter lots of crashes. I am using SNOM 320 and
eyebeam clients. I think crashes are mostly correlated with SNOM 320.
Server is CentOS5.4. OpenSSL is openssl-0.9.8e-12.el5_4.1.
First crashed Kamailio always with the same backtrace:
Program terminated with signal 11, Segmentation fault.
[New process 1580]
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
(gdb) bt
#0 0x0017451d in kssl_keytab_is_available () from /lib/libssl.so.6
This one looks like some openssl bug. What's strange is that it's in
some kerberos code, which is supposed to be disabled.
Could you try with another openssl version, or even better compile
openssl with debug info and link with it?
9:40 a.m.
Hi Andrei!
Am 22.02.2010 20:00, schrieb Andrei Pelinescu-Onciul:
What exactly did you need to do for reproducing it
with a snom (any
settings you think there might be relevant both on the phone and in the
tls handling in the script)?
I think the script is not relevant. Seems like it crashes immediately
with the first TLS request before entering script.
[1323]: DEBUG: <core> [ip_addr.c:116]: tcpconn_new: new tcp connection:
83.136.33.3
[1323]: DEBUG: <core> [tcp_main.c:1052]: tcpconn_new: on port 2239, type 3
[1323]: DEBUG: <core> [tcp_main.c:1351]: tcpconn_add: hashes:
638:2058:3048, 1
[1323]: DEBUG: <core> [io_wait.h:361]: DBG: io_watch_add(0x8217f60, 27,
2, 0xb60e6390), fd_no=19
[1323]: DEBUG: <core> [io_wait.h:588]: DBG: io_watch_del (0x8217f60, 27,
-1, 0x0) fd_no=20 called
[1323]: DEBUG: <core> [tcp_main.c:3627]: tcp: DBG: sendig to child, events 1
[1323]: DEBUG: <core> [tcp_main.c:3336]: send2child: to tcp child 0
10(1319), 0xb60e6390
[1319]: DEBUG: <core> [tcp_read.c:884]: received n=4 con=0xb60e6390, fd=8
[1319]: DEBUG: tls [tls_server.c:109]: Using TLS domain TLSs<default>
[1323]: : <core> [pass_fd.c:283]: ERROR: receive_fd: EOF on 18
[1323]: DEBUG: <core> [tcp_main.c:3038]: DBG: handle_ser_child: dead
child 10, pid 1319 (shutting down?)
[1323]: DEBUG: <core> [io_wait.h:588]: DBG: io_watch_del (0x8217f60, 18,
-1, 0x0) fd_no=19 called
[1323]: DEBUG: <core> [tcp_main.c:2826]: DBG: handle_tcp_child: dead tcp
child 0 (pid 1319, no 10) (shutting down?)
[1323]: DEBUG: <core> [io_wait.h:588]: DBG: io_watch_del (0x8217f60, 20,
-1, 0x0) fd_no=18 called
[1297]: ALERT: <core> [main.c:722]: child process 1319 exited by a signal 11
Attached is the complete log file from kamailio startup until crash with
first message.
Further I have attached the bt of the generated cores (2 cores were
generated). As you already said this looks like a bug in openssl.
# cat /etc/redhat-release
CentOS release 5.4 (Final)
# rpm -qa|grep ssl
openssl-devel-0.9.8e-12.el5_4.1
openssl-0.9.8e-12.el5_4.1
Strangly, the ciphers announced by SNOM do not indicate Kerberos at all:
SSL Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.0 (0x0301)
Length: 63
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 59
Version: TLS 1.0 (0x0301)
Random
Session ID Length: 0
Cipher Suites Length: 20
Cipher Suites (10 suites)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_NULL_MD5 (0x0001)
Cipher Suite: TLS_RSA_WITH_NULL_SHA (0x0002)
Cipher Suite: TLS_DH_anon_WITH_3DES_EDE_CBC_SHA (0x001b)
Cipher Suite: TLS_DH_anon_WITH_RC4_128_MD5 (0x0018)
Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
Cipher Suite: TLS_RSA_EXPORT1024_WITH_RC4_56_SHA (0x0064)
Cipher Suite: TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA (0x0062)
Cipher Suite: TLS_DH_anon_WITH_DES_CBC_SHA (0x001a)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Attached is also the pcap file of the SNOM phone.
regards
Klaus
5386
days inactive
5387
days old
5 comments
2 participants
participants (2)
-
Andrei Pelinescu-Onciul -
Klaus Darilion