in tls.cfg, i made ca_list to point to ca-certificates.crt file that comes with debian. when i then started sip router, i got errors
Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:447]: TLSc<default>: Unable to load CA list '/etc/ssl/certs/ca-certificates.crt' Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:0906D041:PEM routines:PEM_read_bio:malloc failure Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:0B084009:x509 certificate routines:X509_load_cert_crl_file:PEM lib Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: <core> [sr_module.c:889]: init_mod_child(): Error while initializing module tls (/usr/lib/sip-proxy/modules/tls.so) Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: <core> [main.c:1554]: ERROR: main: error in init_child(PROC_INT) -- exiting
is that because there are too many (= 139) entries in ca-certificates.crt? if so, how to solve the problem? would it be a good idea that sip router would be default be able to handle this correctly?
-- juha
Hello,
On 3/19/12 7:57 AM, Juha Heinanen wrote:
in tls.cfg, i made ca_list to point to ca-certificates.crt file that comes with debian. when i then started sip router, i got errors
Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:447]: TLSc<default>: Unable to load CA list '/etc/ssl/certs/ca-certificates.crt' Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:07064041:memory buffer routines:BUF_MEM_grow:malloc failure Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:0906D041:PEM routines:PEM_read_bio:malloc failure Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR: tls [tls_domain.c:448]: load_ca_list:error:0B084009:x509 certificate routines:X509_load_cert_crl_file:PEM lib Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR:<core> [sr_module.c:889]: init_mod_child(): Error while initializing module tls (/usr/lib/sip-proxy/modules/tls.so) Mar 19 08:45:42 test /usr/sbin/sip-proxy[28093]: ERROR:<core> [main.c:1554]: ERROR: main: error in init_child(PROC_INT) -- exiting
is that because there are too many (= 139) entries in ca-certificates.crt? if so, how to solve the problem? would it be a good idea that sip router would be default be able to handle this correctly?
is this with default mem size?
TLS requires lot more memory that usual UDP to deal with a fair enough amount of active connections. In debian, there is /etc/default/kamailio that is used to set -m and -M parameters for using with /etc/init.d/kamailio.
Cheers, Daniel
Daniel-Constantin Mierla writes:
is this with default mem size?
i would just like to add that once i increased shared memory from 32m to 64m, sip proxy started, but only about 10M was left as free:
Shared Memory
free_size 8510256 max_used_size 58623504 real_used_size 58598608 total_size 67108864
it means that debian's ca_list consumes more than 20M of shared memory, which seems like a lot for 139 entries.
-- juha
Hello,
On 3/19/12 10:03 AM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
is this with default mem size?
i would just like to add that once i increased shared memory from 32m to 64m, sip proxy started, but only about 10M was left as free:
afaik, the ca_list is loaded by libssl internally. Kamailio just sets custom memory manager and gives the path to ca_list.
Shared Memory
free_size 8510256 max_used_size 58623504 real_used_size 58598608 total_size 67108864
it means that debian's ca_list consumes more than 20M of shared memory, which seems like a lot for 139 entries.
Is this by one ca_list only, or do you have many tls domains, each with own profile?
Cheers, Daniel
Daniel-Constantin Mierla writes:
Is this by one ca_list only, or do you have many tls domains, each with own profile?
just a single ca_list and tls domain. my tls.cfg looks like this:
[client:default] verify_certificate = yes require_certificate = yes tls_method = TLSv1 private_key = /etc/sip-proxy/certs/sip-proxy/key.pem certificate = /etc/sip-proxy/certs/sip-proxy/cert.pem #ca_list = /etc/sip-proxy/certs/demoCA/cert.pem ca_list = /etc/ssl/certs/ca-certificates.crt
[server:default] verify_certificate = no require_certificate = no tls_method = SSLv23 private_key = /etc/sip-proxy/certs/sip-proxy/key.pem certificate = /etc/sip-proxy/certs/sip-proxy/cert.pem #ca_list = /etc/sip-proxy/certs/demoCA/cert.pem ca_list = /etc/ssl/certs/ca-certificates.crt
sending over tls works fine if i use line
ca_list = /etc/sip-proxy/certs/demoCA/cert.pem
where cert.pem contains just three ca certs one of them being cacert.org ca cert that i use in this test.
-- juha
On 3/19/12 10:47 AM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
Is this by one ca_list only, or do you have many tls domains, each with own profile?
just a single ca_list and tls domain. my tls.cfg looks like this:
[client:default] verify_certificate = yes require_certificate = yes tls_method = TLSv1 private_key = /etc/sip-proxy/certs/sip-proxy/key.pem certificate = /etc/sip-proxy/certs/sip-proxy/cert.pem #ca_list = /etc/sip-proxy/certs/demoCA/cert.pem ca_list = /etc/ssl/certs/ca-certificates.crt
[server:default] verify_certificate = no require_certificate = no tls_method = SSLv23 private_key = /etc/sip-proxy/certs/sip-proxy/key.pem certificate = /etc/sip-proxy/certs/sip-proxy/cert.pem #ca_list = /etc/sip-proxy/certs/demoCA/cert.pem ca_list = /etc/ssl/certs/ca-certificates.crt
sending over tls works fine if i use line
ca_list = /etc/sip-proxy/certs/demoCA/cert.pem
where cert.pem contains just three ca certs one of them being cacert.org ca cert that i use in this test.
I guess it is loaded two time, for the server and client profiles. Try to set it via dedicated module parameter and see if you get better memory usage:
http://kamailio.org/docs/modules/stable/modules/tls.html#ca_list
Overall, libssl and tls in general are known to be very greedy with memory. Maybe Andrei or Jan are available to write more details about it, as they implemented specific parts (also workarounds for libssl bugs) of tls module.
For a heavy loaded tls sip server, lot memory is important.
Cheers, Daniel
Daniel-Constantin Mierla writes:
I guess it is loaded two time, for the server and client profiles. Try to set it via dedicated module parameter and see if you get better memory usage:
http://kamailio.org/docs/modules/stable/modules/tls.html#ca_list
i tried and it turned out that it is not possible to mix and match tls config file and module params. if config file param file is given, then mod param ca_list is ignored.
also, it looks like it is not possible to share the same ca_list between different tls.cfg sections, but each section needs to have its own ca_list entry, which then increases memory requirement.
-- juha
I know this is not a solution, but you can also try to run
dpkg-reconfigure ca-certificates
and select only a couple of CA certificates you trust. That should make the list much smaller. Debian includes a lot of CA certificates in its default list and I am not sure whether it is a good idea to trust them all blindly, given some of the recent issues with bad CAs..
-Jan
On Mon, Mar 19, 2012 at 07:59, Juha Heinanen jh@tutpro.com wrote:
Daniel-Constantin Mierla writes:
I guess it is loaded two time, for the server and client profiles. Try to set it via dedicated module parameter and see if you get better memory usage:
http://kamailio.org/docs/modules/stable/modules/tls.html#ca_list
i tried and it turned out that it is not possible to mix and match tls config file and module params. if config file param file is given, then mod param ca_list is ignored.
also, it looks like it is not possible to share the same ca_list between different tls.cfg sections, but each section needs to have its own ca_list entry, which then increases memory requirement.
-- juha
btw, the devel branch as well as last stable releases are compiled with memory debugger on, you can compile with F_MALLOC and try to see if you get some improvements, but I guess they will not be really relevant over all.
Cheers, Daniel
On 3/19/12 4:04 PM, Jan Janak wrote:
I know this is not a solution, but you can also try to run
dpkg-reconfigure ca-certificates
and select only a couple of CA certificates you trust. That should make the list much smaller. Debian includes a lot of CA certificates in its default list and I am not sure whether it is a good idea to trust them all blindly, given some of the recent issues with bad CAs..
-Jan
On Mon, Mar 19, 2012 at 07:59, Juha Heinanenjh@tutpro.com wrote:
Daniel-Constantin Mierla writes:
I guess it is loaded two time, for the server and client profiles. Try to set it via dedicated module parameter and see if you get better memory usage:
http://kamailio.org/docs/modules/stable/modules/tls.html#ca_list
i tried and it turned out that it is not possible to mix and match tls config file and module params. if config file param file is given, then mod param ca_list is ignored.
also, it looks like it is not possible to share the same ca_list between different tls.cfg sections, but each section needs to have its own ca_list entry, which then increases memory requirement.
-- juha
On 3/19/12 12:59 PM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
I guess it is loaded two time, for the server and client profiles. Try to set it via dedicated module parameter and see if you get better memory usage:
http://kamailio.org/docs/modules/stable/modules/tls.html#ca_list
i tried and it turned out that it is not possible to mix and match tls config file and module params. if config file param file is given, then mod param ca_list is ignored.
right, would have been hard to distinguish if one wants to use default one or none at all when ca_list is not specified in the server profile.
Maybe we can improve a bit in the way that if there are same ca_list and crl path, to load once and do references from the other profiles. I don't know if there is lot of coding, but should be kept in mind for future optimizations. Perhaps could be added in the tracker not to forget about.
also, it looks like it is not possible to share the same ca_list between different tls.cfg sections, but each section needs to have its own ca_list entry, which then increases memory requirement.
At some point I thought about having a global section, for fallback purposes, like if not specified per server, use global options. But then gets into the issue presented in previous paragraph -- how to specify not to use one at all for some servers.
Cheers, Daniel
Daniel-Constantin Mierla writes:
Maybe we can improve a bit in the way that if there are same ca_list and crl path, to load once and do references from the other profiles. I don't know if there is lot of coding, but should be kept in mind for future optimizations. Perhaps could be added in the tracker not to forget about.
yes, it would be an improvement if the same ca_list would be loaded only once no matter in how many profiles it exists.
i "solved" the problem my removing ca_list from client profile, because it does not need it (verify_certificate = no).
-- juha
On Mon, Mar 19, 2012 at 05:31, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 3/19/12 10:03 AM, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:
is this with default mem size?
i would just like to add that once i increased shared memory from 32m to 64m, sip proxy started, but only about 10M was left as free:
afaik, the ca_list is loaded by libssl internally. Kamailio just sets custom memory manager and gives the path to ca_list.
That may be the reason why it consumes so much memory. The custom memory manager is not exactly memory efficient, it trades speed for efficiency.
-Jan
Shared Memory
free_size 8510256 max_used_size 58623504 real_used_size 58598608 total_size 67108864
it means that debian's ca_list consumes more than 20M of shared memory, which seems like a lot for 139 entries.
Is this by one ca_list only, or do you have many tls domains, each with own profile?
Cheers, Daniel
-- Daniel-Constantin Mierla Kamailio Advanced Training, April 23-26, 2012, Berlin, Germany http://www.asipto.com/index.php/kamailio-advanced-training/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
as i mentioned, with shared memory of 64MB, sip proxy started fine. however, it failed to send a request over tls:
Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: INFO: Routing INVITE sip:044234567@192.98.102.10:8001;transport=tls Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: WARNING: <core> [forward.c:268]: WARNING: get_send_socket: protocol/port mismatch (forced udp:192.98.101.10:5060, to tls:192.98.102.10:8001) Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tls [tls_server.c:150]: tls: ssl bug #1491 workaround: not enough memory for safe operation: 8473952 Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tm [../../forward.h:192]: msg_send: ERROR: tcp_send failed Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tm [t_fwd.c:1385]: ERROR: t_send_branch: sending request on branch 0 failed
this works ok when there is only a couple of certs in ca_list.
there is 10M of free shared memory available. is that not enough for "safe operation" or is this now pkg memory issue (i have the default 4M configured)?
-- juha
On Mon, Mar 19, 2012 at 05:32, Juha Heinanen jh@tutpro.com wrote:
as i mentioned, with shared memory of 64MB, sip proxy started fine. however, it failed to send a request over tls:
Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: INFO: Routing INVITE sip:044234567@192.98.102.10:8001;transport=tls Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: WARNING: <core> [forward.c:268]: WARNING: get_send_socket: protocol/port mismatch (forced udp:192.98.101.10:5060, to tls:192.98.102.10:8001) Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tls [tls_server.c:150]: tls: ssl bug #1491 workaround: not enough memory for safe operation: 8473952 Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tm [../../forward.h:192]: msg_send: ERROR: tcp_send failed Mar 19 11:21:41 test /usr/sbin/sip-proxy[30293]: ERROR: tm [t_fwd.c:1385]: ERROR: t_send_branch: sending request on branch 0 failed
this works ok when there is only a couple of certs in ca_list.
there is 10M of free shared memory available. is that not enough for "safe operation" or is this now pkg memory issue (i have the default 4M configured)?
I think you're seeing this because you don't have enough free shared memory. When you load the whole Debian CA certificate list, it probably does not leave enough memory left for openssl. The tls module fails preemptively if there is not enough free memory left to prevent openssl from crashing or causing memory leaks.
There are two configuration options that you can change: low_mem_threshold1 (default 512 * 1024 * get_max_procs) and low_mem_threshold2 (256 * 1024 * get_max_procs).
So you can either try to change one of the options or give the sip proxy more shared memory.
Note that openssl is really memory hungry. By default it consumes over 65kB of memory per connection for various buffers.
-Jan