Re: [sr-dev] kamailio 3.1.0 crash on ssl-dos attack

Hi, As i mentioned in my previous mail that i tested limiting the TLS connection but didn't help. The problem was high frequent renegotiation on existing TLS connections. This was causing the kamailio to restart. So i was thinking to disable the renegoitation until OPENSSL comeup with a solution to this issue. Thanks Jijo On Wed, Dec 14, 2011 at 5:45 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > I committed yesterday in master branch the code that adds a new global > parameter tls_max_connections which sets a limit on how many active tls > connections are on sip server. Since tls connections are over tcp, > practically tls_max_connections is not effective if greater than > tcp_max_connections, since the last will be reached first. > > Can you give it a try and see if works for you. > > Cheers, > Daniel > > > On 12/10/11 12:53 AM, Jijo wrote: > > Hi > > Since TLS run on TCP, the max TLS connections accepted is based on > tcp_max_connections. I thought its better to limit the max tls > connections, since tls require more memory for each tls connection. > > I did look futher into the issue and the kamailio crashes only when the > tool(*thc-ssl-dos)* is ran with periodic renegotiation. Kamailo doesn't > crash if the renegotiation is disabled in the tool. > > The htppd has fixed this issue by providing a flag to disable TLS > renegotation in the server from any client. Please find the httpd patch at > > http://mail-archives.apache.org/mod_mbox/httpd-dev/200911.mbox/raw/%3c20091… > > So I was thinking to apply the same patch in kamailio also. We could have > a new flag in tls_domain structure to enable or disable renegotation. > > Please have a look and let me know your findings. > > Thanks > Jijo > > > On Fri, Dec 9, 2011 at 1:53 AM, Daniel-Constantin Mierla < > miconda(a)gmail.com> >> Hello, >> >> my plan is to look at it as I get back to office (being few days off). >> >> But, isn't tcp_max_connections applied to tls connections? If not, adding >> such limit will be good anyhow. >> >> Cheers, >> Daniel >> >> >> >> On 12/7/11 8:57 PM, Jijo wrote: >> >> Hi All.. >> >> Any comments/suggestions on this issue? >> >> Thanks >> Jijo >> >> On Mon, Dec 5, 2011 at 6:56 PM, Jijo &lt;realjijo(a)gmail.com>> >>> Hello >>> >>> I tested using 3.2 and i got the same error. I couldn't get the memlog, >>> when i enable , the system doesn't come up. >>> >>> The latest update from http://www.thc.org/thc-ssl-dos/ says that >>> >>> "2011-OCT-24 UPDATE: >>> SSL-DOS released. Some organizations already found out about this >>> release a while ago and mistakenly identified it as an SSL-RENEGOTIATION >>> BUG. This is not true. The tool can be modified to work without >>> SSL-RENEGOTIATION by just establishing a new TCP connection for every new >>> handshake. " >>> >>> So this issue could happen if we do TCP connection for every new TLS >>> connection with renegotiation. I believe this could be fixed if we could >>> block based on max active TLS connections, probably a similar flag for max >>> TLS connections like "tcp_max_connections" >>> >>> >>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 160 -f >>> /etc/kamailio/kamailio.cfg'. >>> >>> Program terminated with signal 11, Segmentation fault. >>> #0 0xb61cc2e3 in ?? () from /usr/lib/kamailio/modules/tls.so >>> (gdb) bt >>> #0 0xb61cc2e3 in ?? () from /usr/lib/kamailio/modules/tls.so >>> #1 0xb65199fd in CRYPTO_lock () from /lib/libcrypto.so.1.0.0 >>> #2 0xb658c133 in ?? () from /lib/libcrypto.so.1.0.0 >>> #3 0xb658d240 in RAND_add () from /lib/libcrypto.so.1.0.0 >>> #4 0xb6690d50 in ssl3_accept () from /lib/libssl.so.1.0.0 >>> #5 0xb669bd2f in ssl3_read_bytes () from /lib/libssl.so.1.0.0 >>> #6 0xb66989b9 in ?? () from /lib/libssl.so.1.0.0 >>> #7 0xb66ae499 in SSL_read () from /lib/libssl.so.1.0.0 >>> #8 0xb61e1e33 in tls_read_f () from /usr/lib/kamailio/modules/tls.so >>> #9 0x08177036 in tcp_read_headers () >>> #10 0x081771f6 in tcp_read_req () >>> #11 0x08178480 in ?? () >>> #12 0x0817af77 in tcp_receive_loop () >>> #13 0x08173597 in tcp_init_children () >>> #14 0x080b20bf in main_loop () >>> #15 0x080b38c3 in main () >>> (gdb) >>> >>> >>> >>> On Wed, Nov 23, 2011 at 10:35 AM, Daniel-Constantin Mierla < >>> miconda(a)gmail.com>>> >>>> Hello, >>>> >>>> 3.1.0 is not the right choice in 3.1 series, there were many fixes that >>>> were added to that release series branch. The latest stable version there >>>> is 3.1.5. Try with it (even better, try with latest version from git branch >>>> 3.1 -- see >>>> http://www.kamailio.org/dokuwiki/doku.php/install:kamailio-3.1.x-from-git >>>> ) >>>> >>>> In this way we are sure it is not a bug that was fixed after the time >>>> of releasing 3.1.0 -- config file and database is the same as for 3.1.0. >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> On 11/23/11 4:01 PM, Jijo wrote: >>>> >>>> Thanks I will attach the logs soon..meanwhile here is the kamailio and >>>> openssl version >>>> >>>> OB151:~ # /usr/sbin/kamailio -V >>>> version: kamailio 3.1.0 (i386/linux) 21a375 >>>> flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, >>>> USE_RAW_SOCKS, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, >>>> DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, >>>> USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES >>>> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >>>> MAX_URI_SIZE 1024, BUF_SIZE 65535, PKG_SIZE 15MB >>>> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >>>> id: 21a375 >>>> compiled on 09:22:51 Nov 4 2011 with gcc 4.5.0 >>>> >>>> >>>> OB151:~ # openssl version -a >>>> OpenSSL 1.0.0 29 Mar 2010 >>>> built on: 2011-05-31 07:52:17.000000000 +0000 >>>> platform: linux-elf >>>> options: bn(64,32) rc4(4x,int) des(ptr,risc1,16,long) blowfish(idx) >>>> compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT >>>> -DDSO_DLFCN -DHAVE_DLFCN_H >>>> -DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -Wall -fomit-frame-pointer >>>> -fmessage-length=0 -O2 -Wall >>>> -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables >>>> -fasynchronous-unwind-tables -g -Wa,--noexecstack -fomit-frame-pointer >>>> -fno-strict-aliasing >>>> -DTERMIO -Wall -fstack-protector -DOPENSSL_BN_ASM_PART_WORDS >>>> -DOPENSSL_IA32_SSE2 >>>> -DOPENSSL_BN_ASM_MONT -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM >>>> -DRMD160_ASM -DAES_ASM -DWHIRLPOOL_ASM >>>> OPENSSLDIR: "/etc/ssl" >>>> >>>> >>>> On Wed, Nov 23, 2011 at 4:44 AM, Daniel-Constantin Mierla < >>>> miconda(a)gmail.com>>>> >>>>> Hello, >>>>> >>>>> (discussion kept only on sr-dev as it is very likely going to require >>>>> mostly devel interaction). >>>>> >>>>> What is the version of kamailio (-V command line option). Also, send >>>>> the verision of openssl library -- there were many bugs in various lib >>>>> versions that had to be workarounded in the module, maybe this is a new one >>>>> that has to be fixed. >>>>> >>>>> Do you get any error message in the syslog at the moment of the crash? >>>>> >>>>> What would be useful is to get the memory operations log, you can get >>>>> it by setting: >>>>> >>>>> memdbg=1 >>>>> memlog=1 >>>>> >>>>> in config file. >>>>> >>>>> Then repeat the test and make the log available for download somehow >>>>> (if it is too big), from start to the moment of the crash. >>>>> >>>>> Cheers, >>>>> Daniel >>>>> >>>>> >>>>> On 11/22/11 11:30 PM, Jijo wrote: >>>>> >>>>> Hi All, >>>>> >>>>> Kamailio is resetting when we do TLS renegotiation dos attack using >>>>> the tool available at http://www.thc.org/thc-ssl-dos/. >>>>> >>>>> Anybody looked at this issue? How we could resolve it. Any idea? >>>>> >>>>> The core generated for 3 pid's as below >>>>> >>>>> Pid 1: >>>>> >>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f >>>>> /etc/kamailio/kamailio.cfg'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 atomic_inc_int () at atomic/atomic_x86.h:225 >>>>> (gdb) bt >>>>> #0 atomic_inc_int () at atomic/atomic_x86.h:225 >>>>> #1 cfg_update_local () at cfg/cfg_struct.h:228 >>>>> #2 timer_main () at timer.c:994 >>>>> #3 0x080b0579 in main_loop () at main.c:1632 >>>>> #4 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446 >>>>> >>>>> >>>>> Pid 2: >>>>> >>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f >>>>> /etc/kamailio/kamailio.cfg'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 0x0819bfe8 in qm_insert_free (qm=0xaf6c5000, p=0xb05eec30, >>>>> file=0xb6fb4140 "tls: tls_init.c", func=0xb6fb4ce0 "ser_free", line=296) >>>>> at mem/q_malloc.c:184 >>>>> 184 if (frag->size <= f->size) break; >>>>> (gdb) bt >>>>> #0 0x0819bfe8 in qm_insert_free (qm=0xaf6c5000, p=0xb05eec30, >>>>> file=0xb6fb4140 "tls: tls_init.c", func=0xb6fb4ce0 "ser_free", line=296) >>>>> at mem/q_malloc.c:184 >>>>> #1 qm_free (qm=0xaf6c5000, p=0xb05eec30, file=0xb6fb4140 "tls: >>>>> tls_init.c", func=0xb6fb4ce0 "ser_free", line=296) at mem/q_malloc.c:518 >>>>> #2 0xb6f95404 in ser_free (ptr=0xb05eec30) at tls_init.c:296 >>>>> #3 0xb732e9ba in CRYPTO_free (str=0xb05eec30) at mem.c:391 >>>>> #4 0xb7330bee in int_new_ex_data (class_index=5, obj=0xbfd414f4, >>>>> ad=0xbfd41574) at ex_data.c:440 >>>>> #5 0xb7330443 in CRYPTO_new_ex_data (class_index=5, obj=0xbfd414f4, >>>>> ad=0xbfd41574) at ex_data.c:575 >>>>> #6 0xb73dfde3 in X509_STORE_CTX_init (ctx=0xbfd414f4, >>>>> store=0xafd8b3d0, x509=0xafe08ff0, chain=0x0) at x509_vfy.c:2114 >>>>> #7 0xb74b0f31 in ssl3_output_cert_chain (s=0xb0553a10, x=0xafe08ff0) >>>>> at s3_both.c:349 >>>>> #8 0xb74a4728 in ssl3_send_server_certificate (s=0xb0553a10) at >>>>> s3_srvr.c:3034 >>>>> #9 0xb74a5879 in ssl3_accept (s=0xb0553a10) at s3_srvr.c:353 >>>>> #10 0xb74afa8f in ssl3_read_bytes (s=0xb0553a10, type=23, >>>>> buf=0xb0ad44ec "", len=4095, peek=0) at s3_pkt.c:1266 >>>>> #11 0xb74ac9c9 in ssl3_read_internal (s=0xb0553a10, buf=0xb0ad44ec, >>>>> len=4095, peek=0) at s3_lib.c:3265 >>>>> #12 0xb74c24a9 in SSL_read (s=0xb0553a10, buf=0xb0ad44ec, num=4095) at >>>>> ssl_lib.c:954 >>>>> #13 0xb6fad1c3 in tls_read_f (c=0xb0ad431c, flags=0xbfd619c4) at >>>>> tls_server.c:1058 >>>>> #14 0x08171c0e in tcp_read_headers (c=0xb0ad431c, >>>>> read_flags=0xbfd619c4) at tcp_read.c:406 >>>>> #15 0x08171db8 in tcp_read_req (con=0xb0ad431c, bytes_read=0xbfd619cc, >>>>> read_flags=0xbfd619c4) at tcp_read.c:885 >>>>> #16 0x08172f67 in handle_io (fm=<value optimized out>, events=1, >>>>> idx=<value optimized out>) at tcp_read.c:1234 >>>>> #17 0x0817583b in io_wait_loop_epoll (unix_sock=89) at io_wait.h:1092 >>>>> #18 tcp_receive_loop (unix_sock=89) at tcp_read.c:1345 >>>>> #19 0x0816e2e9 in tcp_init_children () at tcp_main.c:4867 >>>>> #20 0x080affb1 in main_loop () at main.c:1646 >>>>> #21 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446 >>>>> >>>>> Pid 3: >>>>> >>>>> Core was generated by `/usr/sbin/kamailio -u swrun -g sw -m 120 -f >>>>> /etc/kamailio/kamailio.cfg'. >>>>> Program terminated with signal 11, Segmentation fault. >>>>> #0 0xb76c9e7c in memmove () from /lib/libc.so.6 >>>>> (gdb) bt >>>>> #0 0xb76c9e7c in memmove () from /lib/libc.so.6 >>>>> #1 0x081724e7 in tcp_read_req (con=0xb022c8f0, bytes_read=0xbfd619cc, >>>>> read_flags=0xbfd619c4) at tcp_read.c:1026 >>>>> #2 0x08172f67 in handle_io (fm=<value optimized out>, events=1, >>>>> idx=<value optimized out>) at tcp_read.c:1234 >>>>> #3 0x0817583b in io_wait_loop_epoll (unix_sock=93) at io_wait.h:1092 >>>>> #4 tcp_receive_loop (unix_sock=93) at tcp_read.c:1345 >>>>> #5 0x0816e2e9 in tcp_init_children () at tcp_main.c:4867 >>>>> #6 0x080affb1 in main_loop () at main.c:1646 >>>>> #7 0x080b1be4 in main (argc=9, argv=0xbfd61e54) at main.c:2446 >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev >>>>> >>>>> >>>>> -- >>>>> Daniel-Constantin Mierla -- http://www.asipto.com >>>>> Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda >>>>> >>>>> >>>> >>>> -- >>>> Daniel-Constantin Mierla -- http://www.asipto.com >>>> Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda >>>> >>>> >>> >> >> -- >> Daniel-Constantin Mierla -- http://www.asipto.comhttp://linkedin.com/in/miconda -- http://twitter.com/miconda >> >> > > > _______________________________________________ > sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev > > > -- > Daniel-Constantin Mierla -- http://www.asipto.comhttp://linkedin.com/in/miconda -- http://twitter.com/miconda > >