TLS performance testing - sr-users

20 Dec 2011


      Hello everyone,
I'm doing some TLS performance testing on Kamailio 3.2.1.  Here's my setup:
kamailio -V
version: kamailio 3.2.1 (x86_64/linux) 31c991
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS,
USE_RAW_SOCKS, USE_STUN, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK,
SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 31c991
compiled on 19:38:03 Dec 20 2011 with gcc 4.4.6
uname -a
Linux null.null.com 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6 19:48:22
GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
CentOS 6.2 on a Dell PowerEdge R610 with 24 Intel X5650 Cores at
2.67GHz and 12GB of RAM (I could use more).
Kamailio is the default config with a few changes:
- WITH_TLS defined
- TLS is using self-generated CAs/certs (essentially openvpn easy-rsa)
with 1024 bit key size
- TLS is *not* configured to verify client OR server certs by default
- I'm using TLS v1 (SSL 3.1)
- TLS cipher suites are set to any (although my simulated UAs only
offer AES 256+SHA)
- Various changes to Kamailio children (up to 256 at times) and memory
sizes (up to 2048mb and even 4096mb at times)
- One DNS based alias added
- Maximum TCP connections increased to 65000
- Kamailio is configured to only listen on tested IP (UDP, TCP, TLS
sockets active)
- Syslog has been configured to log local0 (Kamailio) asynchronously
My test rig/call generator is an Ixia Xcellon-Ultra NP load module
with IXLoad.  My call scenario does the following:
- Registers two simulated user agents (100000, 200000) to Kamailio with TLS
- Places call from 100000 to 200000 via Kamailio with TLS
- Increments both user agents by 1 and continues as quickly (cps) as I
like up to a channel limit (also configurable)
- The Ixia generates a valid SDP but no RTP is generated (although
that's certainly possible at these call levels)
Two 1 gig ports on the Ixia are connected to the Broadcom NICs on
the Dell R610 via a Cisco Catalyst 4948 switch.  One port on the Ixia
emulates the 100000 agents (A leg) and the other emulates the 200000
agents (B leg).  Of course I can provide more information if needed.
Here are some test numbers:
With TLS at 20cps, 120 sec calls, up to a total of 2470 calls (4940
registrations) life is good.  Very good - call setup time averages
23ms, the cps rate holds indefinitely, and not a single call or
registration fails over long term tests.
UDP and TCP numbers are excellent (bordering ridiculous) - usually
around 500cps with practically no reasonable upper limit on
simultaneous calls.  This doesn't need any further discussion :).
The TLS numbers start falling apart pretty quickly after 20cps,
however.  If I change the TLS test to 40cps, 120 sec calls, up to a
total of 4940 calls (9,880 registrations) Kamailio starts to
(seriously) struggle.  The rate starts fluctuating all over the place,
call setup time averages jump to 8000ms (or more) and things just
generally get ugly.  Interestingly enough all of the user agents are
able to register, the logs look fine (to my eye at this log level) and
the system (CPU, network, etc) doesn't appear to be under stress at
all.
I have a few questions:
1)  Is there something obviously wrong or stupid I'm doing here?
2)  Why are the TLS tests so much worse than TCP and UDP?  Am I
missing something here?
Thanks (in advance) for any advice anyone might be able to offer!
-- 
Kristian Kielhofner