2 Mar
2012
2 Mar
'12
7:01 a.m.
Hi everyone,
I have a problem with Kamailio consuming memory. I started testing with
Kamailio 3.2.1 upgraded to 3.2.2 and also tested 3.1.5, everywhere the
same problem. Whereas Kamailio 1.5.0 is working fine.
Kamailio -V shows the following:
kamailio@test:~$ kamailio -V
version: kamailio 3.2.2 (i386/linux) 98ba92
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 98ba92
compiled on 17:51:30 Feb 29 2012 with gcc 4.2.4
kamailio@test:~$
My test scenario consists of two SIPp servers (a UAC and a UAS) and the
Kamailio in the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases the calls (BYE, 200) after 20 sec call duration. UAS is simply
sending back the right responds. Protocol is UDP.
After some minutes the Kamailio floods the syslog file with error
messages indicating that he is running out of memory. The SIPp shows
retransmissions and also timeouts. When I stop the test and do a single
call I get some lines of error messages. I found three different sets of
error messages for a single call so far:
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [parser/msg_parser.c:172]:
get_hdr_field: out of memory
Mar 2 09:35:44 test P-CSCFtest[21251]: INFO:<core> [parser/msg_parser.c:353]:
ERROR: bad header field [To:<sip:00498111111]
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR: tm [t_lookup.c:1118]: ERROR: reply cannot
be parsed
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [forward.c:794]: ERROR:
forward_reply: building failed
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply:
no mem for outbound reply buffer
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:2036]:
ERROR: build_res_buf_from_sip_req: out of memory ; needs 311
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply:
no mem for outbound reply buffer
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:2036]:
ERROR: build_res_buf_from_sip_req: out of memory ; needs 311
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR:
do_action: memory allocation failure
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action
error at: /usr/local/etc/kamailio//kamailio.cfg:28
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR:
do_action: memory allocation failure
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action
error at: /usr/local/etc/kamailio//kamailio.cfg:32
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [parser/parse_from.c:71]:
ERROR:parse_from_header: out of pkg_memory
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1326]: ERROR: new_t: no
valid From in INVITE
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran:
new_t failed
Mar 2 10:20:16 test P-CSCFtest[21328]: WARNING:<core> [receive.c:209]: WARNING:
receive_msg: error while trying script
I repeated the test various times, it always generates a memory problem.
The "top" command shows that the free memory is decreasing constantly
during the test. Stopping the test and waiting for some time (1 hour)
doesn't increase the free memory showed by "top". Only a restart of the
Kamailio frees the memory again.
I executed the test with 30 calls per second too and got the same
behaviour (after some more minutes).
I finally simplified the Kamailio config as far as possible and came to
this minimum:
#!KAMAILIO
listen=10.10.10.10:5060
children=30
user="kamailio"
group="kamailio"
debug=2
disable_core_dump=yes
log_facility=LOG_LOCAL0
log_name="P-CSCFtest"
server_signature=no
sip_warning=0
syn_branch=0
disable_tcp=yes
####### Modules Section #######
mpath="/usr/local/lib/kamailio/modules_k/:/usr/local/lib/kamailio/modules/"
loadmodule "tm.so"
####### Main Routing Logic ########
route{
if (src_ip==1.2.3.4) {
rewritehost("5.6.7.8");
} else {
rewritehost("1.2.3.4");
};
rewriteport("5061");
t_relay();
}
This config still generates the memory problem. After replacing the
"t_relay()" with the stateless "forward()" the problem is gone.
I played around with the number of child processes too, but no difference.
Is this a memory leak of the TM module or is there a different explanation?
Regards Fred