2 Mar
2012
2 Mar
'12
2:01 p.m.
Hi everyone,
I have a problem with Kamailio consuming memory. I started testing with
Kamailio 3.2.1 upgraded to 3.2.2 and also tested 3.1.5, everywhere the
same problem. Whereas Kamailio 1.5.0 is working fine.
Kamailio -V shows the following:
kamailio@test:~$ kamailio -V
version: kamailio 3.2.2 (i386/linux) 98ba92
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST,
HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 98ba92
compiled on 17:51:30 Feb 29 2012 with gcc 4.2.4
kamailio@test:~$
My test scenario consists of two SIPp servers (a UAC and a UAS) and the
Kamailio in the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases the calls (BYE, 200) after 20 sec call duration. UAS is simply
sending back the right responds. Protocol is UDP.
After some minutes the Kamailio floods the syslog file with error
messages indicating that he is running out of memory. The SIPp shows
retransmissions and also timeouts. When I stop the test and do a single
call I get some lines of error messages. I found three different sets of
error messages for a single call so far:
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [parser/msg_parser.c:172]:
get_hdr_field: out of memory
Mar 2 09:35:44 test P-CSCFtest[21251]: INFO:<core> [parser/msg_parser.c:353]:
ERROR: bad header field [To:<sip:00498111111]
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR: tm [t_lookup.c:1118]: ERROR: reply cannot
be parsed
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 09:35:44 test P-CSCFtest[21251]: ERROR:<core> [forward.c:794]: ERROR:
forward_reply: building failed
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply:
no mem for outbound reply buffer
Mar 2 09:35:44 test P-CSCFtest[21228]: ERROR:<core> [msg_translator.c:2036]:
ERROR: build_res_buf_from_sip_req: out of memory ; needs 311
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:1884]:
ERROR: build_res_buf_from_sip_res: out of mem
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR: tm [t_reply.c:1786]: ERROR: relay_reply:
no mem for outbound reply buffer
Mar 2 10:19:43 test P-CSCFtest[21346]: ERROR:<core> [msg_translator.c:2036]:
ERROR: build_res_buf_from_sip_req: out of memory ; needs 311
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR:
do_action: memory allocation failure
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action
error at: /usr/local/etc/kamailio//kamailio.cfg:28
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:852]: ERROR:
do_action: memory allocation failure
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [action.c:1568]: run action
error at: /usr/local/etc/kamailio//kamailio.cfg:32
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR:<core> [parser/parse_from.c:71]:
ERROR:parse_from_header: out of pkg_memory
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1326]: ERROR: new_t: no
valid From in INVITE
Mar 2 10:20:16 test P-CSCFtest[21328]: ERROR: tm [t_lookup.c:1478]: ERROR: t_newtran:
new_t failed
Mar 2 10:20:16 test P-CSCFtest[21328]: WARNING:<core> [receive.c:209]: WARNING:
receive_msg: error while trying script
I repeated the test various times, it always generates a memory problem.
The "top" command shows that the free memory is decreasing constantly
during the test. Stopping the test and waiting for some time (1 hour)
doesn't increase the free memory showed by "top". Only a restart of the
Kamailio frees the memory again.
I executed the test with 30 calls per second too and got the same
behaviour (after some more minutes).
I finally simplified the Kamailio config as far as possible and came to
this minimum:
#!KAMAILIO
listen=10.10.10.10:5060
children=30
user="kamailio"
group="kamailio"
debug=2
disable_core_dump=yes
log_facility=LOG_LOCAL0
log_name="P-CSCFtest"
server_signature=no
sip_warning=0
syn_branch=0
disable_tcp=yes
####### Modules Section #######
mpath="/usr/local/lib/kamailio/modules_k/:/usr/local/lib/kamailio/modules/"
loadmodule "tm.so"
####### Main Routing Logic ########
route{
if (src_ip==1.2.3.4) {
rewritehost("5.6.7.8");
} else {
rewritehost("1.2.3.4");
};
rewriteport("5061");
t_relay();
}
This config still generates the memory problem. After replacing the
"t_relay()" with the stateless "forward()" the problem is gone.
I played around with the number of child processes too, but no difference.
Is this a memory leak of the TM module or is there a different explanation?
Regards Fred
2 Mar
2 Mar
2:26 p.m.
See http://kamailio.org/dokuwiki/doku.php/troubleshooting:memory
how to debug memory problems.
Using "top" does not reveal memory problems, as Kamailio allocates the
defined memory (shared + private per process) during startup. Thus, the
system memory consumptions should be constant (except some modules use
libraries which use system memory, eg. postgres library)
regards
Klaus
On 02.03.2012 13:01, Martin Tichatschek wrote:
rio consists of two SIPp servers (a UAC and a UAS) and
the Kamailio in
the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases t
4:31 p.m.
You can attempt to give Kamailio more memory to use, by adding the options
-M and -m . If you have enough memory a -M 10240 and -m 1024, should give
you more transaction capabilities.
Op 2 maart 2012 13:26 schreef Klaus Darilion
<klaus.mailinglists(a)pernau.at>het volgende:
See
http://kamailio.org/dokuwiki/**doku.php/troubleshooting:**memory<http://…
how to debug memory problems.
Using "top" does not reveal memory problems, as Kamailio allocates the
defined memory (shared + private per process) during startup. Thus, the
system memory consumptions should be constant (except some modules use
libraries which use system memory, eg. postgres library)
regards
Klaus
On 02.03.2012 13:01, Martin Tichatschek wrote:
rio consists of two SIPp servers (a UAC and a
UAS) and the Kamailio in
the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases t
______________________________**_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/**cgi-bin/mailman/listinfo/sr-**users<http:/…
6:32 p.m.
Or check if it is a memory leak and fix it ;-)
On 02.03.2012 15:31, davy van de moere wrote:
You can attempt to give Kamailio more memory to use,
by adding the
options -M and -m . If you have enough memory a -M 10240 and -m 1024,
should give you more transaction capabilities.
Op 2 maart 2012 13:26 schreef Klaus Darilion
<klaus.mailinglists(a)pernau.at <mailto:klaus.mailinglists@pernau.at>> het
volgende:
See http://kamailio.org/dokuwiki/__doku.php/troubleshooting:__memory
<http://kamailio.org/dokuwiki/doku.php/troubleshooting:memory>
how to debug memory problems.
Using "top" does not reveal memory problems, as Kamailio allocates
the defined memory (shared + private per process) during startup.
Thus, the system memory consumptions should be constant (except some
modules use libraries which use system memory, eg. postgres library)
regards
Klaus
On 02.03.2012 13:01, Martin Tichatschek wrote:
rio consists of two SIPp servers (a UAC and a UAS) and the
Kamailio in
the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases t
_________________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
15 Mar
15 Mar
12:26 p.m.
Thanks for your suggestions, I finally found my problem.
I used a xml file for the SIPp were no branch parameter was included in
the VIA header (which is not RFC compliant).
With such messages the Kamailio 3.x.x seems to have problems whereas
Kamailio 1.5.0 is doing fine.
I will protect my Kamailio against such messages.
On 02.03.2012 17:32, Klaus Darilion wrote:
Or check if it is a memory leak and fix it ;-)
On 02.03.2012 15:31, davy van de moere wrote:
You can attempt to give Kamailio more memory to
use, by adding the
options -M and -m . If you have enough memory a -M 10240 and -m 1024,
should give you more transaction capabilities.
Op 2 maart 2012 13:26 schreef Klaus Darilion
<klaus.mailinglists(a)pernau.at <mailto:klaus.mailinglists@pernau.at>> het
volgende:
See http://kamailio.org/dokuwiki/__doku.php/troubleshooting:__memory
<http://kamailio.org/dokuwiki/doku.php/troubleshooting:memory>
how to debug memory problems.
Using "top" does not reveal memory problems, as Kamailio allocates
the defined memory (shared + private per process) during startup.
Thus, the system memory consumptions should be constant (except some
modules use libraries which use system memory, eg. postgres library)
regards
Klaus
On 02.03.2012 13:01, Martin Tichatschek wrote:
rio consists of two SIPp servers (a UAC and a UAS) and the
Kamailio in
the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200,
ACK) and
releases t
_________________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12:57 p.m.
If you have more details please post them. If a malformed SIP message
can cause a memory leak it needs to be fixed.
thanks
Klaus
On 15.03.2012 11:26, Martin Tichatschek wrote:
Thanks for your suggestions, I finally found my
problem.
I used a xml file for the SIPp were no branch parameter was included in
the VIA header (which is not RFC compliant).
With such messages the Kamailio 3.x.x seems to have problems whereas
Kamailio 1.5.0 is doing fine.
I will protect my Kamailio against such messages.
On 02.03.2012 17:32, Klaus Darilion wrote:
Or check if it is a memory leak and fix it ;-)
On 02.03.2012 15:31, davy van de moere wrote:
You can attempt to give Kamailio more memory to
use, by adding the
options -M and -m . If you have enough memory a -M 10240 and -m 1024,
should give you more transaction capabilities.
Op 2 maart 2012 13:26 schreef Klaus Darilion
<klaus.mailinglists(a)pernau.at <mailto:klaus.mailinglists@pernau.at>> het
volgende:
See http://kamailio.org/dokuwiki/__doku.php/troubleshooting:__memory
<http://kamailio.org/dokuwiki/doku.php/troubleshooting:memory>
how to debug memory problems.
Using "top" does not reveal memory problems, as Kamailio allocates
the defined memory (shared + private per process) during startup.
Thus, the system memory consumptions should be constant (except some
modules use libraries which use system memory, eg. postgres library)
regards
Klaus
On 02.03.2012 13:01, Martin Tichatschek wrote:
rio consists of two SIPp servers (a UAC and a UAS) and the
Kamailio in
the middle proxying the SIP traffic.
The UAC sends 300 calls per second (INVITE, 100, 180, 200, ACK) and
releases t
_________________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/__cgi-bin/mailman/listinfo/sr-__users
<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users>
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4639
days inactive
4652
days old
5 comments
3 participants
participants (3)
-
davy van de moere -
Klaus Darilion -
Martin Tichatschek