29 Nov
2013
29 Nov
'13
2:30 p.m.
Hi,
I have a Kamailio 3.3.3 server that currently reports the following
errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core>
[sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]:
ERROR: new_t: out of mem:
daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]:
ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1].
However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem
{
total: 67108864
free: 43189728
used: 20505784
real_used: 23919136
max_used: 65948568
fragments: 36887
}
These figures are quite similar to another server with the same
configuration and Kamailio version, which is not reporting out-of-memory
errors.
# sercmd core.shmmem
{
total: 67108864
free: 40219536
used: 23241952
real_used: 26889328
max_used: 66306288
fragments: 37832
}
I recently saw similar symptoms on a Kamailio 4.0.4 server, but
Kamailio unfortunately crashed before I could get further details. In
this case, I can consistently duplicate the problem with certain
INVITEs. It seems that the failure is somehow related to the size of the
SIP headers. For example, I have an INVITE from a Polycom phone which
t_relay() routes successfully with 741 bytes of headers (total SIP
length 1039 bytes). However, it seems that if I add one character to any
of its SIP headers (742 bytes of headers; total SIP length 1039 bytes)
the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on
getting it re-compiled). However, since I don't know what caused the
problem, I would still like to get as much information as I can out of
the running system. I have attached the output of mi_statistics, as well
as mem_dump_pkg for the external UDP receivers and the ctl handler. Are
there any other diagnostics that could be useful?
-Andrew
[1]
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_er…
29 Nov
29 Nov
3:37 p.m.
Hi Andrew
A very simple place to start -- have you tried upping the memory
allocations in /etc/default/kamailio to see if it fixes the issues? We had
some issues with shared memory that went away when we changed around these
settings.
All the best.
Will Ferrer
On Fri, Nov 29, 2013 at 11:30 AM, <elactrum(a)jamailca.com> wrote:
Hi,
I have a Kamailio 3.3.3 server that currently reports the following errors
upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core> [sip_msg_clone.c:505]:
ERROR: sip_msg_cloner: cannot allocate memory
daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]: ERROR:
new_t: out of mem:
daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]: ERROR:
t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1].
However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem
{
total: 67108864
free: 43189728
used: 20505784
real_used: 23919136
max_used: 65948568
fragments: 36887
}
These figures are quite similar to another server with the same
configuration and Kamailio version, which is not reporting out-of-memory
errors.
# sercmd core.shmmem
{
total: 67108864
free: 40219536
used: 23241952
real_used: 26889328
max_used: 66306288
fragments: 37832
}
I recently saw similar symptoms on a Kamailio 4.0.4 server, but Kamailio
unfortunately crashed before I could get further details. In this case, I
can consistently duplicate the problem with certain INVITEs. It seems that
the failure is somehow related to the size of the SIP headers. For example,
I have an INVITE from a Polycom phone which t_relay() routes successfully
with 741 bytes of headers (total SIP length 1039 bytes). However, it seems
that if I add one character to any of its SIP headers (742 bytes of
headers; total SIP length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on getting
it re-compiled). However, since I don't know what caused the problem, I
would still like to get as much information as I can out of the running
system. I have attached the output of mi_statistics, as well as
mem_dump_pkg for the external UDP receivers and the ctl handler. Are there
any other diagnostics that could be useful?
-Andrew
[1] http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_
does_errorsip_msg_clonercannot_allocate_memory_
and_errornew_tout_of_memmean_and_how_do_i_fix_them
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
3 Dec
3 Dec
11:10 a.m.
On 2013-11-29 14:37, Will Ferrer wrote:
Hi Andrew
A very simple place to start -- have you tried upping the memory
allocations in /etc/default/kamailio to see if it fixes the issues? We
had some issues with shared memory that went away when we changed
around these settings.
All the best.
Will Ferrer
Will,
Thanks for the suggestion. What sort of issues did you have with shared
memory?
On my other servers, I am already increasing the shared memory
allocation from 64M to 512M in case that might reduce the risk of this.
However, is it normal to have out-of-memory errors when core.shmmem does
not report the memory being full? If not, what could cause this?
-Andrew
On Fri, Nov 29, 2013 at 11:30 AM,
<elactrum(a)jamailca.com> wrote:
Hi,
I have a Kamailio 3.3.3 server that currently reports the following
errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core>
[sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]:
ERROR: new_t: out of mem:
daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]:
ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on [1].
However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem
{
total: 67108864
free: 43189728
used: 20505784
real_used: 23919136
max_used: 65948568
fragments: 36887
}
These figures are quite similar to another server with the same
configuration and Kamailio version, which is not reporting
out-of-memory errors.
# sercmd core.shmmem
{
total: 67108864
free: 40219536
used: 23241952
real_used: 26889328
max_used: 66306288
fragments: 37832
}
I recently saw similar symptoms on a Kamailio 4.0.4 server, but
Kamailio unfortunately crashed before I could get further details. In
this case, I can consistently duplicate the problem with certain
INVITEs. It seems that the failure is somehow related to the size of
the SIP headers. For example, I have an INVITE from a Polycom phone
which t_relay() routes successfully with 741 bytes of headers (total
SIP length 1039 bytes). However, it seems that if I add one character
to any of its SIP headers (742 bytes of headers; total SIP length 1039
bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on
getting it re-compiled). However, since I don't know what caused the
problem, I would still like to get as much information as I can out of
the running system. I have attached the output of mi_statistics, as
well as mem_dump_pkg for the external UDP receivers and the ctl
handler. Are there any other diagnostics that could be useful?
-Andrew
[1]
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_er…
[1]
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
Links:
------
[1]
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_er…
[2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users 
4 Dec
4 Dec
7:17 a.m.
Hello,
do you have mem_join set in the config file?
I saw there is quite a lot of fragments, which means that there could
demand for large size of memory, but available packets have lower size.
mem_join should help in this case.
Cheers,
Daniel
On 12/3/13 5:10 PM, elactrum(a)jamailca.com wrote:
On 2013-11-29 14:37, Will Ferrer wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi Andrew
A very simple place to start -- have you tried upping the memory
allocations in /etc/default/kamailio to see if it fixes the issues? We
had some issues with shared memory that went away when we changed
around these settings.
All the best.
Will Ferrer
Will,
Thanks for the suggestion. What sort of issues did you have with
shared memory?
On my other servers, I am already increasing the shared memory
allocation from 64M to 512M in case that might reduce the risk of
this. However, is it normal to have out-of-memory errors when
core.shmmem does not report the memory being full? If not, what could
cause this?
-Andrew
On Fri, Nov 29, 2013 at 11:30 AM,
<elactrum(a)jamailca.com> wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users Hi,
I have a Kamailio 3.3.3 server that currently reports the following
errors upon execution of t_relay() for certain initial INVITEs.
daemon.err /usr/sbin/kamailio[26916]: ERROR: <core>
[sip_msg_clone.c:505]: ERROR: sip_msg_cloner: cannot allocate memory
daemon.err /usr/sbin/kamailio[26916]: ERROR: tm [t_lookup.c:1338]:
ERROR: new_t: out of mem:
daemon.err /usr/sbin/kamailio[26917]: ERROR: tm [t_lookup.c:1478]:
ERROR: t_newtran: new_t failed
I understand this error to be related to shared memory, based on
[1]. However, sercmd reports normal shared memory utilization:
# sercmd core.shmmem
{
total: 67108864
free: 43189728
used: 20505784
real_used: 23919136
max_used: 65948568
fragments: 36887
}
These figures are quite similar to another server with the same
configuration and Kamailio version, which is not reporting
out-of-memory errors.
# sercmd core.shmmem
{
total: 67108864
free: 40219536
used: 23241952
real_used: 26889328
max_used: 66306288
fragments: 37832
}
I recently saw similar symptoms on a Kamailio 4.0.4 server, but
Kamailio unfortunately crashed before I could get further details.
In this case, I can consistently duplicate the problem with certain
INVITEs. It seems that the failure is somehow related to the size of
the SIP headers. For example, I have an INVITE from a Polycom phone
which t_relay() routes successfully with 741 bytes of headers (total
SIP length 1039 bytes). However, it seems that if I add one
character to any of its SIP headers (742 bytes of headers; total SIP
length 1039 bytes) the t_relay() will consistently fail.
This binary doesn't have DBG_QM_MALLOC configured (I am working on
getting it re-compiled). However, since I don't know what caused the
problem, I would still like to get as much information as I can out
of the running system. I have attached the output of mi_statistics,
as well as mem_dump_pkg for the external UDP receivers and the ctl
handler. Are there any other diagnostics that could be useful?
-Andrew
[1]
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_er…
[1]
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users [2]
Links:
------
[1]
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:faq#qwhat_does_er…
[2] http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4072
days inactive
4077
days old
3 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
elactrum＠jamailca.com -
Will Ferrer