Hello,
I would say that perl_exec() is the one
with the highest chances to be the
reason for the leak. Next is line would
be db_mysql module, if liked with some
custom mysql client library, although
even in this case will be unlikely.
Back to perl, the module itself does not
call any malloc, so it might be the
embedding Perl API that is not used
properly in the module.
Can you use some testbed, set children=1
and run kamailio under valgrind, then do
some calls and see if it detects the
source of the leak?
I'm not using the perl module, I will
try to check it whenever I get a chance
in the next days. What version of perl
do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David
Cunningham wrote:
Hello,
We don't do any kamctl commands at
all. We do have various modules
loaded, as follows. The primary
functions we use Kamailio for are
phone registrations through
usrloc, and routing calls to
Asterisk through logic contained
in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24
July 2013 16:33,
Daniel-Constantin Mierla
<miconda@gmail.com>
wrote:
Hello,
On 7/24/13 4:24 AM,
David Cunningham wrote:
Hello,
Thank you very much
for the email. In reply:
1. The system ran out of
memory. Linux's
oom-killer killed
Kamailio.
then all the instructions I
gave are useless, they are
for debugging kamailio's
internal memory manager,
which handles pkg and shm
mallocs.
The chances to be from
kamailio itself are very low
now. Do you do lot of mi
commands (e.g., kamctl ...)?
The mi api uses system
malloc, but the rest of code
should use internal memory
manager which does not go
beyond the limits set with
-m and -M, thus not causing
an OS memory exhaustion.
Can you list what modules
are you loading? At some
point it was a leak in
libssl, in case you use tls
a lot. But could be another
external library...
Cheers,
Daniel
2. You're right,
DEBUG_MEMORY is a
local configuration
setting. If defined it
sets memdbg to -2, and
memlog to -2. The
debug setting is -1.
3. We'll try setting
mem_summary=12,
thanks.
4. We'll try setting
asynchronous syslog,
thanks.
5. Our configuration
totals 338 lines, or
approx 8.5kb. Is that
a lot?
6. We'll try setting
mem_join=1, thanks.
On
23 July 2013 16:53,
Daniel-Constantin
Mierla
<miconda@gmail.com>
wrote:
Hello,
first, to clarify,
is the system
memory or
kamailio's pkg/shm
memory running
out? If the
operating system
runs out of
memory, then
should be a leak
in a library,
because kamailio
modules uses only
from a
pre-allocated
chunk, not going
over it.
On 7/23/13 7:33
AM, David
Cunningham
wrote:
Hello,
We're running
a Kamailio
3.3.4 system,
and Kamailio
is slowly
using more and
more memory.
Over a couple
of weeks it
will run out
of system
memory.
We tried to
enable memory
debugging
doing the
following, but
it resulted in
Kamailio not
responding to
any SIP
packets. Would
anyone have
advice please
on how to
debug the
situation?
1. In
Makefile.defs
set MEMDBG to
1 and
recompile
Kamailio.
2. In
kamailio.cfg
add the line:
#!define
DEBUG_MEMORY 1
do you set
something special
in config when
DEBUG_MEMORY is 1?
It is not by
default there, so
I assume you added
some rules based
on this
pre-processor
directive.
For memory
troubleshooting,
set memlog to a
value lower than
debug parameter in
config file and
try with
mem_summary=12 for
a more compact
output. See more
about these
parameters in the
wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a
while in normal
conditions, then
restart it to get
the memory usage
summaries. There
should be
indication if
there is some
leak, by seeing
memory chunks
allocated many
times from a
function used at
runtime. You can
send the memory
summary for a
process here, we
can look at it.
While this was
running and
Kamailio
didn't respond
to packets, it
logged lots of
lines like
this:
Do you have syslog
to be configured
in asynchronous
mode? See the
notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less
than debug value,
that means
printing few log
messages for each
memory operation.
You can make
memdbg higher and
rely on memlog for
memory summaries,
otherwise will be
lot of log
messages related
to memory.
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x40048918
frag.
0x40048900
(size=128) on
1 -th hit
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008,
128) called
from
<core>:
cfg.lex:
addstr(1438)
Jul 22
21:32:22
hostname
kamailio: :
<core>
[mem/q_malloc.c:413]:
qm_malloc(0x4000e008,
128) returns
address
0x400489c8
frag.
0x400489b0
(size=128) on
1 -th hit
addstr() is a
function used only
for parsing
configuration
file, as long as
you can still see
them, the
configuration file
parsing was not
finish. addstr()
is not a source of
leaks because it
is not used at
runtime.
If you have large
config file, then
you can get close
to the limits of
the private
memory, which is
set to 4MB. You
can increase its
value using -M
parameter (e.g.,
start kamailio
with -M 8 to set
it to use 8MB of
memory).
Over the time, the
private memory can
get used due to
fragmentation, you
can set the
mem_join parameter
in config file to
avoid it (works
when compiled with
MEMDBG=1).
To monitor usage
of internal pkg
memory, then you
can use sercmd
with pkg.stats
command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory
stats are printed
by 'kamctl fifo
get_statistics
shmem:'
When you see
significant
increase of the
memory usage, then
you can restart to
get the summaries.
You should run
these commands
after start, just
to see the initial
usage of memory.
Cheers,
Daniel
--
Daniel-Constantin
Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
- http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express
Router (SER)
and Kamailio
(OpenSER) -
sr-users
mailing list
sr-users@lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham,
Voisonics
http://voisonics.com/
USA: +1
213 221 1092
UK: +44
(0) 20 3298 1642
Australia: +61
(0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298
1642
Australia: +61 (0) 2 8063
9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda