23 Jul
2013
23 Jul
'13
8:33 a.m.
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more
and more memory. Over a couple of weeks it will run out of system memory.
We tried to enable memory debugging doing the following, but it resulted in
Kamailio not responding to any SIP packets. Would anyone have advice please
on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
While this was running and Kamailio didn't respond to packets, it logged
lots of lines like this:
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
Thanks in advance,
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
23 Jul
23 Jul
9:53 a.m.
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be
a leak in a library, because kamailio modules uses only from a
pre-allocated chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
more and more memory. Over a couple of weeks it will run out of system
memory.
We tried to enable memory debugging doing the following, but it
resulted in Kamailio not responding to any SIP packets. Would anyone
have advice please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is
1? It is not
by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to get
the memory usage summaries. There should be indication if there is some
leak, by seeing memory chunks allocated many times from a function used
at runtime. You can send the memory summary for a process here, we can
look at it.
While this was running and Kamailio didn't respond to packets, it
logged lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the notes
from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log
messages for each memory operation. You can make memdbg higher and rely
on memlog for memory summaries, otherwise will be lot of log messages
related to memory.
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing configuration
file, as long
as you can still see them, the configuration file parsing was not
finish. addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of
the private memory, which is set to 4MB. You can increase its value
using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB
of memory).
Over the time, the private memory can get used due to fragmentation, you
can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial usage
of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
24 Jul
24 Jul
5:24 a.m.
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
2. You're right, DEBUG_MEMORY is a local configuration setting. If defined
it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be a
leak in a library, because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more
and more memory. Over a couple of weeks it will run out of system memory.
We tried to enable memory debugging doing the following, but it resulted
in Kamailio not responding to any SIP packets. Would anyone have advice
please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It is not
by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
-
http://www.kamailio.org/wiki/**cookbooks/3.3.x/core#memlog<http://www.ka…
Run kamailio for a while in normal conditions, then restart it to get the
memory usage summaries. There should be indication if there is some leak,
by seeing memory chunks allocated many times from a function used at
runtime. You can send the memory summary for a process here, we can look at
it.
While this was running and Kamailio didn't
respond to packets, it logged
lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the notes
from:
-
http://www.kamailio.org/wiki/**tutorials/3.2.x/syslog<http://www.kamaili…
The memdbg is less than debug value, that means printing few log messages
for each memory operation. You can make memdbg higher and rely on memlog
for memory summaries, otherwise will be lot of log messages related to
memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file, as long
as you can still see them, the configuration file parsing was not finish.
addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of the
private memory, which is set to 4MB. You can increase its value using -M
parameter (e.g., start kamailio with -M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used due to fragmentation, you
can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/**modules/3.3.x/modules_k/kex.**html#idp16972640&l…
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial usage
of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda&…
______________________________**_________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/**cgi-bin/mailman/listinfo/sr-**users<http:/…
9:33 a.m.
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all
the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot
of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but
the rest of code should use internal memory manager which does not go
beyond the limits set with -m and -M, thus not causing an OS memory
exhaustion.
Can you list what modules are you loading? At some point it was a leak
in libssl, in case you use tls a lot. But could be another external
library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If
defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm
memory running out? If the operating system runs out of memory,
then should be a leak in a library, because kamailio modules uses
only from a pre-allocated chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly
using more and more memory. Over a couple of weeks it will run
out of system memory.
We tried to enable memory debugging doing the following, but
it resulted in Kamailio not responding to any SIP packets.
Would anyone have advice please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It
is not by default there, so I assume you added some rules based on
this pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more
compact output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to
get the memory usage summaries. There should be indication if
there is some leak, by seeing memory chunks allocated many times
from a function used at runtime. You can send the memory summary
for a process here, we can look at it.
While this was running and Kamailio didn't respond to packets,
it logged lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the
notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log
messages for each memory operation. You can make memdbg higher and
rely on memlog for memory summaries, otherwise will be lot of log
messages related to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from
<core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called from
<core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file,
as long as you can still see them, the configuration file parsing
was not finish. addstr() is not a source of leaks because it is
not used at runtime.
If you have large config file, then you can get close to the
limits of the private memory, which is set to 4MB. You can
increase its value using -M parameter (e.g., start kamailio with
-M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in config file
to avoid it (works when compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd
with pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you
can restart to get the summaries.
You should run these commands after start, just to see the initial
usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
list
sr-users(a)lists.sip-router.org <mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
11:31 a.m.
Hello,
We don't do any kamctl commands at all. We do have various modules loaded,
as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot of
mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest
of code should use internal memory manager which does not go beyond the
limits set with -m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a leak in
libssl, in case you use tls a lot. But could be another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If defined
it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be a
leak in a library, because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more
and more memory. Over a couple of weeks it will run out of system memory.
We tried to enable memory debugging doing the following, but it resulted
in Kamailio not responding to any SIP packets. Would anyone have advice
please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It is not
by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to get the
memory usage summaries. There should be indication if there is some leak,
by seeing memory chunks allocated many times from a function used at
runtime. You can send the memory summary for a process here, we can look at
it.
While this was running and Kamailio didn't
respond to packets, it logged
lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the notes
from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log messages
for each memory operation. You can make memdbg higher and rely on memlog
for memory summaries, otherwise will be lot of log messages related to
memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file, as long
as you can still see them, the configuration file parsing was not finish.
addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of
the private memory, which is set to 4MB. You can increase its value using
-M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
memory).
Over the time, the private memory can get used due to fragmentation, you
can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial usage
of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
9:55 p.m.
Hello,
I would say that perl_exec() is the one with the highest chances to be
the reason for the leak. Next is line would be db_mysql module, if liked
with some custom mysql client library, although even in this case will
be unlikely.
Back to perl, the module itself does not call any malloc, so it might be
the embedding Perl API that is not used properly in the module.
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of the leak?
I'm not using the perl module, I will try to check it whenever I get a
chance in the next days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various modules
loaded, as follows. The primary functions we use Kamailio for are
phone registrations through usrloc, and routing calls to Asterisk
through logic contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for
debugging kamailio's internal memory manager, which handles pkg
and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do
lot of mi commands (e.g., kamctl ...)? The mi api uses system
malloc, but the rest of code should use internal memory manager
which does not go beyond the limits set with -m and -M, thus not
causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a
leak in libssl, in case you use tls a lot. But could be another
external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting.
If defined it sets memdbg to -2, and memlog to -2. The debug
setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that
a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm
memory running out? If the operating system runs out of
memory, then should be a leak in a library, because kamailio
modules uses only from a pre-allocated chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is
slowly using more and more memory. Over a couple of weeks
it will run out of system memory.
We tried to enable memory debugging doing the following,
but it resulted in Kamailio not responding to any SIP
packets. Would anyone have advice please on how to debug
the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is
1? It is not by default there, so I assume you added some
rules based on this pre-processor directive.
For memory troubleshooting, set memlog to a value lower than
debug parameter in config file and try with mem_summary=12
for a more compact output. See more about these parameters in
the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart
it to get the memory usage summaries. There should be
indication if there is some leak, by seeing memory chunks
allocated many times from a function used at runtime. You can
send the memory summary for a process here, we can look at it.
While this was running and Kamailio didn't respond to
packets, it logged lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See
the notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few
log messages for each memory operation. You can make memdbg
higher and rely on memlog for memory summaries, otherwise
will be lot of log messages related to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x40048918 frag. 0x40048900 (size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128) called
from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128) returns
address 0x400489c8 frag. 0x400489b0 (size=128) on 1 -th hit
addstr() is a function used only for parsing configuration
file, as long as you can still see them, the configuration
file parsing was not finish. addstr() is not a source of
leaks because it is not used at runtime.
If you have large config file, then you can get close to the
limits of the private memory, which is set to 4MB. You can
increase its value using -M parameter (e.g., start kamailio
with -M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in config
file to avoid it (works when compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use
sercmd with pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo
get_statistics shmem:'
When you see significant increase of the memory usage, then
you can restart to get the summaries.
You should run these commands after start, just to see the
initial usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
25 Jul
25 Jul
2:29 a.m.
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release
5.4. If I remember right programs running under Valgrind can have issues,
so I'm not sure if the customer will want to do that. Ideally we'd do it on
a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
I would say that perl_exec() is the one with the highest chances to be the
reason for the leak. Next is line would be db_mysql module, if liked with
some custom mysql client library, although even in this case will be
unlikely.
Back to perl, the module itself does not call any malloc, so it might be
the embedding Perl API that is not used properly in the module.
Can you use some testbed, set children=1 and run kamailio under valgrind,
then do some calls and see if it detects the source of the leak?
I'm not using the perl module, I will try to check it whenever I get a
chance in the next days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various modules loaded,
as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot of
mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest
of code should use internal memory manager which does not go beyond the
limits set with -m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a leak in
libssl, in case you use tls a lot. But could be another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If
defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be a
leak in a library, because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
more and more memory. Over a couple of weeks it will run out of system
memory.
We tried to enable memory debugging doing the following, but it
resulted in Kamailio not responding to any SIP packets. Would anyone have
advice please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY is 1? It is
not by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to get
the memory usage summaries. There should be indication if there is some
leak, by seeing memory chunks allocated many times from a function used at
runtime. You can send the memory summary for a process here, we can look at
it.
While this was running and Kamailio didn't
respond to packets, it
logged lots of lines like this:
Do you have syslog to be configured in asynchronous mode? See the notes
from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log
messages for each memory operation. You can make memdbg higher and rely on
memlog for memory summaries, otherwise will be lot of log messages related
to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing configuration file, as
long as you can still see them, the configuration file parsing was not
finish. addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of
the private memory, which is set to 4MB. You can increase its value using
-M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
memory).
Over the time, the private memory can get used due to fragmentation, you
can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial usage
of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
8:45 a.m.
Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the
perl embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can
have issues, so I'm not sure if the customer will want to do that.
Ideally we'd do it on a test system, but I'm not sure if we have any
RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
I would say that perl_exec() is the one with the highest chances
to be the reason for the leak. Next is line would be db_mysql
module, if liked with some custom mysql client library, although
even in this case will be unlikely.
Back to perl, the module itself does not call any malloc, so it
might be the embedding Perl API that is not used properly in the
module.
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of
the leak?
I'm not using the perl module, I will try to check it whenever I
get a chance in the next days. What version of perl do you have
installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hello,
We don't do any kamctl commands at all. We do have various
modules loaded, as follows. The primary functions we use Kamailio
for are phone registrations through usrloc, and routing calls to
Asterisk through logic contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed
Kamailio.
then all the instructions I gave are useless, they are
for
debugging kamailio's internal memory manager, which handles
pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do
you do lot of mi commands (e.g., kamctl ...)? The mi api uses
system malloc, but the rest of code should use internal
memory manager which does not go beyond the limits set with
-m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it
was a leak in libssl, in case you use tls a lot. But could be
another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration
setting. If defined it sets memdbg to -2, and memlog to -2.
The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is
that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
first, to clarify, is the system memory or kamailio's
pkg/shm memory running out? If the operating system runs
out of memory, then should be a leak in a library,
because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and Kamailio
is slowly using more and more memory. Over a couple
of weeks it will run out of system memory.
We tried to enable memory debugging doing the
following, but it resulted in Kamailio not
responding to any SIP packets. Would anyone have
advice please on how to debug the situation?
1. In Makefile.defs set MEMDBG to 1 and recompile
Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when DEBUG_MEMORY
is 1? It is not by default there, so I assume you added
some rules based on this pre-processor directive.
For memory troubleshooting, set memlog to a value lower
than debug parameter in config file and try with
mem_summary=12 for a more compact output. See more about
these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then
restart it to get the memory usage summaries. There
should be indication if there is some leak, by seeing
memory chunks allocated many times from a function used
at runtime. You can send the memory summary for a
process here, we can look at it.
While this was running and Kamailio didn't respond
to packets, it logged lots of lines like this:
Do you have syslog to be configured in asynchronous
mode? See the notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing
few log messages for each memory operation. You can make
memdbg higher and rely on memlog for memory summaries,
otherwise will be lot of log messages related to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
returns address 0x40048918 frag. 0x40048900
(size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008, 128)
called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008, 128)
returns address 0x400489c8 frag. 0x400489b0
(size=128) on 1 -th hit
addstr() is a function used only for parsing
configuration file, as long as you can still see them,
the configuration file parsing was not finish. addstr()
is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to
the limits of the private memory, which is set to 4MB.
You can increase its value using -M parameter (e.g.,
start kamailio with -M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used due to
fragmentation, you can set the mem_join parameter in
config file to avoid it (works when compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can
use sercmd with pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo
get_statistics shmem:'
When you see significant increase of the memory usage,
then you can restart to get the summaries.
You should run these commands after start, just to see
the initial usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) -
sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
10:11 a.m.
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the perl
embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can have
issues, so I'm not sure if the customer will want to do that. Ideally we'd
do it on a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
I would say that perl_exec() is the one with the highest chances to be
the reason for the leak. Next is line would be db_mysql module, if liked
with some custom mysql client library, although even in this case will be
unlikely.
Back to perl, the module itself does not call any malloc, so it might be
the embedding Perl API that is not used properly in the module.
Can you use some testbed, set children=1 and run kamailio under valgrind,
then do some calls and see if it detects the source of the leak?
I'm not using the perl module, I will try to check it whenever I get a
chance in the next days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various modules
loaded, as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot
of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the
rest of code should use internal memory manager which does not go beyond
the limits set with -m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a leak
in libssl, in case you use tls a lot. But could be another external
library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If
defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
first, to clarify, is the system memory or kamailio's pkg/shm memory
running out? If the operating system runs out of memory, then should be a
leak in a library, because kamailio modules uses only from a pre-allocated
chunk, not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
> Hello,
>
> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
> more and more memory. Over a couple of weeks it will run out of system
> memory.
>
> We tried to enable memory debugging doing the following, but it
> resulted in Kamailio not responding to any SIP packets. Would anyone have
> advice please on how to debug the situation?
>
> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
> 2. In kamailio.cfg add the line:
> #!define DEBUG_MEMORY 1
>
do you set something special in config when DEBUG_MEMORY is 1? It is
not by default there, so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a value lower than debug
parameter in config file and try with mem_summary=12 for a more compact
output. See more about these parameters in the wiki:
- http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions, then restart it to get
the memory usage summaries. There should be indication if there is some
leak, by seeing memory chunks allocated many times from a function used at
runtime. You can send the memory summary for a process here, we can look at
it.
> While this was running and Kamailio didn't respond to packets, it
> logged lots of lines like this:
>
Do you have syslog to be configured in asynchronous mode? See the
notes from:
- http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means printing few log
messages for each memory operation. You can make memdbg higher and rely on
memlog for memory summaries, otherwise will be lot of log messages related
to memory.
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
> (size=128) on 1 -th hit
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
> (size=128) on 1 -th hit
>
addstr() is a function used only for parsing configuration file, as
long as you can still see them, the configuration file parsing was not
finish. addstr() is not a source of leaks because it is not used at runtime.
If you have large config file, then you can get close to the limits of
the private memory, which is set to 4MB. You can increase its value using
-M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
memory).
Over the time, the private memory can get used due to fragmentation,
you can set the mem_join parameter in config file to avoid it (works when
compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then you can use sercmd with
pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
When you see significant increase of the memory usage, then you can
restart to get the summaries.
You should run these commands after start, just to see the initial
usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
31 Jul
31 Jul
9:26 a.m.
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the following.
The undefined value in question is $m, which should be the SIP message.
Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com> wrote:
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the perl
embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can have
issues, so I'm not sure if the customer will want to do that. Ideally we'd
do it on a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
I would say that perl_exec() is the one with the highest chances to be
the reason for the leak. Next is line would be db_mysql module, if liked
with some custom mysql client library, although even in this case will be
unlikely.
Back to perl, the module itself does not call any malloc, so it might be
the embedding Perl API that is not used properly in the module.
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of the leak?
I'm not using the perl module, I will try to check it whenever I get a
chance in the next days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various modules
loaded, as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio.
then all the instructions I gave are useless, they are for debugging
kamailio's internal memory manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low now. Do you do lot
of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the
rest of code should use internal memory manager which does not go beyond
the limits set with -m and -M, thus not causing an OS memory exhaustion.
Can you list what modules are you loading? At some point it was a leak
in libssl, in case you use tls a lot. But could be another external
library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local configuration setting. If
defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
> Hello,
>
> first, to clarify, is the system memory or kamailio's pkg/shm memory
> running out? If the operating system runs out of memory, then should be a
> leak in a library, because kamailio modules uses only from a pre-allocated
> chunk, not going over it.
>
>
> On 7/23/13 7:33 AM, David Cunningham wrote:
>
>> Hello,
>>
>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
>> more and more memory. Over a couple of weeks it will run out of system
>> memory.
>>
>> We tried to enable memory debugging doing the following, but it
>> resulted in Kamailio not responding to any SIP packets. Would anyone have
>> advice please on how to debug the situation?
>>
>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>> 2. In kamailio.cfg add the line:
>> #!define DEBUG_MEMORY 1
>>
> do you set something special in config when DEBUG_MEMORY is 1? It is
> not by default there, so I assume you added some rules based on this
> pre-processor directive.
>
> For memory troubleshooting, set memlog to a value lower than debug
> parameter in config file and try with mem_summary=12 for a more compact
> output. See more about these parameters in the wiki:
>
> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>
> Run kamailio for a while in normal conditions, then restart it to get
> the memory usage summaries. There should be indication if there is some
> leak, by seeing memory chunks allocated many times from a function used at
> runtime. You can send the memory summary for a process here, we can look at
> it.
>
>
>
>> While this was running and Kamailio didn't respond to packets, it
>> logged lots of lines like this:
>>
>
> Do you have syslog to be configured in asynchronous mode? See the
> notes from:
>
> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>
> The memdbg is less than debug value, that means printing few log
> messages for each memory operation. You can make memdbg higher and rely on
> memlog for memory summaries, otherwise will be lot of log messages related
> to memory.
>
>
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>> (size=128) on 1 -th hit
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>> (size=128) on 1 -th hit
>>
> addstr() is a function used only for parsing configuration file, as
> long as you can still see them, the configuration file parsing was not
> finish. addstr() is not a source of leaks because it is not used at runtime.
>
> If you have large config file, then you can get close to the limits of
> the private memory, which is set to 4MB. You can increase its value using
> -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
> memory).
>
> Over the time, the private memory can get used due to fragmentation,
> you can set the mem_join parameter in config file to avoid it (works when
> compiled with MEMDBG=1).
>
> To monitor usage of internal pkg memory, then you can use sercmd with
> pkg.stats command:
>
> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>
> Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
>
> When you see significant increase of the memory usage, then you can
> restart to get the summaries.
>
> You should run these commands after start, just to see the initial
> usage of memory.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla - http://www.asipto.com
> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
1:54 p.m.
Hello,
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs, the
variable should have been initialized after initializing the environment
-- what I tried to do with previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the line with
SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be the
SIP message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com
<mailto:dcunningham@voisonics.com>> wrote:
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
can you try the attached patch? It's the same patch, just for
two versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl
after creating the temporary environment, so it is actually
destroyed by the perl embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux
Server release 5.4. If I remember right programs running
under Valgrind can have issues, so I'm not sure if the
customer will want to do that. Ideally we'd do it on a test
system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I would say that perl_exec() is the one with the highest
chances to be the reason for the leak. Next is line would
be db_mysql module, if liked with some custom mysql
client library, although even in this case will be unlikely.
Back to perl, the module itself does not call any malloc,
so it might be the embedding Perl API that is not used
properly in the module.
Can you use some testbed, set children=1 and run kamailio
under valgrind, then do some calls and see if it detects
the source of the leak?
I'm not using the perl module, I will try to check it
whenever I get a chance in the next days. What version of
perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019> Hello,
We don't do any kamctl commands at all. We do have
various modules loaded, as follows. The primary
functions we use Kamailio for are phone registrations
through usrloc, and routing calls to Asterisk through
logic contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019> Hello,
Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer
killed Kamailio.
then all the instructions I gave
are useless, they
are for debugging kamailio's internal memory
manager, which handles pkg and shm mallocs.
The chances to be from kamailio itself are very low
now. Do you do lot of mi commands (e.g., kamctl
...)? The mi api uses system malloc, but the rest of
code should use internal memory manager which does
not go beyond the limits set with -m and -M, thus
not causing an OS memory exhaustion.
Can you list what modules are you loading? At some
point it was a leak in libssl, in case you use tls a
lot. But could be another external library...
Cheers,
Daniel
2. You're right, DEBUG_MEMORY is a local
configuration setting. If defined it sets memdbg to
-2, and memlog to -2. The debug setting is -1.
3. We'll try setting mem_summary=12, thanks.
4. We'll try setting asynchronous syslog, thanks.
5. Our configuration totals 338 lines, or approx
8.5kb. Is that a lot?
6. We'll try setting mem_join=1, thanks.
On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
first, to clarify, is the system memory or
kamailio's pkg/shm memory running out? If the
operating system runs out of memory, then
should be a leak in a library, because kamailio
modules uses only from a pre-allocated chunk,
not going over it.
On 7/23/13 7:33 AM, David Cunningham wrote:
Hello,
We're running a Kamailio 3.3.4 system, and
Kamailio is slowly using more and more
memory. Over a couple of weeks it will run
out of system memory.
We tried to enable memory debugging doing
the following, but it resulted in Kamailio
not responding to any SIP packets. Would
anyone have advice please on how to debug
the situation?
1. In Makefile.defs set MEMDBG to 1 and
recompile Kamailio.
2. In kamailio.cfg add the line:
#!define DEBUG_MEMORY 1
do you set something special in config when
DEBUG_MEMORY is 1? It is not by default there,
so I assume you added some rules based on this
pre-processor directive.
For memory troubleshooting, set memlog to a
value lower than debug parameter in config file
and try with mem_summary=12 for a more compact
output. See more about these parameters in the
wiki:
-
http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
Run kamailio for a while in normal conditions,
then restart it to get the memory usage
summaries. There should be indication if there
is some leak, by seeing memory chunks allocated
many times from a function used at runtime. You
can send the memory summary for a process here,
we can look at it.
While this was running and Kamailio didn't
respond to packets, it logged lots of lines
like this:
Do you have syslog to be configured in
asynchronous mode? See the notes from:
-
http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
The memdbg is less than debug value, that means
printing few log messages for each memory
operation. You can make memdbg higher and rely
on memlog for memory summaries, otherwise will
be lot of log messages related to memory.
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008,
128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008,
128) returns address 0x40048918 frag.
0x40048900 (size=128) on 1 -th hit
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]: qm_malloc(0x4000e008,
128) called from <core>: cfg.lex: addstr(1438)
Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]: qm_malloc(0x4000e008,
128) returns address 0x400489c8 frag.
0x400489b0 (size=128) on 1 -th hit
addstr() is a function used only for parsing
configuration file, as long as you can still
see them, the configuration file parsing was
not finish. addstr() is not a source of leaks
because it is not used at runtime.
If you have large config file, then you can get
close to the limits of the private memory,
which is set to 4MB. You can increase its value
using -M parameter (e.g., start kamailio with
-M 8 to set it to use 8MB of memory).
Over the time, the private memory can get used
due to fragmentation, you can set the mem_join
parameter in config file to avoid it (works
when compiled with MEMDBG=1).
To monitor usage of internal pkg memory, then
you can use sercmd with pkg.stats command:
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
Shared memory stats are printed by 'kamctl fifo
get_statistics shmem:'
When you see significant increase of the memory
usage, then you can restart to get the summaries.
You should run these commands after start, just
to see the initial usage of memory.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER)
- sr-users mailing list
sr-users(a)lists.sip-router.org
<mailto:sr-users@lists.sip-router.org>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
2:44 p.m.
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs, the
variable should have been initialized after initializing the environment --
what I tried to do with previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the line with
SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be the SIP
message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the perl
embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can have
issues, so I'm not sure if the customer will want to do that. Ideally we'd
do it on a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
Hello,
I would say that perl_exec() is the one with the highest chances to be
the reason for the leak. Next is line would be db_mysql module, if liked
with some custom mysql client library, although even in this case will be
unlikely.
Back to perl, the module itself does not call any malloc, so it might
be the embedding Perl API that is not used properly in the module.
Can you use some testbed, set children=1 and run kamailio under
valgrind, then do some calls and see if it detects the source of the leak?
I'm not using the perl module, I will try to check it whenever I get a
chance in the next days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
Hello,
We don't do any kamctl commands at all. We do have various modules
loaded, as follows. The primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls to Asterisk through logic
contained in Perl via perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
> Hello,
>
>
> On 7/24/13 4:24 AM, David Cunningham wrote:
>
> Hello,
>
> Thank you very much for the email. In reply:
>
> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>
> then all the instructions I gave are useless, they are for debugging
> kamailio's internal memory manager, which handles pkg and shm mallocs.
>
> The chances to be from kamailio itself are very low now. Do you do lot
> of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the
> rest of code should use internal memory manager which does not go beyond
> the limits set with -m and -M, thus not causing an OS memory exhaustion.
>
> Can you list what modules are you loading? At some point it was a leak
> in libssl, in case you use tls a lot. But could be another external
> library...
>
> Cheers,
> Daniel
>
>
>
> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>
> 3. We'll try setting mem_summary=12, thanks.
>
> 4. We'll try setting asynchronous syslog, thanks.
>
> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
> lot?
>
> 6. We'll try setting mem_join=1, thanks.
>
>
>
> On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>
>> Hello,
>>
>> first, to clarify, is the system memory or kamailio's pkg/shm memory
>> running out? If the operating system runs out of memory, then should be a
>> leak in a library, because kamailio modules uses only from a pre-allocated
>> chunk, not going over it.
>>
>>
>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>
>>> Hello,
>>>
>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
>>> more and more memory. Over a couple of weeks it will run out of system
>>> memory.
>>>
>>> We tried to enable memory debugging doing the following, but it
>>> resulted in Kamailio not responding to any SIP packets. Would anyone have
>>> advice please on how to debug the situation?
>>>
>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>> 2. In kamailio.cfg add the line:
>>> #!define DEBUG_MEMORY 1
>>>
>> do you set something special in config when DEBUG_MEMORY is 1? It is
>> not by default there, so I assume you added some rules based on this
>> pre-processor directive.
>>
>> For memory troubleshooting, set memlog to a value lower than debug
>> parameter in config file and try with mem_summary=12 for a more compact
>> output. See more about these parameters in the wiki:
>>
>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>
>> Run kamailio for a while in normal conditions, then restart it to get
>> the memory usage summaries. There should be indication if there is some
>> leak, by seeing memory chunks allocated many times from a function used at
>> runtime. You can send the memory summary for a process here, we can look at
>> it.
>>
>>
>>
>>> While this was running and Kamailio didn't respond to packets, it
>>> logged lots of lines like this:
>>>
>>
>> Do you have syslog to be configured in asynchronous mode? See the
>> notes from:
>>
>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>
>> The memdbg is less than debug value, that means printing few log
>> messages for each memory operation. You can make memdbg higher and rely on
>> memlog for memory summaries, otherwise will be lot of log messages related
>> to memory.
>>
>>
>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>>> (size=128) on 1 -th hit
>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438)
>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>>> (size=128) on 1 -th hit
>>>
>> addstr() is a function used only for parsing configuration file, as
>> long as you can still see them, the configuration file parsing was not
>> finish. addstr() is not a source of leaks because it is not used at runtime.
>>
>> If you have large config file, then you can get close to the limits
>> of the private memory, which is set to 4MB. You can increase its value
>> using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
>> memory).
>>
>> Over the time, the private memory can get used due to fragmentation,
>> you can set the mem_join parameter in config file to avoid it (works when
>> compiled with MEMDBG=1).
>>
>> To monitor usage of internal pkg memory, then you can use sercmd with
>> pkg.stats command:
>>
>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>
>> Shared memory stats are printed by 'kamctl fifo get_statistics shmem:'
>>
>> When you see significant increase of the memory usage, then you can
>> restart to get the summaries.
>>
>> You should run these commands after start, just to see the initial
>> usage of memory.
>>
>> Cheers,
>> Daniel
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.com
>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>
>>
>> _______________________________________________
>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>> list
>> sr-users(a)lists.sip-router.org
>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
2:48 p.m.
Hi Daniel,
In 3.3.x the code is arranged a little differently and I'd like to make
sure we get it right. Could you please advise?
m = sv_newmortal();
sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
On 31 July 2013 21:44, David Cunningham <dcunningham(a)voisonics.com> wrote:
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs, the
variable should have been initialized after initializing the environment --
what I tried to do with previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the line with
SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be the SIP
message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
can you try the attached patch? It's the same patch, just for two
versions, one is for 3.3.x and the other for devel version
It initializes the SIP message variable that is passed to perl after
creating the temporary environment, so it is actually destroyed by the perl
embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
Hi Daniel,
The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
release 5.4. If I remember right programs running under Valgrind can have
issues, so I'm not sure if the customer will want to do that. Ideally we'd
do it on a test system, but I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
> Hello,
>
> I would say that perl_exec() is the one with the highest chances to be
> the reason for the leak. Next is line would be db_mysql module, if liked
> with some custom mysql client library, although even in this case will be
> unlikely.
>
> Back to perl, the module itself does not call any malloc, so it might
> be the embedding Perl API that is not used properly in the module.
>
> Can you use some testbed, set children=1 and run kamailio under
> valgrind, then do some calls and see if it detects the source of the leak?
>
> I'm not using the perl module, I will try to check it whenever I get a
> chance in the next days. What version of perl do you have installed?
>
> Cheers,
> Daniel
>
>
> On 7/24/13 10:31 AM, David Cunningham wrote:
>
> Hello,
>
> We don't do any kamctl commands at all. We do have various modules
> loaded, as follows. The primary functions we use Kamailio for are phone
> registrations through usrloc, and routing calls to Asterisk through logic
> contained in Perl via perl_exec().
> Thanks for all your advice so far!
>
> loadmodule "tm.so"
> loadmodule "tmx.so"
> loadmodule "usrloc.so"
> loadmodule "auth.so"
> loadmodule "auth_db.so"
> loadmodule "ctl.so"
> loadmodule "db_mysql.so"
> loadmodule "kex.so"
> loadmodule "maxfwd.so"
> loadmodule "mi_fifo.so"
> loadmodule "mi_rpc.so"
> loadmodule "nathelper.so"
> loadmodule "perl.so"
> loadmodule "pv.so"
> loadmodule "registrar.so"
> loadmodule "rr.so"
> loadmodule "sanity.so"
> loadmodule "siputils.so"
> loadmodule "sl.so"
> loadmodule "textops.so"
> loadmodule "xlog.so"
>
>
> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>
>> Hello,
>>
>>
>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>
>> Hello,
>>
>> Thank you very much for the email. In reply:
>>
>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>
>> then all the instructions I gave are useless, they are for debugging
>> kamailio's internal memory manager, which handles pkg and shm mallocs.
>>
>> The chances to be from kamailio itself are very low now. Do you do
>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but
>> the rest of code should use internal memory manager which does not go
>> beyond the limits set with -m and -M, thus not causing an OS memory
>> exhaustion.
>>
>> Can you list what modules are you loading? At some point it was a
>> leak in libssl, in case you use tls a lot. But could be another external
>> library...
>>
>> Cheers,
>> Daniel
>>
>>
>>
>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>
>> 3. We'll try setting mem_summary=12, thanks.
>>
>> 4. We'll try setting asynchronous syslog, thanks.
>>
>> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
>> lot?
>>
>> 6. We'll try setting mem_join=1, thanks.
>>
>>
>>
>> On 23 July 2013 16:53, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>>
>>> Hello,
>>>
>>> first, to clarify, is the system memory or kamailio's pkg/shm memory
>>> running out? If the operating system runs out of memory, then should be a
>>> leak in a library, because kamailio modules uses only from a pre-allocated
>>> chunk, not going over it.
>>>
>>>
>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>
>>>> Hello,
>>>>
>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using
>>>> more and more memory. Over a couple of weeks it will run out of system
>>>> memory.
>>>>
>>>> We tried to enable memory debugging doing the following, but it
>>>> resulted in Kamailio not responding to any SIP packets. Would anyone
have
>>>> advice please on how to debug the situation?
>>>>
>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>> 2. In kamailio.cfg add the line:
>>>> #!define DEBUG_MEMORY 1
>>>>
>>> do you set something special in config when DEBUG_MEMORY is 1? It
>>> is not by default there, so I assume you added some rules based on this
>>> pre-processor directive.
>>>
>>> For memory troubleshooting, set memlog to a value lower than debug
>>> parameter in config file and try with mem_summary=12 for a more compact
>>> output. See more about these parameters in the wiki:
>>>
>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>
>>> Run kamailio for a while in normal conditions, then restart it to
>>> get the memory usage summaries. There should be indication if there is some
>>> leak, by seeing memory chunks allocated many times from a function used at
>>> runtime. You can send the memory summary for a process here, we can look at
>>> it.
>>>
>>>
>>>
>>>> While this was running and Kamailio didn't respond to packets, it
>>>> logged lots of lines like this:
>>>>
>>>
>>> Do you have syslog to be configured in asynchronous mode? See the
>>> notes from:
>>>
>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>
>>> The memdbg is less than debug value, that means printing few log
>>> messages for each memory operation. You can make memdbg higher and rely on
>>> memlog for memory summaries, otherwise will be lot of log messages related
>>> to memory.
>>>
>>>
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900
>>>> (size=128) on 1 -th hit
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]:
>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]:
>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0
>>>> (size=128) on 1 -th hit
>>>>
>>> addstr() is a function used only for parsing configuration file, as
>>> long as you can still see them, the configuration file parsing was not
>>> finish. addstr() is not a source of leaks because it is not used at runtime.
>>>
>>> If you have large config file, then you can get close to the limits
>>> of the private memory, which is set to 4MB. You can increase its value
>>> using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of
>>> memory).
>>>
>>> Over the time, the private memory can get used due to fragmentation,
>>> you can set the mem_join parameter in config file to avoid it (works when
>>> compiled with MEMDBG=1).
>>>
>>> To monitor usage of internal pkg memory, then you can use sercmd
>>> with pkg.stats command:
>>>
>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>
>>> Shared memory stats are printed by 'kamctl fifo get_statistics
>>> shmem:'
>>>
>>> When you see significant increase of the memory usage, then you can
>>> restart to get the summaries.
>>>
>>> You should run these commands after start, just to see the initial
>>> usage of memory.
>>>
>>> Cheers,
>>> Daniel
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.com
>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>
>>>
>>> _______________________________________________
>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>> list
>>> sr-users(a)lists.sip-router.org
>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>
>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
2:54 p.m.
Hello,
I pushed the commit to branch 3.3 now -- forgot about it.
The code you pasted below becomes (after the two patches, thus is the
final version that should be used):
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
m = sv_newmortal();
sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
Cheers,
Daniel
On 7/31/13 1:48 PM, David Cunningham wrote:
Hi Daniel,
In 3.3.x the code is arranged a little differently and I'd like to
make sure we get it right. Could you please advise?
m = sv_newmortal();
sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
ENTER; /* everything created after
here */
SAVETMPS; /* ...is a temporary
variable. */
PUSHMARK(SP); /* remember the stack
pointer */
XPUSHs(m); /* Our reference to the
stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
On 31 July 2013 21:44, David Cunningham <dcunningham(a)voisonics.com
<mailto:dcunningham@voisonics.com>> wrote:
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
revising the patch I noticed I was moving the initialization
of the variable after pushing it to perl environment (from the
perl docs, the variable should have been initialized after
initializing the environment -- what I tried to do with
previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the
line with SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like
the following. The undefined value in question is $m, which
should be the SIP message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method
"pseudoVar" on an undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham
<dcunningham(a)voisonics.com
<mailto:dcunningham@voisonics.com>> wrote:
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
can you try the attached patch? It's the same patch,
just for two versions, one is for 3.3.x and the other
for devel version
It initializes the SIP message variable that is
passed to perl after creating the temporary
environment, so it is actually destroyed by the perl
embedded interpreter.
Cheers,
Daniel
On 7/25/13 1:29 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <tel:%2B61%20%280%29%202%208063%209019>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hi Daniel,
The system is running Perl 5.8.8 on Red Hat
Enterprise Linux Server release 5.4. If I remember
right programs running under Valgrind can have
issues, so I'm not sure if the customer will want to
do that. Ideally we'd do it on a test system, but
I'm not sure if we have any RHEL available.
I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
I would say that perl_exec() is the one with the
highest chances to be the reason for the leak.
Next is line would be db_mysql module, if liked
with some custom mysql client library, although
even in this case will be unlikely.
Back to perl, the module itself does not call
any malloc, so it might be the embedding Perl
API that is not used properly in the module.
Can you use some testbed, set children=1 and run
kamailio under valgrind, then do some calls and
see if it detects the source of the leak?
I'm not using the perl module, I will try to
check it whenever I get a chance in the next
days. What version of perl do you have installed?
Cheers,
Daniel
On 7/24/13 10:31 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019> Hello,
We don't do any kamctl commands at all. We do
have various modules loaded, as follows. The
primary functions we use Kamailio for are phone
registrations through usrloc, and routing calls
to Asterisk through logic contained in Perl via
perl_exec().
Thanks for all your advice so far!
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "ctl.so"
loadmodule "db_mysql.so"
loadmodule "kex.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "mi_rpc.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "pv.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "sanity.so"
loadmodule "siputils.so"
loadmodule "sl.so"
loadmodule "textops.so"
loadmodule "xlog.so"
On 24 July 2013 16:33, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>>
wrote:
Hello,
On 7/24/13 4:24 AM, David Cunningham wrote:
> Hello,
>
> Thank you very much for the email. In reply:
>
> 1. The system ran out of memory. Linux's
> oom-killer killed Kamailio.
then all the instructions I gave are
useless, they are for debugging kamailio's
internal memory manager, which handles pkg
and shm mallocs.
The chances to be from kamailio itself are
very low now. Do you do lot of mi commands
(e.g., kamctl ...)? The mi api uses system
malloc, but the rest of code should use
internal memory manager which does not go
beyond the limits set with -m and -M, thus
not causing an OS memory exhaustion.
Can you list what modules are you loading?
At some point it was a leak in libssl, in
case you use tls a lot. But could be
another external library...
Cheers,
Daniel
>
> 2. You're right, DEBUG_MEMORY is a local
> configuration setting. If defined it sets
> memdbg to -2, and memlog to -2. The debug
> setting is -1.
>
> 3. We'll try setting mem_summary=12, thanks.
>
> 4. We'll try setting asynchronous syslog,
> thanks.
>
> 5. Our configuration totals 338 lines, or
> approx 8.5kb. Is that a lot?
>
> 6. We'll try setting mem_join=1, thanks.
>
>
>
> On 23 July 2013 16:53, Daniel-Constantin
> Mierla <miconda(a)gmail.com
> <mailto:miconda@gmail.com>> wrote:
>
> Hello,
>
> first, to clarify, is the system
> memory or kamailio's pkg/shm memory
> running out? If the operating system
> runs out of memory, then should be a
> leak in a library, because kamailio
> modules uses only from a pre-allocated
> chunk, not going over it.
>
>
> On 7/23/13 7:33 AM, David Cunningham
> wrote:
>
> Hello,
>
> We're running a Kamailio 3.3.4
> system, and Kamailio is slowly
> using more and more memory. Over a
> couple of weeks it will run out of
> system memory.
>
> We tried to enable memory
> debugging doing the following, but
> it resulted in Kamailio not
> responding to any SIP packets.
> Would anyone have advice please on
> how to debug the situation?
>
> 1. In Makefile.defs set MEMDBG to
> 1 and recompile Kamailio.
> 2. In kamailio.cfg add the line:
> #!define DEBUG_MEMORY 1
>
> do you set something special in config
> when DEBUG_MEMORY is 1? It is not by
> default there, so I assume you added
> some rules based on this pre-processor
> directive.
>
> For memory troubleshooting, set memlog
> to a value lower than debug parameter
> in config file and try with
> mem_summary=12 for a more compact
> output. See more about these
> parameters in the wiki:
>
> -
> http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>
> Run kamailio for a while in normal
> conditions, then restart it to get the
> memory usage summaries. There should
> be indication if there is some leak,
> by seeing memory chunks allocated many
> times from a function used at runtime.
> You can send the memory summary for a
> process here, we can look at it.
>
>
>
> While this was running and
> Kamailio didn't respond to
> packets, it logged lots of lines
> like this:
>
>
> Do you have syslog to be configured in
> asynchronous mode? See the notes from:
>
> -
> http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>
> The memdbg is less than debug value,
> that means printing few log messages
> for each memory operation. You can
> make memdbg higher and rely on memlog
> for memory summaries, otherwise will
> be lot of log messages related to memory.
>
>
> Jul 22 21:32:22 hostname kamailio:
> : <core> [mem/q_malloc.c:369]:
> qm_malloc(0x4000e008, 128) called
> from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio:
> : <core> [mem/q_malloc.c:413]:
> qm_malloc(0x4000e008, 128) returns
> address 0x40048918 frag.
> 0x40048900 (size=128) on 1 -th hit
> Jul 22 21:32:22 hostname kamailio:
> : <core> [mem/q_malloc.c:369]:
> qm_malloc(0x4000e008, 128) called
> from <core>: cfg.lex: addstr(1438)
> Jul 22 21:32:22 hostname kamailio:
> : <core> [mem/q_malloc.c:413]:
> qm_malloc(0x4000e008, 128) returns
> address 0x400489c8 frag.
> 0x400489b0 (size=128) on 1 -th hit
>
> addstr() is a function used only for
> parsing configuration file, as long as
> you can still see them, the
> configuration file parsing was not
> finish. addstr() is not a source of
> leaks because it is not used at runtime.
>
> If you have large config file, then
> you can get close to the limits of the
> private memory, which is set to 4MB.
> You can increase its value using -M
> parameter (e.g., start kamailio with
> -M 8 to set it to use 8MB of memory).
>
> Over the time, the private memory can
> get used due to fragmentation, you can
> set the mem_join parameter in config
> file to avoid it (works when compiled
> with MEMDBG=1).
>
> To monitor usage of internal pkg
> memory, then you can use sercmd with
> pkg.stats command:
>
>
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>
> Shared memory stats are printed by
> 'kamctl fifo get_statistics shmem:'
>
> When you see significant increase of
> the memory usage, then you can restart
> to get the summaries.
>
> You should run these commands after
> start, just to see the initial usage
> of memory.
>
> Cheers,
> Daniel
>
> --
> Daniel-Constantin Mierla -
> http://www.asipto.com
> http://twitter.com/#!/miconda
> <http://twitter.com/#%21/miconda> -
> http://www.linkedin.com/in/miconda
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio
> (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> <mailto:sr-users@lists.sip-router.org>
>
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>
>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092
> <tel:%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642
> <tel:%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019
> <tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
2 Aug
2 Aug
4:34 a.m.
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 21:54, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
I pushed the commit to branch 3.3 now -- forgot about it.
The code you pasted below becomes (after the two patches, thus is the
final version that should be used):
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
m = sv_newmortal();
sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
Cheers,
Daniel
On 7/31/13 1:48 PM, David Cunningham wrote:
Hi Daniel,
In 3.3.x the code is arranged a little differently and I'd like to make
sure we get it right. Could you please advise?
m = sv_newmortal();
sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
On 31 July 2013 21:44, David Cunningham <dcunningham(a)voisonics.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs, the
variable should have been initialized after initializing the environment --
what I tried to do with previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the line with
SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be the SIP
message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
Hi Daniel,
I'll suggest that to the customer. Thank you!
On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
> Hello,
>
> can you try the attached patch? It's the same patch, just for two
> versions, one is for 3.3.x and the other for devel version
>
> It initializes the SIP message variable that is passed to perl after
> creating the temporary environment, so it is actually destroyed by the perl
> embedded interpreter.
>
> Cheers,
> Daniel
>
>
> On 7/25/13 1:29 AM, David Cunningham wrote:
>
> Hi Daniel,
>
> The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
> release 5.4. If I remember right programs running under Valgrind can have
> issues, so I'm not sure if the customer will want to do that. Ideally we'd
> do it on a test system, but I'm not sure if we have any RHEL available.
> I'll see what we can do. Thanks again.
>
>
> On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>
>> Hello,
>>
>> I would say that perl_exec() is the one with the highest chances to
>> be the reason for the leak. Next is line would be db_mysql module, if liked
>> with some custom mysql client library, although even in this case will be
>> unlikely.
>>
>> Back to perl, the module itself does not call any malloc, so it might
>> be the embedding Perl API that is not used properly in the module.
>>
>> Can you use some testbed, set children=1 and run kamailio under
>> valgrind, then do some calls and see if it detects the source of the leak?
>>
>> I'm not using the perl module, I will try to check it whenever I get
>> a chance in the next days. What version of perl do you have installed?
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/24/13 10:31 AM, David Cunningham wrote:
>>
>> Hello,
>>
>> We don't do any kamctl commands at all. We do have various modules
>> loaded, as follows. The primary functions we use Kamailio for are phone
>> registrations through usrloc, and routing calls to Asterisk through logic
>> contained in Perl via perl_exec().
>> Thanks for all your advice so far!
>>
>> loadmodule "tm.so"
>> loadmodule "tmx.so"
>> loadmodule "usrloc.so"
>> loadmodule "auth.so"
>> loadmodule "auth_db.so"
>> loadmodule "ctl.so"
>> loadmodule "db_mysql.so"
>> loadmodule "kex.so"
>> loadmodule "maxfwd.so"
>> loadmodule "mi_fifo.so"
>> loadmodule "mi_rpc.so"
>> loadmodule "nathelper.so"
>> loadmodule "perl.so"
>> loadmodule "pv.so"
>> loadmodule "registrar.so"
>> loadmodule "rr.so"
>> loadmodule "sanity.so"
>> loadmodule "siputils.so"
>> loadmodule "sl.so"
>> loadmodule "textops.so"
>> loadmodule "xlog.so"
>>
>>
>> On 24 July 2013 16:33, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>>
>>> Hello,
>>>
>>>
>>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>>
>>> Hello,
>>>
>>> Thank you very much for the email. In reply:
>>>
>>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>>
>>> then all the instructions I gave are useless, they are for debugging
>>> kamailio's internal memory manager, which handles pkg and shm mallocs.
>>>
>>> The chances to be from kamailio itself are very low now. Do you do
>>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc, but
>>> the rest of code should use internal memory manager which does not go
>>> beyond the limits set with -m and -M, thus not causing an OS memory
>>> exhaustion.
>>>
>>> Can you list what modules are you loading? At some point it was a
>>> leak in libssl, in case you use tls a lot. But could be another external
>>> library...
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>>
>>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>>
>>> 3. We'll try setting mem_summary=12, thanks.
>>>
>>> 4. We'll try setting asynchronous syslog, thanks.
>>>
>>> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
>>> lot?
>>>
>>> 6. We'll try setting mem_join=1, thanks.
>>>
>>>
>>>
>>> On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda(a)gmail.com>wrote;wrote:
>>>
>>>> Hello,
>>>>
>>>> first, to clarify, is the system memory or kamailio's pkg/shm
>>>> memory running out? If the operating system runs out of memory, then
should
>>>> be a leak in a library, because kamailio modules uses only from a
>>>> pre-allocated chunk, not going over it.
>>>>
>>>>
>>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly
>>>>> using more and more memory. Over a couple of weeks it will run out
of
>>>>> system memory.
>>>>>
>>>>> We tried to enable memory debugging doing the following, but it
>>>>> resulted in Kamailio not responding to any SIP packets. Would anyone
have
>>>>> advice please on how to debug the situation?
>>>>>
>>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>>> 2. In kamailio.cfg add the line:
>>>>> #!define DEBUG_MEMORY 1
>>>>>
>>>> do you set something special in config when DEBUG_MEMORY is 1? It
>>>> is not by default there, so I assume you added some rules based on this
>>>> pre-processor directive.
>>>>
>>>> For memory troubleshooting, set memlog to a value lower than debug
>>>> parameter in config file and try with mem_summary=12 for a more compact
>>>> output. See more about these parameters in the wiki:
>>>>
>>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>
>>>> Run kamailio for a while in normal conditions, then restart it to
>>>> get the memory usage summaries. There should be indication if there is
some
>>>> leak, by seeing memory chunks allocated many times from a function used
at
>>>> runtime. You can send the memory summary for a process here, we can look
at
>>>> it.
>>>>
>>>>
>>>>
>>>>> While this was running and Kamailio didn't respond to packets,
it
>>>>> logged lots of lines like this:
>>>>>
>>>>
>>>> Do you have syslog to be configured in asynchronous mode? See the
>>>> notes from:
>>>>
>>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>
>>>> The memdbg is less than debug value, that means printing few log
>>>> messages for each memory operation. You can make memdbg higher and rely
on
>>>> memlog for memory summaries, otherwise will be lot of log messages
related
>>>> to memory.
>>>>
>>>>
>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]:
>>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag.
0x40048900
>>>>> (size=128) on 1 -th hit
>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]:
>>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag.
0x400489b0
>>>>> (size=128) on 1 -th hit
>>>>>
>>>> addstr() is a function used only for parsing configuration file,
>>>> as long as you can still see them, the configuration file parsing was
not
>>>> finish. addstr() is not a source of leaks because it is not used at
runtime.
>>>>
>>>> If you have large config file, then you can get close to the limits
>>>> of the private memory, which is set to 4MB. You can increase its value
>>>> using -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB
of
>>>> memory).
>>>>
>>>> Over the time, the private memory can get used due to
>>>> fragmentation, you can set the mem_join parameter in config file to
avoid
>>>> it (works when compiled with MEMDBG=1).
>>>>
>>>> To monitor usage of internal pkg memory, then you can use sercmd
>>>> with pkg.stats command:
>>>>
>>>>
>>>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>
>>>> Shared memory stats are printed by 'kamctl fifo get_statistics
>>>> shmem:'
>>>>
>>>> When you see significant increase of the memory usage, then you can
>>>> restart to get the summaries.
>>>>
>>>> You should run these commands after start, just to see the initial
>>>> usage of memory.
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>> --
>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>> _______________________________________________
>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>> list
>>>> sr-users(a)lists.sip-router.org
>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>
>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda
- http://www.linkedin.com/in/miconda
>>>
>>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>
>
> --
> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
1 Oct
1 Oct
1:03 p.m.
Hello,
Following up on this after a long break. It seems we are still having a
memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap
size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used=
191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: dumping
all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 0.
N address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 1.
N address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 2.
N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 3.
N address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 4.
N address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 5.
N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 6.
N address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 7.
N address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 8.
N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 9.
N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 10.
N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 11.
N address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 12.
N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 13.
N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: 14.
N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
On 2 August 2013 11:34, David Cunningham <dcunningham(a)voisonics.com> wrote:
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 21:54, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
I pushed the commit to branch 3.3 now -- forgot about it.
The code you pasted below becomes (after the two patches, thus is the
final version that should be used):
ENTER; /* everything created after here */
SAVETMPS; /* ...is a temporary variable. */
PUSHMARK(SP); /* remember the stack pointer */
m = sv_newmortal();
sv_setref_pv(m, "Kamailio::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
XPUSHs(m); /* Our reference to the stack... */
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
Cheers,
Daniel
On 7/31/13 1:48 PM, David Cunningham wrote:
Hi Daniel,
In 3.3.x the code is arranged a little differently and I'd like to make
sure we get it right. Could you please advise?
m = sv_newmortal();
sv_setref_pv(m, "OpenSER::Message", (void *)_msg);
SvREADONLY_on(SvRV(m));
ENTER; /* everything created after here
*/
SAVETMPS; /* ...is a temporary variable.
*/
PUSHMARK(SP); /* remember the stack pointer
*/
XPUSHs(m); /* Our reference to the stack...
*/
if (mystr)
XPUSHs(sv_2mortal(newSVpv(mystr, strlen(mystr))));
/* Our string to the stack... */
On 31 July 2013 21:44, David Cunningham <dcunningham(a)voisonics.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
Thank you, we will try that.
On 31 July 2013 20:54, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Hello,
revising the patch I noticed I was moving the initialization of the
variable after pushing it to perl environment (from the perl docs, the
variable should have been initialized after initializing the environment --
what I tried to do with previous patch).
There is a new smaller patch to be added:
http://git.sip-router.org/cgi-bin/gitweb.cgi?p=sip-router;a=commitdiff_plai…
Practically, the line XPUSHs(m); has to be moved after the line with
SvREADONLY_on(SvRV(m));
Cheers,
Daniel
On 7/31/13 8:26 AM, David Cunningham wrote:
Hi Daniel,
We tried that patch, but Kamailio logged lots of errors like the
following. The undefined value in question is $m, which should be the SIP
message. Would you have any advice? Thanks.
Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl
[openserxs.xs:1022]: perl error: Can't call method "pseudoVar" on an
undefined value at Foo.pm line 247.#012
On 25 July 2013 17:11, David Cunningham <dcunningham(a)voisonics.com>wrote;wrote:
> Hi Daniel,
>
> I'll suggest that to the customer. Thank you!
>
>
>
> On 25 July 2013 15:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>
>> Hello,
>>
>> can you try the attached patch? It's the same patch, just for two
>> versions, one is for 3.3.x and the other for devel version
>>
>> It initializes the SIP message variable that is passed to perl after
>> creating the temporary environment, so it is actually destroyed by the perl
>> embedded interpreter.
>>
>> Cheers,
>> Daniel
>>
>>
>> On 7/25/13 1:29 AM, David Cunningham wrote:
>>
>> Hi Daniel,
>>
>> The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server
>> release 5.4. If I remember right programs running under Valgrind can have
>> issues, so I'm not sure if the customer will want to do that. Ideally
we'd
>> do it on a test system, but I'm not sure if we have any RHEL available.
>> I'll see what we can do. Thanks again.
>>
>>
>> On 25 July 2013 04:55, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
>>
>>> Hello,
>>>
>>> I would say that perl_exec() is the one with the highest chances to
>>> be the reason for the leak. Next is line would be db_mysql module, if liked
>>> with some custom mysql client library, although even in this case will be
>>> unlikely.
>>>
>>> Back to perl, the module itself does not call any malloc, so it
>>> might be the embedding Perl API that is not used properly in the module.
>>>
>>> Can you use some testbed, set children=1 and run kamailio under
>>> valgrind, then do some calls and see if it detects the source of the leak?
>>>
>>> I'm not using the perl module, I will try to check it whenever I get
>>> a chance in the next days. What version of perl do you have installed?
>>>
>>> Cheers,
>>> Daniel
>>>
>>>
>>> On 7/24/13 10:31 AM, David Cunningham wrote:
>>>
>>> Hello,
>>>
>>> We don't do any kamctl commands at all. We do have various modules
>>> loaded, as follows. The primary functions we use Kamailio for are phone
>>> registrations through usrloc, and routing calls to Asterisk through logic
>>> contained in Perl via perl_exec().
>>> Thanks for all your advice so far!
>>>
>>> loadmodule "tm.so"
>>> loadmodule "tmx.so"
>>> loadmodule "usrloc.so"
>>> loadmodule "auth.so"
>>> loadmodule "auth_db.so"
>>> loadmodule "ctl.so"
>>> loadmodule "db_mysql.so"
>>> loadmodule "kex.so"
>>> loadmodule "maxfwd.so"
>>> loadmodule "mi_fifo.so"
>>> loadmodule "mi_rpc.so"
>>> loadmodule "nathelper.so"
>>> loadmodule "perl.so"
>>> loadmodule "pv.so"
>>> loadmodule "registrar.so"
>>> loadmodule "rr.so"
>>> loadmodule "sanity.so"
>>> loadmodule "siputils.so"
>>> loadmodule "sl.so"
>>> loadmodule "textops.so"
>>> loadmodule "xlog.so"
>>>
>>>
>>> On 24 July 2013 16:33, Daniel-Constantin Mierla
<miconda(a)gmail.com>wrote;wrote:
>>>
>>>> Hello,
>>>>
>>>>
>>>> On 7/24/13 4:24 AM, David Cunningham wrote:
>>>>
>>>> Hello,
>>>>
>>>> Thank you very much for the email. In reply:
>>>>
>>>> 1. The system ran out of memory. Linux's oom-killer killed Kamailio.
>>>>
>>>> then all the instructions I gave are useless, they are for
>>>> debugging kamailio's internal memory manager, which handles pkg and
shm
>>>> mallocs.
>>>>
>>>> The chances to be from kamailio itself are very low now. Do you do
>>>> lot of mi commands (e.g., kamctl ...)? The mi api uses system malloc,
but
>>>> the rest of code should use internal memory manager which does not go
>>>> beyond the limits set with -m and -M, thus not causing an OS memory
>>>> exhaustion.
>>>>
>>>> Can you list what modules are you loading? At some point it was a
>>>> leak in libssl, in case you use tls a lot. But could be another external
>>>> library...
>>>>
>>>> Cheers,
>>>> Daniel
>>>>
>>>>
>>>>
>>>> 2. You're right, DEBUG_MEMORY is a local configuration setting. If
>>>> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1.
>>>>
>>>> 3. We'll try setting mem_summary=12, thanks.
>>>>
>>>> 4. We'll try setting asynchronous syslog, thanks.
>>>>
>>>> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a
>>>> lot?
>>>>
>>>> 6. We'll try setting mem_join=1, thanks.
>>>>
>>>>
>>>>
>>>> On 23 July 2013 16:53, Daniel-Constantin Mierla
<miconda(a)gmail.com>wrote;wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> first, to clarify, is the system memory or kamailio's pkg/shm
>>>>> memory running out? If the operating system runs out of memory, then
should
>>>>> be a leak in a library, because kamailio modules uses only from a
>>>>> pre-allocated chunk, not going over it.
>>>>>
>>>>>
>>>>> On 7/23/13 7:33 AM, David Cunningham wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> We're running a Kamailio 3.3.4 system, and Kamailio is
slowly
>>>>>> using more and more memory. Over a couple of weeks it will run
out of
>>>>>> system memory.
>>>>>>
>>>>>> We tried to enable memory debugging doing the following, but it
>>>>>> resulted in Kamailio not responding to any SIP packets. Would
anyone have
>>>>>> advice please on how to debug the situation?
>>>>>>
>>>>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio.
>>>>>> 2. In kamailio.cfg add the line:
>>>>>> #!define DEBUG_MEMORY 1
>>>>>>
>>>>> do you set something special in config when DEBUG_MEMORY is 1? It
>>>>> is not by default there, so I assume you added some rules based on
this
>>>>> pre-processor directive.
>>>>>
>>>>> For memory troubleshooting, set memlog to a value lower than debug
>>>>> parameter in config file and try with mem_summary=12 for a more
compact
>>>>> output. See more about these parameters in the wiki:
>>>>>
>>>>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog
>>>>>
>>>>> Run kamailio for a while in normal conditions, then restart it to
>>>>> get the memory usage summaries. There should be indication if there
is some
>>>>> leak, by seeing memory chunks allocated many times from a function
used at
>>>>> runtime. You can send the memory summary for a process here, we can
look at
>>>>> it.
>>>>>
>>>>>
>>>>>
>>>>>> While this was running and Kamailio didn't respond to
packets, it
>>>>>> logged lots of lines like this:
>>>>>>
>>>>>
>>>>> Do you have syslog to be configured in asynchronous mode? See the
>>>>> notes from:
>>>>>
>>>>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog
>>>>>
>>>>> The memdbg is less than debug value, that means printing few log
>>>>> messages for each memory operation. You can make memdbg higher and
rely on
>>>>> memlog for memory summaries, otherwise will be lot of log messages
related
>>>>> to memory.
>>>>>
>>>>>
>>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]:
>>>>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag.
0x40048900
>>>>>> (size=128) on 1 -th hit
>>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:369]:
>>>>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex:
addstr(1438)
>>>>>> Jul 22 21:32:22 hostname kamailio: : <core>
[mem/q_malloc.c:413]:
>>>>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag.
0x400489b0
>>>>>> (size=128) on 1 -th hit
>>>>>>
>>>>> addstr() is a function used only for parsing configuration file,
>>>>> as long as you can still see them, the configuration file parsing was
not
>>>>> finish. addstr() is not a source of leaks because it is not used at
runtime.
>>>>>
>>>>> If you have large config file, then you can get close to the
>>>>> limits of the private memory, which is set to 4MB. You can increase
its
>>>>> value using -M parameter (e.g., start kamailio with -M 8 to set it to
use
>>>>> 8MB of memory).
>>>>>
>>>>> Over the time, the private memory can get used due to
>>>>> fragmentation, you can set the mem_join parameter in config file to
avoid
>>>>> it (works when compiled with MEMDBG=1).
>>>>>
>>>>> To monitor usage of internal pkg memory, then you can use sercmd
>>>>> with pkg.stats command:
>>>>>
>>>>>
>>>>>
http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640
>>>>>
>>>>> Shared memory stats are printed by 'kamctl fifo get_statistics
>>>>> shmem:'
>>>>>
>>>>> When you see significant increase of the memory usage, then you
>>>>> can restart to get the summaries.
>>>>>
>>>>> You should run these commands after start, just to see the initial
>>>>> usage of memory.
>>>>>
>>>>> Cheers,
>>>>> Daniel
>>>>>
>>>>> --
>>>>> Daniel-Constantin Mierla - http://www.asipto.com
>>>>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
>>>>> list
>>>>> sr-users(a)lists.sip-router.org
>>>>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> David Cunningham, Voisonics
>>>> http://voisonics.com/
>>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>>> Australia: +61 (0) 2 8063 9019
>>>>
>>>>
>>>> --
>>>> Daniel-Constantin Mierla -
http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
>>>>
>>>
>>>
>>> --
>>> David Cunningham, Voisonics
>>> http://voisonics.com/
>>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>>> Australia: +61 (0) 2 8063 9019
>>>
>>>
>>> --
>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda
- http://www.linkedin.com/in/miconda
>>>
>>>
>>
>>
>> --
>> David Cunningham, Voisonics
>> http://voisonics.com/
>> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
>> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
>> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
>>
>>
>> --
>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
>>
>>
>
>
> --
> David Cunningham, Voisonics
> http://voisonics.com/
> USA: +1 213 221 1092 <%2B1%20213%20221%201092>
> UK: +44 (0) 20 3298 1642
> Australia: +61 (0) 2 8063 9019
>
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
2:57 p.m.
Hello,
you gave only the first part of memory log output. The chunks listed
there are from startup, thus not part of a leak, because they are not
allocated again at runtime.
Can you give the full logs with messages from qm_status? The leaks are
typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl module,
so it might be something else there. I found one, which is related to an
error case, but perhaps you don't get that, otherwise you will notice
the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems we are still having
a memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
used= 191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
3:35 p.m.
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on
the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
you gave only the first part of memory log output. The chunks listed there
are from startup, thus not part of a leak, because they are not allocated
again at runtime.
Can you give the full logs with messages from qm_status? The leaks are
typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl module,
so it might be something else there. I found one, which is related to an
error case, but perhaps you don't get that, otherwise you will notice the
error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
Following up on this after a long break. It seems we are still having a
memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap
size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: used=
191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda&…
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
3:45 p.m.
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP listener). The
main process of kamailio doesn't process SIP messages, so it is not
exposed to leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage of pkg
per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill
-SIGUSR1 on the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
you gave only the first part of memory log output. The chunks
listed there are from startup, thus not part of a leak, because
they are not allocated again at runtime.
Can you give the full logs with messages from qm_status? The leaks
are typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl
module, so it might be something else there. I found one, which is
related to an error case, but perhaps you don't get that,
otherwise you will notice the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems we are still
having a memory leak. We've added the memory debugging options
as per the documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
<core> [main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: (0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: used= 191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414 frag=0x401cf3fc
size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644 frag=0x401cf62c
size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc
size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824 frag=0x401cf80c
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894 frag=0x401cf87c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c frag=0x401cf924
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984 frag=0x401cf96c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c frag=0x401cfa14
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac frag=0x401cfa94
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c frag=0x401cfb04
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c frag=0x401cfb84
size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20,
2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
2 Oct
2 Oct
3:30 a.m.
Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any ideas there?
The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP listener). The main
process of kamailio doesn't process SIP messages, so it is not exposed to
leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per
each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill -SIGUSR1
on the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
you gave only the first part of memory log output. The chunks listed
there are from startup, thus not part of a leak, because they are not
allocated again at runtime.
Can you give the full logs with messages from qm_status? The leaks are
typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl module,
so it might be something else there. I found one, which is related to an
error case, but perhaps you don't get that, otherwise you will notice the
error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
Following up on this after a long break. It seems we are still having a
memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: heap
size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
used= 191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
10:17 a.m.
Hello,
ahh, right, forgot that it was disabled because it was not safe to use
at runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any ideas
there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP listener).
The main process of kamailio doesn't process SIP messages, so it
is not exposed to leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage of
pkg per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill
-SIGUSR1 on the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
you gave only the first part of memory log output. The chunks
listed there are from startup, thus not part of a leak,
because they are not allocated again at runtime.
Can you give the full logs with messages from qm_status? The
leaks are typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using
perl module, so it might be something else there. I found
one, which is related to an error case, but perhaps you don't
get that, otherwise you will notice the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems we are
still having a memory leak. We've added the memory
debugging options as per the documentation with the
following output.
I'm not sure how to read it though - can anyone help?
Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
<core> [main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: (0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: used= 191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414 frag=0x401cf3fc
size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644 frag=0x401cf62c
size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4 frag=0x401cf6dc
size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824 frag=0x401cf80c
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: counters.c:
init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894 frag=0x401cf87c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc frag=0x401cf8b4
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c frag=0x401cf924
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984 frag=0x401cf96c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc frag=0x401cf9a4
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c frag=0x401cfa14
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74 frag=0x401cfa5c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: db_id.c:
dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac frag=0x401cfa94
size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: str_hash.h:
str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c frag=0x401cfb04
size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64 frag=0x401cfb4c
size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c frag=0x401cfb84
size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov
18-20, 2013
- more details about Kamailio trainings at
http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
10:26 a.m.
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it to
this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
ahh, right, forgot that it was disabled because it was not safe to use at
runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all processes.
Another alternative is to call pkg_status() or pkg_summary() from cfgutils
module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any ideas
there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP listener). The
main process of kamailio doesn't process SIP messages, so it is not exposed
to leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per
each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill -SIGUSR1
on the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
you gave only the first part of memory log output. The chunks listed
there are from startup, thus not part of a leak, because they are not
allocated again at runtime.
Can you give the full logs with messages from qm_status? The leaks are
typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl module,
so it might be something else there. I found one, which is related to an
error case, but perhaps you don't get that, otherwise you will notice the
error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
Following up on this after a long break. It seems we are still having a
memory leak. We've added the memory debugging options as per the
documentation with the following output.
I'm not sure how to read it though - can anyone help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
[main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
(0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
used= 191100, used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status: max
used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
alloc'd from <core>: route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
11:19 a.m.
Hello,
the leak is due to keep defining counters or statistics. That should
happen only at startup, but if you execute from perl script some
function from a kamailio module, it may trigger this issue. Can you
share what functions from kamailio are you execute inside the perl script?
Cheers,
Daniel
On 10/2/13 9:26 AM, David Cunningham wrote:
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it
to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
ahh, right, forgot that it was disabled because it was not safe to
use at runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process SIP
messages, so it is not exposed to leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage
of pkg per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hi Daniel,
What I copied and pasted is all that's listed when I do a
kill -SIGUSR1 on the Kamailio master process. How do I get
fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
you gave only the first part of memory log output. The
chunks listed there are from startup, thus not part of a
leak, because they are not allocated again at runtime.
Can you give the full logs with messages from qm_status?
The leaks are typically listed towards the end of those
messages.
Yesterday someone else reported a potential leak when
using perl module, so it might be something else there.
I found one, which is related to an error case, but
perhaps you don't get that, otherwise you will notice
the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems we
are still having a memory leak. We've added the
memory debugging options as per the documentation
with the following output.
I'm not sure how to read it though - can anyone
help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: <core> [main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: (0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: used= 191100,
used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: db_id.c:
dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>: route.c:
init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end check=
c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013
- more details about Kamailio trainings at
http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
11:39 a.m.
Hello,
on a second check, those counter chunks for counters/statistics seems to
be from startup, thus not part of a leak.
How long have you been running kamailio? Looking at the pkg stats from
previous email, the free part is still a lot, with only about 200k used.
It could mean the leak was not making any effect yet.
Cheers,
Daniel
On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:
Hello,
the leak is due to keep defining counters or statistics. That should
happen only at startup, but if you execute from perl script some
function from a kamailio module, it may trigger this issue. Can you
share what functions from kamailio are you execute inside the perl script?
Cheers,
Daniel
On 10/2/13 9:26 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it
to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
ahh, right, forgot that it was disabled because it was not safe
to use at runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all
processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda -http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com - Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any
ideas there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process SIP
messages, so it is not exposed to leaks.
You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hi Daniel,
What I copied and pasted is all that's listed when I do a
kill -SIGUSR1 on the Kamailio master process. How do I get
fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
you gave only the first part of memory log output. The
chunks listed there are from startup, thus not part of
a leak, because they are not allocated again at runtime.
Can you give the full logs with messages from
qm_status? The leaks are typically listed towards the
end of those messages.
Yesterday someone else reported a potential leak when
using perl module, so it might be something else there.
I found one, which is related to an error case, but
perhaps you don't get that, otherwise you will notice
the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
Hello,
Following up on this after a long break. It seems
we are still having a memory leak. We've added the
memory debugging options as per the documentation
with the following output.
I'm not sure how to read it though - can anyone
help? Thank you.
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: <core> [main.c:754]: Memory status (pkg):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: (0x4019b008):
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: heap size= 4194304
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: used= 191100,
used+overhead=462800, free=3731504
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: max used (+overhead)= 480224
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: dumping all alloc'ed. fragments:
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: cnt_hash_add(331)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
counters.c: init_counters(121)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
db_id.c: dupl_string(50)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
str_hash.h: str_hash_alloc(69)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: route_add(159)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: alloc'd from <core>:
route.c: init_rlist(180)
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]:
NOTICE: qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda
<http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013
- more details about Kamailio trainings at
http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642
<tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
3:09 p.m.
Hi Daniel,
Kamailio had only been running for about 24 hours with very little traffic
on that server. Should we wait for longer and throw some calls at it?
Thanks again.
On 2 October 2013 18:39, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
on a second check, those counter chunks for counters/statistics seems to
be from startup, thus not part of a leak.
How long have you been running kamailio? Looking at the pkg stats from
previous email, the free part is still a lot, with only about 200k used. It
could mean the leak was not making any effect yet.
Cheers,
Daniel
On 10/2/13 10:19 AM, Daniel-Constantin Mierla wrote:
Hello,
the leak is due to keep defining counters or statistics. That should
happen only at startup, but if you execute from perl script some function
from a kamailio module, it may trigger this issue. Can you share what
functions from kamailio are you execute inside the perl script?
Cheers,
Daniel
On 10/2/13 9:26 AM, David Cunningham wrote:
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory
information. 10 megabytes of it! I've compressed that and attached it to
this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
ahh, right, forgot that it was disabled because it was not safe to use at
runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
Hi Daniel,
When I send SIGUSR1 to a worker process nothing is logged. Any ideas
there? The following is the output of "sercmd pkg.stats".
{
entry: 0
pid: 14570
rank: 0
used: 191100
free: 3731504
real_used: 462800
}
{
entry: 1
pid: 14571
rank: 1
used: 202656
free: 3714236
real_used: 480068
}
{
entry: 2
pid: 14572
rank: 2
used: 202720
free: 3714220
real_used: 480084
}
{
entry: 3
pid: 14573
rank: 3
used: 202664
free: 3714324
real_used: 479980
}
{
entry: 4
pid: 14574
rank: 4
used: 202744
free: 3714100
real_used: 480204
}
{
entry: 5
pid: 14575
rank: 5
used: 201908
free: 3715176
real_used: 479128
}
{
entry: 6
pid: 14576
rank: 6
used: 202572
free: 3714368
real_used: 479936
}
{
entry: 7
pid: 14577
rank: 7
used: 202740
free: 3714488
real_used: 479816
}
{
entry: 8
pid: 14578
rank: 8
used: 199760
free: 3717084
real_used: 477220
}
{
entry: 9
pid: 14579
rank: 9
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 10
pid: 14580
rank: 10
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 11
pid: 14581
rank: 11
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 12
pid: 14582
rank: 12
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 13
pid: 14583
rank: 13
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 14
pid: 14584
rank: 14
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 15
pid: 14585
rank: 15
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 16
pid: 14586
rank: 16
used: 199440
free: 3722828
real_used: 471476
}
{
entry: 17
pid: 14587
rank: -1
used: 199400
free: 3722868
real_used: 471436
}
{
entry: 18
pid: 14588
rank: -1
used: 2105844
free: 3722868
real_used: 2337944
}
{
entry: 19
pid: 14589
rank: -2
used: 198552
free: 3723236
real_used: 471068
}
{
entry: 20
pid: 0
rank: 0
used: 209756
free: 3712368
real_used: 481936
}
{
entry: 21
pid: 14591
rank: -4
used: 253660
free: 3668704
real_used: 525600
}
On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
send SIGUSR1 to one of the worker processes (e.g., UDP listener). The
main process of kamailio doesn't process SIP messages, so it is not exposed
to leaks.
You can used 'kamcmd pkg.stats" to see statistics about usage of pkg
per each process (older versions have sercmd).
'kamctl ps' gives the list of processes.
Cheers,
Daniel
On 10/1/13 2:35 PM, David Cunningham wrote:
Hi Daniel,
What I copied and pasted is all that's listed when I do a kill -SIGUSR1
on the Kamailio master process. How do I get fuller logs?
Thanks for your help.
On 1 October 2013 21:57, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
you gave only the first part of memory log output. The chunks listed
there are from startup, thus not part of a leak, because they are not
allocated again at runtime.
Can you give the full logs with messages from qm_status? The leaks are
typically listed towards the end of those messages.
Yesterday someone else reported a potential leak when using perl
module, so it might be something else there. I found one, which is related
to an error case, but perhaps you don't get that, otherwise you will notice
the error message.
Cheers,
Daniel
On 10/1/13 12:03 PM, David Cunningham wrote:
> Hello,
>
> Following up on this after a long break. It seems we are still having
> a memory leak. We've added the memory debugging options as per the
> documentation with the following output.
> I'm not sure how to read it though - can anyone help? Thank you.
>
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: <core>
> [main.c:754]: Memory status (pkg):
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> (0x4019b008):
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> heap size= 4194304
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> used= 191100, used+overhead=462800, free=3731504
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> max used (+overhead)= 480224
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> dumping all alloc'ed. fragments:
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 0. N address=0x401cf414 frag=0x401cf3fc size=512 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 1. N address=0x401cf644 frag=0x401cf62c size=128 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 2. N address=0x401cf6f4 frag=0x401cf6dc size=256 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: counters.c: cnt_hash_add(331)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 3. N address=0x401cf824 frag=0x401cf80c size=64 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: counters.c: init_counters(121)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 4. N address=0x401cf894 frag=0x401cf87c size=8 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: init_rlist(180)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 5. N address=0x401cf8cc frag=0x401cf8b4 size=64 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 6. N address=0x401cf93c frag=0x401cf924 size=24 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: route_add(159)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 7. N address=0x401cf984 frag=0x401cf96c size=8 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: init_rlist(180)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 8. N address=0x401cf9bc frag=0x401cf9a4 size=64 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 9. N address=0x401cfa2c frag=0x401cfa14 size=24 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: route_add(159)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 10. N address=0x401cfa74 frag=0x401cfa5c size=8 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: db_id.c: dupl_string(50)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 11. N address=0x401cfaac frag=0x401cfa94 size=64 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: str_hash.h: str_hash_alloc(69)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 12. N address=0x401cfb1c frag=0x401cfb04 size=24 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: route_add(159)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 13. N address=0x401cfb64 frag=0x401cfb4c size=8 used=1
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> alloc'd from <core>: route.c: init_rlist(180)
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
> Oct 1 05:58:48 sip0-test /sbin/kamailio[14570]: NOTICE: qm_status:
> 14. N address=0x401cfb9c frag=0x401cfb84 size=64 used=1
>
>
>
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
3:51 p.m.
Hello,
yes, you should run it for a while, with traffic, otherwise the leak
cannot be discovered.
You can watch the processes with rank 1,2,3 and see when the free memory
is decreasing. Then get the pkg status logs. Get also the list of
running processes with 'kamctl ps' so I know at what pid to look for the
leaks.
Cheers,
Daniel
On 10/2/13 2:09 PM, David Cunningham wrote:
Hi Daniel,
Kamailio had only been running for about 24 hours with very little
traffic on that server. Should we wait for longer and throw some calls
at it?
Thanks again.
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
3 Oct
3 Oct
4:14 a.m.
Thanks Daniel. We will provide feedback when we have let it run for a while.
On 2 October 2013 22:51, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
yes, you should run it for a while, with traffic, otherwise the leak
cannot be discovered.
You can watch the processes with rank 1,2,3 and see when the free memory
is decreasing. Then get the pkg status logs. Get also the list of running
processes with 'kamctl ps' so I know at what pid to look for the leaks.
Cheers,
Daniel
On 10/2/13 2:09 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
Kamailio had only been running for about 24 hours with very little
traffic on that server. Should we wait for longer and throw some calls at
it?
Thanks again.
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda&…
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
4 Oct
4 Oct
5:25 a.m.
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran
for lots of REGISTER packets came through and memory use definitely
increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060
Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060
Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060
Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060
Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060
Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060
Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060
Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060
Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061
Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=67.58.182.98:5061
Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=67.58.182.98:5061
Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=67.58.182.98:5061
Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=67.58.182.98:5061
Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=67.58.182.98:5061
Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=67.58.182.98:5061
Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=67.58.182.98:5061
Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060
Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060
Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060
Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060
Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060
Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060
Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060
Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>: parser/msg_parser.c:
get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>: parser/parse_via.c:
parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>: parser/msg_parser.c:
get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: parser/parse_to.c:
parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: parser/msg_parser.c:
get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 328 bytes from <core>: parser/msg_parser.c:
parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c: route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
15 size= 228 bytes from textops: textops.c: fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 9000 bytes from <core>: rvalue.c: mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
18 size= 16200 bytes from <core>: rvalue.c: mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
58 size= 52200 bytes from <core>: rvalue.c: mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
180 size= 22032 bytes from <core>: route_struct.c: mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
23 size= 1656 bytes from <core>: sr_module.c: register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
23 size= 856 bytes from <core>: sr_module.c: load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 312 bytes from <core>: modparam.c: set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c: subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c: subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c: subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
14 size= 276 bytes from <core>: cfg.lex: pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
63 size= 7048 bytes from <core>: rpc_lookup.c: rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 10240 bytes from <core>: pvapi.c: pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 248 bytes from <core>: db_id.c: dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
10 size= 240 bytes from <core>: route.c: route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c: init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
73 size= 9044 bytes from <core>: counters.c: cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
On 3 October 2013 11:14, David Cunningham <dcunningham(a)voisonics.com> wrote:
Thanks Daniel. We will provide feedback when we have
let it run for a
while.
On 2 October 2013 22:51, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
yes, you should run it for a while, with traffic, otherwise the leak
cannot be discovered.
You can watch the processes with rank 1,2,3 and see when the free memory
is decreasing. Then get the pkg status logs. Get also the list of running
processes with 'kamctl ps' so I know at what pid to look for the leaks.
Cheers,
Daniel
On 10/2/13 2:09 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
Kamailio had only been running for about 24 hours with very little
traffic on that server. Should we wait for longer and throw some calls at
it?
Thanks again.
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda&…
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
9:42 a.m.
Hello,
none of the chunks there are indication of a leak. Can you give the
output of pkg.stats for pid 6824? Which of the processes has the highest
value of used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours it
ran for lots of REGISTER packets came through and memory use
definitely increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=2 PID=6825 Type=udp receiver child=1
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=3 PID=6826 Type=udp receiver child=2
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=4 PID=6827 Type=udp receiver child=3
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=5 PID=6828 Type=udp receiver child=4
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=6 PID=6829 Type=udp receiver child=5
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=7 PID=6830 Type=udp receiver child=6
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=8 PID=6831 Type=udp receiver child=7
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=9 PID=6832 Type=udp receiver child=0
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=10 PID=6833 Type=udp receiver child=1
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=11 PID=6834 Type=udp receiver child=2
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=12 PID=6835 Type=udp receiver child=3
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=13 PID=6836 Type=udp receiver child=4
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=14 PID=6837 Type=udp receiver child=5
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=15 PID=6838 Type=udp receiver child=6
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=16 PID=6839 Type=udp receiver child=7
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=17 PID=6840 Type=udp receiver child=0
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=18 PID=6841 Type=udp receiver child=1
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=19 PID=6842 Type=udp receiver child=2
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=20 PID=6843 Type=udp receiver child=3
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=21 PID=6844 Type=udp receiver child=4
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=22 PID=6845 Type=udp receiver child=5
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=23 PID=6846 Type=udp receiver child=6
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=24 PID=6847 Type=udp receiver child=7
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h:
add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h:
add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c:
receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>: parser/msg_parser.c:
get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>: parser/parse_via.c:
parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>: parser/msg_parser.c:
get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: parser/parse_to.c:
parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: parser/msg_parser.c:
get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 328 bytes from <core>: parser/msg_parser.c:
parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c:
route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c:
fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c:
fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 15 size= 228 bytes from textops: textops.c:
fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 9000 bytes from <core>: rvalue.c:
mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c:
mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 18 size= 16200 bytes from <core>: rvalue.c:
mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 58 size= 52200 bytes from <core>: rvalue.c:
mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 180 size= 22032 bytes from <core>: route_struct.c:
mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c:
xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 86 size= 1752 bytes from <core>: rvalue.c:
rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 46 size= 1592 bytes from <core>: sr_module.c:
fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c:
tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 1656 bytes from <core>: sr_module.c:
register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 856 bytes from <core>: sr_module.c:
load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 312 bytes from <core>: modparam.c:
set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c: subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c: subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c: subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 63 size= 7048 bytes from <core>: rpc_lookup.c:
rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 10240 bytes from <core>: pvapi.c:
pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 240 bytes from <core>: route.c: route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c: init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 73 size= 9044 bytes from <core>: counters.c:
cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
10 Oct
10 Oct
7:48 a.m.
Hi Daniel,
How do I get that information please? The information above includes what I
got from the pkgstats module. Is it by restarting Kamailio?
On 4 October 2013 16:42, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
none of the chunks there are indication of a leak. Can you give the output
of pkg.stats for pid 6824? Which of the processes has the highest value of
used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran
for lots of REGISTER packets came through and memory use definitely
increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060
Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060
Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060
Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060
Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060
Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060
Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060
Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060
Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061
Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=67.58.182.98:5061
Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=67.58.182.98:5061
Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=67.58.182.98:5061
Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=67.58.182.98:5061
Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=67.58.182.98:5061
Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=67.58.182.98:5061
Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=67.58.182.98:5061
Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060
Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060
Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060
Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060
Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060
Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060
Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060
Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>: parser/msg_parser.c:
get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>: parser/parse_via.c:
parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>: parser/msg_parser.c:
get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: parser/parse_to.c:
parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: parser/msg_parser.c:
get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 328 bytes from <core>: parser/msg_parser.c:
parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c: route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 15 size= 228 bytes from textops: textops.c:
fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 9000 bytes from <core>: rvalue.c:
mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 18 size= 16200 bytes from <core>: rvalue.c:
mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 58 size= 52200 bytes from <core>: rvalue.c:
mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
180 size= 22032 bytes from <core>: route_struct.c: mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 1656 bytes from <core>: sr_module.c:
register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 856 bytes from <core>: sr_module.c:
load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 312 bytes from <core>: modparam.c:
set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c: subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c: subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c: subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 63 size= 7048 bytes from <core>: rpc_lookup.c:
rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums: count=
133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 10240 bytes from <core>: pvapi.c:
pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 240 bytes from <core>: route.c: route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c: init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 73 size= 9044 bytes from <core>: counters.c:
cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
1:41 p.m.
Hello,
getting pkg stats for a process:
sercmd pkg.stats pid 1234
replace 1234 with the real PID value. There are options to use rank or
index, see:
- http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872
Is it what you were looking for or did I misunderstood?
Cheers,
Daniel
On 10/10/13 6:48 AM, David Cunningham wrote:
Hi Daniel,
How do I get that information please? The information above includes
what I got from the pkgstats module. Is it by restarting Kamailio?
On 4 October 2013 16:42, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
none of the chunks there are indication of a leak. Can you give
the output of pkg.stats for pid 6824? Which of the processes has
the highest value of used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours
it ran for lots of REGISTER packets came through and memory use
definitely increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=2 PID=6825 Type=udp receiver child=1
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=3 PID=6826 Type=udp receiver child=2
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=4 PID=6827 Type=udp receiver child=3
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=5 PID=6828 Type=udp receiver child=4
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=6 PID=6829 Type=udp receiver child=5
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=7 PID=6830 Type=udp receiver child=6
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=8 PID=6831 Type=udp receiver child=7
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=9 PID=6832 Type=udp receiver child=0
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=10 PID=6833 Type=udp receiver child=1
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=11 PID=6834 Type=udp receiver child=2
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=12 PID=6835 Type=udp receiver child=3
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=13 PID=6836 Type=udp receiver child=4
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=14 PID=6837 Type=udp receiver child=5
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=15 PID=6838 Type=udp receiver child=6
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=16 PID=6839 Type=udp receiver child=7
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=17 PID=6840 Type=udp receiver child=0
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=18 PID=6841 Type=udp receiver child=1
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=19 PID=6842 Type=udp receiver child=2
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=20 PID=6843 Type=udp receiver child=3
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=21 PID=6844 Type=udp receiver child=4
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=22 PID=6845 Type=udp receiver child=5
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=23 PID=6846 Type=udp receiver child=6
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=24 PID=6847 Type=udp receiver child=7
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c:
init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c:
cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h:
add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h:
add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c:
receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>:
parser/msg_parser.c: get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>:
parser/parse_via.c: parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>:
parser/msg_parser.c: get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>:
parser/parse_to.c: parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>:
parser/msg_parser.c: get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 328 bytes from <core>:
parser/msg_parser.c: parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c:
route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c:
new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c:
fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y:
mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c:
db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c:
fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 15 size= 228 bytes from textops: textops.c:
fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 9000 bytes from <core>: rvalue.c:
mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c:
mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c:
mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 18 size= 16200 bytes from <core>: rvalue.c:
mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 58 size= 52200 bytes from <core>: rvalue.c:
mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 180 size= 22032 bytes from <core>: route_struct.c:
mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c:
xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 86 size= 1752 bytes from <core>: rvalue.c:
rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 46 size= 1592 bytes from <core>: sr_module.c:
fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c:
tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 1656 bytes from <core>: sr_module.c:
register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 856 bytes from <core>: sr_module.c:
load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 312 bytes from <core>: modparam.c:
set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c:
subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c:
subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c:
subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 14 size= 276 bytes from <core>: cfg.lex:
pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c:
pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 63 size= 7048 bytes from <core>: rpc_lookup.c:
rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 133 size= 5856 bytes from <core>: pvapi.c:
pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 10240 bytes from <core>: pvapi.c:
pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 248 bytes from <core>: db_id.c:
dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 240 bytes from <core>: route.c:
route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c:
init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 73 size= 9044 bytes from <core>: counters.c:
cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
2:14 p.m.
Hi Daniel,
I was looking to give you the information you requested in your previous
reply for PID 6824. Hopefully this is it:
# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
On 10 October 2013 21:41, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
Hello,
getting pkg stats for a process:
sercmd pkg.stats pid 1234
replace 1234 with the real PID value. There are options to use rank or
index, see:
- http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872
Is it what you were looking for or did I misunderstood?
Cheers,
Daniel
On 10/10/13 6:48 AM, David Cunningham wrote:
Hi Daniel,
How do I get that information please? The information above includes what
I got from the pkgstats module. Is it by restarting Kamailio?
On 4 October 2013 16:42, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
none of the chunks there are indication of a leak. Can you give the
output of pkg.stats for pid 6824? Which of the processes has the highest
value of used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours it
ran for lots of REGISTER packets came through and memory use definitely
increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060
Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=67.58.182.98:5060
Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=67.58.182.98:5060
Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=67.58.182.98:5060
Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=67.58.182.98:5060
Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=67.58.182.98:5060
Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=67.58.182.98:5060
Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=67.58.182.98:5060
Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=67.58.182.98:5061
Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=
67.58.182.98:5061
Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=
67.58.182.98:5061
Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=
67.58.182.98:5061
Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=
67.58.182.98:5061
Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=
67.58.182.98:5061
Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=
67.58.182.98:5061
Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=
67.58.182.98:5061
Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060
Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060
Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060
Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060
Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060
Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060
Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060
Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>: parser/msg_parser.c:
get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>: parser/parse_via.c:
parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>: parser/msg_parser.c:
get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: parser/parse_to.c:
parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: parser/msg_parser.c:
get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 328 bytes from <core>: parser/msg_parser.c:
parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c: route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 15 size= 228 bytes from textops: textops.c:
fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 9000 bytes from <core>: rvalue.c:
mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 18 size= 16200 bytes from <core>: rvalue.c:
mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 58 size= 52200 bytes from <core>: rvalue.c:
mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 180 size= 22032 bytes from <core>: route_struct.c:
mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 1656 bytes from <core>: sr_module.c:
register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 856 bytes from <core>: sr_module.c:
load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 312 bytes from <core>: modparam.c:
set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c: subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c: subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c: subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 63 size= 7048 bytes from <core>: rpc_lookup.c:
rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 10240 bytes from <core>: pvapi.c:
pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 240 bytes from <core>: route.c: route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c: init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 73 size= 9044 bytes from <core>: counters.c:
cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
4:49 p.m.
Hello,
On 10/10/13 1:14 PM, David Cunningham wrote:
Hi Daniel,
I was looking to give you the information you requested in your
previous reply for PID 6824. Hopefully this is it:
# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
used memory is like 200k out of 4MB, so it is not big and the log you
sent with memory chunks didn't reveal a potential leaking. Maybe it is a
corner case not hit so far.
You need to monitor for a while and when you see that used memory value
has increased with few more 100k (or getting at 1MB) and doesn't go back
low, then dump again the log with memory chunks and sent it over.
Cheers,
Daniel
On 10 October 2013 21:41, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hello,
getting pkg stats for a process:
sercmd pkg.stats pid 1234
replace 1234 with the real PID value. There are options to use
rank or index, see:
- http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872
Is it what you were looking for or did I misunderstood?
Cheers,
Daniel
On 10/10/13 6:48 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
How do I get that information please? The information above
includes what I got from the pkgstats module. Is it by restarting
Kamailio?
On 4 October 2013 16:42, Daniel-Constantin Mierla
<miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
none of the chunks there are indication of a leak. Can you
give the output of pkg.stats for pid 6824? Which of the
processes has the highest value of used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019 Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24
hours it ran for lots of REGISTER packets came through and
memory use definitely increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=2 PID=6825 Type=udp receiver child=1
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=3 PID=6826 Type=udp receiver child=2
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=4 PID=6827 Type=udp receiver child=3
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=5 PID=6828 Type=udp receiver child=4
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=6 PID=6829 Type=udp receiver child=5
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=7 PID=6830 Type=udp receiver child=6
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=8 PID=6831 Type=udp receiver child=7
sock=67.58.182.98:5060 <http://67.58.182.98:5060>
Process:: ID=9 PID=6832 Type=udp receiver child=0
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=10 PID=6833 Type=udp receiver child=1
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=11 PID=6834 Type=udp receiver child=2
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=12 PID=6835 Type=udp receiver child=3
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=13 PID=6836 Type=udp receiver child=4
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=14 PID=6837 Type=udp receiver child=5
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=15 PID=6838 Type=udp receiver child=6
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=16 PID=6839 Type=udp receiver child=7
sock=67.58.182.98:5061 <http://67.58.182.98:5061>
Process:: ID=17 PID=6840 Type=udp receiver child=0
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=18 PID=6841 Type=udp receiver child=1
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=19 PID=6842 Type=udp receiver child=2
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=20 PID=6843 Type=udp receiver child=3
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=21 PID=6844 Type=udp receiver child=4
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=22 PID=6845 Type=udp receiver child=5
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=23 PID=6846 Type=udp receiver child=6
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=24 PID=6847 Type=udp receiver child=7
sock=127.0.0.1:5060 <http://127.0.0.1:5060>
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 8192 bytes from mi_fifo:
mi_writer.c: mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 964 bytes from db_mysql:
km_my_con.c: db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 4100 bytes from xlog: xlog.c:
mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 512 bytes from <core>:
counters.c: cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 24 bytes from kex: mi_core.c:
init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 60 bytes from <core>:
script_cb.c: add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 288 bytes from <core>:
cfg/cfg_struct.c: cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 2904 bytes from <core>:
cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 228 bytes from <core>:
socket_info.c: fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 28 bytes from <core>:
udp_server.c: udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 28 bytes from db_mysql:
km_my_con.c: db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 84 bytes from <core>:
name_alias.h: add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 112 bytes from <core>:
name_alias.h: add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 44 bytes from <core>:
socket_info.c: fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 9 size= 392 bytes from <core>:
counters.c: grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 24 bytes from <core>:
socket_info.c: fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 900 bytes from <core>:
receive.c: receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 336 bytes from <core>:
parser/msg_parser.c: get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 96 bytes from <core>:
parser/parse_via.c: parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 116 bytes from <core>:
parser/msg_parser.c: get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 12 bytes from rtpproxy:
rtpproxy.c: child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 32 bytes from <core>:
parser/parse_to.c: parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 24 bytes from <core>:
parser/msg_parser.c: get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 328 bytes from <core>:
parser/msg_parser.c: parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 240 bytes from sanity:
sanity.c: check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 16 bytes from <core>: route.c:
route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 64 bytes from <core>: db_id.c:
new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 32 bytes from <core>:
sr_module.c: fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 124 bytes from <core>: cfg.y:
mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 32 bytes from <core>: db.c:
db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 4 size= 248 bytes from <core>:
sr_module.c: fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 72 bytes from <core>:
mod_fix.c: fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 15 size= 228 bytes from textops:
textops.c: fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 4 size= 140 bytes from textops:
textops.c: hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 9000 bytes from <core>:
rvalue.c: mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 64 bytes from <core>:
route_struct.c: mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 96 bytes from <core>:
ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 18 size= 16200 bytes from <core>:
rvalue.c: mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 58 size= 52200 bytes from <core>:
rvalue.c: mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 180 size= 22032 bytes from <core>:
route_struct.c: mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 28 bytes from <core>: re.c:
subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 24 bytes from xlog: xlog.c:
xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 86 size= 1752 bytes from <core>:
rvalue.c: rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 46 size= 1592 bytes from <core>:
sr_module.c: fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 24 bytes from <core>:
sr_module.c: init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 7 size= 140 bytes from <core>: pvapi.c:
tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 21 size= 9828 bytes from <core>:
sr_module.c: sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 23 size= 1656 bytes from <core>:
sr_module.c: register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 23 size= 856 bytes from <core>:
sr_module.c: load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 312 bytes from <core>:
modparam.c: set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 24 bytes from <core>: re.c:
subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 80 bytes from <core>: re.c:
subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 32 bytes from <core>: re.c:
subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 293 size= 38296 bytes from <core>:
cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 14 size= 276 bytes from <core>: cfg.lex:
pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 20 bytes from <core>: ppcfg.c:
pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 44 bytes from <core>:
socket_info.c: new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 528 bytes from <core>:
socket_info.c: new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 3 size= 48 bytes from <core>: cfg.y:
yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 120 bytes from <core>: cfg.y:
yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 256 bytes from <core>:
rpc_lookup.c: rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 63 size= 7048 bytes from <core>:
rpc_lookup.c: rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 256 bytes from <core>:
str_hash.h: str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 133 size= 5856 bytes from <core>:
pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 10240 bytes from <core>:
pvapi.c: pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 12 bytes from <core>:
nonsip_hooks.c: init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 248 bytes from <core>: db_id.c:
dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 10 size= 240 bytes from <core>: route.c:
route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 6 size= 384 bytes from <core>:
str_hash.h: str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 5 size= 36 bytes from <core>: route.c:
init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 1 size= 64 bytes from <core>:
counters.c: init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 73 size= 9044 bytes from <core>:
counters.c: cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: count= 2 size= 640 bytes from <core>:
str_hash.h: str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE:
qm_sums: -----------------------------
--
Daniel-Constantin Mierla -http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda>
-http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings athttp://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <tel:%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <tel:%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019
<tel:%2B61%20%280%29%202%208063%209019>
11 Oct
11 Oct
12:36 a.m.
Hi Daniel,
Thanks for the reply. Perhaps what we're seeing is normal, and the memory
use is meant to increase as time progresses. Would you expect to see an
ongoing memory use increase, or when should it stop increasing?
On 11 October 2013 00:49, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
Hello,
On 10/10/13 1:14 PM, David Cunningham wrote:
Hi Daniel,
I was looking to give you the information you requested in your previous
reply for PID 6824. Hopefully this is it:
# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
used memory is like 200k out of 4MB, so it is not big and the log you sent
with memory chunks didn't reveal a potential leaking. Maybe it is a corner
case not hit so far.
You need to monitor for a while and when you see that used memory value
has increased with few more 100k (or getting at 1MB) and doesn't go back
low, then dump again the log with memory chunks and sent it over.
Cheers,
Daniel
On 10 October 2013 21:41, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hello,
getting pkg stats for a process:
sercmd pkg.stats pid 1234
replace 1234 with the real PID value. There are options to use rank or
index, see:
- http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872
Is it what you were looking for or did I misunderstood?
Cheers,
Daniel
On 10/10/13 6:48 AM, David Cunningham wrote:
Hi Daniel,
How do I get that information please? The information above includes
what I got from the pkgstats module. Is it by restarting Kamailio?
On 4 October 2013 16:42, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hello,
none of the chunks there are indication of a leak. Can you give the
output of pkg.stats for pid 6824? Which of the processes has the highest
value of used pkg memory?
Cheers,
Daniel
On 10/4/13 4:25 AM, David Cunningham wrote:
Hi Daniel,
Here are the 'kamctl ps' and pkg status logs. During the 24 hours it
ran for lots of REGISTER packets came through and memory use definitely
increased. Thank you.
Process:: ID=0 PID=6822 Type=attendant
Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=
67.58.182.98:5060
Process:: ID=2 PID=6825 Type=udp receiver child=1 sock=
67.58.182.98:5060
Process:: ID=3 PID=6826 Type=udp receiver child=2 sock=
67.58.182.98:5060
Process:: ID=4 PID=6827 Type=udp receiver child=3 sock=
67.58.182.98:5060
Process:: ID=5 PID=6828 Type=udp receiver child=4 sock=
67.58.182.98:5060
Process:: ID=6 PID=6829 Type=udp receiver child=5 sock=
67.58.182.98:5060
Process:: ID=7 PID=6830 Type=udp receiver child=6 sock=
67.58.182.98:5060
Process:: ID=8 PID=6831 Type=udp receiver child=7 sock=
67.58.182.98:5060
Process:: ID=9 PID=6832 Type=udp receiver child=0 sock=
67.58.182.98:5061
Process:: ID=10 PID=6833 Type=udp receiver child=1 sock=
67.58.182.98:5061
Process:: ID=11 PID=6834 Type=udp receiver child=2 sock=
67.58.182.98:5061
Process:: ID=12 PID=6835 Type=udp receiver child=3 sock=
67.58.182.98:5061
Process:: ID=13 PID=6836 Type=udp receiver child=4 sock=
67.58.182.98:5061
Process:: ID=14 PID=6837 Type=udp receiver child=5 sock=
67.58.182.98:5061
Process:: ID=15 PID=6838 Type=udp receiver child=6 sock=
67.58.182.98:5061
Process:: ID=16 PID=6839 Type=udp receiver child=7 sock=
67.58.182.98:5061
Process:: ID=17 PID=6840 Type=udp receiver child=0 sock=127.0.0.1:5060
Process:: ID=18 PID=6841 Type=udp receiver child=1 sock=127.0.0.1:5060
Process:: ID=19 PID=6842 Type=udp receiver child=2 sock=127.0.0.1:5060
Process:: ID=20 PID=6843 Type=udp receiver child=3 sock=127.0.0.1:5060
Process:: ID=21 PID=6844 Type=udp receiver child=4 sock=127.0.0.1:5060
Process:: ID=22 PID=6845 Type=udp receiver child=5 sock=127.0.0.1:5060
Process:: ID=23 PID=6846 Type=udp receiver child=6 sock=127.0.0.1:5060
Process:: ID=24 PID=6847 Type=udp receiver child=7 sock=127.0.0.1:5060
Process:: ID=25 PID=6848 Type=slow timer
Process:: ID=26 PID=6849 Type=timer
Process:: ID=27 PID=6850 Type=ctl handler
Process:: ID=28 PID=6851 Type=MI FIFO
Process:: ID=29 PID=6852 Type=tcp main process
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
summarizing all alloc'ed. fragments:
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8192 bytes from mi_fifo: mi_writer.c:
mi_writer_init(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 964 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(62)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 4100 bytes from xlog: xlog.c: mod_init(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 512 bytes from <core>: counters.c:
cnt_hash_add(382)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from kex: mi_core.c: init_mi_uptime(74)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from auth: auth_mod.c:
generate_random_secret(221)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 76 bytes from auth: auth_mod.c:
generate_random_secret(220)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 60 bytes from <core>: script_cb.c:
add_callback(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 8 bytes from <core>: select.c:
register_select_table(461)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 288 bytes from <core>: cfg/cfg_struct.c:
cfg_new_group(86)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 2904 bytes from <core>: cfg/cfg.c: cfg_declare(52)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 228 bytes from <core>: socket_info.c:
fix_sock_str(436)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: udp_server.c:
udp_rcv_loop(434)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from db_mysql: km_my_con.c:
db_mysql_new_connection(53)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 84 bytes from <core>: name_alias.h: add_alias(101)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 112 bytes from <core>: name_alias.h: add_alias(99)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
fix_hostname(1377)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 9 size= 392 bytes from <core>: counters.c:
grp_hash_add(233)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 24 bytes from <core>: socket_info.c:
fix_socket_list(1581)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 900 bytes from <core>: receive.c: receive_msg(120)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 336 bytes from <core>: parser/msg_parser.c:
get_hdr_field(171)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 96 bytes from <core>: parser/parse_via.c:
parse_via(2429)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 116 bytes from <core>: parser/msg_parser.c:
get_hdr_field(131)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 12 bytes from rtpproxy: rtpproxy.c:
child_init(907)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: parser/parse_to.c:
parse_to_param(287)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: parser/msg_parser.c:
get_hdr_field(150)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 328 bytes from <core>: parser/msg_parser.c:
parse_headers(341)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 240 bytes from sanity: sanity.c:
check_parse_uris(767)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 16 bytes from <core>: route.c: route_new_list(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: db_id.c: new_db_id(242)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: sr_module.c: fix_param(1142)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 124 bytes from <core>: cfg.y: mk_case_stm(3877)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 32 bytes from <core>: db.c: db_do_init2(300)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 56 bytes from <core>: switch.c:
mk_switch_cond_table(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 248 bytes from <core>: sr_module.c: fix_param(1203)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 72 bytes from <core>: mod_fix.c:
fixup_regexp_null(219)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 15 size= 228 bytes from textops: textops.c:
fixup_method(1705)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 4 size= 140 bytes from textops: textops.c:
hname_fixup(1634)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 8 size= 448 bytes from <core>: pvapi.c:
pv_parse_format(954)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 44 bytes from <core>: rvalue.c:
mk_rval_expr_v(2545)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 9000 bytes from <core>: rvalue.c:
mk_rval_expr2(2669)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 64 bytes from <core>: route_struct.c: mk_elem(127)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 96 bytes from <core>: ip_addr.c: mk_new_net(66)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 18 size= 16200 bytes from <core>: rvalue.c:
mk_rval_expr1(2611)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 58 size= 52200 bytes from <core>: rvalue.c:
mk_rval_expr_v(2534)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 180 size= 22032 bytes from <core>: route_struct.c:
mk_action(155)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 28 bytes from <core>: re.c: subst_str(533)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from xlog: xlog.c: xdbg_fixup_helper(346)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 86 size= 1752 bytes from <core>: rvalue.c: rval_get_str(1260)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 46 size= 1592 bytes from <core>: sr_module.c: fix_param(1108)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: sr_module.c:
init_modules(971)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 7 size= 140 bytes from <core>: pvapi.c: tr_table_add(1660)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 21 size= 9828 bytes from <core>: sr_module.c:
sr_cmd_exports_convert(234)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 1656 bytes from <core>: sr_module.c:
register_module(278)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 23 size= 856 bytes from <core>: sr_module.c:
load_module(485)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 312 bytes from <core>: modparam.c:
set_mod_param_regex(124)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: cfg.lex:
sr_push_yy_state(1692)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 24 bytes from <core>: re.c: subst_parser(315)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 80 bytes from <core>: re.c: subst_parser(306)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 32 bytes from <core>: re.c: subst_parser(289)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 293 size= 38296 bytes from <core>: cfg.lex: addstr(1439)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 14 size= 276 bytes from <core>: cfg.lex: pp_define(1799)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: ppcfg.c: pp_subst_add(54)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 20 bytes from <core>: cfg.lex:
sr_push_yy_state(1682)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 44 bytes from <core>: socket_info.c:
new_sock_info(251)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 528 bytes from <core>: socket_info.c:
new_sock_info(246)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 3 size= 48 bytes from <core>: cfg.y: yyparse(722)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 120 bytes from <core>: cfg.y: yyparse(1820)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: rpc_lookup.c:
rpc_hash_add(156)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 63 size= 7048 bytes from <core>: rpc_lookup.c:
rpc_hash_add(110)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 256 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 133 size= 5856 bytes from <core>: pvapi.c: pv_table_add(231)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 10240 bytes from <core>: pvapi.c:
pv_init_buffer(1831)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 40 bytes from <core>: pvapi.c:
pv_init_buffer(1821)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 12 bytes from <core>: nonsip_hooks.c:
init_nonsip_hooks(57)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 248 bytes from <core>: db_id.c: dupl_string(50)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 10 size= 240 bytes from <core>: route.c: route_add(159)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 6 size= 384 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 5 size= 36 bytes from <core>: route.c: init_rlist(180)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 1 size= 64 bytes from <core>: counters.c:
init_counters(121)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 73 size= 9044 bytes from <core>: counters.c:
cnt_hash_add(331)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
count= 2 size= 640 bytes from <core>: str_hash.h:
str_hash_alloc(69)
Oct 3 22:23:21 sip0-test /sbin/kamailio[6844]: NOTICE: qm_sums:
-----------------------------
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092 <%2B1%20213%20221%201092>
UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642>
Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019>
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
4:23 p.m.
Hi David,
On 10/10/13 11:36 PM, David Cunningham wrote:
Hi Daniel,
Thanks for the reply. Perhaps what we're seeing is normal, and the
memory use is meant to increase as time progresses. Would you expect
to see an ongoing memory use increase, or when should it stop increasing?
private memory (pkg) should stay rather constant. It increases when
there is a sip message processed, but once is sent out, it should come
back around the average.
There are couple of functions that can fill the private memory and keep
it up, such as doing an sql_query() that returns a big data and the
result is not freed (sql_result_free()). It is not actually a leak as
the next sql_query() will free previous result, but in case you have
such query for some corner case that is not executed frequently, then
the memory can stay filled in. Another example will be storing very
large value in a $var(...) (e.g., $var(x) ).
This is private memory, per process, which is meant for temporary
operations. Shared memory (shm) can increase over the time, being the
place where the dynamic data required at runtime is stored (e.g.,
location records, hash tables, transactions) - so as you get more
traffic or more phones using kamailio, more shm is used. But your
problem was reported for pkg.
Anyhow, keep an eye on the pkg.stats and if you see constant increase
which is substantial, then get a mem log dump.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
15 Oct
15 Oct
6:12 a.m.
Hi Daniel,
Thanks for the reply again. Looking at the email history, I'm not sure how
we decided it was definitely a pkg memory problem. What we see is the
output of "ps aux" as follows:
root@sip0-test:~# ps aux | egrep -i "kamailio|mem"
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12
/opt/testuser/current/sbin/testuser_safe_kamailio
testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
You'll see for example that process 6824 is using 8.7% of memory, which is
much more than it was using 5 days ago. Yet if I run the same sercmd again
I get (exactly!) the same numbers:
root@sip0-test:~# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
Any ideas?
On 12 October 2013 00:23, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
Hi David,
On 10/10/13 11:36 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi Daniel,
Thanks for the reply. Perhaps what we're seeing is normal, and the memory
use is meant to increase as time progresses. Would you expect to see an
ongoing memory use increase, or when should it stop increasing?
private memory (pkg) should stay rather constant. It increases when
there is a
sip message processed, but once is sent out, it should come back
around the average.
There are couple of functions that can fill the private memory and keep it
up, such as doing an sql_query() that returns a big data and the result is
not freed (sql_result_free()). It is not actually a leak as the next
sql_query() will free previous result, but in case you have such query for
some corner case that is not executed frequently, then the memory can stay
filled in. Another example will be storing very large value in a $var(...)
(e.g., $var(x) ).
This is private memory, per process, which is meant for temporary
operations. Shared memory (shm) can increase over the time, being the place
where the dynamic data required at runtime is stored (e.g., location
records, hash tables, transactions) - so as you get more traffic or more
phones using kamailio, more shm is used. But your problem was reported for
pkg.
Anyhow, keep an eye on the pkg.stats and if you see constant increase
which is substantial, then get a mem log dump.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda&…
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
16 Oct
16 Oct
12:28 p.m.
Hello,
indeed, it looks like a system memory leak, for what so ever reason the
discussion was misled to pkg.
Can you send me the list of modules you load (loadmodule lines) as well
as the functions from your perl script that are related to kamailio
internal functions or variables? You can send directly to me, without
mailing list if there is something you want to protect from public eyes.
Cheers,
Daniel
On 10/15/13 5:12 AM, David Cunningham wrote:
Hi Daniel,
Thanks for the reply again. Looking at the email history, I'm not sure
how we decided it was definitely a pkg memory problem. What we see is
the output of "ps aux" as follows:
root@sip0-test:~# ps aux | egrep -i "kamailio|mem"
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12
/opt/testuser/current/sbin/testuser_safe_kamailio
testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
You'll see for example that process 6824 is using 8.7% of memory,
which is much more than it was using 5 days ago. Yet if I run the same
sercmd again I get (exactly!) the same numbers:
root@sip0-test:~# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
Any ideas?
On 12 October 2013 00:23, Daniel-Constantin Mierla <miconda(a)gmail.com
<mailto:miconda@gmail.com>> wrote:
Hi David,
On 10/10/13 11:36 PM, David Cunningham wrote:
Hi Daniel,
Thanks for the reply. Perhaps what we're seeing is normal, and
the memory use is meant to increase as time progresses. Would
you expect to see an ongoing memory use increase, or when
should it stop increasing?
private memory (pkg) should stay rather constant. It increases
when there is a sip message processed, but once is sent out, it
should come back around the average.
There are couple of functions that can fill the private memory and
keep it up, such as doing an sql_query() that returns a big data
and the result is not freed (sql_result_free()). It is not
actually a leak as the next sql_query() will free previous result,
but in case you have such query for some corner case that is not
executed frequently, then the memory can stay filled in. Another
example will be storing very large value in a $var(...) (e.g.,
$var(x) ).
This is private memory, per process, which is meant for temporary
operations. Shared memory (shm) can increase over the time, being
the place where the dynamic data required at runtime is stored
(e.g., location records, hash tables, transactions) - so as you
get more traffic or more phones using kamailio, more shm is used.
But your problem was reported for pkg.
Anyhow, keep an eye on the pkg.stats and if you see constant
increase which is substantial, then get a mem log dump.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20,
2013
- more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
17 Oct
17 Oct
5:23 a.m.
Hi Daniel,
Configuration sent to you privately. Thanks.
On 16 October 2013 20:28, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
Hello,
indeed, it looks like a system memory leak, for what so ever reason the
discussion was misled to pkg.
Can you send me the list of modules you load (loadmodule lines) as well as
the functions from your perl script that are related to kamailio internal
functions or variables? You can send directly to me, without mailing list
if there is something you want to protect from public eyes.
Cheers,
Daniel
On 10/15/13 5:12 AM, David Cunningham wrote:
Hi Daniel,
Thanks for the reply again. Looking at the email history, I'm not sure
how we decided it was definitely a pkg memory problem. What we see is the
output of "ps aux" as follows:
root@sip0-test:~# ps aux | egrep -i "kamailio|mem"
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12
/opt/testuser/current/sbin/testuser_safe_kamailio
testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02
/sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid
You'll see for example that process 6824 is using 8.7% of memory, which
is much more than it was using 5 days ago. Yet if I run the same sercmd
again I get (exactly!) the same numbers:
root@sip0-test:~# sercmd pkg.stats pid 6824
{
entry: 1
pid: 6824
rank: 1
used: 209836
free: 3704080
real_used: 490224
}
Any ideas?
On 12 October 2013 00:23, Daniel-Constantin Mierla <miconda(a)gmail.com>wrote;wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
Hi David,
On 10/10/13 11:36 PM, David Cunningham wrote:
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
Hi Daniel,
Thanks for the reply. Perhaps what we're seeing is normal, and the
memory use is meant to increase as time progresses. Would you expect to see
an ongoing memory use increase, or when should it stop increasing?
private memory (pkg) should stay rather constant. It increases when
there is a
sip message processed, but once is sent out, it should come back
around the average.
There are couple of functions that can fill the private memory and keep
it up, such as doing an sql_query() that returns a big data and the result
is not freed (sql_result_free()). It is not actually a leak as the next
sql_query() will free previous result, but in case you have such query for
some corner case that is not executed frequently, then the memory can stay
filled in. Another example will be storing very large value in a $var(...)
(e.g., $var(x) ).
This is private memory, per process, which is meant for temporary
operations. Shared memory (shm) can increase over the time, being the place
where the dynamic data required at runtime is stored (e.g., location
records, hash tables, transactions) - so as you get more traffic or more
phones using kamailio, more shm is used. But your problem was reported for
pkg.
Anyhow, keep an eye on the pkg.stats and if you see constant increase
which is substantial, then get a mem log dump.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013
- more details about Kamailio trainings at http://www.asipto.com -
2 Oct
2 Oct
1:04 p.m.
Hi Daniel
Just want to make you aware of something on kamailio 4.0.3 .
I loaded cfgutils and used pkg_summary() . But I run into the following:
0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fbad3882010,
0x7fbad3ab7220), called from <core>: rpc_lookup.c: destroy_rpcs(79)
0(14112) NOTICE: <core> [mem/q_malloc.c:472]: qm_free(): qm_free: freeing frag.
0x7fbad3ab71f0 alloc'ed from <core>: rpc_lookup.c: rpc_hash_add(156)
0(14112) NOTICE: <core> [mem/q_malloc.c:437]: qm_free(): qm_free(0x7fba8923e000,
0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806)
0(14112) : <core> [mem/q_malloc.c:466]: qm_free(): BUG: qm_free: freeing already
freed pointer (0x7fba894e62b0), called from cfgutils: cfgutils.c: mod_destroy(806), first
free cfgutils: cfgutils.c: mod_destroy(806) - aborting
Regards,
Dragos
________________________________
From: David Cunningham <dcunningham(a)voisonics.com>
To: Daniel-Constantin Mierla <miconda(a)gmail.com>
Cc: Kamailio (SER) - Users Mailing List <sr-users(a)lists.sip-router.org>
Sent: Wednesday, October 2, 2013 9:26 AM
Subject: Re: [SR-Users] Memory leak in 3.3.4
Hi Daniel,
Thank you - I used the restart method and got a whole load of memory information. 10
megabytes of it! I've compressed that and attached it to this email.
Thanks for your help.
On 2 October 2013 17:17, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
Hello,
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
ahh, right, forgot that it was disabled because it was not safe to
use at
runtime for worker processes. You have to use:
kamcmd cfg.set_now_int core mem_dump_pkg <pid>
then wait for a bit so that process is receiving a SIP message.
Or, if you restart, the stats are printed in syslog for all
processes.
Another alternative is to call pkg_status() or pkg_summary() from
cfgutils
module in config file, like:
if(src_ip==127.0.0.1 && is_method("OPTIONS") &&
$rU=="pkgstats") {
send_reply("200", "OK");
pkg_summary();
exit;
}
You can use sipsak to send an options request to "sip:pkgstats@..."
Cheers,
Daniel
On 10/2/13 2:30 AM, David Cunningham wrote:
Hi Daniel,
>
>When I send SIGUSR1 to a worker process nothing is logged. Any
ideas
there? The following is the output of "sercmd pkg.stats".
>
>{
> entry: 0
> pid: 14570
> rank: 0
> used: 191100
> free: 3731504
> real_used: 462800
>}
>{
> entry: 1
> pid: 14571
> rank: 1
> used: 202656
> free: 3714236
> real_used: 480068
>}
>{
> entry: 2
> pid: 14572
> rank: 2
> used: 202720
> free: 3714220
> real_used: 480084
>}
>{
> entry: 3
> pid: 14573
> rank: 3
> used: 202664
> free: 3714324
> real_used: 479980
>}
>{
> entry: 4
> pid: 14574
> rank: 4
> used: 202744
> free: 3714100
> real_used: 480204
>}
>{
> entry: 5
> pid: 14575
> rank: 5
> used: 201908
> free: 3715176
> real_used: 479128
>}
>{
> entry: 6
> pid: 14576
> rank: 6
> used: 202572
> free: 3714368
> real_used: 479936
>}
>{
> entry: 7
> pid: 14577
> rank: 7
> used: 202740
> free: 3714488
> real_used: 479816
>}
>{
> entry: 8
> pid: 14578
> rank: 8
> used: 199760
> free: 3717084
> real_used: 477220
>}
>{
> entry: 9
> pid: 14579
> rank: 9
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 10
> pid: 14580
> rank: 10
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 11
> pid: 14581
> rank: 11
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 12
> pid: 14582
> rank: 12
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 13
> pid: 14583
> rank: 13
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 14
> pid: 14584
> rank: 14
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 15
> pid: 14585
> rank: 15
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 16
> pid: 14586
> rank: 16
> used: 199440
> free: 3722828
> real_used: 471476
>}
>{
> entry: 17
> pid: 14587
> rank: -1
> used: 199400
> free: 3722868
> real_used: 471436
>}
>{
> entry: 18
> pid: 14588
> rank: -1
> used: 2105844
> free: 3722868
> real_used: 2337944
>}
>{
> entry: 19
> pid: 14589
> rank: -2
> used: 198552
> free: 3723236
> real_used: 471068
>}
>{
> entry: 20
> pid: 0
> rank: 0
> used: 209756
> free: 3712368
> real_used: 481936
>}
>{
> entry: 21
> pid: 14591
> rank: -4
> used: 253660
> free: 3668704
> real_used: 525600
>}
>
>
>
>
>
>
>On 1 October 2013 22:45, Daniel-Constantin Mierla <miconda(a)gmail.com> wrote:
>
>Hello,
>>
>>send SIGUSR1 to one of the worker processes (e.g., UDP
listener). The main process of kamailio doesn't process
SIP messages, so it is not exposed to leaks.
>>
>>You can used 'kamcmd pkg.stats" to see statistics about
usage of pkg per each process (older versions have
sercmd).
>>
>>'kamctl ps' gives the list of processes.
>>
>>Cheers,
>>Daniel
>>
>>
>>
>>On 10/1/13 2:35 PM, David Cunningham wrote:
>>
>>Hi Daniel,
>>>
>>>
What I copied and pasted is all that's listed when I do a kill
-SIGUSR1 on the Kamailio master process. How do I get fuller logs?
>>>
>>>
Thanks for your help.
>>>
>>>
>>>
>>>
>>>>
>>>>On 1 October 2013 21:57, Daniel-Constantin Mierla
<miconda(a)gmail.com> wrote:
>>>>
>>>>Hello,
>>>>>
>>>>>you gave only the first part of memory log
output. The chunks listed there are from
startup, thus not part of a leak, because they
are not allocated again at runtime.
>>>>
>>>>Can you give the full logs with messages from
qm_status? The leaks are typically listed
towards the end of those messages.
>>>>
>>>>Yesterday someone else reported a potential
leak when using perl module, so it might be
something else there. I found one, which is
related to an error case, but perhaps you
don't get that, otherwise you will notice the
error message.
>>>>
>>>>Cheers,
>>>>Daniel
>>>>
>>>>
>>>>On 10/1/13 12:03 PM, David Cunningham
wrote:
>>>>
>>>>Hello,
>>>>>
>>>>>Following up on this after a long break.
It seems we are still having a memory
leak. We've added the memory debugging
options as per the documentation with
the following output.
>>>>>I'm not sure how to read it
though - can
anyone help? Thank you.
>>>>>
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
<core> [main.c:754]: Memory status
(pkg):
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: (0x4019b008):
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: heap size= 4194304
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: used= 191100,
used+overhead=462800, free=3731504
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: max used (+overhead)= 480224
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: dumping all alloc'ed.
fragments:
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 0. N address=0x401cf414
frag=0x401cf3fc size=512 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 1. N address=0x401cf644
frag=0x401cf62c size=128 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 2. N address=0x401cf6f4
frag=0x401cf6dc size=256 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
cnt_hash_add(331)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 3. N address=0x401cf824
frag=0x401cf80c size=64 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: counters.c:
init_counters(121)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 4. N address=0x401cf894
frag=0x401cf87c size=8 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 5. N address=0x401cf8cc
frag=0x401cf8b4 size=64 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 6. N address=0x401cf93c
frag=0x401cf924 size=24 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 7. N address=0x401cf984
frag=0x401cf96c size=8 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 8. N address=0x401cf9bc
frag=0x401cf9a4 size=64 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 9. N address=0x401cfa2c
frag=0x401cfa14 size=24 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 10. N address=0x401cfa74
frag=0x401cfa5c size=8 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: db_id.c: dupl_string(50)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 11. N address=0x401cfaac
frag=0x401cfa94 size=64 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: str_hash.h:
str_hash_alloc(69)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 12. N address=0x401cfb1c
frag=0x401cfb04 size=24 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: route_add(159)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 13. N address=0x401cfb64
frag=0x401cfb4c size=8 used=1
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: alloc'd from
<core>: route.c: init_rlist(180)
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: start
check=f0f0f0f0, end check= c0c0c0c0,
abcdefed
>>>>>Oct 1 05:58:48 sip0-test
/sbin/kamailio[14570]: NOTICE:
qm_status: 14. N address=0x401cfb9c
frag=0x401cfb84 size=64 used=1
>>>>>
>>>>>
>>>>>
>>>>
>>>>--
>>>>Daniel-Constantin Mierla - http://www.asipto.com
>>>>http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
>>>>
Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20,
2013
>>>>> - more details about Kamailio trainings at http://www.asipto.com -
>>>>>
> >>>
>>>
>>>>
>>>>--
>>>>David Cunningham, Voisonics
>>>>http://voisonics.com/
>>>>USA: +1 213 221 1092
>>>>UK: +44 (0) 20 3298 1642
>>>>Australia: +61 (0) 2 8063 9019
>>>>
>>>
>>>--
Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami,
Nov 18-20, 2013 - more details about Kamailio trainings at http://www.asipto.com -
--
David Cunningham, Voisonics
http://voisonics.com/
USA: +1 213 221 1092
UK: +44 (0) 20 3298 1642
Australia: +61 (0) 2 8063 9019
--
4054
days inactive
4140
days old
38 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
David Cunningham -
Dragos Oancea