Re: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
8 May
2008
8 May
'08
10 a.m.
Bogdan,
I'll need to check out the shmem statistics.
I mean, this is the memory output for the OpenSER processes having run
for about 24 hours. Does it look right?:
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
Here are the modules I'm using. Not sure if any have a dependency on libxml:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
Thanks,
Julian
On Wed, May 7, 2008 at 5:22 AM, Bogdan-Andrei Iancu
<bogdan(a)voice-system.ro> wrote:
Hi Julian,
Have you tried to use the the shmem statistics to monitor the usage of SHM
memory?
Both for PKG and SHM memory, openser allocs at startup the whole memory
(depending of how much you configured). Any leak will not be vizible for any
external tools.
Also, are you using any module that have dependency of libxml library?
The dump you posted shows more or less nothing more than init phase -
before sending the signal, be sure you select a worker process (use fifo
command "ps" to get the list and types of processes).
Regards,
Bogdan
Julian Yap wrote:
On Mon, May 5, 2008 at 10:25 PM, Henning Westerholt
<henning.westerholt(a)1und1.de> wrote:
> On Saturday 03 May 2008, Julian Yap wrote:
> > I'm currently running OpenSER 1.3.1.
> >
> > It seems to slowly leaks memory.
> >
> > I've recompiled it as per for debug support in the memory manager:
> > http://www.openser.org/dokuwiki/doku.php/troubleshooting:memory
> >
> > Here is the memory manager dump of the memory status:
> > http://julianyap.com/download/openser1.3.1_memory_dump20080502.txt
> >
> > Any suggestions would be very welcome.
>
> Hi Julian,
>
> can you give a little more details about your problem? Why do you think
it
leaks
memory? Do you get memory allocation errors in the logs?
Cheers,
Henning
Henning,
I don't get any memory allocation errors.
I've recently updated to OpenSER 1.3 branch SVN r4107 but the 'leak'
still remains.
On a 4GB box, free memory just seems to deplete.
I have a cron job which restarts OpenSER every night but it seems like
not all the memory is free'd up. Eventually the box needs to be
rebooted. In the past, running out of memory has led to oom-killer
killing OpenSER.
For example, I have this cron job:
*/5 * * * * root echo "`date` `free | grep 'buffers/cache'`" >>
/var/log/free_memory.log
Memory just depletes. Here's where OpenSER was restarted around midnight:
Sun May 4 23:10:01 HST 2008 -/+ buffers/cache: 2014972 2095204
Sun May 4 23:15:01 HST 2008 -/+ buffers/cache: 2027184 2082992
Sun May 4 23:20:01 HST 2008 -/+ buffers/cache: 2027604 2082572
Sun May 4 23:25:01 HST 2008 -/+ buffers/cache: 2038064 2072112
Sun May 4 23:30:01 HST 2008 -/+ buffers/cache: 2037868 2072308
Sun May 4 23:35:01 HST 2008 -/+ buffers/cache: 2053184 2056992
Sun May 4 23:40:01 HST 2008 -/+ buffers/cache: 2055216 2054960
Sun May 4 23:45:01 HST 2008 -/+ buffers/cache: 2075612 2034564
Sun May 4 23:50:01 HST 2008 -/+ buffers/cache: 2103232 2006944
Sun May 4 23:55:01 HST 2008 -/+ buffers/cache: 2114384 1995792
Mon May 5 00:00:01 HST 2008 -/+ buffers/cache: 2117320 1992856
Mon May 5 00:05:01 HST 2008 -/+ buffers/cache: 860304 3249872
Mon May 5 00:10:01 HST 2008 -/+ buffers/cache: 861208 3248968
Mon May 5 00:15:01 HST 2008 -/+ buffers/cache: 868968 3241208
Here's where I just restarted OpenSER:
Mon May 5 23:30:01 HST 2008 -/+ buffers/cache: 2041740 2068436
Mon May 5 23:35:01 HST 2008 -/+ buffers/cache: 2049572 2060604
Mon May 5 23:40:01 HST 2008 -/+ buffers/cache: 2054856 2055320
Mon May 5 23:45:01 HST 2008 -/+ buffers/cache: 2065316 2044860
Mon May 5 23:50:01 HST 2008 -/+ buffers/cache: 2062248 2047928
Mon May 5 23:55:01 HST 2008 -/+ buffers/cache: 2076148 2034028
Tue May 6 00:00:01 HST 2008 -/+ buffers/cache: 2087256 2022920
Tue May 6 00:05:01 HST 2008 -/+ buffers/cache: 2091300 2018876
Tue May 6 00:10:01 HST 2008 -/+ buffers/cache: 2093244 2016932
Tue May 6 00:15:01 HST 2008 -/+ buffers/cache: 1382632 2727544
Thanks,
Julian
_______________________________________________
Devel mailing list
Devel(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/devel
8 May
8 May
6:05 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
Julian,
how much private and shared memory are you using and how many process
you have?
Again, note that openser is not dynamically getting memory from OS at
runtime - all memory that is used by openser is requested from OS at
startup.
Regards,
Bogdan
Julian Yap wrote:
Bogdan,
I'll need to check out the shmem statistics.
I mean, this is the memory output for the OpenSER processes having run
for about 24 hours. Does it look right?:
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
Here are the modules I'm using. Not sure if any have a dependency on libxml:
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
Thanks,
Julian
On Wed, May 7, 2008 at 5:22 AM, Bogdan-Andrei Iancu
<bogdan(a)voice-system.ro> wrote:
Hi Julian,
Have you tried to use the the shmem statistics to monitor the usage of SHM
memory?
Both for PKG and SHM memory, openser allocs at startup the whole memory
(depending of how much you configured). Any leak will not be vizible for any
external tools.
Also, are you using any module that have dependency of libxml library?
The dump you posted shows more or less nothing more than init phase -
before sending the signal, be sure you select a worker process (use fifo
command "ps" to get the list and types of processes).
Regards,
Bogdan
Julian Yap wrote:
On Mon, May 5, 2008 at 10:25 PM, Henning Westerholt
<henning.westerholt(a)1und1.de> wrote:
> On Saturday 03 May 2008, Julian Yap wrote:
> > I'm currently running OpenSER 1.3.1.
> >
> > It seems to slowly leaks memory.
> >
> > I've recompiled it as per for debug support in the memory manager:
> > http://www.openser.org/dokuwiki/doku.php/troubleshooting:memory
> >
> > Here is the memory manager dump of the memory status:
> > http://julianyap.com/download/openser1.3.1_memory_dump20080502.txt
> >
> > Any suggestions would be very welcome.
>
> Hi Julian,
>
> can you give a little more details about your problem? Why do you think
>
it
leaks
memory? Do you get memory allocation errors in the logs?
Cheers,
Henning
Henning,
I don't get any memory allocation errors.
I've recently updated to OpenSER 1.3 branch SVN r4107 but the 'leak'
still remains.
On a 4GB box, free memory just seems to deplete.
I have a cron job which restarts OpenSER every night but it seems like
not all the memory is free'd up. Eventually the box needs to be
rebooted. In the past, running out of memory has led to oom-killer
killing OpenSER.
For example, I have this cron job:
*/5 * * * * root echo "`date` `free | grep 'buffers/cache'`" >>
/var/log/free_memory.log
Memory just depletes. Here's where OpenSER was restarted around midnight:
Sun May 4 23:10:01 HST 2008 -/+ buffers/cache: 2014972 2095204
Sun May 4 23:15:01 HST 2008 -/+ buffers/cache: 2027184 2082992
Sun May 4 23:20:01 HST 2008 -/+ buffers/cache: 2027604 2082572
Sun May 4 23:25:01 HST 2008 -/+ buffers/cache: 2038064 2072112
Sun May 4 23:30:01 HST 2008 -/+ buffers/cache: 2037868 2072308
Sun May 4 23:35:01 HST 2008 -/+ buffers/cache: 2053184 2056992
Sun May 4 23:40:01 HST 2008 -/+ buffers/cache: 2055216 2054960
Sun May 4 23:45:01 HST 2008 -/+ buffers/cache: 2075612 2034564
Sun May 4 23:50:01 HST 2008 -/+ buffers/cache: 2103232 2006944
Sun May 4 23:55:01 HST 2008 -/+ buffers/cache: 2114384 1995792
Mon May 5 00:00:01 HST 2008 -/+ buffers/cache: 2117320 1992856
Mon May 5 00:05:01 HST 2008 -/+ buffers/cache: 860304 3249872
Mon May 5 00:10:01 HST 2008 -/+ buffers/cache: 861208 3248968
Mon May 5 00:15:01 HST 2008 -/+ buffers/cache: 868968 3241208
Here's where I just restarted OpenSER:
Mon May 5 23:30:01 HST 2008 -/+ buffers/cache: 2041740 2068436
Mon May 5 23:35:01 HST 2008 -/+ buffers/cache: 2049572 2060604
Mon May 5 23:40:01 HST 2008 -/+ buffers/cache: 2054856 2055320
Mon May 5 23:45:01 HST 2008 -/+ buffers/cache: 2065316 2044860
Mon May 5 23:50:01 HST 2008 -/+ buffers/cache: 2062248 2047928
Mon May 5 23:55:01 HST 2008 -/+ buffers/cache: 2076148 2034028
Tue May 6 00:00:01 HST 2008 -/+ buffers/cache: 2087256 2022920
Tue May 6 00:05:01 HST 2008 -/+ buffers/cache: 2091300 2018876
Tue May 6 00:10:01 HST 2008 -/+ buffers/cache: 2093244 2016932
Tue May 6 00:15:01 HST 2008 -/+ buffers/cache: 1382632 2727544
Thanks,
Julian
_______________________________________________
Devel mailing list
Devel(a)lists.openser.org
http://lists.openser.org/cgi-bin/mailman/listinfo/devel
6:10 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
On Thursday 08 May 2008, Bogdan-Andrei Iancu wrote:
how much private and shared memory are you using and
how many process
you have?
Again, note that openser is not dynamically getting memory from OS at
runtime - all memory that is used by openser is requested from OS at
startup.
Hi Bogdan,
(quoted from a previous mail from him)
Private memory is default. Should I increase this?
Is there a
recommended rule of thumb?
Shared memory is set for 128Mb. There were memory errors at 32Mb but
none since this was increased to 128Mb.
Using the setting 'children = 8'.
Cheers,
Henning
6:50 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
Hi Henning,
Thanks for the hint.
So, your openser processes should use as private memory:
(8/*udp*/ + 1/*timer*/ + 1/*attendant*/ + 1/*fifo*/) * 1 = 11M
So, overall, all openser processes will use 11 + 128 = 139M plus memory
requested directly from the OS.
looking at your numbers:
<quote>
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
</quote>
it is clearly that you have 8 workers (2 - 9) - the rest of the
processes have a similar VSZ and RSS, so no more mem is required form
the system.
For the workers, I guess there are some script function (from some
modules) that are directly requesting memory (and not via openser mem
manager). Looking at your list:
<quote>
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
</quote>
there are none of the modules that use libxml2 or expat, but I see you
are using perl - there were some past problems with mem leaks due this
module, so you may try to take out the module and see if the leak is
still present.
Regards,
Bogdan
Henning Westerholt wrote:
On Thursday 08 May 2008, Bogdan-Andrei Iancu wrote:
how much private and shared memory are you using
and how many process
you have?
Again, note that openser is not dynamically getting memory from OS at
runtime - all memory that is used by openser is requested from OS at
startup.
Hi Bogdan,
(quoted from a previous mail from him)
Private memory is default. Should I increase
this? Is there a
recommended rule of thumb?
Shared memory is set for 128Mb. There were memory errors at 32Mb but
none since this was increased to 128Mb.
Using the setting 'children = 8'.
Cheers,
Henning
10:25 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
Bogdan,
Thanks for the clarification and your suggestion.
Julian
On Thu, May 8, 2008 at 5:50 AM, Bogdan-Andrei Iancu
<bogdan(a)voice-system.ro> wrote:
Hi Henning,
Thanks for the hint.
So, your openser processes should use as private memory:
(8/*udp*/ + 1/*timer*/ + 1/*attendant*/ + 1/*fifo*/) * 1 = 11M
So, overall, all openser processes will use 11 + 128 = 139M plus memory
requested directly from the OS.
looking at your numbers:
<quote>
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
</quote>
it is clearly that you have 8 workers (2 - 9) - the rest of the processes
have a similar VSZ and RSS, so no more mem is required form the system.
For the workers, I guess there are some script function (from some modules)
that are directly requesting memory (and not via openser mem manager).
Looking at your list:
<quote>
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
</quote>
there are none of the modules that use libxml2 or expat, but I see you are
using perl - there were some past problems with mem leaks due this module,
so you may try to take out the module and see if the leak is still present.
Regards,
Bogdan
Henning Westerholt wrote:
On Thursday 08 May 2008, Bogdan-Andrei Iancu
wrote:
how much private and shared memory are you using
and how many process
you have?
Again, note that openser is not dynamically getting memory from OS at
runtime - all memory that is used by openser is requested from OS at
startup.
Hi Bogdan,
(quoted from a previous mail from him)
Private memory is default. Should I increase
this? Is there a
recommended rule of thumb?
Shared memory is set for 128Mb. There were memory errors at 32Mb but
none since this was increased to 128Mb.
Using the setting 'children = 8'.
Cheers,
Henning
5 Jun
5 Jun
4:06 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
Bogdan-Andrei Iancu schrieb:
Hi Henning,
Thanks for the hint.
So, your openser processes should use as private memory:
(8/*udp*/ + 1/*timer*/ + 1/*attendant*/ + 1/*fifo*/) * 1 = 11M
So, overall, all openser processes will use 11 + 128 = 139M plus memory
requested directly from the OS.
looking at your numbers:
<quote>
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
</quote>
it is clearly that you have 8 workers (2 - 9) - the rest of the
processes have a similar VSZ and RSS, so no more mem is required form
the system.
For the workers, I guess there are some script function (from some
modules) that are directly requesting memory (and not via openser mem
manager). Looking at your list:
<quote>
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
</quote>
there are none of the modules that use libxml2 or expat, but I see you
are using perl - there were some past problems with mem leaks due this
module, so you may try to take out the module and see if the leak is
still present.
what about mysql? It uses the mysql library which probably will use OS
memory allocator and not openser's memory allocator.
regards
klaus
6 Jun
6 Jun
6:15 p.m.
New subject: [OpenSER-Users] [OpenSER-Devel] Memory dump for OpenSER 1.3.1 which slowly leaks memory
Hi Klaus,
I discarded the mysql option as a potential problem as in this module,
all the mem ops are transparently done by the mysql client library - as
a user of the lib, there are very small chances to screw up something
that will lead to mem leak, without having worst side effects (like out
of sync errors).
On the other hand, libxml2 give you (as user of the lib) more
flexibility in playing with data (xml and attr nodes), so the chance to
get a undetected leak is much, much higher.
Regards,
Bogdan
Klaus Darilion wrote:
Bogdan-Andrei Iancu schrieb:
Hi Henning,
Thanks for the hint.
So, your openser processes should use as private memory:
(8/*udp*/ + 1/*timer*/ + 1/*attendant*/ + 1/*fifo*/) * 1 = 11M
So, overall, all openser processes will use 11 + 128 = 139M plus
memory requested directly from the OS.
looking at your numbers:
<quote>
# ps -C openser -o vsz,rss
VSZ RSS
234124 12636
369940 159392
369940 158840
369940 158860
370340 159220
369804 158696
370068 158932
370204 159048
370464 159344
234124 20904
234124 9484
234128 7952
234124 7948
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 7880
234124 8332
</quote>
it is clearly that you have 8 workers (2 - 9) - the rest of the
processes have a similar VSZ and RSS, so no more mem is required form
the system.
For the workers, I guess there are some script function (from some
modules) that are directly requesting memory (and not via openser mem
manager). Looking at your list:
<quote>
loadmodule "mysql.so"
loadmodule "sl.so"
loadmodule "tm.so"
loadmodule "usrloc.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "maxfwd.so"
loadmodule "mi_fifo.so"
loadmodule "nathelper.so"
loadmodule "perl.so"
loadmodule "registrar.so"
loadmodule "rr.so"
loadmodule "textops.so"
loadmodule "uri.so"
</quote>
there are none of the modules that use libxml2 or expat, but I see
you are using perl - there were some past problems with mem leaks due
this module, so you may try to take out the module and see if the
leak is still present.
what about mysql? It uses the mysql library which probably will use OS
memory allocator and not openser's memory allocator.
regards
klaus
6043
days inactive
6072
days old
6 comments
4 participants
participants (4)
-
Bogdan-Andrei Iancu -
Henning Westerholt -
Julian Yap -
Klaus Darilion