Hey Daniel, When I looked at the ChangeLog, it does not seem to indicate that the fix is there.

Should I be concerned about this?

Should I just use the daily snapshot instead? How stable is the daily snapshot?

Tim On 4/23/07, Daniel-Constantin Mierla <daniel(a)voice-system.ro> wrote: > Hello, > > daily snapshots are enabled now for 1.2.x as well: > > http://www.openser.org/downloads/snapshots/openser-1.2.x/ > > Cheers, > Daniel > > On 04/23/07 17:47, Ovidiu Sas wrote: > > Hi Tim, > > > > Check the download page from openser website: > > http://www.openser.org/mos/view/Download/: > > > > The command that you need to run: > > svn co

> > openser > > > > Make sure that you have svn installed. > > > > > > Regards, > > Ovidiu Sas > > > > On 4/23/07, Tim Madorma <tmadorma(a)gmail.com> wrote: > >> Hi Daniel, > >> > >> I have run into a leak in 1.2 and I assume it is the same one that > >> Ovidiu ran into. I see in your response that it was "backported to > >> 1.2", but I'm not sure how to get the fix. When I look at the SVN > >> repository at: > >> http://www.openser.org/pub/openser/latest-1.2.x/, the date is

> >> than the date of your email exchange so I don't think the fix has > been > >> added there. Can you please let me know how I can get it? > >> > >> thanks, > >> Tim > >> > >> On 3/23/07, Daniel-Constantin Mierla <daniel(a)voice-system.ro>

> >> > Hello Ovidiu, > >> > > >> > On 03/23/07 17:04, Ovidiu Sas wrote: > >> > > Hi Daniel, > >> > > > >> > > Can we backport this one to 1.2? > >> > already done, two minutes after the commit in trunk. > >> > > >> > Cheers, > >> > Daniel > >> > > >> > > > >> > > > >> > > Regards, > >> > > Ovidiu Sas > >> > > > >> > > On 3/22/07, Daniel-Constantin Mierla <daniel(a)voice-system.ro> > wrote: > >> > >> Hello, > >> > >> > >> > >> the supposed fragmentation turned out to be a mem leak in

> >> Please > >> > >> take the latest SVN version and try again to see if you

> >> > >> results. > >> > >> > >> > >> Thanks, > >> > >> Daniel > >> > >> > >> > >> On 03/19/07 18:52, Christian Schlatter wrote: > >> > >> > ... > >> > >> >>> The memory statistics indeed show a high number of memory > >> fragments: > >> > >> >>> > >> > >> >>> before 'out of memory': > >> > >> >>> > >> > >> >>> shmem:total_size = 536870912 > >> > >> >>> shmem:used_size = 59607040 > >> > >> >>> shmem:real_used_size = 60106488 > >> > >> >>> shmem:max_used_size = 68261536 > >> > >> >>> shmem:free_size = 476764424 > >> > >> >>> shmem:fragments = 9897 > >> > >> >>> > >> > >> >>> after 'out of memory' (about 8000 calls per process): > >> > >> >>> > >> > >> >>> shmem:total_size = 536870912 > >> > >> >>> shmem:used_size = 4171160 > >> > >> >>> shmem:real_used_size = 4670744 > >> > >> >>> shmem:max_used_size = 68261536 > >> > >> >>> shmem:free_size = 532200168 > >> > >> >>> shmem:fragments = 57902 > >> > >> >>> > >> > >> >>>> > >> > >> >>>> You can try to compile openser with -DQM_JOIN_FREE

> >> in DEFS > >> > >> >>>> variable of Makefile.defs) and test again. Free

> >> should be > >> > >> >>>> merged and fragmentation should not occur -- processing > >> will be > >> > >> >>>> slower. We will try for next release to provide a better > >> solution > >> > >> >>>> for that. > >> > >> >>> > >> > >> >>> Compiling openser with -DQM_JOIN_FREE did not help.

> >> sure how > >> > >> >>> big of a problem this fragmentation issue is. > >> > >> >> What is the number of fragments with QM_JOIN_FREE after > >> flooding? > >> > >> > > >> > >> > The numbers included above are with QM_JOIN_FREE enabled. > >> > >> > > >> > >> >>> Do you think it would make sense to restart our

> >> openser > >> > >> >>> instances from time to time just to make sure they're not > >> running > >> > >> >>> into this memory fragmentation limits? > >> > >> >> The issue will occur only when the call rate reaches the > >> limits of > >> > >> >> the proxy's memory. Otherwise the chunks are reused. > >> Transactions and > >> > >> >> avps are rounded up to be sure there will be minimized the > >> number of > >> > >> >> different sizes for memory chunks. It wasn't reported too > often, > >> > >> >> maybe that's why no big attention was paid to it. This

> >> system > >> > >> >> is in place since the beginning of ser. Alternative is

> >> sysv > >> > >> >> shared memory, but is much slower, along with libc private > >> memory > >> > >> >> manager. > >> > >> > > >> > >> > I've done some more testing and the same out-of-memory

> >> happens > >> > >> > when I run sipp with 10 calls per second only. I tested

> >> > >> > 'children=1' and I only could get through about 8200 calls > (again > >> > >> > those 8000 calls / process). And this is with QM_JOIN_FREE > >> enabled. > >> > >> > > >> > >> > Memory statistics: > >> > >> > > >> > >> > before: > >> > >> > shmem:total_size = 536870912 > >> > >> > shmem:used_size = 2311976 > >> > >> > shmem:real_used_size = 2335720 > >> > >> > shmem:max_used_size = 2465816 > >> > >> > shmem:free_size = 534535192 > >> > >> > shmem:fragments = 183 > >> > >> > > >> > >> > after: > >> > >> > shmem:total_size = 536870912 > >> > >> > shmem:used_size = 1853472 > >> > >> > shmem:real_used_size = 1877224 > >> > >> > shmem:max_used_size = 2465816 > >> > >> > shmem:free_size = 534993688 > >> > >> > shmem:fragments = 547 > >> > >> > > >> > >> > So I'm not sure if this is really a fragmentation issue. 10 > >> cps surely > >> > >> > doesn't reach the proxy's memory. > >> > >> > > >> > >> > Thoughts? > >> > >> > > >> > >> > Christian > >> > >> > > >> > >> > > >> > >> > > >> > >> >> Cheers, > >> > >> >> Daniel > >> > >> >> > >> > >> >>> > >> > >> >>> thanks, > >> > >> >>> Christian > >> > >> >>> > >> > >> >>>> > >> > >> >>>> Cheers, > >> > >> >>>> Daniel > >> > >> >>>> > >> > >> >>>> On 03/18/07 01:21, Christian Schlatter wrote: > >> > >> >>>>> Christian Schlatter wrote: > >> > >> >>>>> ... > >> > >> >>>>>> > >> > >> >>>>>> I always had 768MB shared memory configured though,

> >> still > >> > >> >>>>>> can't explain the memory allocation errors I got. Some > >> more test > >> > >> >>>>>> runs revealed that I only get these errors when using > a more > >> > >> >>>>>> production oriented config that loads more modules

> >> the one > >> > >> >>>>>> posted in my earlier email. I now try to figure out

> >> exactly > >> > >> >>>>>> causes these memory allocation errors that happen > >> reproducibly > >> > >> >>>>>> after about 220s at 400 cps. > >> > >> >>>>> > >> > >> >>>>> I think I found the cause for the memory allocation > >> errors. As > >> > >> >>>>> soon as I include an AVP write operation in the routing > >> script, I > >> > >> >>>>> get 'out of memory' messages after a certain number of > calls > >> > >> >>>>> generated with sipp. > >> > >> >>>>> > >> > >> >>>>> The routing script to reproduce this behavior looks

> >> (full > >> > >> >>>>> config available at > >> > >> >>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg): > >> > >> >>>>> > >> > >> >>>>> route{ > >> > >> >>>>> $avp(s:ct) = $ct; # commenting this line solves > >> > >> >>>>> # the memory problem > >> > >> >>>>> > >> > >> >>>>> if (!method=="REGISTER") record_route(); > >> > >> >>>>> if (loose_route()) route(1); > >> > >> >>>>> > >> > >> >>>>> if (uri==myself) rewritehost("xx.xx.xx.xx"); > >> > >> >>>>> route(1); > >> > >> >>>>> } > >> > >> >>>>> > >> > >> >>>>> route[1] { > >> > >> >>>>> if (!t_relay()) sl_reply_error(); > >> > >> >>>>> exit; > >> > >> >>>>> } > >> > >> >>>>> > >> > >> >>>>> An example log file showing the 'out of memory' > messages is > >> > >> >>>>> available at > >> http://www.unc.edu/~cschlatt/openser/openser.log . > >> > >> >>>>> > >> > >> >>>>> Some observations: > >> > >> >>>>> > >> > >> >>>>> - The 'out of memory' messages always appear after

> >> 8000 test > >> > >> >>>>> calls per worker process. One call consists of two SIP > >> > >> >>>>> transactions and six end-to-end SIP messages. An

> >> with 8 > >> > >> >>>>> children handles about 64'000 calls, whereas 4 children > only > >> > >> >>>>> handle about 32'000 calls. The sipp call rate doesn't > >> matter, only > >> > >> >>>>> number of calls. > >> > >> >>>>> > >> > >> >>>>> - The 8000 calls per worker process are independent > from the > >> > >> >>>>> amount of shared memory available. Running openser

> >> 128 or > >> > >> >>>>> -m 768 does not make a difference. > >> > >> >>>>> > >> > >> >>>>> - The more AVP writes are done in the script, the less > >> calls go > >> > >> >>>>> through. It looks like each AVP write is leaking memory > >> (unnoticed > >> > >> >>>>> by the memory statistics). > >> > >> >>>>> > >> > >> >>>>> - The fifo memory statistics do not reflect the 'out of > >> memory' > >> > >> >>>>> syslog messages. Even if openser does not route a > single SIP > >> > >> >>>>> message because of memory issues, the statistics still > >> show a lot > >> > >> >>>>> of 'free' memory. > >> > >> >>>>> > >> > >> >>>>> > >> > >> >>>>> All tests were done with openser SVN 1.2 branch on