Re: [Users] memory issues

Hello, the supposed fragmentation turned out to be a mem leak in pkg. Please take the latest SVN version and try again to see if you got same results.

Thanks, Daniel On 03/19/07 18:52, Christian Schlatter wrote: > ... >>> The memory statistics indeed show a high number of memory fragments: >>> >>> before 'out of memory': >>> >>> shmem:total_size = 536870912 >>> shmem:used_size = 59607040 >>> shmem:real_used_size = 60106488 >>> shmem:max_used_size = 68261536 >>> shmem:free_size = 476764424 >>> shmem:fragments = 9897 >>> >>> after 'out of memory' (about 8000 calls per process): >>> >>> shmem:total_size = 536870912 >>> shmem:used_size = 4171160 >>> shmem:real_used_size = 4670744 >>> shmem:max_used_size = 68261536 >>> shmem:free_size = 532200168 >>> shmem:fragments = 57902 >>> >>>> >>>> You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS >>>> variable of Makefile.defs) and test again. Free fragments should be >>>> merged and fragmentation should not occur -- processing will be >>>> slower. We will try for next release to provide a better solution >>>> for that. >>> >>> Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how >>> big of a problem this fragmentation issue is. >> What is the number of fragments with QM_JOIN_FREE after flooding? > > The numbers included above are with QM_JOIN_FREE enabled. > >>> Do you think it would make sense to restart our production openser >>> instances from time to time just to make sure they're not running >>> into this memory fragmentation limits? >> The issue will occur only when the call rate reaches the limits of >> the proxy's memory. Otherwise the chunks are reused. Transactions and >> avps are rounded up to be sure there will be minimized the number of >> different sizes for memory chunks. It wasn't reported too often, >> maybe that's why no big attention was paid to it. This memory system >> is in place since the beginning of ser. Alternative is to use sysv >> shared memory, but is much slower, along with libc private memory >> manager. > > I've done some more testing and the same out-of-memory stuff happens > when I run sipp with 10 calls per second only. I tested with > 'children=1' and I only could get through about 8200 calls (again > those 8000 calls / process). And this is with QM_JOIN_FREE enabled. > > Memory statistics: > > before: > shmem:total_size = 536870912 > shmem:used_size = 2311976 > shmem:real_used_size = 2335720 > shmem:max_used_size = 2465816 > shmem:free_size = 534535192 > shmem:fragments = 183 > > after: > shmem:total_size = 536870912 > shmem:used_size = 1853472 > shmem:real_used_size = 1877224 > shmem:max_used_size = 2465816 > shmem:free_size = 534993688 > shmem:fragments = 547 > > So I'm not sure if this is really a fragmentation issue. 10 cps surely > doesn't reach the proxy's memory. > > Thoughts? > > Christian > > > >> Cheers, >> Daniel >> >>> >>> thanks, >>> Christian >>> >>>> >>>> Cheers, >>>> Daniel >>>> >>>> On 03/18/07 01:21, Christian Schlatter wrote: >>>>> Christian Schlatter wrote: >>>>> ... >>>>>> >>>>>> I always had 768MB shared memory configured though, so I still >>>>>> can't explain the memory allocation errors I got. Some more test >>>>>> runs revealed that I only get these errors when using a more >>>>>> production oriented config that loads more modules than the one >>>>>> posted in my earlier email. I now try to figure out what exactly >>>>>> causes these memory allocation errors that happen reproducibly >>>>>> after about 220s at 400 cps. >>>>> >>>>> I think I found the cause for the memory allocation errors. As >>>>> soon as I include an AVP write operation in the routing script, I >>>>> get 'out of memory' messages after a certain number of calls >>>>> generated with sipp. >>>>> >>>>> The routing script to reproduce this behavior looks like (full >>>>> config available at >>>>> http://www.unc.edu/~cschlatt/openser/openser.cfg): >>>>> >>>>> route{ >>>>> $avp(s:ct) = $ct; # commenting this line solves >>>>> # the memory problem >>>>> >>>>> if (!method=="REGISTER") record_route(); >>>>> if (loose_route()) route(1); >>>>> >>>>> if (uri==myself) rewritehost("xx.xx.xx.xx"); >>>>> route(1); >>>>> } >>>>> >>>>> route[1] { >>>>> if (!t_relay()) sl_reply_error(); >>>>> exit; >>>>> } >>>>> >>>>> An example log file showing the 'out of memory' messages is >>>>> available at http://www.unc.edu/~cschlatt/openser/openser.log . >>>>> >>>>> Some observations: >>>>> >>>>> - The 'out of memory' messages always appear after about 8000 test >>>>> calls per worker process. One call consists of two SIP >>>>> transactions and six end-to-end SIP messages. An openser with 8 >>>>> children handles about 64'000 calls, whereas 4 children only >>>>> handle about 32'000 calls. The sipp call rate doesn't matter, only >>>>> number of calls. >>>>> >>>>> - The 8000 calls per worker process are independent from the >>>>> amount of shared memory available. Running openser with -m 128 or >>>>> -m 768 does not make a difference. >>>>> >>>>> - The more AVP writes are done in the script, the less calls go >>>>> through. It looks like each AVP write is leaking memory (unnoticed >>>>> by the memory statistics). >>>>> >>>>> - The fifo memory statistics do not reflect the 'out of memory' >>>>> syslog messages. Even if openser does not route a single SIP >>>>> message because of memory issues, the statistics still show a lot >>>>> of 'free' memory. >>>>> >>>>> >>>>> All tests were done with openser SVN 1.2 branch on Ubuntu dapper >>>>> x86. I think the same is true for 1.1 version but I haven't tested >>>>> that yet. >>>>> >>>>> >>>>> Christian >>>>> >>> >>> > > > _______________________________________________ > Users mailing list > Users(a)openser.org > http://openser.org/cgi-bin/mailman/listinfo/users >