Re: [SR-Users] reload to memory is much slower and has problms after upgrade to 3.3.2 and using mem_join=1

Hi, yes, the reload time is constant - 56 sec. i tested for 100 reloads in an hour and it was OK. i configured kamailio to start with 4Gb and after reload the shmem (real_used) take around 30% of it. but, after 20 reloads it grows in 1%. so, after the 100 reloads the real_used take around 34%-35% of shmem. i made the choise to compile again with f_malloc and not use mem_join. the reloads are faster, it uses less real_size (12% and not 30%) and the increasment of it is around 1% for 5 reloads (i do 5 reloads a week). i will keep track on it and update. thanks a lot. Btw, what do you think? would you use f_malloc with no mem_join or q_malloc with join? BR, Uri On Wed, Nov 21, 2012 at 8:55 PM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com > Hello, > > > On 11/21/12 1:33 PM, Uri Shacked wrote: > > Hi, > > I recompiled with MEMDBG=1 and installed. > here are the results for reloading 5 million rows with MTREE: > > mem_join=1 -->takes 56 seconds and the real_used_size of shmem is around > 1.2Gb. > mem_join=0 --> takes 10 seconds and the real_used_size of shmem is around > 2Gb > does it seems normal? > 56 seconds is a lot of time...... > > the join is done for the free operation, meaning that most of the time is > spent when freeing the old tree from memory. The new values will be used > after loading the database records, then the old tree is destroyed (this > involves the join operation). Also, the sip routing is not affected, > loading the new records and destroying old memory tree is done in the > MI/RPC process. > > In other words, while the MI/RPC process takes care of loading new data > and destroying the old one, the SIP routing is not affected at all. > > Even when the reload command is executed, the old tree is used until all > the records are loaded in a new tree. At that moment, the pointer to the > active tree is changed from the old tree to the new tree (a very simple > sequence of assignments, very fast). Routing will use the new tree and the > Mi/RPC process starts destroying the old tree. > > > > by the way, when the f_malloc was used, the size of the real_used shmem > was twice smaller. > > > The overhead when storing small values is significant for q_malloc, each > fragment keeping references (pointers) to file name and line where it was > allocated and freed. In addition it keeps information to get to previews > and next fragment, resulting in faster join. > > It is some space to improve, in order to make less overhead (like a > compile time option not to keep info about file and line of malloc/free). I > will think about what can be done for the next release. > > Btw, is the reload time constant now? Even if you run couple of times? > > What are the values for 'kamctl mi get_statistics shmem:'? > > Cheers, > Daniel > > > Thanks. > On Tue, Nov 20, 2012 at 9:45 PM, Daniel-Constantin Mierla < > miconda(a)gmail.com> >> Hello, >> >> >> On 11/20/12 7:34 PM, Uri Shacked wrote: >> >> Hi, >> >> can you be a litle more specific of the steps of the install and where do >> i make the changes? >> >> >> in the source tree, edit the file Makefile.defs and set: >> >> MEMDBG=1 >> >> then run: >> >> make all >> make install >> >> >> some words of what is the diff between f_malloc and q_malloc will be >> great :-). >> >> >> q_malloc is more debugging purposes, keeping more information for each >> chunk, therefore the overhead is a bit higher than with f_malloc, but >> because keeps more details, it is faster to find the fragments that can be >> joined. >> >> Cheers, >> Daniel >> >> >> >> thanks, >> Uri >> >> On Tue, Nov 20, 2012 at 6:26 PM, Daniel-Constantin Mierla < >> miconda(a)gmail.com>> >>> Hello, >>> >>> ok, I will look over it. At this moment the f_malloc (which is enabled >>> for 3.3) has a pretty inefficient mem join implementation, can you try with >>> q_malloc? Edit Makefile.defs and set: >>> >>> MEMDBG=1 >>> >>> Then compile and install. >>> >>> The join operation should be faster, let's see if you get blocking >>> issues with this one. >>> >>> Cheers, >>> Daniel >>> >>> >>> On 11/20/12 2:57 PM, Uri Shacked wrote: >>> >>> Daniel hi, >>> >>> I attached 2 txt files. >>> One with mem_join=1, the other with mem_join=0, and the info you asked >>> for. >>> Let me know if it is OK. >>> >>> Thanks, >>> Uri >>> >>> On Mon, Nov 19, 2012 at 10:50 AM, Daniel-Constantin Mierla < >>> miconda(a)gmail.com>>> >>>> Hello, >>>> >>>> if you set memjoin to 0, do you see any difference? >>>> >>>> Can you try again (with memjoin 1 as well as 0) and send the output of: >>>> >>>> kamctl mi get_statistics shmem: >>>> >>>> before executing the reload commands? >>>> >>>> When it gets to 100%, can you see which process is using the cpu and >>>> attach to it with: >>>> >>>> gdb /path/to/kamailio PID >>>> >>>> then do: >>>> >>>> bt full >>>> >>>> and send output here? >>>> >>>> Cheers, >>>> Daniel >>>> >>>> >>>> On 11/18/12 4:09 PM, Uri Shacked wrote: >>>> >>>> After some testing I notice the following: >>>> First reload of 5 million records after kamailio started took about 9 >>>> sec. >>>> Second reload (4 minutes after the first one) took 60 sec. >>>> The third one (again about 4 minutes after the secind) got kamailio to >>>> use 100% cpu and after 13 minutes! i killed it..... >>>> >>>> I can understand that the memory manger works harder, still, any ideas >>>> on how to use mem_join and keep on reloading data. >>>> (in real life our data loads 5 million records once a day when almost >>>> no traffic. still after a few days it stops...) >>>> >>>> Thanks, >>>> Uri >>>> >>>> >>>> >>>> On Sun, Nov 18, 2012 at 11:52 AM, Uri Shacked &lt;ushacked(a)gmail.com&gt;wrote;wrote: >>>> >>>>> Hi, >>>>> >>>>> I am using MTREE and DIALPLAN modules to load lots of info to >>>>> kamailio. (6 million rows). >>>>> >>>>> When kamailio was running with 3.2.1 (no mem_join=1 option), the used >>>>> size was increasing but the process of loading the data was fast eanough. >>>>> >>>>> I upgraded to 3.3.2 and set mem_join=1. Now the loading process take >>>>> about 10 time longer and sometimes stops kamailio from responding to >>>>> traffic. >>>>> >>>>> Any ideas? >>>>> >>>>> Thanks, >>>>> >>>>> Uri >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>>> >>>> >>>> -- >>>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>>> >>>> >>> >>> -- >>> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> >>> >> >> -- >> Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >> >> > > -- > Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > >