Re: [SR-Users] Slowly increasing PKG memory usage

On 5 Aug 2019, at 11:30 pm, Andrew White &lt;andrew(a)uconnected.com.au <mailto:andrew@uconnected.com.au>> wrote: Hey Henning, Unfortunately I’ve had some troubles with app_ruby in dev-7. Interestingly, dev-7 shows as “dirty” too: version: kamailio 5.3.0-dev7 (x86_64/linux) bd477f-dirty Here’s the ansible playbook I wrote to pull the specific version, there’s no fancy patching or anything going on: - name: Download kamailio git: repo: https://github.com/kamailio/kamailio.git version: "{{ kamailio_version }}" dest: /usr/src/kamailio - name: Roll back to specific commit command: "git reset --hard {{ kamailio_commit }}" args: chdir: /usr/src/kamailio when: kamailio_commit is defined - name: Build kamailio from source command: "{{ item }} chdir=/usr/src/kamailio" with_items: - make include_modules="app_ruby db_redis" cfg - make all - make install args: creates: /usr/local/sbin/kamailio Either way, I’ve removed the topoh module from my production (dev-4) machine, and re-taken memory information. I’ll give it about 12 hours and see where memory is sitting. I figure this should isolate if it’s a topoh issue or not. Thanks! Andrew > On 5 Aug 2019, at 10:46 am, Andrew White &lt;andrew(a)uconnected.com.au > <mailto:andrew@uconnected.com.au>> wrote: > > Hi Henning, > > That is interesting, I don’t know what the dirty version is. My > deploy script I use for Kamailio > pulls 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6 from GitHub, as > this had a fix in place Daniel did for app_ruby a while back. I > don’t recall ever putting on any custom patches or modules. I > compile from source on build from that git commit using: > > makeinclude_modules="app_rubydb_redis"cfg > > I’ll deploy dev7 tonight and report back! > > Thanks, > > Andrew > >> On 4 Aug 2019, at 8:28 pm, Henning Westerholt &lt;hw(a)skalatan.de >> <mailto:hw@skalatan.de>> wrote: >> >> Hi Andrew, >> >> dev4 version - I see. I quickly looked into the topos module >> commits, but there have been not that much changes there. But can >> you try nevertheless to update to the latest git master (e.g. >> dev7) version on your test setup? >> >> What about the "dirty" version flag, do you have some own patches >> or modules that you use? >> >> Cheers, >> >> Henning >> >> Am 04.08.19 um 01:42 schrieb Andrew White: >>> Hey Henning, >>> >>> It’s currently version: kamailio 5.3.0-dev4 (x86_64/linux) >>> 6d43ea-dirty.  >>> >>> I don’t think there’s any extra messages. The actual dispatching >>> is simple, here’s the code: >>> >>> 1. if destination then >>> 2. KSR::PV.sets("$du", destination['sip_uri']) >>> 3. pbx = destination['display_name'] >>> 4. KSR.info <http://ksr.info/>("Chose #{pbx} as inbound PBX, as >>> it's received >>> #{destination['count'].to_i}/#{destination['total_count'].to_i} >>> (#{destination['calculated_percentage'].to_i}%) of calls in >>> this ruleset, where its max is >>> #{destination['percentage'].to_i}%") >>> 5. end >>> >>> >>> The code before that just pulls data from redis and returns a >>> ruby hash with where we should send the call, so no kamailio >>> variables set at all. >>> >>> Thanks! >>> >>> Andrew >>> >>>> On 3 Aug 2019, at 8:03 pm, Henning Westerholt &lt;hw(a)skalatan.de >>>> <mailto:hw@skalatan.de>> wrote: >>>> >>>> Hi Andrew, >>>> >>>> interesting insight from the logs indeed. According to your >>>> changes this looks indeed like related to the topoh module or >>>> custom dispatcher code. Which version of Kamailio do you use? >>>> >>>> Are you sending out some messages /replies as well from your >>>> custom dispatcher app_ruby logic? >>>> >>>> Cheers, >>>> >>>> Henning >>>> >>>> Am 03.08.19 um 09:32 schrieb Andrew White: >>>>> Hi all, >>>>> >>>>> I’ve done two outputs about 8 hours/3200 calls in between: >>>>> >>>>> First one (about 5 hours since last restart): >>>>> >>>>> https://paste.ubuntu.com/p/Y6Wvd9hsV2/ >>>>> >>>>> Second one (about 8 hours after the first, 3200 odd cals later): >>>>> >>>>> https://paste.ubuntu.com/p/85KYg7gD2z/ >>>>> >>>>> It looks like roughly 7MB of extra memory is being used on >>>>> that process, which is 1/4 active workers (I can see all of >>>>> them have grown roughly the same amount in that time). Looking >>>>> at the memory status, there appear to be about 14,000 more >>>>> lines. A cursory glance shows about 6k of those lines >>>>> with msg_translator.c (compared to 1k in the first). >>>>> >>>>> I wonder if maybe this could be related to my use of the topoh >>>>> module, given every message is now being touched by it to mask >>>>> the origin? >>>>> >>>>> Thanks! >>>>> >>>>> Andrew >>>>> >>>>>> On 3 Aug 2019, at 10:07 am, Andrew White >>>>>> &lt;andrew(a)uconnected.com.au <mailto:andrew@uconnected.com.au>> >>>>>> wrote: >>>>>> >>>>>> Hi Daniel/Henning, >>>>>> >>>>>> Thanks so much for your quick responses, they’re very much >>>>>> appreciated! >>>>>> >>>>>> Daniel: I don’t believe so. I’ve just checked over the code, >>>>>> and I only have 9 instances of PV.sets throughout the entire >>>>>> script. Additionally, this issue appears to have only cropped >>>>>> up in the last week or two (I didn’t monitor PKG memory at >>>>>> the time, but we didn’t have any memory related crashes >>>>>> before that point), which leads me to believe it’s related to >>>>>> my config code, the adding of the topoh module or the removal >>>>>> of the dispatcher module.  >>>>>> >>>>>> Within that config code, there’s only one additional Kamailio >>>>>> variable set, which is part of my replacement/custom >>>>>> dispatcher (app_ruby): >>>>>> >>>>>> KSR::PV.sets("$du", destination['sip_uri’]) >>>>>> For now, I’ve taken the output of a mem dump of one of the >>>>>> production workers, ps output, etc. I’ll leave that few a few >>>>>> hours, monitor to ensure there is a measurable memory >>>>>> increase, and send through the logs if that’s ok? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Andrew >>>>>> >>>>>>> On 2 Aug 2019, at 11:44 pm, Daniel-Constantin Mierla >>>>>>> &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> are you defining a lot of kamailio variables in your ruby >>>>>>> script? In other words, are you using variables with >>>>>>> different name, like $var(xyz) or $sht(a=>xyz), where xyz is >>>>>>> passed from/computed in ruby script and is changing >>>>>>> depending of the sip message? >>>>>>> >>>>>>> Cheers, >>>>>>> Daniel >>>>>>> >>>>>>> On 01.08.19 15:34, Andrew White wrote: >>>>>>>> Thanks Daniel, you’re fantastic! >>>>>>>> >>>>>>>> I have 4 children/workers configured with -m 128 -M 32. The >>>>>>>> machine in question has 512MB of memory, 1 core and 1GB >>>>>>>> swap on an SSD. >>>>>>>> >>>>>>>> I restarted Kamailio with memlog=1 and I’ve been sending >>>>>>>> batches of 30 calls in. I’ve noticed 4 of the 13 Kamailio >>>>>>>> processes going up in memory after each batch, which I >>>>>>>> suspect to be the primary children/workers. Immediately >>>>>>>> post restart: >>>>>>>> >>>>>>>> root     28531  0.7  5.5 329368 27196 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28532  0.6  4.9 329368 24528 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28533  0.6  5.5 329368 27244 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28534  0.7  5.4 329368 26788 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> >>>>>>>> After about 90 calls: >>>>>>>> >>>>>>>> root     28531  0.0  6.7 330688 32948 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28532  0.0  6.5 330560 32264 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28533  0.0  6.5 330556 32272 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28534  0.0  6.6 330564 32592 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> >>>>>>>> None of the other 9 Kamailio processes are increasing at all. >>>>>>>> >>>>>>>> I ran corex.pkg_summary against one of them and got the >>>>>>>> following dump: >>>>>>>> >>>>>>>> https://paste.ubuntu.com/p/SqTF3K5knK/ >>>>>>>> >>>>>>>> I can see a lot of allocation to pvapi.c, does this >>>>>>>> indicate maybe I’m setting PVs that need to be unset? >>>>>>>> >>>>>>>> Here’s another after another 60 calls: >>>>>>>> >>>>>>>> https://paste.ubuntu.com/p/9WQXqZtfT2/ >>>>>>>> >>>>>>>> root     28531  0.0  6.9 330820 33928 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28532  0.0  6.7 330692 33352 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28533  0.0  6.7 330688 33280 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> root     28534  0.0  6.7 330696 33192 ?        Sl   22:48   >>>>>>>> 0:00 /usr/local/sbin/kamailio -DD -P >>>>>>>> /var/run/kamailio/kamailio.pid -f >>>>>>>> /etc/kamailio/kamailio.cfg -m 128 -M 32 >>>>>>>> >>>>>>>> The only changes I’ve made on this config over the last >>>>>>>> couple of weeks (since I saw this issue) is removing the >>>>>>>> dispatcher module and adding in a small function in >>>>>>>> app_ruby (which I already use) to query redis (which I also >>>>>>>> already use from app_ruby and make a heap of queries per >>>>>>>> call) for some values and write $du manually. I also added >>>>>>>> in the topoh module. >>>>>>>> >>>>>>>> It also makes a lot of sense to me to monitor the >>>>>>>> individual processes rather than the aggregate. Is there a >>>>>>>> way to identify simply from bash what processes are workers >>>>>>>> programmatically? I’d like to monitor just those >>>>>>>> individually in my monitoring. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> >>>>>>>>> On 1 Aug 2019, at 8:24 pm, Daniel-Constantin Mierla >>>>>>>>> &lt;miconda(a)gmail.com <mailto:miconda@gmail.com>> wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> if it is pkg, then you have to see which process is >>>>>>>>> increasing the use of memory, because it is private >>>>>>>>> memory, specific for each process. The sum is an >>>>>>>>> indicator, but the debugging has to be done for a specific >>>>>>>>> process/pid. >>>>>>>>> >>>>>>>>> Once you indentify a process that is leaking pkg, execute >>>>>>>>> the rpc command: >>>>>>>>> >>>>>>>>>   - >>>>>>>>> https://www.kamailio.org/docs/modules/devel/modules/corex.html#corex.rpc.pk… >>>>>>>>> >>>>>>>>> When that process is doing some runtime work (e.g., >>>>>>>>> handling of a sip message), the syslog will get a summary >>>>>>>>> with used pkg chunks. Send those log messages here for >>>>>>>>> analysis. You have to set memlog core parameter to a value >>>>>>>>> smaller than debug. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Daniel >>>>>>>>> >>>>>>>>> >>>>>>>>> On 01.08.19 03:43, Andrew White wrote: >>>>>>>>>> Hi all, >>>>>>>>>> >>>>>>>>>> I had a Kamailio crash the other day, and some debugging >>>>>>>>>> showed I ran out of PKG memory. >>>>>>>>>> >>>>>>>>>> Since then I’ve run a simple bash script to compile the >>>>>>>>>> amount of memory used by all child processes, >>>>>>>>>> effective /usr/local/sbin/kamcmd pkg.stats | grep >>>>>>>>>> real_used summed together. I’ve graphed out the data, and >>>>>>>>>> there’s a clear growth of PKG memory going on, mostly >>>>>>>>>> increasing during our busier daytime hours. >>>>>>>>>> >>>>>>>>>> https://i.imgur.com/UTzx2k1.png >>>>>>>>>> >>>>>>>>>> Based on this, I suspect either a module loaded or >>>>>>>>>> something within my app_ruby conf is leaking memory. >>>>>>>>>> >>>>>>>>>> I’ve been reading >>>>>>>>>> through https://www.kamailio.org/wiki/tutorials/troubleshooting/memory, >>>>>>>>>> but I’m a bit nervous, as I’m not really a C/deep memory >>>>>>>>>> type of guy. I can see a GDB script I can attach to >>>>>>>>>> Kamailio, but is that going to use significant resources >>>>>>>>>> to run or impact the running process? Is there a >>>>>>>>>> newer/better/alternative way to do this, and to help me >>>>>>>>>> break this down? >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Andrew >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Kamailio (SER) - Users Mailing List >>>>>>>>>> sr-users(a)lists.kamailio.org >>>>>>>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>>>>>>> -- >>>>>>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>>>>>> >>>>>>> -- >>>>>>> Daniel-Constantin Mierla -- www.asipto.com >>>>>>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Kamailio (SER) - Users Mailing List >>>>> sr-users(a)lists.kamailio.org >>>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> -- >>>> Henning Westerholt - https://skalatan.de/blog/ >>>> Kamailio services - https://skalatan.de/services >>> >> -- >> Henning Westerholt - https://skalatan.de/blog/ >> Kamailio services - https://skalatan.de/services >