<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment). -->
### Description
I've noticed PKG memory increasing since enabling the topoh module last week in a significant manner (~7MB per worker over 3000 calls split between 4 workers). Removing the topoh module removed the memory usage.
Here is a pkg dump of one of the process post 3000 calls:
https://paste.ubuntu.com/p/85KYg7gD2z/
### Additional Information
```kamailio 5.3.0-dev4 (x86_64/linux) 6d43ea-dirty. ```
I'm unsure why it's tagged dirty, as I simply cloned master and reset to 6d43eacd6d5b78fe857267e70b7fa85519a7d5b6.
I do plan to test this on dev7, however an unrelated issue is preventing me from testing this with my app_ruby based code.
Update: app_ruby issues have been fixed in latest master. I'll build a new machine on latest master tonight and split some live calls over there with topoh enabled to see if we can replicate the issue in master. Will report back in ~ 24 hours
Tested with kamailio 5.3.0-dev7 (x86_64/linux) 4c537a (4c537a618949eb48ffed9297f3abc8cdc879b70d).
Initial (after a couple of hundred test calls)l: https://paste.ubuntu.com/p/3PNrTRf8Kw/
3881 calls later: https://paste.ubuntu.com/p/Pg6dsQ94qt/
It looks like this is still happening in the current topoh.
This may have been a result of having two modules using the same callback to update the content of the outbound buffer, in your case the topoh (hiding header attributes) and dialog (updating the cseq).
I pushed a bunch of patches to core and modules using this callback. They are in master branch for now. Can you test? If all ok I can backport.
Ah, this might be the cause that I did not managed to reproduce it with only one module..
Hi @miconda. Thanks so much for this!
I tested with **kamailio 5.3.0-dev7 (x86_64/linux) 4c537a**
Unfortunately after ~ 20 hours/10k calls:
https://paste.ubuntu.com/p/vBmpQXKztj/
I somehow lost the dump I took initially, however please find attached graph using the flawed methodology I used previously of summing together all children memory usage. Even though it's not a great measurement, it does prove the continued growth:
<img width="1415" alt="Screen Shot 2019-08-14 at 9 17 04 am" src="https://user-images.githubusercontent.com/287821/62983951-995cf280-be74-11e9-8b00-3d3d88520356.png">
The version you used is before the fix -- the commit id in your version points to:
``` commit 4c537a618949eb48ffed9297f3abc8cdc879b70d Author: Daniel-Constantin Mierla miconda@gmail.com Date: Tue Aug 6 21:37:56 2019 +0200
misc_radius: increase MAX_EXTRA from 4 to 8 ```
To have the version with the fix, it has to be after the commit:
``` commit c79dfbeab0bfefaa4dd5cefc41cba3ba157da0ce Author: Daniel-Constantin Mierla miconda@gmail.com Date: Mon Aug 12 11:18:39 2019 +0200
corex: free old outbound buffer inside SREV_NET_DATA_OUT callback ```
Whoops! Thanks Daniel, looks like I forgot to make install on my UAT server.
I'll re-test and update ASAP.
Hi @miconda . I can confirm that this patch works. I put through ~ 4000 calls and roughly a 100kb increase in memory across all children (25kb each), and only as the first couple of calls came in. After that, 0 growth.
I believe this is fixed, thanks so much! I'll leave this open as you said you wish to backport.
Patches are backported to branch 5.2.
Closed #2027.
-
Daniel-Constantin Mierla -
Henning Westerholt
-
Whitey