Hi! Happy new year to all!!!
Look like I am first in this year wit hthe questions in this list :-).
I'm using stateless kamailio and RTPengnine to build some kind of the stateless cluster I found that kamailio keeps some data in the SHMEM in case of using RTPengine module even if it is not a rtpengine_manage function but offer/answer/delete
In this case if INVITE (offer) handled by 1-st kamailio in my cluster, and BYE/CANCEL handled by second kamailio in the cluster - 1-st kamailio (which has been used for offer) will hase kinda internal "memory leak" (in SHMEM it never decrased)
At the rtpengine module source I found some transation dependencies for the rtpengine_manage function but did not find for the offer/answer/delete I supposed these 3 functions just sending requests to the rtpnegine with keys and not storing anything
So my question - is it possible to use RTPengine module in stateless way to avoid internal "memory leak" because in my scenario I have big chance never receive BYE/CANCEL on the machine that started handle particular call
On 02/01/2019 07.45, Yuriy Gorlichenko wrote:
Hi! Happy new year to all!!!
Look like I am first in this year wit hthe questions in this list :-).
I'm using stateless kamailio and RTPengnine to build some kind of the stateless cluster I found that kamailio keeps some data in the SHMEM in case of using RTPengine module even if it is not a rtpengine_manage function but offer/answer/delete
In this case if INVITE (offer) handled by 1-st kamailio in my cluster, and BYE/CANCEL handled by second kamailio in the cluster - 1-st kamailio (which has been used for offer) will hase kinda internal "memory leak" (in SHMEM it never decrased)
At the rtpengine module source I found some transation dependencies for the rtpengine_manage function but did not find for the offer/answer/delete I supposed these 3 functions just sending requests to the rtpnegine with keys and not storing anything
So my question - is it possible to use RTPengine module in stateless way to avoid internal "memory leak" because in my scenario I have big chance never receive BYE/CANCEL on the machine that started handle particular call
This is probably the module-internal hash table that is used to make sure that signalling relating to the same call is always sent to the same rtpengine instance. This hash table does have a configurable timeout value (`hash_table_tout`, defaults to 1 hour), which makes it possible to still use it in a way you've described. However, from the code it's my understanding that timed out hash table entries are only deleted when they're encountered during processing, so it's not entirely deterministic that old entries are always deleted after they've timed out. The RPC command `rtpengine.get_hash_total` can be used to inspect the current size of the hash table.
Cheers
Hello,
to see the source of the leak in shared memory, the best is to generate usage summary.
First set memlog lower than debug parameter, you can do in the kamailio.cfg or via rpc:
kamcmd cfg.set_now_int core memlog 1
The above sets memlog to 1, so choose a value lower than what you have for debug.
The generate the summary via rpc:
kamcmd corex.shm_summary
Look in syslog for printed messages related to use of shared memory.
Cheers, Daniel
On Wed, Jan 2, 2019 at 2:45 PM Richard Fuchs rfuchs@sipwise.com wrote:
On 02/01/2019 07.45, Yuriy Gorlichenko wrote:
Hi! Happy new year to all!!!
Look like I am first in this year wit hthe questions in this list :-).
I'm using stateless kamailio and RTPengnine to build some kind of the stateless cluster I found that kamailio keeps some data in the SHMEM in case of using RTPengine module even if it is not a rtpengine_manage function but offer/answer/delete
In this case if INVITE (offer) handled by 1-st kamailio in my cluster, and BYE/CANCEL handled by second kamailio in the cluster - 1-st kamailio (which has been used for offer) will hase kinda internal "memory leak" (in SHMEM it never decrased)
At the rtpengine module source I found some transation dependencies for the rtpengine_manage function but did not find for the offer/answer/delete I supposed these 3 functions just sending requests to the rtpnegine with keys and not storing anything
So my question - is it possible to use RTPengine module in stateless way to avoid internal "memory leak" because in my scenario I have big chance never receive BYE/CANCEL on the machine that started handle particular call
This is probably the module-internal hash table that is used to make sure that signalling relating to the same call is always sent to the same rtpengine instance. This hash table does have a configurable timeout value (`hash_table_tout`, defaults to 1 hour), which makes it possible to still use it in a way you've described. However, from the code it's my understanding that timed out hash table entries are only deleted when they're encountered during processing, so it's not entirely deterministic that old entries are always deleted after they've timed out. The RPC command `rtpengine.get_hash_total` can be used to inspect the current size of the hash table.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel! thx for the reply too! Richard was correct about hash table (I tried a couple of experiments) So it i not a Leak in bug point of view but behavior that gives growing hash of rtpengine module
If I looking at the kamcmd rtpengine.get_hash_total with watch every second - I see it growing Also if I disabling rtpengine_<> calling -I see shmem is not growing at all
So for now question only how to reset/not use/drop hash table for the rtpengine modue
ср, 2 янв. 2019 г. в 17:32, Daniel-Constantin Mierla miconda@gmail.com:
Hello,
to see the source of the leak in shared memory, the best is to generate usage summary.
First set memlog lower than debug parameter, you can do in the kamailio.cfg or via rpc:
kamcmd cfg.set_now_int core memlog 1
The above sets memlog to 1, so choose a value lower than what you have for debug.
The generate the summary via rpc:
kamcmd corex.shm_summary
Look in syslog for printed messages related to use of shared memory.
Cheers, Daniel
On Wed, Jan 2, 2019 at 2:45 PM Richard Fuchs rfuchs@sipwise.com wrote:
On 02/01/2019 07.45, Yuriy Gorlichenko wrote:
Hi! Happy new year to all!!!
Look like I am first in this year wit hthe questions in this list :-).
I'm using stateless kamailio and RTPengnine to build some kind of the stateless cluster I found that kamailio keeps some data in the SHMEM in case of using RTPengine module even if it is not a rtpengine_manage function but offer/answer/delete
In this case if INVITE (offer) handled by 1-st kamailio in my cluster, and BYE/CANCEL handled by second kamailio in the cluster - 1-st kamailio (which has been used for offer) will hase kinda internal "memory leak" (in SHMEM it never decrased)
At the rtpengine module source I found some transation dependencies for the rtpengine_manage function but did not find for the offer/answer/delete I supposed these 3 functions just sending requests to the rtpnegine with keys and not storing anything
So my question - is it possible to use RTPengine module in stateless way to avoid internal "memory leak" because in my scenario I have big chance never receive BYE/CANCEL on the machine that started handle particular call
This is probably the module-internal hash table that is used to make sure that signalling relating to the same call is always sent to the same rtpengine instance. This hash table does have a configurable timeout value (`hash_table_tout`, defaults to 1 hour), which makes it possible to still use it in a way you've described. However, from the code it's my understanding that timed out hash table entries are only deleted when they're encountered during processing, so it's not entirely deterministic that old entries are always deleted after they've timed out. The RPC command `rtpengine.get_hash_total` can be used to inspect the current size of the hash table.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla - http://www.asipto.com http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Thx for the reply Yes Internal hash table diffenentelly stores info But even it case of putting timeout to 0 it still grows in synthetic tests. So looks like it will grows alsways because of deletes entries but creates new and so on and so on... So means it decrases "leak" but not fully
Is there some hidden function maybe to drop hast table o some ticky to do this? (we are using oru ow algorinthm that garanties to use same node in case of transaction)
ср, 2 янв. 2019 г. в 16:45, Richard Fuchs rfuchs@sipwise.com:
On 02/01/2019 07.45, Yuriy Gorlichenko wrote:
Hi! Happy new year to all!!!
Look like I am first in this year wit hthe questions in this list :-).
I'm using stateless kamailio and RTPengnine to build some kind of the stateless cluster I found that kamailio keeps some data in the SHMEM in case of using RTPengine module even if it is not a rtpengine_manage function but offer/answer/delete
In this case if INVITE (offer) handled by 1-st kamailio in my cluster, and BYE/CANCEL handled by second kamailio in the cluster - 1-st kamailio (which has been used for offer) will hase kinda internal "memory leak" (in SHMEM it never decrased)
At the rtpengine module source I found some transation dependencies for the rtpengine_manage function but did not find for the offer/answer/delete I supposed these 3 functions just sending requests to the rtpnegine with keys and not storing anything
So my question - is it possible to use RTPengine module in stateless way to avoid internal "memory leak" because in my scenario I have big chance never receive BYE/CANCEL on the machine that started handle particular call
This is probably the module-internal hash table that is used to make sure that signalling relating to the same call is always sent to the same rtpengine instance. This hash table does have a configurable timeout value (`hash_table_tout`, defaults to 1 hour), which makes it possible to still use it in a way you've described. However, from the code it's my understanding that timed out hash table entries are only deleted when they're encountered during processing, so it's not entirely deterministic that old entries are always deleted after they've timed out. The RPC command `rtpengine.get_hash_total` can be used to inspect the current size of the hash table.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 02/01/2019 09.32, Yuriy Gorlichenko wrote:
Thx for the reply Yes Internal hash table diffenentelly stores info But even it case of putting timeout to 0 it still grows in synthetic tests. So looks like it will grows alsways because of deletes entries but creates new and so on and so on... So means it decrases "leak" but not fully
Is there some hidden function maybe to drop hast table o some ticky to do this? (we are using oru ow algorinthm that garanties to use same node in case of transaction)
You should be able to set the timeout to zero and the size of the hash table to one (variable `hash_table_size`). That should make the code purge out all entries on every operation, keeping the number of entries minimal.
Cheers
Yep that works with size of 1 it stores minimal data and not grows more than a cuple of bytes Thx a lot
P.S. Anyway this is a tricky (hack) and suppose in the future will be cool to add function to disable hash at all. I understand that this is not ususal case but anyway it makes sence to be
ср, 2 янв. 2019 г. в 18:02, Richard Fuchs rfuchs@sipwise.com:
On 02/01/2019 09.32, Yuriy Gorlichenko wrote:
Thx for the reply Yes Internal hash table diffenentelly stores info But even it case of putting timeout to 0 it still grows in synthetic tests. So looks like it will grows alsways because of deletes entries but creates new and so on and so on... So means it decrases "leak" but not fully
Is there some hidden function maybe to drop hast table o some ticky to do this? (we are using oru ow algorinthm that garanties to use same node in case of transaction)
You should be able to set the timeout to zero and the size of the hash table to one (variable `hash_table_size`). That should make the code purge out all entries on every operation, keeping the number of entries minimal.
Cheers
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Am Mittwoch, 2. Januar 2019, 16:01:30 CET schrieb Richard Fuchs:
Internal hash table diffenentelly stores info But even it case of putting timeout to 0 it still grows in synthetic tests. So looks like it will grows alsways because of deletes entries but creates new and so on and so on... So means it decrases "leak" but not fully
Is there some hidden function maybe to drop hast table o some ticky to do this? (we are using oru ow algorinthm that garanties to use same node in case of transaction)
You should be able to set the timeout to zero and the size of the hash table to one (variable `hash_table_size`). That should make the code purge out all entries on every operation, keeping the number of entries minimal.
Hi Richard,
would this information something that is useful to be added to the module README?
Best regards,
Henning