I have some questions regarding the shared memory. I'm working with the version 1.5.2 and its environment is quite simple. It does'nt storage any informations in the memory, like dialplan, contacts of subscribers, carrierroute, LCR and etc. The only purpose of this server is being a redirect server. Some calls have several possible destinations, reaching easily 12 contacts. The config.h's constant MAX_BRANCHES is set in 12, the default number. Ok. Like I said, almost the requests have more than 12 possible contacts. When it happens, Kamailio logs these messages:
/usr/local/kamailio-1.5.2/sbin/kamailio[26188]: ERROR:core:append_branch: max nr of branches exceeded /usr/local/kamailio-1.5.2/sbin/kamailio[26188]: ERROR:pv:pv_set_branch: append_branch action failed
Ok. It is not a problem. It is an error but it does not stop the system. The SIP messages still flowing normally. The server works for several days, 24/7 without functional errors, receiving five hundred thousand requests per day . But, usually at the weekends and the clock tells 2:30 am, the company's cell phone rings bringing a message: 'the system is down'. Kamailio shows:
/usr/local/kamailio-1.5.2/sbin/kamailio[25492]: ERROR:core:add_avp: no more shm mem /usr/local/kamailio-1.5.2/sbin/kamailio[25492]: ERROR:exec:exec_avp: unable to add avp
Kamailio stills running but it is not possible to route any request. All the new incoming calls generates the same error: 'no more shm mem'. I changed the size of shared memory, running Kamailio with '-m 64', but the server stopped again after some days. I see the options:
- increase MAX_BRANCHES - increase again 'm' parameter (but it will just add more some days in the running time) - reduce the maximum number of contacts to 11 of the script that returns them to Kamailio
So, my doubts are:
- is it possible to have a memory leak when the message "max nr of branches exceeded" is showed? - changing '-m parameter' to a higher value, like 256, could affect the performance?
tks friends bruno machado
____________________________________________________________________________________ Veja quais são os assuntos do momento no Yahoo! +Buscados http://br.maisbuscados.yahoo.com
On Monday 22 March 2010, Bruno Machado wrote:
I have some questions regarding the shared memory. I'm working with the version 1.5.2 and its environment is quite simple. It does'nt storage any informations in the memory, like dialplan, contacts of subscribers, carrierroute, LCR and etc. The only purpose of this server is being a redirect server. Some calls have several possible destinations, reaching easily 12 contacts. The config.h's constant MAX_BRANCHES is set in 12, the default number. Ok. Like I said, almost the requests have more than 12 possible contacts. When it happens, Kamailio logs these messages:
/usr/local/kamailio-1.5.2/sbin/kamailio[26188]: ERROR:core:append_branch: max nr of branches exceeded /usr/local/kamailio-1.5.2/sbin/kamailio[26188]: ERROR:pv:pv_set_branch: append_branch action failed [..] /usr/local/kamailio-1.5.2/sbin/kamailio[25492]: ERROR:core:add_avp: no more shm mem /usr/local/kamailio-1.5.2/sbin/kamailio[25492]: ERROR:exec:exec_avp: unable to add avp
Kamailio stills running but it is not possible to route any request. All the new incoming calls generates the same error: 'no more shm mem'. I changed the size of shared memory, running Kamailio with '-m 64', but the server stopped again after some days. I see the options:
- increase MAX_BRANCHES
- increase again 'm' parameter (but it will just add more some days in the
running time) - reduce the maximum number of contacts to 11 of the script that returns them to Kamailio
So, my doubts are:
- is it possible to have a memory leak when the message "max nr of branches
exceeded" is showed? - changing '-m parameter' to a higher value, like 256, could affect the performance?
Hi Bruno,
this looks indeed like an memory leak in the code path that is executed during the error. Also possible is that there is (really small) mem leak in another code path, that gets executed a lot.
Another (unlikely) option could be that this problem is caused from shared memory fragmentation, as you noticed that the server gets some traffic. How often this error happens? Do you can clearly correlate this to the error logs you notice?
Anyway, you should increase the shared memory setting to 512M or higher, depending on the system memory. It has no negative performance impact.
Henning
Hi Henning
Thanks for your answer. This memory error happened in a interval of almost 10 days. Considering the high traffic that the server handles, we can say that it is a lot of time. Following your hint, I set the shared memory to work with 512 megabytes. If we consider that the usage of memory is a linear function of the number of calls, the server will work ininterruptly for more than 70 days.
I have been looking for a way to monitor how much of shared memory was already used. But I didn't find anything, except the troubleshooting section on the Devel Docs. I'm not using Kamailio v3 yet, but I saw that it has a memory manager. I think that these features (Doug Lea and LL) are going to be used mainly for developers. The third one, 'print memory status summary in debug mode', is more oriented to users like me. It is what I think.
Despite anything, it is already time to migrate to version 3.
tks bruno machado
____________________________________________________________________________________ Veja quais são os assuntos do momento no Yahoo! +Buscados http://br.maisbuscados.yahoo.com
On Tuesday 23 March 2010, Bruno Machado wrote:
Thanks for your answer. This memory error happened in a interval of almost 10 days. Considering the high traffic that the server handles, we can say that it is a lot of time.
Hello Bruno,
10 days is not that long, this looks more like a memory leak.
Following your hint, I set the shared memory to work with 512 megabytes. If we consider that the usage of memory is a linear function of the number of calls, the server will work ininterruptly for more than 70 days.
I have been looking for a way to monitor how much of shared memory was already used. But I didn't find anything, except the troubleshooting section on the Devel Docs. I'm not using Kamailio v3 yet, but I saw that it has a memory manager. I think that these features (Doug Lea and LL) are going to be used mainly for developers. The third one, 'print memory status summary in debug mode', is more oriented to users like me. It is what I think.
Kamailio has (more or less) the same internal memory manager. You could use the statistics API (kamctl fifo get_statistics all). This should give you plenty of statistics, including the shared memory usage.
Regards,
Henning
-
Bruno Machado -
Henning Westerholt