8 Jun
2012
8 Jun
'12
3:09 p.m.
Hi,
We have 2 Kamailio 3.0.3 servers that has been running with
carrierroute for about 2 years without any problems. They have 128 MB
shared memory and
modparam("carrierroute", "config_source", "db")
modparam("carrierroute", "db_url", "<DBURL>")
modparam("carrierroute", "fetch_rows", 500)
The carrierroute table is about 91K lines and have been growing slowly.
Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
allocate shared memory from available pool after a few "kamctl cr
reload".
I increased the shared memory to 256 MB but with the same result. I
have now increased it to 512 MB and it seems to work better now.
I have noticed this. After a restart the shmem counters is like this:
shmem:total_size = 536870912
shmem:used_size = 28486752
shmem:real_used_size = 40147128
shmem:max_used_size = 41135424
shmem:free_size = 496723784
shmem:fragments = 555
And after the first "kamctl cr reload" it is like this:
shmem:total_size = 536870912
shmem:used_size = 28619016
shmem:real_used_size = 51842768
shmem:max_used_size = 76993616
shmem:free_size = 485028144
shmem:fragments = 722063
Notice the increase in fragments. Sequentials "kamctl cr reload" does
not change the fragments allot.
Any ideas?
--
Morten Isaksen
8 Jun
8 Jun
3:53 p.m.
Hello,
On 6/8/12 2:09 PM, Morten Isaksen wrote:
Hi,
We have 2 Kamailio 3.0.3 servers that has been running with
carrierroute for about 2 years without any problems. They have 128 MB
shared memory and
modparam("carrierroute", "config_source", "db")
modparam("carrierroute", "db_url", "<DBURL>")
modparam("carrierroute", "fetch_rows", 500)
The carrierroute table is about 91K lines and have been growing slowly.
Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
allocate shared memory from available pool after a few "kamctl cr
reload".
I increased the shared memory to 256 MB but with the same result. I
have now increased it to 512 MB and it seems to work better now.
I have noticed this. After a restart the shmem counters is like this:
shmem:total_size = 536870912
shmem:used_size = 28486752
shmem:real_used_size = 40147128
shmem:max_used_size = 41135424
shmem:free_size = 496723784
shmem:fragments = 555
And after the first "kamctl cr reload" it is like this:
shmem:total_size = 536870912
shmem:used_size = 28619016
shmem:real_used_size = 51842768
shmem:max_used_size = 76993616
shmem:free_size = 485028144
shmem:fragments = 722063
Notice the increase in fragments. Sequentials "kamctl cr reload" does
not change the fragments allot.
Any ideas?
be aware that a reload command usually requires at least twice memory
size than the size needed to store the records. That's because the old
structure in memory is kept until the database records are all loaded
and packed in shared memory in a new structure. Then the old and new
structures are swapped for routing usage and the old one is deallocated.
I havent really checked for carrier route module, but it is how happens
with the other modules that do caching (dispatcher, mtree, permissions,
...).
So if you are at the edge of available memory for storing cr rules, then
could happen to get out of memory easy, as shared memory can be used for
other tasks at that time. If number of fragments don't increase even
with 512MB, then this is likely what happens.
Regarding the fragmentation, new release has options for doing
defragmentation automatically:
https://www.kamailio.org/wiki/cookbooks/devel/core#mem_join
The code was actually pretty much all there, so I guess backporting will
not be big task if you want to do it on your local copy.
Also, testing the devel with this feature and giving feedback will be
really appreciated, I had no time to play with it and large number of
records. The feature is off by default, you have to set mem_join=1 in
config to get it on.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
5:51 p.m.
On 06/08/2012 03:09 PM, Morten Isaksen wrote:
Hi,
Hello,
We have 2 Kamailio 3.0.3 servers that has been running
with
carrierroute for about 2 years without any problems. They have 128 MB
shared memory and
The shared memory is too little. Keep in mind that this is shared
by all
children, so it will not increase with the creating of more children.
Also keep in mind that the Linux memory alocator (like for mmap) is
lazy, so allocating 1 GB by the application will not use 1 GB of RAM,
but instead pages will be delivered to the application as it needs it.
To force allocation there is the shm_force_alloc core parameter to
really force allocation of memory by the system bye writing 0's to the
memory reagion. Also there is the mlock_pages parameter that will force
the allocated memory pages to not be swappable.
modparam("carrierroute",
"config_source", "db")
modparam("carrierroute", "db_url", "<DBURL>")
modparam("carrierroute", "fetch_rows", 500)
Fetch_rows is affects the private memory used when loading data from a
data source that has the fetch result capability. This decreases the
private memory requirements for reloading the table.
The carrierroute table is about 91K lines and have been growing slowly.
From experience this takes around 60-70 MB of shared memory to hold
it. This depends a lot based on system (32 vs 64 for example),
structure etc.
Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
allocate shared memory from available pool after a few "kamctl cr
reload".
I increased the shared memory to 256 MB but with the same result. I
have now increased it to 512 MB and it seems to work better now.
I have noticed this. After a restart the shmem counters is like this:
shmem:total_size = 536870912
shmem:used_size = 28486752
shmem:real_used_size = 40147128
shmem:max_used_size = 41135424
shmem:free_size = 496723784
shmem:fragments = 555
And after the first "kamctl cr reload" it is like this:
shmem:total_size = 536870912
shmem:used_size = 28619016
shmem:real_used_size = 51842768
shmem:max_used_size = 76993616
shmem:free_size = 485028144
shmem:fragments = 722063
I would worry if the used_size or real_used_size would have changed a
lot. The difference from used_size to real_used_size is due to
fragmentation but this will be manageable if the number of fragments is
kept constant. As Daniel suggested, for carrierroute reload we need
double the amount of memory as it holds the old and the new tables until
it swaps the pointers (so that the applications doesn't block when
reloading the table).
Notice the increase in fragments. Sequentials
"kamctl cr reload" does
not change the fragments allot.
Any ideas?
I would recommend using kamailio with at least 1 GB of shared memory.
Also monitor the used_size and real_used_size if they are increasing in
a `weird` way. The real_used_size might increase because of
fragmentation during cr_reload_route commands, but it should not fail,
as chunks will be reallocated to other usages of the command.
Cheers,
Marius
--
Zbihlei Marius
Head of
Linux Development Services Romania
1&1 Internet Development srl Tel KA: 754-9152
Str Mircea Eliade 18 Tel RO: +40-31-223-9152
Sect 1, Bucuresti mailto: marius.zbihlei(a)1and1.ro
71295, Romania
11 Jun
11 Jun
11:03 a.m.
Hi Daniel and Marius,
Thanks for your feedback.
I hope I have solved the issue. Due to a bug in the perl script that
generated the route table some destinations got added instead of
updated to the table. The result of this bug was that about 50
destinations each was 1082 times in the route table.
/Morten
On Fri, Jun 8, 2012 at 4:51 PM, Marius Zbihlei <marius.zbihlei(a)1and1.ro> wrote:
On 06/08/2012 03:09 PM, Morten Isaksen wrote:
--
Morten Isaksen
Hi,
Hello,
We have 2 Kamailio 3.0.3 servers that has been
running with
carrierroute for about 2 years without any problems. They have 128 MB
shared memory and
The shared memory is too little. Keep in mind that this is shared by all
children, so it will not increase with the creating of more children. Also
keep in mind that the Linux memory alocator (like for mmap) is lazy, so
allocating 1 GB by the application will not use 1 GB of RAM, but instead
pages will be delivered to the application as it needs it. To force
allocation there is the shm_force_alloc core parameter to really force
allocation of memory by the system bye writing 0's to the memory reagion.
Also there is the mlock_pages parameter that will force the allocated memory
pages to not be swappable.
modparam("carrierroute",
"config_source", "db")
modparam("carrierroute", "db_url", "<DBURL>")
modparam("carrierroute", "fetch_rows", 500)
Fetch_rows is affects the private memory used when loading data from a data
source that has the fetch result capability. This decreases the private
memory requirements for reloading the table.
The carrierroute table is about 91K lines and have been growing slowly.
From experience this takes around 60-70 MB of shared memory to hold it.
This depends a lot based on system (32 vs 64 for example), structure etc.
Suddenly we get this ERROR: carrierroute [cr_data.c:585]: could not
allocate shared memory from available pool after a few "kamctl cr
reload".
I increased the shared memory to 256 MB but with the same result. I
have now increased it to 512 MB and it seems to work better now.
I have noticed this. After a restart the shmem counters is like this:
shmem:total_size = 536870912
shmem:used_size = 28486752
shmem:real_used_size = 40147128
shmem:max_used_size = 41135424
shmem:free_size = 496723784
shmem:fragments = 555
And after the first "kamctl cr reload" it is like this:
shmem:total_size = 536870912
shmem:used_size = 28619016
shmem:real_used_size = 51842768
shmem:max_used_size = 76993616
shmem:free_size = 485028144
shmem:fragments = 722063
I would worry if the used_size or real_used_size would have changed a lot.
The difference from used_size to real_used_size is due to fragmentation but
this will be manageable if the number of fragments is kept constant. As
Daniel suggested, for carrierroute reload we need double the amount of
memory as it holds the old and the new tables until it swaps the pointers
(so that the applications doesn't block when reloading the table).
Notice the increase in fragments. Sequentials
"kamctl cr reload" does
not change the fragments allot.
Any ideas?
I would recommend using kamailio with at least 1 GB of shared memory. Also
monitor the used_size and real_used_size if they are increasing in a
`weird` way. The real_used_size might increase because of fragmentation
during cr_reload_route commands, but it should not fail, as chunks will be
reallocated to other usages of the command.
Cheers,
Marius
--
Zbihlei Marius
Head of
Linux Development Services Romania
1&1 Internet Development srl Tel KA: 754-9152
Str Mircea Eliade 18 Tel RO: +40-31-223-9152
Sect 1, Bucuresti mailto: marius.zbihlei(a)1and1.ro
71295, Romania
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
12 Jun
12 Jun
12:29 p.m.
Hello,
On 6/11/12 10:03 AM, Morten Isaksen wrote:
Hi Daniel and Marius,
Thanks for your feedback.
I hope I have solved the issue. Due to a bug in the perl script that
generated the route table some destinations got added instead of
updated to the table. The result of this bug was that about 50
destinations each was 1082 times in the route table.
ok, good that you found it!
Cheers,
Daniel
/Morten
On Fri, Jun 8, 2012 at 4:51 PM, Marius Zbihlei <marius.zbihlei(a)1and1.ro> wrote:
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio Advanced Training, Seattle, USA, Sep 23-26, 2012 - http://asipto.com/u/katu
Kamailio Practical Workshop, Netherlands, Sep 10-12, 2012 - http://asipto.com/u/kpw
On 06/08/2012 03:09 PM, Morten Isaksen wrote:
Hi,
Hello,
We have 2 Kamailio 3.0.3 servers that has been
running with
carrierroute for about 2 years without any problems. They have 128 MB
shared memory and
The shared memory is too little. Keep in mind that this is
shared by all
children, so it will not increase with the creating of more children. Also
keep in mind that the Linux memory alocator (like for mmap) is lazy, so
allocating 1 GB by the application will not use 1 GB of RAM, but instead
pages will be delivered to the application as it needs it. To force
allocation there is the shm_force_alloc core parameter to really force
allocation of memory by the system bye writing 0's to the memory reagion.
Also there is the mlock_pages parameter that will force the allocated memory
pages to not be swappable.
modparam("carrierroute",
"config_source", "db")
modparam("carrierroute", "db_url", "<DBURL>")
modparam("carrierroute", "fetch_rows", 500)
Fetch_rows is affects the private memory used when loading data from a data
source that has the fetch result capability. This decreases the private
memory requirements for reloading the table.
The carrierroute table is about 91K lines and
have been growing slowly.
From experience this takes around 60-70 MB of shared memory to hold it.
This depends a lot based on system (32 vs 64 for example), structure etc.
Suddenly we get this ERROR: carrierroute
[cr_data.c:585]: could not
allocate shared memory from available pool after a few "kamctl cr
reload".
I increased the shared memory to 256 MB but with the same result. I
have now increased it to 512 MB and it seems to work better now.
I have noticed this. After a restart the shmem counters is like this:
shmem:total_size = 536870912
shmem:used_size = 28486752
shmem:real_used_size = 40147128
shmem:max_used_size = 41135424
shmem:free_size = 496723784
shmem:fragments = 555
And after the first "kamctl cr reload" it is like this:
shmem:total_size = 536870912
shmem:used_size = 28619016
shmem:real_used_size = 51842768
shmem:max_used_size = 76993616
shmem:free_size = 485028144
shmem:fragments = 722063
I would worry if the used_size or real_used_size would have changed a lot.
The difference from used_size to real_used_size is due to fragmentation but
this will be manageable if the number of fragments is kept constant. As
Daniel suggested, for carrierroute reload we need double the amount of
memory as it holds the old and the new tables until it swaps the pointers
(so that the applications doesn't block when reloading the table).
Notice the increase in fragments. Sequentials
"kamctl cr reload" does
not change the fragments allot.
Any ideas?
I would recommend using kamailio with at least 1 GB of shared memory. Also
monitor the used_size and real_used_size if they are increasing in a
`weird` way. The real_used_size might increase because of fragmentation
during cr_reload_route commands, but it should not fail, as chunks will be
reallocated to other usages of the command.
Cheers,
Marius
--
Zbihlei Marius
Head of
Linux Development Services Romania
1&1 Internet Development srl Tel KA: 754-9152
Str Mircea Eliade 18 Tel RO: +40-31-223-9152
Sect 1, Bucuresti mailto: marius.zbihlei(a)1and1.ro
71295, Romania
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
4549
days inactive
4553
days old
4 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Marius Zbihlei -
Morten Isaksen