Hello,
I'm using LCR module where I have 44 million rules and 13 million rule targets (not all rules have destinations).
Noting that I'm running Ubuntu on virtual machine with 8GB allocated memory.
When I'm setting lcr_rule_hash_size parameter to 268435456 (2 power 28) I got the following error : "no memory for rules hash table" in the log ,
noting that when I run ipcs -lm command I got max total shared memory shared memory (kbytes) : 8388608.
When I'm setting lcr_rule_hash_size parameter to 1024,meaning 1024 rules will be fetched from the database (please correct if I'm wrong) Kamailio takes around 2 minutes and 30 seconds to start , which is worrying for me :(
Please advise how many rules Kamailio can support (if 8 GB RAM are allocated to the server) , and what size every rule takes in the hash table built in the shared memory.
Thanks,
Ali
Ali Taher writes:
I'm using LCR module where I have 44 million rules and 13 million rule targets (not all rules have destinations).
When I'm setting lcr_rule_hash_size parameter to 268435456 (2 power 28) I got the following error : "no memory for rules hash table" in the log
it makes no sense to use hash table of 268+ million entries for 44 million rules.
When I'm setting lcr_rule_hash_size parameter to 1024,meaning 1024 rules will be fetched from the database (please correct if I'm wrong) Kamailio takes around 2 minutes and 30 seconds to start , which is worrying for me :(
lcr_rule_hash_size has nothing to do with fetching rules from database. fetch_rows is for that purpose. read README.
-- juha
Hi Juha,
Can you please tell me what memory should be allocated to handle 44 million rules.
Thanks, Ali
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Juha Heinanen Sent: Wednesday, June 10, 2015 6:13 PM To: Kamailio (SER) - Users Mailing List Subject: [SR-Users] Kamailio LCR module memory concern
Ali Taher writes:
I'm using LCR module where I have 44 million rules and 13 million rule targets (not all rules have destinations).
When I'm setting lcr_rule_hash_size parameter to 268435456 (2 power 28) I got the following error : "no memory for rules hash table" in the log
it makes no sense to use hash table of 268+ million entries for 44 million rules.
When I'm setting lcr_rule_hash_size parameter to 1024,meaning 1024 rules will be fetched from the database (please correct if I'm wrong) Kamailio takes around 2 minutes and 30 seconds to start , which is worrying for me :(
lcr_rule_hash_size has nothing to do with fetching rules from database. fetch_rows is for that purpose. read README.
-- juha
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 11:33 AM, Ali Taher wrote:
Can you please tell me what memory should be allocated to handle 44 million rules.
How much memory, or how many hash buckets?
Hi Alex,
I don't know a lot about hash tables , and how hash buckets and memory are related. I just need to know the specs needed for a server in order to be able to handle 44 million or rules using LCR module , and what configuration (parameters) should be applied.
Thanks, Ali
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Alex Balashov Sent: Wednesday, June 10, 2015 6:35 PM To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio LCR module memory concern
On 06/10/2015 11:33 AM, Ali Taher wrote:
Can you please tell me what memory should be allocated to handle 44 million rules.
How much memory, or how many hash buckets?
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 11:48 AM, Ali Taher wrote:
Hi Alex,
I don't know a lot about hash tables , and how hash buckets and memory are related. I just need to know the specs needed for a server in order to be able to handle 44 million or rules using LCR module , and what configuration (parameters) should be applied.
Well, an ideal hash function would require 44 million buckets for 44 million rules. :-) Given that this ideal is probably not attainable, your safest bet is to go one power of 2 bigger, i.e. 2^26 (~67m buckets).
You should take the time to learn about hash algorithms and other O(1) data structures. Their whole purpose is to keep lookup time of constant computational complexity irrespectively of the number of entries. The trade-off to make that possible is memory.
Thanks Alex,
So what memory should be allocated to handle 67m buckets ? Is there any documentation for hash tables for kamailio?
Ali
Sent from my iPhone
On Jun 10, 2015, at 6:52 PM, Alex Balashov abalashov@evaristesys.com wrote:
On 06/10/2015 11:48 AM, Ali Taher wrote:
Hi Alex,
I don't know a lot about hash tables , and how hash buckets and memory are related. I just need to know the specs needed for a server in order to be able to handle 44 million or rules using LCR module , and what configuration (parameters) should be applied.
Well, an ideal hash function would require 44 million buckets for 44 million rules. :-) Given that this ideal is probably not attainable, your safest bet is to go one power of 2 bigger, i.e. 2^26 (~67m buckets).
You should take the time to learn about hash algorithms and other O(1) data structures. Their whole purpose is to keep lookup time of constant computational complexity irrespectively of the number of entries. The trade-off to make that possible is memory.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 12:00 PM, Ali Taher wrote:
Is there any documentation for hash tables for kamailio?
http://en.wikipedia.org/wiki/Hash_table
On 06/10/2015 12:00 PM, Ali Taher wrote:
So what memory should be allocated to handle 67m buckets ?
A better question would be: why do you want to store 44m rules in memory? That's a lot of rules.
This is the number of rules generated from our system for LCR routing (we are dealing with transit traffic)
Ali Sent from my iPhone
On Jun 10, 2015, at 7:10 PM, Alex Balashov abalashov@evaristesys.com wrote:
On 06/10/2015 12:00 PM, Ali Taher wrote:
So what memory should be allocated to handle 67m buckets ?
A better question would be: why do you want to store 44m rules in memory? That's a lot of rules.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 12:44 PM, Ali Taher wrote:
This is the number of rules generated from our system for LCR routing (we are dealing with transit traffic)
Indeed, but if you have a system of that size and complexity, I would encourage you to consider an LCR approach that does not rely on having the entire data set in memory, but instead utilises an efficient, lightweight, but still database-bound query approach.
What if your route table doubles in size?
Even estimating 1 kB per rule and exactly 44m rules, you're looking at 42 GB of memory. That's not realistic. I'd go back to the architectural drawing board.
Hi Alex,
What do you mean by "consider an LCR approach that does not rely on having the entire data set in memory, but instead utilises an efficient, lightweight, but still database-bound query approach."
Can I configure kamailio to read directly from the database without setting them in the memory? or you mean trying to compress the rules
Thanks, Ali
Sent from my iPhone
On Jun 10, 2015, at 7:48 PM, Alex Balashov abalashov@evaristesys.com wrote:
On 06/10/2015 12:44 PM, Ali Taher wrote:
This is the number of rules generated from our system for LCR routing (we are dealing with transit traffic)
Indeed, but if you have a system of that size and complexity, I would encourage you to consider an LCR approach that does not rely on having the entire data set in memory, but instead utilises an efficient, lightweight, but still database-bound query approach.
What if your route table doubles in size?
Even estimating 1 kB per rule and exactly 44m rules, you're looking at 42 GB of memory. That's not realistic. I'd go back to the architectural drawing board.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 03:53 PM, Ali Taher wrote:
Can I configure kamailio to read directly from the database without setting them in the memory?
Sure...
http://kamailio.org/docs/modules/4.3.x/modules/sqlops.html
Thanks Alex you saved my life! will check and try to implement this module
Ali Sent from my iPhone
On Jun 10, 2015, at 10:55 PM, Alex Balashov abalashov@evaristesys.com wrote:
On 06/10/2015 03:53 PM, Ali Taher wrote:
Can I configure kamailio to read directly from the database without setting them in the memory?
Sure...
http://kamailio.org/docs/modules/4.3.x/modules/sqlops.html
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Ali Taher writes:
This is the number of rules generated from our system for LCR routing (we are dealing with transit traffic).
i wonder how it is possible to have so many distinct e.164 prefixes for transit unless you need to deal with individual subscriber numbers.
if you deal with subscriber numbers, one possibility would be to put them in enum.
-- juha
On 06/10/2015 12:52 PM, Juha Heinanen wrote:
i wonder how it is possible to have so many distinct e.164 prefixes for transit unless you need to deal with individual subscriber numbers.
It's quite possible for folks who run dialler traffic[1], at least in the US. The reasons are:
- Most rate centres in the US are subject to number pooling, which allocates numbers in blocks of 1000 using 7-digit prefixes. This yields a stupefyingly large number of prefixes just for domestic USA routing.
- Wholesale ITSPs, particularly ones specialising in short-duration traffic, typically utilise a large number of vendors, since many of them will fail the call due to race-to-the-bottom LCR. We once had a customer with 135 million routes from around 30 different vendors.
-- Alex
[1] Incidentally, the subject of my recent Kamailio World talk!
https://www.youtube.com/watch?v=u30Fyp3QanE
The case mentioned is exactly our case where we deal with class 4, wholesale carriers with many vendors; just as explained in the provided short duration dialer video where we are using Kamailio lcr as redirect server. Hence, what can we do in order to afford such Huge number of rules? How did your customer handle 137 million using kamailio lcr?
Thanks for your cooperation, Ali Sent from my iPhone
On Jun 10, 2015, at 7:55 PM, Alex Balashov abalashov@evaristesys.com wrote:
On 06/10/2015 12:52 PM, Juha Heinanen wrote:
i wonder how it is possible to have so many distinct e.164 prefixes for transit unless you need to deal with individual subscriber numbers.
It's quite possible for folks who run dialler traffic[1], at least in the US. The reasons are:
Most rate centres in the US are subject to number pooling, which allocates numbers in blocks of 1000 using 7-digit prefixes. This yields a stupefyingly large number of prefixes just for domestic USA routing.
Wholesale ITSPs, particularly ones specialising in short-duration traffic, typically utilise a large number of vendors, since many of them will fail the call due to race-to-the-bottom LCR. We once had a customer with 135 million routes from around 30 different vendors.
-- Alex
[1] Incidentally, the subject of my recent Kamailio World talk!
https://www.youtube.com/watch?v=u30Fyp3QanE
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 04:30 PM, Ali Taher wrote:
How did your customer handle 137 million using kamailio lcr?
Database query to PostgreSQL.
On 06/10/2015 04:30 PM, Ali Taher wrote:
How did your customer handle 137 million using kamailio lcr?
I suppose I should clarify; they did not use Kamailio's lcr module. As I said earlier, I don't think the Kamailio lcr module is appropriate for such a case.
Hi Alex, What do you recommend for such giant 137 million routes or entries ?
Cheers, Abdul Hakeem
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Alex Balashov Sent: Wednesday, June 10, 2015 9:41 PM To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio LCR module memory concern
On 06/10/2015 04:30 PM, Ali Taher wrote:
How did your customer handle 137 million using kamailio lcr?
I suppose I should clarify; they did not use Kamailio's lcr module. As I said earlier, I don't think the Kamailio lcr module is appropriate for such a case.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
On 06/10/2015 05:13 PM, Abdul Hakeem wrote:
What do you recommend for such giant 137 million routes or entries ?
The same that anyone would recommend for any application that aims to store 137 million rows of anything: a relational database, queried manually from Kamailio route script using the 'sqlops' module. :-)
-- Alex
Hello,
you can consider mtree module if you don't need to restart very often. Eventually you can run it in an instance of kamailio that is used as redirect server, having the main proxy instances routing to it and handling the redirect responses.
Another option is using some nosql system, perhaps Redis fits well in this case, but could be also memcached or mongodb. In kamailio you have connectors to interact with these systems directly from configuration file.
Of course, if you prefer sql, Alex gave already the answer to use sqlops module.
Cheers, Daniel
On 10/06/15 23:13, Abdul Hakeem wrote:
Hi Alex, What do you recommend for such giant 137 million routes or entries ?
Cheers, Abdul Hakeem
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Alex Balashov Sent: Wednesday, June 10, 2015 9:41 PM To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio LCR module memory concern
On 06/10/2015 04:30 PM, Ali Taher wrote:
How did your customer handle 137 million using kamailio lcr?
I suppose I should clarify; they did not use Kamailio's lcr module. As I said earlier, I don't think the Kamailio lcr module is appropriate for such a case.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Thanks Daniel. I was thinking SQL is an overkill for route lookup. If an instance of Kamailio is setup as a Registrar/Redirect with a Redis (Pub/Sub) backend, is there a way for a Kamailio proxy instance to subscribe to dynamic routes from the Redis which can be injected into call variables?
Cheers, Abdul Hakeem
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of Daniel-Constantin Mierla Sent: Thursday, June 11, 2015 12:11 PM To: Kamailio (SER) - Users Mailing List Subject: Re: [SR-Users] Kamailio LCR module memory concern
Hello,
you can consider mtree module if you don't need to restart very often. Eventually you can run it in an instance of kamailio that is used as redirect server, having the main proxy instances routing to it and handling the redirect responses.
Another option is using some nosql system, perhaps Redis fits well in this case, but could be also memcached or mongodb. In kamailio you have connectors to interact with these systems directly from configuration file.
Of course, if you prefer sql, Alex gave already the answer to use sqlops module.
Cheers, Daniel
On 10/06/15 23:13, Abdul Hakeem wrote:
Hi Alex, What do you recommend for such giant 137 million routes or entries ?
Cheers, Abdul Hakeem
-----Original Message----- From: sr-users [mailto:sr-users-bounces@lists.sip-router.org] On Behalf Of
Alex
Balashov Sent: Wednesday, June 10, 2015 9:41 PM To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio LCR module memory concern
On 06/10/2015 04:30 PM, Ali Taher wrote:
How did your customer handle 137 million using kamailio lcr?
I suppose I should clarify; they did not use Kamailio's lcr module. As I said earlier, I don't think the Kamailio lcr module is appropriate for such a case.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Why would you go through the trouble of encapsulating that in some sort of presence layer? Just query Redis directly with redis_cmd() from the ndb_redis module.
Whether you need an SQL database or can get by with Redis is a complex topic. It's ultimately a question of how complicated a feature set you need and the structure of your data and workflow as a whole.
When viewed in isolation, as a single objective, routing with Redis appears to be much more performant. But most call routing platforms consist of a lot more database interactions than just a single route lookup. That's where the added value of a full-on RDBM becomes relevant.
-- Alex Balashov | Principal | Evariste Systems LLC 303 Perimeter Center North, Suite 300 Atlanta, GA 30346 United States
Tel: +1-800-250-5920 (toll-free) / +1-678-954-0671 (direct) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Sent from my BlackBerry.
Ali Taher writes:
I just need to know the specs needed for a server in order to be able to handle 44 million or rules using LCR module , and what configuration (parameters) should be applied.
I have already answered this. Calculate from lcr_mod.h rule_info struct how much memory one rule requires. On top of that you need memory for the hash table. How much depends on your collision requirement.
-- Juha
-
Abdul Hakeem -
Alex Balashov -
Ali Taher -
Daniel-Constantin Mierla -
Juha Heinanen