15 Nov
2011
15 Nov
'11
11:09 p.m.
Hello
coming back to the topic related to mtree, to be able to set values via
mi/rpc -- it won't be that difficult to add such functionality. Usually
with a tree is mainly reading, due to fast matching on tree indexing.
The question is how many times/often do you need to change values and
how many of them at the same time (more or less).
I was also thinking about that: our application works in such a way that
the full
tree is recalculated every 15 minutes. Currently we have aound 60
branches with 150000 entries each.
I assume many times the changes will be somewhere down
the tree, since
the first part of the number is usually the same (e.g., country code and
operator prefix). To update the tree at runtime, while there are reads
on it, there must be used a lock to be safe an consistent. If you do lot
of writes and very often, then you keep the tree structure locked a lot,
slowing the search.
That's the case, tname is rarely updated, it's tvalue the column that we
normally update.
Can you estimate the number of writes and how often they would be on a
daily basis? There might be other solutions to look at, if writes are
very often.
From the numbers above, let's say that we need to
update around 8M records
every 15 minutes (we expect this number will keep steadily
increasing...)
As a side note, I've looked at how to implement the mt_match equivalent in
redis and it does't look that hard using kamailio s.prefixed transformation
(as you suggested) and redis sorted sets. I'll need to make more tests to
check the performance, I'll share the results.
Thanks!
Javi
Cheers,
Daniel
Regards
Javi
16 Nov
16 Nov
9:13 a.m.
New subject: mtree on db Berkeley DB?
Hello,
On 11/15/11 10:09 PM, Javier Gallart wrote:
Hello
coming back to the topic related to mtree, to be able to set
values via
mi/rpc -- it won't be that difficult to add such functionality.
Usually
with a tree is mainly reading, due to fast matching on tree indexing.
The question is how many times/often do you need to change values and
how many of them at the same time (more or less).
I was also thinking about that: our application works in such a way
that the full tree is recalculated every 15 minutes. Currently we have
aound 60 branches with 150000 entries each.
I assume many times the changes will be somewhere down the tree, since
the first part of the number is usually the same (e.g., country
code and
operator prefix). To update the tree at runtime, while there are reads
on it, there must be used a lock to be safe an consistent. If you
do lot
of writes and very often, then you keep the tree structure locked
a lot,
slowing the search.
That's the case, tname is rarely updated, it's tvalue the column that
we normally update.
I guess you meant actually tprefix is not updated -- tname is table
name, tprefix is the prefix used to build the tree. What would be the
percentage of tvalue updates, do you have add/remove of tprefix-es?
Can you estimate the number of writes and how often they would be on a
daily basis? There might be other solutions to look at, if writes are
very often.
From the numbers above, let's say that we need to update around 8M
records every 15 minutes (we expect this number will keep steadily
increasing...)
But not all of them change the value, right?
As a side note, I've looked at how to implement the mt_match
equivalent in redis and it does't look that hard using kamailio
s.prefixed transformation (as you suggested) and redis sorted sets.
I'll need to make more tests to check the performance, I'll share the
results.
I have used this solution with mysql (using a table structure in memory
of mysql server), since the changes could have been done in mysql in
normal way, in one update query, which is a sync operation. Does redis
do any caching or is using always the disk file db?
Cheers,
Daniel
Thanks!
Javi
Cheers,
Daniel
--
Daniel-Constantin Mierla -- http://www.asipto.com
Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kat
http://linkedin.com/in/miconda -- http://twitter.com/miconda
Regards
Javi
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
1:45 p.m.
New subject: mtree on db Berkeley DB?
Hi
On Wed, Nov 16, 2011 at 8:13 AM, Daniel-Constantin Mierla <miconda(a)gmail.com
I've been checking this, and on average about 40% are updated in each
reload.
wrote:
Hello,
On 11/15/11 10:09 PM, Javier Gallart wrote:
Hello
You're right, sorry for the wrong names. The tprefixes are only updated
once a day, and only a tiny percentage of them (less than 1%) actually
changes.
coming back to the topic related to mtree, to be able to set values via
mi/rpc -- it won't be that difficult to add such functionality. Usually
with a tree is mainly reading, due to fast matching on tree indexing.
The question is how many times/often do you need to change values and
how many of them at the same time (more or less).
I was also thinking about that: our application works in such a way
that the full
tree is recalculated every 15 minutes. Currently we have
aound 60 branches with 150000 entries each.
I assume many times the changes will be somewhere
down the tree, since
the first part of the number is usually the same (e.g., country code and
operator prefix). To update the tree at runtime, while there are reads
on it, there must be used a lock to be safe an consistent. If you do lot
of writes and very often, then you keep the tree structure locked a lot,
slowing the search.
That's the case, tname is rarely updated, it's tvalue the column that we
normally update.
I guess you meant actually tprefix is not updated -- tname is table name,
tprefix is the prefix used to build the tree. What would be the percentage
of tvalue updates, do you have add/remove of tprefix-es?
Can you estimate the number of writes and how
often they would be on a
daily basis? There might be other solutions to look at, if writes are
very often.
From the numbers above, let's say that we need to update around 8M
records every 15 minutes (we expect this number will keep steadily
increasing...)
But not all of them change the value, right?
As a side note, I've looked at how to implement the mt_match equivalent
in redis and it does't look that hard using kamailio s.prefixed
transformation (as you suggested) and redis sorted sets. I'll need to make
more tests to check the performance, I'll share the results.
I have used this solution with mysql (using a table structure in memory of
mysql server), since the changes could have been done in mysql in normal
way, in one update query, which is a sync operation. Does redis do any
caching or is using always the disk file db?
All the data in Redis stays in memory, you can dump periodic snapshots to
disk. Some time ago they worked in a feature called disktore, with some
cache mechanism between disk and memory, but as far as I know that line was
abandoned.
Cheers,
Daniel
Thanks!
Javi
Cheers,
Daniel
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing
listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- http://www.asipto.com
Kamailio Advanced Training, Dec 5-8, Berlin:
http://asipto.com/u/kathttp://linkedin.com/in/miconda -- http://twitter.com/miconda
Regards
Javi
17 Nov
17 Nov
9:03 a.m.
New subject: mtree on db Berkeley DB?
Hello,
On 11/16/11 12:45 PM, Javier Gallart wrote:
[...]
All the data in Redis stays in memory, you can dump periodic snapshots
to disk. Some time ago they worked in a feature called disktore, with
some cache mechanism between disk and memory, but as far as I know
that line was abandoned.
so if the redis goes down, then the records are lost or is doing dump of
memory to disk at that time? I try to figure out if redis makes more
sense than using one kamailio as a load balancer in front of two
kamailios having the mtree in memory. The two with mtree will be
restarted one after the other every 15 min. Load balancer can be
configured to use only one, with quick fallback (100ms) to the second,
or via kamctl the active kamailio with mtree is set to be used first.
kamailio with mtree will send back a 302 which is handled by load balancer
Cheers,
Daniel
--
Daniel-Constantin Mierla -- http://www.asipto.com
Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kat
http://linkedin.com/in/miconda -- http://twitter.com/miconda
3:55 p.m.
New subject: mtree on db Berkeley DB?
Hi Daniel
On Thu, Nov 17, 2011 at 8:03 AM, Daniel-Constantin Mierla <miconda(a)gmail.com
wrote:
Hello,
On 11/16/11 12:45 PM, Javier Gallart wrote:
kamailio as
load-balancer we are using our own softswitch that sends the
calls to 2 kamailio with mtree in memory. If there is a problem with one of
them, the call is quickly rerouted to the backup kamailio. We're happy with
that configuration; but as the tree keeps growing the time needed for
populating the db + reloading the tree increases (about 4-5 minutes
currently, not really under pressure yet...). That's why we were thinking
about a full in-memory solution. At this point we're going to focus our
tests with this setup:
-1 redis Master server for writing, with frequent snapshots
-2 redis Slave servers
-2 kamailios, each of them reading from a slave redis stored in a shared
variable for quick update via rpc
-If redis_cmd returns an error, kamailio will respond with a 503 Status so
that the softswitch will send the invite to the backup kamailio
Of course this will make sense only if we're able to implement a mt_match
equivalent in redis that gives an acceptable performance...
Regards
Javi
[...]
All the data in Redis stays in memory, you can dump periodic snapshots to
disk. Some time ago they worked in a feature called disktore, with some
cache mechanism between disk and memory, but as far as I know that line was
abandoned.
so if the redis goes down, then the records are lost or is doing dump of
memory to disk at that time? I try to figure out if redis makes more sense
than using one kamailio as a load balancer in front of two kamailios having
the mtree in memory. The two with mtree will be restarted one after the
other every 15 min. Load balancer can be configured to use only one, with
quick fallback (100ms) to the second, or via kamctl the active kamailio
with mtree is set to be used first.
That's pretty much our current configuration; instead of having a kamailio with mtree will send back a 302 which is
handled by load balancer
Cheers,
Daniel
--
Daniel-Constantin Mierla -- http://www.asipto.com
Kamailio Advanced Training, Dec 5-8, Berlin: http://asipto.com/u/kat
http://linkedin.com/in/miconda -- http://twitter.com/miconda
4758
days inactive
4760
days old
4 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
Javier Gallart