Hi,
I have an application as follows:
Multiple Kamailio Nodes
Shared Postgres database
Auth information will be stored in DB and accessed with Authdb.
Here is the behavior I seek:
I would like all the Kamailo nodes to cooperate in updating the database, each updating data as appropriate:
When a registration comes in to any Node, and it authenticated, create or update a database record in location table. When a registration expires on that Node, delete database record in location table. When INVITE comes in on any Node, ignore local cache and always lookup() record from database.
#1 and #2 are used so that a third SIP system can query the database and look up the Node a NAT’d subscriber is ‘connected’ to.
#3 is to let NON-NAT’d subscribers be contacted from any Node.
Now I have something close to this now – using following settings:
modparam("usrloc", "db_mode", 1) # immediately write registrations to the database.
modparam("usrloc", "db_timer_clean", 0) # do not expire DB records separately
modparam("usrloc", "db_check_update", 1)
This takes care of #1 and #2 above. However, #3 seems to still be relying purely on local cache and does not hit the database.
Is there a way to do #3? I can set db_mode to 3 but that would seem to have other effects besides merely making lookup() read from database.
Thanks in advance,
Jawaid
If what you want is what you really want, db_mode 3 is the only way to achieve it. What other effects of mode 3 are you concerned about?
-- Alex
Hi Alex,
Well I guess the other effects if any are unclear.
As to what I want..
The goal is to ensure that all the nodes in a large-scale geographically-distributed system have routing information to locate the dynamic contacts of registered endpoints.
This requires that information to be shared with all other nodes somehow. Either you can store it in some kind of database and look it up, or, you have to proactively notify the other nodes of the information. (e.g. with DMQ or something similar).
DMQ replication is probably only scalable to a very small number of nodes (2-3), and also does not implement message reliability. This will force some other area of the application to shard across many small kamailio clusters.
Is there a different way to achieve a similar result? (Perhaps I have an incorrect assumption somewhere along the line)..
Thanks,
Jawaid
On 10/19/23, 8:57 PM, "Alex Balashov via sr-users" <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org> wrote:
If what you want is what you really want, db_mode 3 is the only way to achieve it. What other effects of mode 3 are you concerned about?
-- Alex
Hello,
at least for some ideas you can have a look at the p_usrloc module. This implements a clustered registration database, where the data location is queried from another database table. In the end you can of course also use native partition means provided from your database and build it by yourself.
Cheers,
Henning
Hi,
This is one of the effects I was remembering:
" For example NAT pinging is a killer since during each ping cycle all nated contact are loaded from the DB;"
On 10/20/23, 9:17 AM, "Jawaid Bazyar" <bazyar@gmail.com mailto:bazyar@gmail.com> wrote:
Hi Alex,
Well I guess the other effects if any are unclear.
As to what I want..
The goal is to ensure that all the nodes in a large-scale geographically-distributed system have routing information to locate the dynamic contacts of registered endpoints.
This requires that information to be shared with all other nodes somehow. Either you can store it in some kind of database and look it up, or, you have to proactively notify the other nodes of the information. (e.g. with DMQ or something similar).
DMQ replication is probably only scalable to a very small number of nodes (2-3), and also does not implement message reliability. This will force some other area of the application to shard across many small kamailio clusters.
Is there a different way to achieve a similar result? (Perhaps I have an incorrect assumption somewhere along the line)..
Thanks,
Jawaid
On 10/19/23, 8:57 PM, "Alex Balashov via sr-users" <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>> wrote:
If what you want is what you really want, db_mode 3 is the only way to achieve it. What other effects of mode 3 are you concerned about?
-- Alex
On 20 Oct 2023, at 09:17, Jawaid Bazyar bazyar@gmail.com wrote:
Hi Alex,
Well I guess the other effects if any are unclear.
As to what I want..
The goal is to ensure that all the nodes in a large-scale geographically-distributed system have routing information to locate the dynamic contacts of registered endpoints.
This requires that information to be shared with all other nodes somehow. Either you can store it in some kind of database and look it up, or, you have to proactively notify the other nodes of the information. (e.g. with DMQ or something similar).
DMQ replication is probably only scalable to a very small number of nodes (2-3), and also does not implement message reliability. This will force some other area of the application to shard across many small kamailio clusters.
Is there a different way to achieve a similar result? (Perhaps I have an incorrect assumption somewhere along the line)..
Not really, you've pretty much covered it.
I might disagree with the idea that DMQ doesn't scale to a larger number of nodes, but no, its scaling is not infinite.
Still sounds like db_mode 3 is the way to go for what you want, which is to use the database as the distribution and synchronisation mechanism. There are of course drawbacks to that, but you seem to be okay with those, and that's not where the pressure of the overall question appears to lie.
Not sure about your concerns about NAT pinging from another message, but they are orthogonal to the storage backend question per se.
db_mode 3 is the simple answer to your question. You might be splitting hairs on some of the other stuff.
-- Alex
Would a DMQ cluster with RAM-based store and a relatively small number of nodes (say, 5 for example) be able to handle a location table measured in the scale of 1 million nodes?
Has anyone done this before?
I assume DMQ has some provision to prevent loops / etc ?
Wondering about scaling a location service vertically and separating into a different layer from the stuff that needs to scale horizontally.
On 10/20/23, 11:16 AM, "Alex Balashov" <abalashov@evaristesys.com mailto:abalashov@evaristesys.com> wrote:
On 20 Oct 2023, at 09:17, Jawaid Bazyar <bazyar@gmail.com mailto:bazyar@gmail.com> wrote:
Hi Alex,
Well I guess the other effects if any are unclear.
As to what I want..
The goal is to ensure that all the nodes in a large-scale geographically-distributed system have routing information to locate the dynamic contacts of registered endpoints.
This requires that information to be shared with all other nodes somehow. Either you can store it in some kind of database and look it up, or, you have to proactively notify the other nodes of the information. (e.g. with DMQ or something similar).
DMQ replication is probably only scalable to a very small number of nodes (2-3), and also does not implement message reliability. This will force some other area of the application to shard across many small kamailio clusters.
Is there a different way to achieve a similar result? (Perhaps I have an incorrect assumption somewhere along the line)..
Not really, you've pretty much covered it.
I might disagree with the idea that DMQ doesn't scale to a larger number of nodes, but no, its scaling is not infinite.
Still sounds like db_mode 3 is the way to go for what you want, which is to use the database as the distribution and synchronisation mechanism. There are of course drawbacks to that, but you seem to be okay with those, and that's not where the pressure of the overall question appears to lie.
Not sure about your concerns about NAT pinging from another message, but they are orthogonal to the storage backend question per se.
db_mode 3 is the simple answer to your question. You might be splitting hairs on some of the other stuff.
-- Alex
On 20 Oct 2023, at 11:34, Jawaid Bazyar bazyar@gmail.com wrote:
Would a DMQ cluster with RAM-based store and a relatively small number of nodes (say, 5 for example) be able to handle a location table measured in the scale of 1 million nodes?
A million rows isn't really that much in contemporary computing scales.
Has anyone done this before?
Yes.
I assume DMQ has some provision to prevent loops / etc ?
Wondering about scaling a location service vertically and separating into a different layer from the stuff that needs to scale horizontally.
I think you might be overthinking it for a million contacts. I wouldn't split these hairs. DMQ is either fundamentally viable for something within that order of magnitude (it is), or it's not. It doesn't magically cave in at 1.2 million.
In thinking of reasons not to do DMQ, I would think about it from an even higher-level perspective: you know that data consistency and consensus are pretty difficult problems, and it's unlikely that a module developed by more or less one person and without significant resources and focus invested in these problems is going to rival a widely-used database (of one sort or another).
In thinking of reasons to do DMQ: if it's fundamentally reliable, edge-cases and failure modes may not be important enough to outweigh the cost savings, simplicity and performance boost of not having to synchronise blocking lookups over a remote database, deal with the problems of making that database highly available, operate and care for and feed the database, etc. Plus, there's a generally correct sense in software engineering that using database as an inter-process communication mechanism for ephemeral, short-term data streams is an antipattern[1][2]. There is room for lively debate about whether is the correct framing of what a SIP location service does.
-- Alex
[1] https://en.wikipedia.org/wiki/Database-as-IPC
[2] https://stackoverflow.com/questions/3815941/database-as-ipc-antipattern
PS. Kamailio gives you lots of low-level building blocks that can be used to stitch together a custom location service, if you want to rewrite usrloc/registrar. There are quite a few ways to replicate data out of Kamailio at little to no cost to an external consumer.
But if you want to use the tools in the Kamailio toolbox -- and I would argue that in your use-case, a rational actor would -- then you really have two choices:
1) db_mode 3 for the database-backed approach;
2) dmq/dmq_usrloc for the distributed, in-memory approach.
That's about it. There may be an honourable mention here for nonrelational DBs that can be coerced into acting like RDBMs, e.g. `db_redis`, but that's really in the weeds.
-- Alex
On 20 Oct 2023, at 11:41, Alex Balashov abalashov@evaristesys.com wrote:
On 20 Oct 2023, at 11:34, Jawaid Bazyar bazyar@gmail.com wrote:
Would a DMQ cluster with RAM-based store and a relatively small number of nodes (say, 5 for example) be able to handle a location table measured in the scale of 1 million nodes?
A million rows isn't really that much in contemporary computing scales.
Has anyone done this before?
Yes.
I assume DMQ has some provision to prevent loops / etc ?
Wondering about scaling a location service vertically and separating into a different layer from the stuff that needs to scale horizontally.
I think you might be overthinking it for a million contacts. I wouldn't split these hairs. DMQ is either fundamentally viable for something within that order of magnitude (it is), or it's not. It doesn't magically cave in at 1.2 million.
In thinking of reasons not to do DMQ, I would think about it from an even higher-level perspective: you know that data consistency and consensus are pretty difficult problems, and it's unlikely that a module developed by more or less one person and without significant resources and focus invested in these problems is going to rival a widely-used database (of one sort or another).
In thinking of reasons to do DMQ: if it's fundamentally reliable, edge-cases and failure modes may not be important enough to outweigh the cost savings, simplicity and performance boost of not having to synchronise blocking lookups over a remote database, deal with the problems of making that database highly available, operate and care for and feed the database, etc. Plus, there's a generally correct sense in software engineering that using database as an inter-process communication mechanism for ephemeral, short-term data streams is an antipattern[1][2]. There is room for lively debate about whether is the correct framing of what a SIP location service does.
-- Alex
[1] https://en.wikipedia.org/wiki/Database-as-IPC
[2] https://stackoverflow.com/questions/3815941/database-as-ipc-antipattern
-- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com Tel: +1-706-510-6800
-
Alex Balashov -
Henning Westerholt -
Jawaid Bazyar