Yesterday at Kamailio World there was an interesting question, raising a concern about replication of user location and DMQ in general, we should track it in the mailing list and use it as a feature request and I can share my initial thoughts on the topic.
The concern was about DMQ reordering transactions.
With SIP registration this seems like a minor concern, high responsiveness at the cost of potential small inconsistency seems like a good option. Considering that, the state will automatically correct itself and that this can be controlled using the expiration timer. This will become even more insignificant when the replicas can only be used as primary server failover. (it becomes a very small concern only when the primary server fails)
With other type of data re-ordering may result in more problematic use cases. My impression is that DMQ should try to be good only with volatile data, data that will expire anyway, like caching with expiration, registration, Dialogs, etc.
I think it may be worth looking at adding support for ordering/queuing in DMQ.
Performance and simplicity could be maintained by doing mainly transactions in parallel with a configurable re-ordering best effort to minimize the impact of the problem.
For example : 1- trying to re-order for a configurable amount of time. (32 seconds to match the RFC for SIP re-transmissions) 2- adding tractability for failed transactions. 3- the possibility to trigger a re-sync.
Another option is to look at preserving strict ordering but I can imagine this could add more problems in some cases ...
I guess, we could simply use tcp with t_relay using only one process in the module.
On Thu, May 9, 2019, 04:52 Julien Chavanton jchavanton@gmail.com wrote:
Yesterday at Kamailio World there was an interesting question, raising a concern about replication of user location and DMQ in general, we should track it in the mailing list and use it as a feature request and I can share my initial thoughts on the topic.
The concern was about DMQ reordering transactions.
With SIP registration this seems like a minor concern, high responsiveness at the cost of potential small inconsistency seems like a good option. Considering that, the state will automatically correct itself and that this can be controlled using the expiration timer. This will become even more insignificant when the replicas can only be used as primary server failover. (it becomes a very small concern only when the primary server fails)
With other type of data re-ordering may result in more problematic use cases. My impression is that DMQ should try to be good only with volatile data, data that will expire anyway, like caching with expiration, registration, Dialogs, etc.
I think it may be worth looking at adding support for ordering/queuing in DMQ.
Performance and simplicity could be maintained by doing mainly transactions in parallel with a configurable re-ordering best effort to minimize the impact of the problem.
For example : 1- trying to re-order for a configurable amount of time. (32 seconds to match the RFC for SIP re-transmissions) 2- adding tractability for failed transactions. 3- the possibility to trigger a re-sync.
Another option is to look at preserving strict ordering but I can imagine this could add more problems in some cases ...
Hi Julien,
It's a good point and I agree it could be a concern in some cases.
I'm not sure, however, the DMQ module itself is the place to do it. DMQ is purely a transport mechanism and has no knowledge or context of what's being relayed. This limits its scope to the ordering of messages only, rather than the replicated data.
Limiting DMQ to a single process could have performance implications, and at the same time does nothing to affect the ordering of messages sent to or received from multiple nodes (in a cluster of more than two) with varying distance/latency between them.
The sensible thing to do, in my opinion, is to implement ordering/version checking within each of the modules that need it. Most of them do anyway, in some way or another, for messages received locally. It would not be a big job to extend that to replicated messages and provide some kind of feedback/reverse-sync to the sending node if the local version is higher or the timestamp is later.
What do you/others think?
Cheers,
Charles
On Thu, 9 May 2019 at 11:57, Julien Chavanton jchavanton@gmail.com wrote:
I guess, we could simply use tcp with t_relay using only one process in the module.
On Thu, May 9, 2019, 04:52 Julien Chavanton jchavanton@gmail.com wrote:
Yesterday at Kamailio World there was an interesting question, raising a concern about replication of user location and DMQ in general, we should track it in the mailing list and use it as a feature request and I can share my initial thoughts on the topic.
The concern was about DMQ reordering transactions.
With SIP registration this seems like a minor concern, high responsiveness at the cost of potential small inconsistency seems like a good option. Considering that, the state will automatically correct itself and that this can be controlled using the expiration timer. This will become even more insignificant when the replicas can only be used as primary server failover. (it becomes a very small concern only when the primary server fails)
With other type of data re-ordering may result in more problematic use cases. My impression is that DMQ should try to be good only with volatile data, data that will expire anyway, like caching with expiration, registration, Dialogs, etc.
I think it may be worth looking at adding support for ordering/queuing in DMQ.
Performance and simplicity could be maintained by doing mainly transactions in parallel with a configurable re-ordering best effort to minimize the impact of the problem.
For example : 1- trying to re-order for a configurable amount of time. (32 seconds to match the RFC for SIP re-transmissions) 2- adding tractability for failed transactions. 3- the possibility to trigger a re-sync.
Another option is to look at preserving strict ordering but I can imagine this could add more problems in some cases ...
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Charles,
By the way, great work implementing all of this ! As far as I am concerned, this is the best solution to create a cluster of registrar, working great with well shared state overall.
Considering the usage of TCP to solve this problem for us, I can see that this may be quite complicated.
If I am correct, modules like dmq_userloc are using the dmq api: bcast_message and/or send_message and the request is actually sent from the current worker process using tmb.t_request. (If we did use TCP, than another process would take care of async connect)
It seems there will be no strait forward way to force the same process in this case because DMQ is not controlling anything.
Maybe DMQ could expose generic a queue and forward functionality, when queuing it would be possible to specify an argument/logic to make sure order is preserved. We can imagine that hashing over call-id would for dialog and registration and htable name, etc. (Then it could become an option to use TCP several connections to several servers, etc.)
I may need to do some test and spend more time to validate my assumptions ... (or you may correct me to save some time)
Anyway I am not thinking about finding a quick solution for this, I beleive there is no urgency. Regards, Julien
On Thu, May 9, 2019 at 5:46 AM Charles Chance charles.chance@sipcentric.com wrote:
Hi Julien,
It's a good point and I agree it could be a concern in some cases.
I'm not sure, however, the DMQ module itself is the place to do it. DMQ is purely a transport mechanism and has no knowledge or context of what's being relayed. This limits its scope to the ordering of messages only, rather than the replicated data.
Limiting DMQ to a single process could have performance implications, and at the same time does nothing to affect the ordering of messages sent to or received from multiple nodes (in a cluster of more than two) with varying distance/latency between them.
The sensible thing to do, in my opinion, is to implement ordering/version checking within each of the modules that need it. Most of them do anyway, in some way or another, for messages received locally. It would not be a big job to extend that to replicated messages and provide some kind of feedback/reverse-sync to the sending node if the local version is higher or the timestamp is later.
What do you/others think?
Cheers,
Charles
On Thu, 9 May 2019 at 11:57, Julien Chavanton jchavanton@gmail.com wrote:
I guess, we could simply use tcp with t_relay using only one process in the module.
On Thu, May 9, 2019, 04:52 Julien Chavanton jchavanton@gmail.com wrote:
Yesterday at Kamailio World there was an interesting question, raising a concern about replication of user location and DMQ in general, we should track it in the mailing list and use it as a feature request and I can share my initial thoughts on the topic.
The concern was about DMQ reordering transactions.
With SIP registration this seems like a minor concern, high responsiveness at the cost of potential small inconsistency seems like a good option. Considering that, the state will automatically correct itself and that this can be controlled using the expiration timer. This will become even more insignificant when the replicas can only be used as primary server failover. (it becomes a very small concern only when the primary server fails)
With other type of data re-ordering may result in more problematic use cases. My impression is that DMQ should try to be good only with volatile data, data that will expire anyway, like caching with expiration, registration, Dialogs, etc.
I think it may be worth looking at adding support for ordering/queuing in DMQ.
Performance and simplicity could be maintained by doing mainly transactions in parallel with a configurable re-ordering best effort to minimize the impact of the problem.
For example : 1- trying to re-order for a configurable amount of time. (32 seconds to match the RFC for SIP re-transmissions) 2- adding tractability for failed transactions. 3- the possibility to trigger a re-sync.
Another option is to look at preserving strict ordering but I can imagine this could add more problems in some cases ...
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB. _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If we were to implement queue and forward, we could use the mqueue module/API This a very simple and elegant module, we could later improve this module, this would be sinergetic.
On Fri, May 10, 2019 at 12:03 PM Julien Chavanton jchavanton@gmail.com wrote:
Hi Charles,
By the way, great work implementing all of this ! As far as I am concerned, this is the best solution to create a cluster of registrar, working great with well shared state overall.
Considering the usage of TCP to solve this problem for us, I can see that this may be quite complicated.
If I am correct, modules like dmq_userloc are using the dmq api: bcast_message and/or send_message and the request is actually sent from the current worker process using tmb.t_request. (If we did use TCP, than another process would take care of async connect)
It seems there will be no strait forward way to force the same process in this case because DMQ is not controlling anything.
Maybe DMQ could expose generic a queue and forward functionality, when queuing it would be possible to specify an argument/logic to make sure order is preserved. We can imagine that hashing over call-id would for dialog and registration and htable name, etc. (Then it could become an option to use TCP several connections to several servers, etc.)
I may need to do some test and spend more time to validate my assumptions ... (or you may correct me to save some time)
Anyway I am not thinking about finding a quick solution for this, I beleive there is no urgency. Regards, Julien
On Thu, May 9, 2019 at 5:46 AM Charles Chance < charles.chance@sipcentric.com> wrote:
Hi Julien,
It's a good point and I agree it could be a concern in some cases.
I'm not sure, however, the DMQ module itself is the place to do it. DMQ is purely a transport mechanism and has no knowledge or context of what's being relayed. This limits its scope to the ordering of messages only, rather than the replicated data.
Limiting DMQ to a single process could have performance implications, and at the same time does nothing to affect the ordering of messages sent to or received from multiple nodes (in a cluster of more than two) with varying distance/latency between them.
The sensible thing to do, in my opinion, is to implement ordering/version checking within each of the modules that need it. Most of them do anyway, in some way or another, for messages received locally. It would not be a big job to extend that to replicated messages and provide some kind of feedback/reverse-sync to the sending node if the local version is higher or the timestamp is later.
What do you/others think?
Cheers,
Charles
On Thu, 9 May 2019 at 11:57, Julien Chavanton jchavanton@gmail.com wrote:
I guess, we could simply use tcp with t_relay using only one process in the module.
On Thu, May 9, 2019, 04:52 Julien Chavanton jchavanton@gmail.com wrote:
Yesterday at Kamailio World there was an interesting question, raising a concern about replication of user location and DMQ in general, we should track it in the mailing list and use it as a feature request and I can share my initial thoughts on the topic.
The concern was about DMQ reordering transactions.
With SIP registration this seems like a minor concern, high responsiveness at the cost of potential small inconsistency seems like a good option. Considering that, the state will automatically correct itself and that this can be controlled using the expiration timer. This will become even more insignificant when the replicas can only be used as primary server failover. (it becomes a very small concern only when the primary server fails)
With other type of data re-ordering may result in more problematic use cases. My impression is that DMQ should try to be good only with volatile data, data that will expire anyway, like caching with expiration, registration, Dialogs, etc.
I think it may be worth looking at adding support for ordering/queuing in DMQ.
Performance and simplicity could be maintained by doing mainly transactions in parallel with a configurable re-ordering best effort to minimize the impact of the problem.
For example : 1- trying to re-order for a configurable amount of time. (32 seconds to match the RFC for SIP re-transmissions) 2- adding tractability for failed transactions. 3- the possibility to trigger a re-sync.
Another option is to look at preserving strict ordering but I can imagine this could add more problems in some cases ...
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB. _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Charles Chance -
Julien Chavanton