Hi Charles, I took the time to find my notes and reproduce this in a lab

https://github.com/kamailio/kamailio/issues/1349

I guess we can follow up from there in the github issue

Cheers !

On Fri, Dec 1, 2017 at 4:06 AM, Charles Chance <charles.chance@sipcentric.com> wrote:
Hi Julien,

Please do and I'll gladly take a closer look.

Best,

Charles

On 1 December 2017 at 11:59, Julien Chavanton <jchavanton@gmail.com> wrote:
Hi Charles, great trick the FQDN + multi_notify

I am guessing that with the multi_notify setting, the problem I was refering too may be solved.

I will document it in a github issue.



On Fri, Dec 1, 2017 at 3:26 AM, Charles Chance <charles.chance@sipcentric.com> wrote:
Hello,

It seems you have already solved your issue, but just to add, it's often better to provide an FQDN as notification_address which resolves to multiple IPs (and set "multi_notify" parameter).

There should be no issues when shutting down multiple nodes at once, as long as the remaining nodes are still in contact with each other - at least, that is my experience across several long-established (and highly dynamic) clusters. When the disabled nodes are re-enabled, providing they have the address of at least one active node (or better, an FQDN resolving to multiple) they'll rejoin seamlessly.

Of course, if others have a different experience, it would be good to learn more about it.

Cheers,

Charles


On 1 December 2017 at 11:00, Julien Chavanton <jchavanton@gmail.com> wrote:
Forgot to mention

192.168.150.225 must have another node as notitifcation peer, not himself else he will not replicate data to other nodes

On Fri, Dec 1, 2017 at 2:33 AM, Julien Chavanton <jchavanton@gmail.com> wrote:
Hi Aydar,

You can have only one notification_address in this case the last one will be used "192.168.150.225" this will be you DMQ master node

I know the bus can break if you shutdown multiple nodes at the same time and one of them is the master node and then you restart them. (not sure if this is your case)

If this happens the only option I can see is to restart all nodes one by one.

I think this case could be handled better maybe by adding a dmq resync command that can recreate the bus .


On Fri, Dec 1, 2017 at 12:32 AM, Aidar Kamalov <aidar.kamalov@gmail.com> wrote:
Hello. I have 4  kamailio nodes with same config:

modparam("dmq", "server_address", "sip:LOCALIP:5050")
modparam("dmq", "notification_address", "sip:192.168.8.213:5050")
modparam("dmq", "notification_address", "sip:192.168.10.54:5050")
modparam("dmq", "notification_address", "sip:192.168.107.195:5050")
modparam("dmq", "notification_address", "sip:192.168.150.225:5050")
modparam("dmq", "num_workers", 6)
modparam("dmq_usrloc", "enable", 1)
modparam("dmq_usrloc", "sync", 1)
modparam("dmq_usrloc", "batch_size", 2000)
modparam("dmq_usrloc", "batch_usleep", 1000)


regularly (don't know why) I've got issues - replication is broken, for example now at all nodes I have:
# kamcmd dmq.list_nodes
{
host: 192.168.8.213
port: 5050
resolved_ip: 192.168.8.213
status: 2
last_notification: 0
local: 0
}
{
host: 192.168.10.54
port: 5050
resolved_ip: 192.168.10.54
status: 2
last_notification: 0
local: 0
}
{
host: 192.168.150.225
port: 5050
resolved_ip: 192.168.150.225
status: 8
last_notification: 0
local: 0
}
{
host: 192.168.107.195
port: 5050
resolved_ip: 192.168.107.195
status: 2
last_notification: 0
local: 1
}

but at node 192.168.150.225:
# kamcmd dmq.list_nodes
{
host: 192.168.150.225
port: 5050
resolved_ip: 192.168.150.225
status: 2
last_notification: 0
local: 0
}
{
host: 192.168.150.225
port: 5050
resolved_ip: 192.168.150.225
status: 2
last_notification: 0
local: 1
}

nodes restart doesn't help. but if i exec
systemctl stop kamailio && sleep 15 && systemctl start kamailio
it is partially solve my issue - dmq.list_nodes show all nodes, but at 192.168.150.225 i see not all user locations: they are added gradually

There is no errors in logs
Dec  1 08:28:37 sipufa /usr/sbin/kamailio[18252]: NOTICE: <script>: ROOT: KDMQ notification_peer [15f3edf9258014d1-7114@192.168.8.213]
Dec  1 08:28:38 sipufa /usr/sbin/kamailio[18271]: NOTICE: <script>: ROOT: KDMQ notification_peer [7acb9eae627482f8-87828@192.168.10.54]
Dec  1 08:28:39 sipufa /usr/sbin/kamailio[18250]: NOTICE: <script>: ROOT: KDMQ notification_peer [621ad9f474b335e2-18272@192.168.150.225]
Dec  1 08:28:40 sipufa /usr/sbin/kamailio[18252]: NOTICE: <script>: ROOT: KDMQ notification_peer [7b817f6a6244f923-8872@192.168.107.195]
Dec  1 08:28:47 sipufa /usr/sbin/kamailio[18271]: NOTICE: <script>: ROOT: KDMQ usrloc [7acb9eae627482f2-87794@192.168.10.54]
Dec  1 08:29:20 sipufa /usr/sbin/kamailio[18251]: NOTICE: <script>: ROOT: KDMQ usrloc [7acb9eae627482ef-87790@192.168.10.54]


--
Aydar A. Kamalov

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users




_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users





Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users



_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users





Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.

_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users