Hi,
I've set up a dmq server for other kamailio nodes with this parameters: ``` modparam("dmq", "server_address","sip:10.0.0.101:5060") modparam("dmq", "notification_address", "sip:127.0.0.1:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15) ```
On the kamailio nodes the config is the following: ``` modparam("dmq", "server_address", "sip:10.0.0.x:5060") modparam("dmq", "notification_address", "sip:10.0.0.101:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15) ```
On the kamailio dmq server I've got this in the logs: ``` 25(41) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request 25(41) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer ```
On the kamailio nodes the same thing with the ip of the dmq server: ``` 25(112) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.0.0.101:5060 because of failed request 25(112) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer ```
The request route on the dmq server is simple as this: ``` request_route { if(is_method("KDMQ")){ dmq_handle_message(); } }#end request_route ```
I've tried with some xlog-s inside the if(is_method("KDMQ")) but they don't show up so the dmq_handle_message() is not even executed.
I've also tried to bind the dmq server to a different port but with no success on making it work.
The dmq.list_nodes shows only the dmq server IP and the current node IP, the IPs of other nodes are not shown: ``` root@proxy-94b6ccf46-t8fx2:/# kamcmd dmq.list_nodes { host: 10.0.0.101 port: 5060 resolved_ip: 10.0.0.101 status: active last_notification: 0 local: 0 } { host: 10.0.0.45 port: 5060 resolved_ip: 10.0.0.45 status: active last_notification: 0 local: 1 } ``` And it's the same on all other kamailio nodes: dmq server ip and local ip.
The kamailio version on all the nodes is: 5.2.0~dev6+0~20180616010152.1138+xenial
Thanks, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
Hello,
Is there anything preventing the messages from reaching Kamailio? If you have a pcap from one of the servers we may be able to see what’s happening.
Also, you have the notification address set to localhost - this should instead point to one of the other nodes.
Cheers,
Charles
On Fri, 22 Jun 2018 at 21:35, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi,
I've set up a dmq server for other kamailio nodes with this parameters:
modparam("dmq", "server_address","sip:10.0.0.101:5060") modparam("dmq", "notification_address", "sip:127.0.0.1:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
Why are you setting the notification address as localhost?
On the kamailio nodes the config is the following:
modparam("dmq", "server_address", "sip:10.0.0.x:5060") modparam("dmq", "notification_address", "sip:10.0.0.101:5060") modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)On the kamailio dmq server I've got this in the logs:
25(41) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request 25(41) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peerOn the kamailio nodes the same thing with the ip of the dmq server:
25(112) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.0.0.101:5060 because of failed request 25(112) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peerThe request route on the dmq server is simple as this:
request_route { if(is_method("KDMQ")){ dmq_handle_message(); } }#end request_routeI've tried with some xlog-s inside the if(is_method("KDMQ")) but they don't show up so the dmq_handle_message() is not even executed.
I've also tried to bind the dmq server to a different port but with no success on making it work.
The dmq.list_nodes shows only the dmq server IP and the current node IP, the IPs of other nodes are not shown:
root@proxy-94b6ccf46-t8fx2:/# kamcmd dmq.list_nodes { host: 10.0.0.101 port: 5060 resolved_ip: 10.0.0.101 status: active last_notification: 0 local: 0 } { host: 10.0.0.45 port: 5060 resolved_ip: 10.0.0.45 status: active last_notification: 0 local: 1 }And it's the same on all other kamailio nodes: dmq server ip and local ip.
The kamailio version on all the nodes is: 5.2.0~dev6+0~20180616010152.1138+xenial
Thanks,
Aleksandar Sosic mail: alex.sosic@evosip.cloud
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Charles,
The notification address is set to localhost only for the server node because I have a mutual architecture and don't know which nodes are up and with which IPs. There could be a possibility that there are no other kamailio nodes beside the dmq-server. I'm pretty sure this configuration was working some time ago, We've since then changed some configuration but no dmq configuration was changed as I'm aware of. The other thing that changes is the kamailio version. I will try to use a stable version and retry.
I've tried a capture with tshark and on the local interface I've got: ``` 3 0.116690338 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 4 0.116771925 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 5 1.116940867 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 6 3.116729798 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 7 4.116611072 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 8 7.116677754 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 9 8.116711422 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) ```
On the public interface there are packets arriving from the other two kamailio nodes: ``` 221 218.212276413 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 222 219.212053642 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 223 219.955864183 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 224 220.955964403 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) ```
But in the logs of all three kamailio nodes I've got: ``` 9(64) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.0.0.102:5060 because of failed request 9(64) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer ``` Also no luck with the dmq.list_nodes.
Kind regards and thank you for your time, I really appreciate it! -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Sat, Jun 23, 2018 at 6:27 PM Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Is there anything preventing the messages from reaching Kamailio? If you have a pcap from one of the servers we may be able to see what’s happening.
Also, you have the notification address set to localhost - this should instead point to one of the other nodes.
Cheers,
Charles
With a more verbose kamailio (not sure if it helps):
```3(32) DEBUG: <core> [core/udp_server.c:491]: udp_rcv_loop(): received on udp socket: (106/100/520) [[KDMQ sip:notification_peer@10.0.0.102:5060 SIP/2.0 0D 0A Via: SIP/2.0/UDP 10.0.0.49;branch=z9hG4bK361]] 3(32) DEBUG: <core> [core/parser/msg_parser.c:604]: parse_msg(): SIP Request: 3(32) DEBUG: <core> [core/parser/msg_parser.c:606]: parse_msg(): method: <KDMQ> 3(32) DEBUG: <core> [core/parser/msg_parser.c:608]: parse_msg(): uri: sip:notification_peer@10.0.0.102:5060 3(32) DEBUG: <core> [core/parser/msg_parser.c:610]: parse_msg(): version: <SIP/2.0> 3(32) DEBUG: <core> [core/parser/parse_via.c:1303]: parse_via_param(): Found param type 232, <branch> = <z9hG4bK3618.4ae6e325000000000000000000000000.0>; state=16 3(32) DEBUG: <core> [core/parser/parse_via.c:2639]: parse_via(): end of header reached, state=5 3(32) DEBUG: <core> [core/parser/msg_parser.c:492]: parse_headers(): Via found, flags=2 3(32) DEBUG: <core> [core/parser/msg_parser.c:494]: parse_headers(): this is the first via 3(32) DEBUG: <core> [core/parser/parse_addr_spec.c:864]: parse_addr_spec(): end of header reached, state=10 3(32) DEBUG: <core> [core/parser/msg_parser.c:171]: get_hdr_field(): <To> [43]; uri=[sip:notification_peer@10.0.0.102:5060] 3(32) DEBUG: <core> [core/parser/msg_parser.c:172]: get_hdr_field(): to body [sip:notification_peer@10.0.0.102:5060 ], to tag [] 3(32) DEBUG: <core> [core/parser/msg_parser.c:152]: get_hdr_field(): cseq <CSeq>: <10> <KDMQ> 3(32) DEBUG: <core> [core/receive.c:232]: receive_msg(): --- received sip message - request - call-id: [3d4783ff3ccce81f-64@10.0.0.49] - cseq: [10 KDMQ] 3(32) DEBUG: <core> [core/receive.c:287]: receive_msg(): preparing to run routing scripts... 3(32) DEBUG: <core> [core/parser/msg_parser.c:183]: get_hdr_field(): content_length=73 3(32) DEBUG: <core> [core/parser/msg_parser.c:89]: get_hdr_field(): found end of header 3(32) DEBUG: <core> [core/receive.c:344]: receive_msg(): request-route executed in: 203 usec 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 3(32) DEBUG: <core> [core/xavp.c:495]: xavp_destroy_list(): destroying xavp list (nil) 3(32) DEBUG: <core> [core/receive.c:447]: receive_msg(): cleaning up ``` ... ``` 9(38) DEBUG: <core> [core/receive.c:232]: receive_msg(): --- received sip message - request - call-id: [0eb7f0c66155bdad-46@10.0.0.102] - cseq: [10 KDMQ] 9(38) DEBUG: <core> [core/receive.c:287]: receive_msg(): preparing to run routing scripts... 9(38) DEBUG: <core> [core/parser/msg_parser.c:183]: get_hdr_field(): content_length=71 9(38) DEBUG: <core> [core/parser/msg_parser.c:89]: get_hdr_field(): found end of header 9(38) DEBUG: <core> [core/receive.c:344]: receive_msg(): request-route executed in: 102 usec 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list (nil) 9(38) DEBUG: <core> [core/xavp.c:495]: xavp_destroy_list(): destroying xavp list (nil) 9(38) DEBUG: <core> [core/receive.c:447]: receive_msg(): cleaning up 17(46) DEBUG: tm [t_reply.c:1262]: t_should_relay_response(): ->>>>>>>>> T_code=0, new_code=408 17(46) DEBUG: tm [t_reply.c:2092]: local_reply(): branch=0, save=0, winner=0 17(46) DEBUG: tm [t_reply.c:2131]: local_reply(): local transaction completed 408/0 (totag retr: 0/1024) 17(46) DEBUG: tm [t_hooks.c:258]: run_trans_callbacks_internal(): DBG: trans=0x7f9093424300, callback type 1024, id 0 entered 17(46) DEBUG: dmq [dmq_funcs.c:61]: dmq_tm_callback(): dmq_tm_callback start 17(46) DEBUG: dmq [notification_peer.c:586]: notification_resp_callback_f(): notification_callback_f triggered [0xffffffffffffffff 408 (nil)] 17(46) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request 17(46) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peer 17(46) DEBUG: dmq [dmq_funcs.c:69]: dmq_tm_callback(): dmq_tm_callback done ``` -- Aleksandar Sosic mail: alex.sosic@evosip.cloud On Sat, Jun 30, 2018 at 8:32 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
The notification address is set to localhost only for the server node because I have a mutual architecture and don't know which nodes are up and with which IPs. There could be a possibility that there are no other kamailio nodes beside the dmq-server. I'm pretty sure this configuration was working some time ago, We've since then changed some configuration but no dmq configuration was changed as I'm aware of. The other thing that changes is the kamailio version. I will try to use a stable version and retry.
I've tried a capture with tshark and on the local interface I've got:
request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 4 0.116771925 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 5 1.116940867 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 6 3.116729798 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 7 4.116611072 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 8 7.116677754 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain) 9 8.116711422 10.0.0.101 ? 127.0.0.1 SIP 557 Unknown request: KDMQ sip:notification_peer@127.0.0.1:5060 | (text/plain)On the public interface there are packets arriving from the other two kamailio nodes:
221 218.212276413 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 222 219.212053642 10.0.0.50 ? 172.22.5.102 SIP 561 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 223 219.955864183 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain) 224 220.955964403 10.0.0.49 ? 172.22.5.102 SIP 562 Unknown request: KDMQ sip:notification_peer@10.0.0.102:5060 | (text/plain)But in the logs of all three kamailio nodes I've got:
9(64) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.0.0.102:5060 because of failed request 9(64) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): not deleting notification_peerAlso no luck with the dmq.list_nodes.
Kind regards and thank you for your time, I really appreciate it!
Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Sat, Jun 23, 2018 at 6:27 PM Charles Chance charles.chance@sipcentric.com wrote:
Hello,
Is there anything preventing the messages from reaching Kamailio? If you have a pcap from one of the servers we may be able to see what’s happening.
Also, you have the notification address set to localhost - this should instead point to one of the other nodes.
Cheers,
Charles
Hi everyone,
so adding some xlogs We've managed to find out what's happening. The request route was using: ``` if(is_method("KDMQ")){ ``` but this if was never triggered(!?) We changed that to ``` if($rm == "KDMQ"){ ``` And now it's working.
So why is that? In the documentation there's an example with is_method: https://kamailio.org/docs/modules/5.2.x/modules/dmq.html#dmq.f.dmq_handle_me... and I can recall it working a month ago.
Everything is working except for this message: ```25(42) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request``` And that's because of the notification address on the kamailio node we use as DMQ server
This portion of DMQ documentation says that `Default value is "NULL".` for the notification_address: https://kamailio.org/docs/modules/5.2.x/modules/dmq.html#dmq.p.notification_... But with NULL or NOT specifying a value kamailio does not start. Am I misunderstanding the documentation? Could this be a feature request or it's a bug?
In a situation like our where kamailio nodes start and stop on demand without any prior defined topology or IP addresses defined it's hard to use DMQ module like that. That's why we introduced a DMQ server with a fixed IP address that it's handling only those packets (but it's a workaround).
So my next question is: Do you guys have an idea on how could we handle DMQ on a very mutational infrastructure?
Example: I have 2 kamailio nodes in the begining (no fixed IPs - using DHCP), then when there's demand the infrastructure starts another 2 nodes, now I have 4 of them, then 3, one of the nodes randomly is stopped. And I want to use DMQ on these node to propagate information without using a DB. How could we do that?
Also another thing that we noticed is that the last node to be added is the one with no prior information regarding usrloc for example, it has only the changes happening from when it starts on. Could there be a method or modparam flag to force dmq sync of all the information upon the startup of another node (another possible feature request)?
Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Sat, Jun 30, 2018 at 11:12 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
With a more verbose kamailio (not sure if it helps):
Hi Aleksandar,
On 1 July 2018 at 07:08, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi everyone,
so adding some xlogs We've managed to find out what's happening. The request route was using:
if(is_method("KDMQ")){but this if was never triggered(!?) We changed that to
if($rm == "KDMQ"){And now it's working.
So why is that? In the documentation there's an example with is_method: https://kamailio.org/docs/modules/5.2.x/modules/dmq. html#dmq.f.dmq_handle_message and I can recall it working a month ago.
Either way should work - there should be no difference between them. We are using is_method everywhere and it works just fine.
Everything is working except for this message:
notification_resp_callback_f(): deleting server sip:127.0.0.1:5060 because of failed request``` And that's because of the notification address on the kamailio node we use as DMQ server
I suspect setting localhost as the notification address is causing you some problems - it's certainly not supported and although I've not tested it, it wouldn't surprise me if this has something to do with your earlier issue.
You do not, in fact, need to maintain a dedicated node who's only role is "DMQ server" as you have described it. I would recommend removing it completely and allowing your other nodes to discover/communicate between themselves.
This portion of DMQ documentation says that `Default value is "NULL".` for the notification_address: https://kamailio.org/docs/modules/5.2.x/modules/dmq. html#dmq.p.notification_address But with NULL or NOT specifying a value kamailio does not start. Am I misunderstanding the documentation? Could this be a feature request or it's a bug?
Omitting the notification address should not cause an issue - what error do you see in your log when it fails to start? If you get rid of the "DMQ server' node as suggested above, though, this should be a none-issue.
In a situation like our where kamailio nodes start and stop on demand without any prior defined topology or IP addresses defined it's hard to use DMQ module like that. That's why we introduced a DMQ server with a fixed IP address that it's handling only those packets (but it's a workaround).
So my next question is: Do you guys have an idea on how could we handle DMQ on a very mutational infrastructure?
Example: I have 2 kamailio nodes in the begining (no fixed IPs - using DHCP), then when there's demand the infrastructure starts another 2 nodes, now I have 4 of them, then 3, one of the nodes randomly is stopped. And I want to use DMQ on these node to propagate information without using a DB. How could we do that?
Use multi_notify - https://www.kamailio.org/docs/modules/stable/modules/dmq.html#dmq.p.multi_no...
If nodes are being added/removed dynamically then you'll probably also want to employ some kind of service discovery mechanism, e.g. consul.
Also another thing that we noticed is that the last node to be added is the one with no prior information regarding usrloc for example, it has only the changes happening from when it starts on. Could there be a method or modparam flag to force dmq sync of all the information upon the startup of another node (another possible feature request)?
See sync parameter - https://www.kamailio.org/docs/modules/stable/modules/dmq_usrloc.html#usrloc_...
Cheers,
Charles
On Mon, Jul 2, 2018 at 2:55 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi Aleksandar, [...] You do not, in fact, need to maintain a dedicated node who's only role is "DMQ server" as you have described it. I would recommend removing it completely and allowing your other nodes to discover/communicate between themselves.
Ok Thanks for your inputs, seems a good idea and I think we're on a good path to solve our issues.
Omitting the notification address should not cause an issue - what error do you see in your log when it fails to start? If you get rid of the "DMQ server' node as suggested above, though, this should be a none-issue.
It does cause an issue: ``` 0(51) DEBUG: <core> [core/parser/parse_uri.c:1254]: parse_uri(): uri too short: <> (0) 0(51) ERROR: dmq [dmq.c:218]: mod_init(): notification address invalid 0(51) ERROR: <core> [core/sr_module.c:990]: init_mod(): Error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) ERROR: error while initializing modules ```
Maybe a feature request could be to just start kamailio even if it does not resolve a dns record for the dmq notification_address? This would solve us some problems in a highly mutable infrastructure like ours:
``` 0(88) ERROR: dmq [notification_peer.c:336]: add_server_and_notify(): error adding notification node 0(88) ERROR: dmq [dmq.c:304]: child_init(): cannot retrieve initial nodelist from sip:proxy-service:5061 0(88) ERROR: <core> [core/sr_module.c:944]: init_mod_child(): error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) (idx: 0 rank: 0 desc: [main]) 0(88) ERROR: <core> [main.c:1711]: main_loop(): error in init_child```
So just to recap: - Not specifying a notification_address DOES give issues - Having a DNS record for the notification_address that does not resolve makes kamailio to crash
Any possibility to have this solved/enhanced?
Thanks, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
I'll take a look - which version are you using?
Cheers,
Charles
On 5 July 2018 at 11:17, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
On Mon, Jul 2, 2018 at 2:55 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi Aleksandar, [...] You do not, in fact, need to maintain a dedicated node who's only role
is "DMQ server" as you have described it. I would recommend removing it completely and allowing your other nodes to discover/communicate between themselves.
Ok Thanks for your inputs, seems a good idea and I think we're on a good path to solve our issues.
Omitting the notification address should not cause an issue - what error
do you see in your log when it fails to start? If you get rid of the "DMQ server' node as suggested above, though, this should be a none-issue.
It does cause an issue:
0(51) DEBUG: <core> [core/parser/parse_uri.c:1254]: parse_uri(): uri too short: <> (0) 0(51) ERROR: dmq [dmq.c:218]: mod_init(): notification address invalid 0(51) ERROR: <core> [core/sr_module.c:990]: init_mod(): Error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) ERROR: error while initializing modulesMaybe a feature request could be to just start kamailio even if it does not resolve a dns record for the dmq notification_address? This would solve us some problems in a highly mutable infrastructure like ours:
add_server_and_notify(): error adding notification node 0(88) ERROR: dmq [dmq.c:304]: child_init(): cannot retrieve initial nodelist from sip:proxy-service:5061 0(88) ERROR: <core> [core/sr_module.c:944]: init_mod_child(): error while initializing module dmq (/usr/lib/x86_64-linux-gnu/kamailio/modules/dmq.so) (idx: 0 rank: 0 desc: [main]) 0(88) ERROR: <core> [main.c:1711]: main_loop(): error in init_child``` So just to recap: - Not specifying a notification_address DOES give issues - Having a DNS record for the notification_address that does not resolve makes kamailio to crash Any possibility to have this solved/enhanced? Thanks, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance charles.chance@sipcentric.com wrote:
I'll take a look - which version are you using?
5.2.0~dev6+0~20180616010152.1138+xenial
Thank you! -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
Hi,
I'm now creating a dns record inside kubernetes with a headless service. Unfortunately I must use a busybox that will start before the kamailio nodes so the dns record will be created before kamailio starts because otherwise it will crash as I told you before. IMHO it will be useful to have a flag that allows me to choose if kamailio should start even if the notification_address of the DMQ module does not resolve. That would allow me to start nodes, populate the dns and then slowly add nodes to the cluster without having a busybox to do that.
As you can see from the logs kamailio notices the busybox IP (10.32.68.202) which obviously not being a kamailio node gives me this errors: ``` 17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no destination sets 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.32.68.202:5061 because of failed request 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:43.100.109.113:5061 because of failed request ``` Being a workaround for creating the DNS record before starting kamailio nodes that's ok. But notice the IP 43.100.109.113 which is not an IP that the DNS record for the notification_address resolves:
``` $ nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53
Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.83 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.84 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.202 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.81 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.82 ```
So why is that?
Then I noticed that on one node I do have all the other IPs: ```kamcmd> dmq.list_nodes { host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 0 } { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.84 port: 5061 resolved_ip: 172.22.5.84 status: active last_notification: 0 local: 1 } ```
But on another one: ``` kamcmd> dmq.list_nodes { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 1 } ``` I miss a node? And all the kamailio instances in this range are the same with the same config. Also the missing node is not the ip of the node itself but another one. -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Fri, Jul 6, 2018 at 9:21 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance charles.chance@sipcentric.com wrote:
I'll take a look - which version are you using?
5.2.0~dev6+0~20180616010152.1138+xenial
Thank you!
Aleksandar Sosic mail: alex.sosic@evosip.cloud
Here's another example:
``` router-0 3/3 Running 0 13m 172.22.5.94 master.alex.cloud.evox.it router-1 3/3 Running 0 13m 172.22.5.3 master.alex.cloud.evox.it router-2 3/3 Running 0 12m 172.22.5.4 master.alex.cloud.evox.it router-3 3/3 Running 0 12m 172.22.5.5 master.alex.cloud.evox.it ```
``` root@router-0:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 1 } ```
``` root@router-1:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 1 } ```
``` root@router-2:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 1 } ```
``` root@router-3:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.5 port: 5062 resolved_ip: 172.22.5.5 status: active last_notification: 0 local: 1 } ```
``` nslookup dmq-router-service Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.5 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.4 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.3 Name: dmq-router-service.default.svc.cluster.local Address: 10.32.68.217 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.94 ```
There is no difference in configuration between the router nodes. -- Aleksandar Sosic mail: alex.sosic@evosip.cloud On Fri, Jul 6, 2018 at 11:42 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi,
I'm now creating a dns record inside kubernetes with a headless service. Unfortunately I must use a busybox that will start before the kamailio nodes so the dns record will be created before kamailio starts because otherwise it will crash as I told you before. IMHO it will be useful to have a flag that allows me to choose if kamailio should start even if the notification_address of the DMQ module does not resolve. That would allow me to start nodes, populate the dns and then slowly add nodes to the cluster without having a busybox to do that.
As you can see from the logs kamailio notices the busybox IP (10.32.68.202) which obviously not being a kamailio node gives me this errors:
17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no destination sets 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.32.68.202:5061 because of failed request 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:43.100.109.113:5061 because of failed requestBeing a workaround for creating the DNS record before starting kamailio nodes that's ok. But notice the IP 43.100.109.113 which is not an IP that the DNS record for the notification_address resolves:
$ nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.83 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.84 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.202 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.81 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.82So why is that?
Then I noticed that on one node I do have all the other IPs:
{ host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 0 } { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.84 port: 5061 resolved_ip: 172.22.5.84 status: active last_notification: 0 local: 1 }But on another one:
kamcmd> dmq.list_nodes { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 1 }I miss a node? And all the kamailio instances in this range are the same with the same config. Also the missing node is not the ip of the node itself but another one. -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Fri, Jul 6, 2018 at 9:21 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance charles.chance@sipcentric.com wrote:
I'll take a look - which version are you using?
5.2.0~dev6+0~20180616010152.1138+xenial
Thank you!
Aleksandar Sosic mail: alex.sosic@evosip.cloud
Similar thing with a different type of nodes:
``` proxy-94b6ccf46-6n49v 3/3 Running 0 1m 172.22.5.99 master.alex.cloud.evox.it proxy-94b6ccf46-7jrgj 3/3 Running 0 1m 172.22.5.98 master.alex.cloud.evox.it proxy-94b6ccf46-rbskb 3/3 Running 0 1m 172.22.5.95 master.alex.cloud.evox.it proxy-94b6ccf46-rtzh9 3/3 Running 0 1m 172.22.5.96 master.alex.cloud.evox.it` ```
``` root@proxy-94b6ccf46-6n49v:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.99 port: 5061 resolved_ip: 172.22.5.99 status: active last_notification: 0 local: 1 } ```
``` root@proxy-94b6ccf46-7jrgj:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.95 port: 5061 resolved_ip: 172.22.5.95 status: active last_notification: 0 local: 0 } { host: 172.22.5.98 port: 5061 resolved_ip: 172.22.5.98 status: active last_notification: 0 local: 1 } ```
``` root@proxy-94b6ccf46-rbskb:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.95 port: 5061 resolved_ip: 172.22.5.95 status: active last_notification: 0 local: 1 } ```
``` root@proxy-94b6ccf46-rtzh9:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.96 port: 5061 resolved_ip: 172.22.5.96 status: active last_notification: 0 local: 1 } ```
``` nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.99 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.98 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.96 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.95 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.218 ```
What am I missing out? -- Aleksandar Sosic mail: alex.sosic@evosip.cloud On Fri, Jul 6, 2018 at 12:26 PM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Here's another example:
router-0 3/3 Running 0 13m 172.22.5.94 master.alex.cloud.evox.it router-1 3/3 Running 0 13m 172.22.5.3 master.alex.cloud.evox.it router-2 3/3 Running 0 12m 172.22.5.4 master.alex.cloud.evox.it router-3 3/3 Running 0 12m 172.22.5.5 master.alex.cloud.evox.itroot@router-0:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 1 }root@router-1:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 1 }root@router-2:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 1 }root@router-3:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.5 port: 5062 resolved_ip: 172.22.5.5 status: active last_notification: 0 local: 1 }nslookup dmq-router-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.5 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.4 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.3 Name: dmq-router-service.default.svc.cluster.local Address: 10.32.68.217 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.94There is no difference in configuration between the router nodes.
Aleksandar Sosic mail: alex.sosic@evosip.cloud On Fri, Jul 6, 2018 at 11:42 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi,
I'm now creating a dns record inside kubernetes with a headless service. Unfortunately I must use a busybox that will start before the kamailio nodes so the dns record will be created before kamailio starts because otherwise it will crash as I told you before. IMHO it will be useful to have a flag that allows me to choose if kamailio should start even if the notification_address of the DMQ module does not resolve. That would allow me to start nodes, populate the dns and then slowly add nodes to the cluster without having a busybox to do that.
As you can see from the logs kamailio notices the busybox IP (10.32.68.202) which obviously not being a kamailio node gives me this errors:
17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no destination sets 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.32.68.202:5061 because of failed request 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:43.100.109.113:5061 because of failed requestBeing a workaround for creating the DNS record before starting kamailio nodes that's ok. But notice the IP 43.100.109.113 which is not an IP that the DNS record for the notification_address resolves:
$ nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.83 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.84 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.202 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.81 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.82So why is that?
Then I noticed that on one node I do have all the other IPs:
{ host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 0 } { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.84 port: 5061 resolved_ip: 172.22.5.84 status: active last_notification: 0 local: 1 }But on another one:
kamcmd> dmq.list_nodes { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 1 }I miss a node? And all the kamailio instances in this range are the same with the same config. Also the missing node is not the ip of the node itself but another one. -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Fri, Jul 6, 2018 at 9:21 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance charles.chance@sipcentric.com wrote:
I'll take a look - which version are you using?
5.2.0~dev6+0~20180616010152.1138+xenial
Thank you!
Aleksandar Sosic mail: alex.sosic@evosip.cloud
Hi,
I have just pushed some changes to master - one of these allows startup to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the outcome?
Cheers,
Charles
On 6 July 2018 at 11:35, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Similar thing with a different type of nodes:
proxy-94b6ccf46-6n49v 3/3 Running 0 1m 172.22.5.99 master.alex.cloud.evox.it proxy-94b6ccf46-7jrgj 3/3 Running 0 1m 172.22.5.98 master.alex.cloud.evox.it proxy-94b6ccf46-rbskb 3/3 Running 0 1m 172.22.5.95 master.alex.cloud.evox.it proxy-94b6ccf46-rtzh9 3/3 Running 0 1m 172.22.5.96 master.alex.cloud.evox.it`root@proxy-94b6ccf46-6n49v:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.99 port: 5061 resolved_ip: 172.22.5.99 status: active last_notification: 0 local: 1 }root@proxy-94b6ccf46-7jrgj:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.95 port: 5061 resolved_ip: 172.22.5.95 status: active last_notification: 0 local: 0 } { host: 172.22.5.98 port: 5061 resolved_ip: 172.22.5.98 status: active last_notification: 0 local: 1 }root@proxy-94b6ccf46-rbskb:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.95 port: 5061 resolved_ip: 172.22.5.95 status: active last_notification: 0 local: 1 }root@proxy-94b6ccf46-rtzh9:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.96 port: 5061 resolved_ip: 172.22.5.96 status: active last_notification: 0 local: 1 }nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.99 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.98 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.96 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.95 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.218What am I missing out?
Aleksandar Sosic mail: alex.sosic@evosip.cloud On Fri, Jul 6, 2018 at 12:26 PM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Here's another example:
router-0 3/3 Running 0 13m 172.22.5.94 master.alex.cloud.evox.it router-1 3/3 Running 0 13m 172.22.5.3 master.alex.cloud.evox.it router-2 3/3 Running 0 12m 172.22.5.4 master.alex.cloud.evox.it router-3 3/3 Running 0 12m 172.22.5.5 master.alex.cloud.evox.itroot@router-0:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 1 }root@router-1:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 0 } { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 1 }root@router-2:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.94 port: 5062 resolved_ip: 172.22.5.94 status: active last_notification: 0 local: 0 } { host: 172.22.5.3 port: 5062 resolved_ip: 172.22.5.3 status: active last_notification: 0 local: 0 } { host: 172.22.5.4 port: 5062 resolved_ip: 172.22.5.4 status: active last_notification: 0 local: 1 }root@router-3:/# kamcmd kamcmd 1.5 Copyright 2006 iptelorg GmbH This is free software with ABSOLUTELY NO WARRANTY. For details type `warranty'. kamcmd> dmq.list_nodes { host: 172.22.5.5 port: 5062 resolved_ip: 172.22.5.5 status: active last_notification: 0 local: 1 }nslookup dmq-router-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.5 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.4 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.3 Name: dmq-router-service.default.svc.cluster.local Address: 10.32.68.217 Name: dmq-router-service.default.svc.cluster.local Address: 172.22.5.94There is no difference in configuration between the router nodes.
Aleksandar Sosic mail: alex.sosic@evosip.cloud On Fri, Jul 6, 2018 at 11:42 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi,
I'm now creating a dns record inside kubernetes with a headless
service.
Unfortunately I must use a busybox that will start before the kamailio nodes so the dns record will be created before kamailio starts because otherwise it will crash as I told you before. IMHO it will be useful to have a flag that allows me to choose if kamailio should start even if the notification_address of the DMQ module does not resolve. That would allow me to start nodes, populate the dns and then slowly add nodes to the cluster without having a busybox to do that.
As you can see from the logs kamailio notices the busybox IP (10.32.68.202) which obviously not being a kamailio node gives me this errors:
17(109) ERROR: dispatcher [dispatch.c:1910]: ds_select_dst_limit(): no destination sets 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:10.32.68.202:5061 because of failed request 33(125) ERROR: dmq [notification_peer.c:596]: notification_resp_callback_f(): deleting server sip:43.100.109.113:5061 because of failed requestBeing a workaround for creating the DNS record before starting kamailio nodes that's ok. But notice the IP 43.100.109.113 which is not an IP that the DNS record for the notification_address resolves:
$ nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.83 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.84 Name: dmq-proxy-service.default.svc.cluster.local Address: 10.32.68.202 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.81 Name: dmq-proxy-service.default.svc.cluster.local Address: 172.22.5.82So why is that?
Then I noticed that on one node I do have all the other IPs:
{ host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 0 } { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.84 port: 5061 resolved_ip: 172.22.5.84 status: active last_notification: 0 local: 1 }But on another one:
kamcmd> dmq.list_nodes { host: 172.22.5.83 port: 5061 resolved_ip: 172.22.5.83 status: active last_notification: 0 local: 0 } { host: 172.22.5.81 port: 5061 resolved_ip: 172.22.5.81 status: active last_notification: 0 local: 0 } { host: 172.22.5.82 port: 5061 resolved_ip: 172.22.5.82 status: active last_notification: 0 local: 1 }I miss a node? And all the kamailio instances in this range are the same with the same config. Also the missing node is not the ip of the node itself but another one. -- Aleksandar Sosic mail: alex.sosic@evosip.cloudOn Fri, Jul 6, 2018 at 9:21 AM Aleksandar Sosic alex.sosic@evosip.cloud wrote:
On Thu, Jul 5, 2018 at 12:35 PM Charles Chance charles.chance@sipcentric.com wrote:
I'll take a look - which version are you using?
5.2.0~dev6+0~20180616010152.1138+xenial
Thank you!
Aleksandar Sosic mail: alex.sosic@evosip.cloud
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running: ``` proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7 ```
In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example: ``` kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 } ```
And after a few seconds only the loca node itself: ``` kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 } ``` And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this: ``` root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4 ``` Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows startup to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the outcome?
Cheers,
Charles
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows startup
to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to
push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the outcome?
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to : - open a tcpdump trace in the instance tracing only the DMQ port defined [5062] - (tcpdump -i eth0 port 5062) - packet coming from nowhere - started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e) - seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){ dmq_handle_message(); exit; }
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance charles.chance@sipcentric.com:
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.1165+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows startup
to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to
push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the
outcome?
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud < paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){ dmq_handle_message(); exit; }
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance charles.chance@sipcentric.com:
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 65+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows
startup to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to
push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the
outcome?
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud < paolo.visintin@evosip.cloud> wrote:
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud < paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){ dmq_handle_message(); exit; }
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance charles.chance@sipcentric.com :
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 65+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows
startup to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning to
push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the
outcome?
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Charles, when I say kamailio "resolves" but does not query DNS I mean that in DEBUG I am able to see 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c luster.local" and starting kamailio at the same time the result is always the correct ip, no 42.100.109.113 coming up so this is very strange!
Cheers
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 15:29 GMT+02:00 Charles Chance charles.chance@sipcentric.com:
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud < paolo.visintin@evosip.cloud> wrote:
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud < paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){ dmq_handle_message(); exit; }
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance@sipcentric.com
:
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 65+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance charles.chance@sipcentric.com wrote:
Hi,
I have just pushed some changes to master - one of these allows
startup to continue even if initial node resolution fails.
There are some other improvements, too, which I have been planning
to push for some time and which should also help in your situation.
Can you try again with these changes applied and let me know the
outcome?
Cheers,
Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin - evosip.cloud:
I am able to see 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c luster.local" and starting kamailio at the same time the result is always the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello,
without looking too much into that, I have two remarks which may worth investigating:
- for DNS names in the configuration file Kamailio does a lookup during startup and "compiles" this to structure which is evaluated during run time
- there is also a DNS caching functionality, which could cause something like you observed. You can deactivate this with a core variable, I think.
Best regards,
Henning
2018-08-02 15:29 GMT+02:00 Charles Chance charles.chance@sipcentric.com:
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud> wrote:
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){
dmq_handle_message(); exit;}
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance@sipcentric.com
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud
wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 65+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
charles.chance@sipcentric.com wrote: > Hi, > > I have just pushed some changes to master - one of these allows
startup to continue even if initial node resolution fails.
> There are some other improvements, too, which I have been planning
to push for some time and which should also help in your situation.
> Can you try again with these changes applied and let me know the
outcome?
> Cheers, > > Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Hello Henning, Thanks for precious indications! I've tried to disable dns caching but nothing changed Also captured everything on the network interface, but nothing discovered! I've changed also dns server (from kube DNs to core dns) without success.
I'll try to reproduce in vm environment with power dns and capturing also on the dns side
Il Gio 2 Ago 2018, 20:40 Henning Westerholt hw@kamailio.org ha scritto:
Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin - evosip.cloud:
I am able to see 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c luster.local" and starting kamailio at the same time the result is always the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello,
without looking too much into that, I have two remarks which may worth investigating:
- for DNS names in the configuration file Kamailio does a lookup during
startup and "compiles" this to structure which is evaluated during run time
- there is also a DNS caching functionality, which could cause something
like you observed. You can deactivate this with a core variable, I think.
Best regards,
Henning
2018-08-02 15:29 GMT+02:00 Charles Chance <charles.chance@sipcentric.com :
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see if you run the following commands?
kamcmd dns.lookup SRV
_sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud> wrote:
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list():
adding
DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
0xf31d
SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query
0x1904
A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port
defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){
dmq_handle_message(); exit;}
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:
42.100.109.113:5062
because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance <
charles.chance@sipcentric.com
Hi Aleksandar,
The initial depopulation of the nodes (following a period of
'pending'
state) is due to no response being received from them. Are you able
to
trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It
must
be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or
more
other nodes. Are you running on a public cloud somewhere or is it your
own,
locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic <alex.sosic@evosip.cloud
wrote: > Hi Charles, > > We're so glad about the improvements you just committed! Thanks! > > Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 > 65+xenial > Kamailio starts even if the DNS record does not exist at first,
that's
> great. I'm having this nodes up and running: > ``` > proxy-66f79498cc-8ws6d 3/3 Running 0 > 4m 172.28.1.4 > proxy-66f79498cc-b68dd 3/3 Running 0 > 4m 172.28.1.5 > proxy-66f79498cc-lhbxg 3/3 Running 0 > 4m 172.28.1.6 > proxy-66f79498cc-png6p 3/3 Running 0 > 4m 172.28.1.7 > ``` > > In those 4-5 minutes I've experienced a population and
depopulation of
> the dmq nodes. > > For example: > ``` > kamcmd> dmq.list_nodes > { > host: 172.28.1.4 > port: 5061 > resolved_ip: 172.28.1.4 > status: pending > last_notification: 0 > local: 0 > } > { > host: 172.28.1.6 > port: 5061 > resolved_ip: 172.28.1.6 > status: pending > last_notification: 0 > local: 0 > } > { > host: 40.100.109.113 > port: 5061 > resolved_ip: 40.100.109.113 > status: pending > last_notification: 0 > local: 0 > } > { > host: 172.28.1.5 > port: 5061 > resolved_ip: 172.28.1.5 > status: active > last_notification: 0 > local: 1 > } > ``` > > And after a few seconds only the loca node itself: > ``` > kamcmd> dmq.list_nodes > { > host: 172.28.1.5 > port: 5061 > resolved_ip: 172.28.1.5 > status: active > last_notification: 0 > local: 1 > } > ``` > And then again the 3 nodes and the 40.100.109.113 which is a
Microsoft
> IP and I don't know what's doing that IP into my dmq server list. > > The DNS record is this: > ``` > root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service > Server: 10.96.0.10 > Address: 10.96.0.10#53 > > Non-authoritative answer: > Name: dmq-proxy-service.alex.svc.cluster.local > Address: 172.28.1.7 > Name: dmq-proxy-service.alex.svc.cluster.local > Address: 172.28.1.6 > Name: dmq-proxy-service.alex.svc.cluster.local > Address: 172.28.1.5 > Name: dmq-proxy-service.alex.svc.cluster.local > Address: 172.28.1.4 > ``` > Any clues why that ip is in the dmq nodes list? And why .7 is
missing?
> Kind regards, > -- > Aleksandar Sosic > mail: alex.sosic@evosip.cloud > > On Wed, Jul 25, 2018 at 3:06 PM Charles Chance > > charles.chance@sipcentric.com wrote: > > Hi, > > > > I have just pushed some changes to master - one of these allows > > startup to continue even if initial node resolution fails. > > > There are some other improvements, too, which I have been
planning
> > to push for some time and which should also help in your situation. > > > Can you try again with these changes applied and let me know the > > outcome? > > > Cheers, > > > > Charles > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
-- Henning Westerholt https://skalatan.de/blog/
-
Aleksandar Sosic -
Charles Chance -
Henning Westerholt -
Paolo Visintin - evosip.cloud