Am Donnerstag, 2. August 2018, 20:33:28 CEST schrieb Paolo Visintin - evosip.cloud:
I am able to see 0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
but in tcpdump no DNS query
Also doing a loop of "kamcmd dns.lookup A dmq-router-service.paolo.svc.c luster.local" and starting kamailio at the same time the result is always the correct ip, no 42.100.109.113 coming up so this is very strange!
Hello,
without looking too much into that, I have two remarks which may worth investigating:
- for DNS names in the configuration file Kamailio does a lookup during startup and "compiles" this to structure which is evaluated during run time
- there is also a DNS caching functionality, which could cause something like you observed. You can deactivate this with a core variable, I think.
Best regards,
Henning
2018-08-02 15:29 GMT+02:00 Charles Chance charles.chance@sipcentric.com:
Hi Paolo,
kamailio "resolves" A host but does not query the DNS
What do you mean by this?
I am not overly familiar with Kamailio's DNS internals, but what do you see if you run the following commands?
kamcmd dns.lookup SRV _sip._udp.dmq-router-service.paolo.svc.cluster.local
kamcmd dns.lookup A dmq-router-service.paolo.svc.cluster.local
Cheers,
Charles
On 2 August 2018 at 13:27, Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud> wrote:
Hello again! Another interesting thing found running in debug=3
0(710) DEBUG: dmq [notification_peer.c:240]: get_dmq_host_list(): adding DMQ node A host dmq-router-service=sip:42.100.109.113:5062
so , kamailio "resolves" A host but does not query the DNS this is dump of dns request inside kamailio container:
147 0.597474 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0xf31d SRV dmq-router-service.paolo.svc.cluster.local 151 0.598306 10.96.0.10 → 10.42.1.251 DNS 155 Standard query response 0xf31d SRV dmq-router-service.paolo.svc.cluster.local SRV 10 100 0 3934376365383035.dmq-router-service.paolo.svc.cluster.local A 172.28.0.240 156 0.598538 10.42.1.251 → 10.96.0.10 DNS 102 Standard query 0x1904 A dmq-router-service.paolo.svc.cluster.local 163 0.599275 10.96.0.10 → 10.42.1.251 DNS 118 Standard query response 0x1904 A dmq-router-service.paolo.svc.cluster.local A 172.28.0.240
*Paolo Visintin* *CTO* evosip.cloud [image: Risultati immagini per evosip]
2018-08-02 0:24 GMT+02:00 Paolo Visintin - evosip.cloud <
paolo.visintin@evosip.cloud>:
Hello Charles, sorry for late reply, we've had some issues rebuilding docker images with kamailo nightly build!
We are running on our local, private on premise kubernetes cluster so connections from outside is simply impossible (in dev we are using a "closed" network system)
I've just tried to :
- open a tcpdump trace in the instance tracing only the DMQ port defined
[5062] - (tcpdump -i eth0 port 5062)
- packet coming from nowhere
- started kamailio ($kamailio -f $PATH_KAMAILIO_CFG -m 512 -DD -E -e)
- seeing this in tcpdump :
root@router-0:/# tcpdump -i eth0 port 5062 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 22:06:44.966991 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:45.572084 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497 22:06:46.571875 IP router-0.router.paolo.svc.cluster.local.5062 > 42.100.109.113.5062: UDP, length 497
so seems that kamailio is trying to contact the foreign ip
if i check inside the configuration: modparam("dmq", "server_address", DMQ_SERVER_ADDRESS) modparam("dmq", "notification_address", DMQ_NOTIFICATION_ADDRESS) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 15)
...
if(is_method("KDMQ") && dst_port == DMQ_PORT){
dmq_handle_message(); exit;}
and in DMQ_NOTIFICATION_ADDRESS I have : #!define DMQ_NOTIFICATION_ADDRESS "sip:dmq-router-service:5062"
dmq-router-service DNS resolves:
dmq-router-service.paolo.svc.cluster.local
Server: 10.96.0.10 Address: 10.96.0.10#53
Non-authoritative answer: Name: dmq-router-service.paolo.svc.cluster.local Address: 172.28.0.240
so -nowhere- i am able to find 42.100.109.113
and in kamailio log: 13(821) ERROR: dmq [notification_peer.c:599]: notification_resp_callback_f(): deleting server sip:42.100.109.113:5062 because of failed request
Cheers,
Paolo
2018-07-26 18:14 GMT+02:00 Charles Chance <charles.chance@sipcentric.com
Hi Aleksandar,
The initial depopulation of the nodes (following a period of 'pending' state) is due to no response being received from them. Are you able to trace the messages to/from one of them to confirm what is happening there?
As for the unrecognised IP, I'm afraid I can't answer that one. It must be coming from somewhere - it's just a case of finding out where. Following the initial population/depopulation of the node list, DMQ simply repeats the process over and over until it eventually connects to one or more other nodes. Are you running on a public cloud somewhere or is it your own, locally running Kubernetes cluster?
Cheers,
Charles
On 26 July 2018 at 15:30, Aleksandar Sosic alex.sosic@evosip.cloud
wrote:
Hi Charles,
We're so glad about the improvements you just committed! Thanks!
Now I'm using the latest nightly: 5.2.0~dev6+0~20180726010431.11 65+xenial Kamailio starts even if the DNS record does not exist at first, that's great. I'm having this nodes up and running:
proxy-66f79498cc-8ws6d 3/3 Running 0 4m 172.28.1.4 proxy-66f79498cc-b68dd 3/3 Running 0 4m 172.28.1.5 proxy-66f79498cc-lhbxg 3/3 Running 0 4m 172.28.1.6 proxy-66f79498cc-png6p 3/3 Running 0 4m 172.28.1.7In those 4-5 minutes I've experienced a population and depopulation of the dmq nodes.
For example:
kamcmd> dmq.list_nodes { host: 172.28.1.4 port: 5061 resolved_ip: 172.28.1.4 status: pending last_notification: 0 local: 0 } { host: 172.28.1.6 port: 5061 resolved_ip: 172.28.1.6 status: pending last_notification: 0 local: 0 } { host: 40.100.109.113 port: 5061 resolved_ip: 40.100.109.113 status: pending last_notification: 0 local: 0 } { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And after a few seconds only the loca node itself:
kamcmd> dmq.list_nodes { host: 172.28.1.5 port: 5061 resolved_ip: 172.28.1.5 status: active last_notification: 0 local: 1 }And then again the 3 nodes and the 40.100.109.113 which is a Microsoft IP and I don't know what's doing that IP into my dmq server list.
The DNS record is this:
root@proxy-66f79498cc-b68dd:/# nslookup dmq-proxy-service Server: 10.96.0.10 Address: 10.96.0.10#53 Non-authoritative answer: Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.7 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.6 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.5 Name: dmq-proxy-service.alex.svc.cluster.local Address: 172.28.1.4Any clues why that ip is in the dmq nodes list? And why .7 is missing? Kind regards, -- Aleksandar Sosic mail: alex.sosic@evosip.cloud
On Wed, Jul 25, 2018 at 3:06 PM Charles Chance
charles.chance@sipcentric.com wrote: > Hi, > > I have just pushed some changes to master - one of these allows
startup to continue even if initial node resolution fails.
> There are some other improvements, too, which I have been planning
to push for some time and which should also help in your situation.
> Can you try again with these changes applied and let me know the
outcome?
> Cheers, > > Charles
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- *Charles Chance* Managing Director
t. 0330 120 1200 m. 07932 063 891
Sipcentric Ltd. Company registered in England & Wales no. 7365592. Registered office: Faraday Wharf, Innovation Birmingham Campus, Holt Street, Birmingham Science Park, Birmingham B7 4BB.