missing BYE when 2 redundant kamailio servers share the same database
Hi, all I'm trying to set up 2 kamailio servers for active-active redundancy. The two kamailio severs share the the same database with db_mode=3, and no registration replication. Use pjsua2 as SIP client for testing. The test setup is as follows: kamailio server 1(k1): 10.0.1.30:5061 <http://10.0.1.30:5061/> kamailio server 2(k2): 10.0.1.32:5061 <http://10.0.1.32:5061/> sip client c1: sip:16317@10.0.1.30 <mailto:sip%3A16317@10.0.1.30>, client ip: 10.0.1.254 sip client c2: sip:72316@10.0.1.30 <mailto:sip%3A72316@10.0.1.30>, client ip: 10.0.1.254 c1 is registered with k1. c2 is registered with k2. When c1 calls c2, the call flow is as follows: INVITE: c1------>k1------>k2------>c2 200 OK: c1<------k1<------k2<------c2 So far, the INVITE works. c1 and c2 can establish a call. However, when c2 hangs up and sends BYE, the BYE is forwarded all the way to k1, and then k1 gives a 500 server error. The log on k1 seems to indicate k1 was attempting to create a TLS connection to itself instead of using the existing TLS connection to c1 to forward the BYE. BYE: c1 X k1<------k2<------c2 | |------->Connection refused/500 Here is the kamailio log on k1: Mar 16 10:34:21 kamailio.sip.com <http://kamailio.sip.com/> /usr/sbin/kamailio[13396]: ERROR: <core> [tcp_main.c:2740]: tcpconn_1st_send(): connect 10.0.1.30:55999 <http://10.0.1.30:55999/> failed (RST) Connection refused Mar 16 10:34:21 kamailio.sip.com <http://kamailio.sip.com/> /usr/sbin/kamailio[13396]: ERROR: <core> [tcp_main.c:2750]: tcpconn_1st_send(): 10.0.1.30:55999 <http://10.0.1.30:55999/>: connect & send for 0x7fc96a68a1a0 failed: Connection refused (111) This is the routing logic for BYE in kamailio.cfg: if (loose_route()) { route(DLGURI); if (is_method("BYE")) { xlog("L_DBG", "=====BYE $ru from $fu $si:$sp to $du=====\n"); dlg_manage(); setflag(FLT_ACC); # do accounting ... setflag(FLT_ACCFAILED); # ... even if the transaction fails } else if ( is_method("ACK") ) { # ACK is forwarded statelessy route(NATMANAGE); } else if ( is_method("NOTIFY") ) { # Add Record-Route for in-dialog NOTIFY as per RFC 6665. record_route(); } route(RELAY); exit; } else {...... If I add location based routing for BYE from peer kamailio, then the BYE seems to be forwarded correctly: if (loose_route()) { route(DLGURI); if (is_method("BYE")) { xlog("L_DBG", "=====BYE $ru from $fu $si:$sp to $du=====\n"); dlg_manage(); setflag(FLT_ACC); # do accounting ... setflag(FLT_ACCFAILED); # ... even if the transaction fails # If BYE coming from Kamailio peer, route BYE by location $var(peerlist)=$sel(cfg_get.trusted.peers); $var(i) = 0; while($var(i)<$(var(peerlist){param.count})) { xlog("L_DBG", "=====$(var(peerlist){param.count})=====$(var(peerlist){param.valueat,$var(i)})=====\n"); if(src_ip==$(var(peerlist){param.valueat,$var(i)})) { lookup("location"); xlog("L_DBG", "=====BYE from $fu $si:$sp to $du=====\n"); break; } $var(i) = $var(i) + 1; } } else if ( is_method("ACK") ) { # ACK is forwarded statelessy route(NATMANAGE); } else if ( is_method("NOTIFY") ) { # Add Record-Route for in-dialog NOTIFY as per RFC 6665. record_route(); } route(RELAY); exit; } else {...... My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug? My next question is whether the above location routing for BYE from peer kamailio a good/safe approach. The SIP traces will be sent later to avoid exceeding email size limit. Thanks
Hello, On 19/03/15 02:54, Ding Ma wrote:
[...]
My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug?
can you get the pcap of such call? We have to see the routing headers to say what is next hop address. Are all the requests within dialog routed via same instance of kamailio?
My next question is whether the above location routing for BYE from peer kamailio a good/safe approach.
The SIP traces will be sent later to avoid exceeding email size limit.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
We use TLS for SIP. The Wireshark pcap would be encrypted. I’ll try to get a pcap anyway. Wonder if there is a way to dump pcap from inside kamailio. All the requests within dialog are routed through 2 kamailio instances. We want to make sure each phone only sends requests through its registrar. I have included pjsua logs in subsequent emails in this thread. Those logs have SIP messages, but only provide client perspective. Thanks for the help, Ding
On Mar 20, 2015, at 3:00 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,
On 19/03/15 02:54, Ding Ma wrote:
[...]
My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug?
can you get the pcap of such call? We have to see the routing headers to say what is next hop address.
Are all the requests within dialog routed via same instance of kamailio?
My next question is whether the above location routing for BYE from peer kamailio a good/safe approach.
The SIP traces will be sent later to avoid exceeding email size limit.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#!/miconda> - http://www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com <http://www.kamailioworld.com/>
We use TLS for SIP. The Wireshark pcap would be encrypted. I’ll try to get a pcap anyway. Wonder if there is a way to dump pcap from inside kamailio. Wireshark can decrypt SIP signalling sent over TLS connections if you provide server's private key to it.
All the requests within dialog are routed through 2 kamailio instances. We want to make sure each phone only sends requests through its registrar.
I have included pjsua logs in subsequent emails in this thread. Those logs have SIP messages, but only provide client perspective.
Thanks for the help,
Ding
On Mar 20, 2015, at 3:00 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
On 19/03/15 02:54, Ding Ma wrote:
[...]
My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug?
can you get the pcap of such call? We have to see the routing headers to say what is next hop address.
Are all the requests within dialog routed via same instance of kamailio?
My next question is whether the above location routing for BYE from peer kamailio a good/safe approach.
The SIP traces will be sent later to avoid exceeding email size limit.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda -http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany -http://www.kamailioworld.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
An alternative would be to test with tcp, from routing point of view should be the same if it is udp-tcp or udp-tls. Cheers, Daniel On 20/03/15 15:09, Vitaliy Aleksandrov wrote:
We use TLS for SIP. The Wireshark pcap would be encrypted. I’ll try to get a pcap anyway. Wonder if there is a way to dump pcap from inside kamailio. Wireshark can decrypt SIP signalling sent over TLS connections if you provide server's private key to it.
All the requests within dialog are routed through 2 kamailio instances. We want to make sure each phone only sends requests through its registrar.
I have included pjsua logs in subsequent emails in this thread. Those logs have SIP messages, but only provide client perspective.
Thanks for the help,
Ding
On Mar 20, 2015, at 3:00 AM, Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>> wrote:
Hello,
On 19/03/15 02:54, Ding Ma wrote:
[...]
My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug?
can you get the pcap of such call? We have to see the routing headers to say what is next hop address.
Are all the requests within dialog routed via same instance of kamailio?
My next question is whether the above location routing for BYE from peer kamailio a good/safe approach.
The SIP traces will be sent later to avoid exceeding email size limit.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
Hi, all It took some time to clean up the pjsua settings and collect the pcaps for both cases. Here are the two pcap files. call_failed_bye.pcap has the call trace when the BYE is handled with the following logic. route[WITHINDLG] { if (has_totag()) { if (loose_route()) { route(DLGURI); if (is_method("BYE")) { xlog("L_DBG", "=====BYE $ru from $fu $si:$sp to $du=====\n"); dlg_manage(); setflag(FLT_ACC); # do accounting ... setflag(FLT_ACCFAILED); # ... even if the transaction fails } } .... route(RELAY); This generated a 500 error. call_extra_bye.pcap is the call trace when the BYE is handled like this: route[WITHINDLG] { if (has_totag()) { if (loose_route()) { route(DLGURI); if (is_method("BYE")) { xlog("L_DBG", "=====BYE $ru from $fu $si:$sp to $du=====\n"); dlg_manage(); setflag(FLT_ACC); # do accounting ... setflag(FLT_ACCFAILED); # ... even if the transaction fails # If BYE is coming from Kamailio peer, route BYE by location $var(peerlist)=$sel(cfg_get.trusted.peers); $var(i) = 0; while($var(i)<$(var(peerlist){param.count})) { xlog("L_DBG", "=====$(var(peerlist){param.count})=====$(var(peerlist){param.valueat,$var(i)})=====\n"); if(src_ip==$(var(peerlist){param.valueat,$var(i)})) { lookup("location"); xlog("L_DBG", "=====BYE from $fu $si:$sp to $du=====\n"); break; } $var(i) = $var(i) + 1; } } ... route(RELAY); This got the BYE and 200 OK for the BYE. But there was an extra BYE at the end, which resulted in 481 Call/Transaction Does Not Exist. Wonder if the 481 was caused by pjsua not sending ACK for the 200 OK of BYE. Just to recap the use case. We're trying to set up two kamailio servers sharing the same database(db_mode=3). kamailio server 1(k1): 10.0.1.30:5060 kamailio server 2(k2): 10.0.1.32:5060 sip client c1: sip:16317@10.0.1.30, client ip: 10.0.1.254 sip client c2: sip:72316@10.0.1.30, client ip: 10.0.1.254 c1 is registered with k1. c2 is registered with k2. The expected call flow is as follows: When c1 calls c2, the expected call flow is as follows: INVITE: c1------>k1------>k2------>c2 200 OK: c1<------k1<------k2<------c2 BYE c1<------k1<------k2<------c2 But we have the above-mentioned issue with BYE handling at k1. Thanks Ding On Fri, Mar 20, 2015 at 1:08 PM, Daniel-Constantin Mierla <miconda@gmail.com
wrote:
An alternative would be to test with tcp, from routing point of view should be the same if it is udp-tcp or udp-tls.
Cheers, Daniel
On 20/03/15 15:09, Vitaliy Aleksandrov wrote:
We use TLS for SIP. The Wireshark pcap would be encrypted. I’ll try to get a pcap anyway. Wonder if there is a way to dump pcap from inside kamailio.
Wireshark can decrypt SIP signalling sent over TLS connections if you provide server's private key to it.
All the requests within dialog are routed through 2 kamailio instances. We want to make sure each phone only sends requests through its registrar.
I have included pjsua logs in subsequent emails in this thread. Those logs have SIP messages, but only provide client perspective.
Thanks for the help,
Ding
On Mar 20, 2015, at 3:00 AM, Daniel-Constantin Mierla <miconda@gmail.com> wrote:
Hello,
On 19/03/15 02:54, Ding Ma wrote:
[...]
My first question is why k1 loose_route sends the BYE to itself instead of the client. Is this a bug?
can you get the pcap of such call? We have to see the routing headers to say what is next hop address.
Are all the requests within dialog routed via same instance of kamailio?
My next question is whether the above location routing for BYE from peer kamailio a good/safe approach.
The SIP traces will be sent later to avoid exceeding email size limit.
Cheers, Daniel
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing listsr-users@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
participants (3)
-
Daniel-Constantin Mierla -
Ding Ma -
Vitaliy Aleksandrov