In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] { # no connect for sending replies set_reply_no_connect(); if(has_totag()) { # no connect for requests within dialog set_forward_no_connect(); }
This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] { # no connect for sending replies set_reply_no_connect(); if(has_totag()) { # no connect for requests within dialog set_forward_no_connect(); }
This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I still ran into some trouble when one side was NAT'd.
Am I correct in thinking that it would be undesirable to maintain a TCP/TLS connection to an upstream TLS gateway that is using the well-known port 5061?
I was thinking this may be a case for TCP_REUSEPORT and force_send_socket, but that seems a little complex seeing as I can just let Kamailio reconnect (when necessary) rather than preventing the outbound TLS from connecting when it would otherwise succeed.
I'll try and work through more detailed configuration scenarios. -A
On Monday, September 9, 2019 2:12:07 AM CDT Daniel-Constantin Mierla wrote:
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] {
# no connect for sending replies set_reply_no_connect(); if(has_totag()) { # no connect for requests within dialog set_forward_no_connect(); }This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
It no longer looks like an issue related to set_reply_no_connect() or set_forward_no_connect() added by the commit you referenced. Those were added to prevent attempting to connect to devices behind the nat (in that case the device has to maintain the connection, otherwise nobody can connect back to it) as well as prevent someone in the wild sending a request then closing the connection, without waiting for the reply, which is typically routed back to via, commonly with an ephemeral port.
The follow up commit I did it in master recently is no longer setting the flag for requests within dialog, but I understand you have connection problems for requests within dialog, am I right?
Cheers, Daniel
On 10.09.19 01:08, Anthony Joseph Messina wrote:
I still ran into some trouble when one side was NAT'd.
Am I correct in thinking that it would be undesirable to maintain a TCP/TLS connection to an upstream TLS gateway that is using the well-known port 5061?
I was thinking this may be a case for TCP_REUSEPORT and force_send_socket, but that seems a little complex seeing as I can just let Kamailio reconnect (when necessary) rather than preventing the outbound TLS from connecting when it would otherwise succeed.
I'll try and work through more detailed configuration scenarios. -A
On Monday, September 9, 2019 2:12:07 AM CDT Daniel-Constantin Mierla wrote:
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] {
# no connect for sending replies set_reply_no_connect(); if(has_totag()) { # no connect for requests within dialog set_forward_no_connect(); }This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
I'm going to keep testing against the issue I originally reported, and probably wait until after 5.3 is released. My issue may also be related to a combination of TCPOPS keepalive not keeping the proper connection open for
UAC -> Kamailio/LCR -> TLS Gateway
The connection that's kept open to the TLS Gateway is the original forward of the INVITE
<Kamailio IP>:<ephemeral port> -> <TLS Gateway>:<Port 5061>
The subsequent in-dialog connections (such as BYE from the UAC to the TLS Gateway) don't use the original TLS connection so they are prevented from re- connecting to the TLS Gateway.
Again, I have to do more testing to clear up the root issue on my end.
Also, for a more compact config, would the following achieve the same thing...
route[NATMANAGE] { #!ifdef WITH_NAT if(is_request()) { if(has_totag()) { if(check_route_param("nat=yes")) { setbflag(FLB_NATB); ### Add the command here.... set_forward_no_connect(); } } }
...
On Wednesday, September 11, 2019 1:21:34 AM CDT Daniel-Constantin Mierla wrote:
It no longer looks like an issue related to set_reply_no_connect() or set_forward_no_connect() added by the commit you referenced. Those were added to prevent attempting to connect to devices behind the nat (in that case the device has to maintain the connection, otherwise nobody can connect back to it) as well as prevent someone in the wild sending a request then closing the connection, without waiting for the reply, which is typically routed back to via, commonly with an ephemeral port.
The follow up commit I did it in master recently is no longer setting the flag for requests within dialog, but I understand you have connection problems for requests within dialog, am I right?
Cheers, Daniel
On 10.09.19 01:08, Anthony Joseph Messina wrote:
I still ran into some trouble when one side was NAT'd.
Am I correct in thinking that it would be undesirable to maintain a TCP/TLS connection to an upstream TLS gateway that is using the well-known port 5061?
I was thinking this may be a case for TCP_REUSEPORT and force_send_socket, but that seems a little complex seeing as I can just let Kamailio reconnect (when necessary) rather than preventing the outbound TLS from connecting when it would otherwise succeed.
I'll try and work through more detailed configuration scenarios. -A
On Monday, September 9, 2019 2:12:07 AM CDT Daniel-Constantin Mierla
wrote:
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] {
# no connect for sending replies set_reply_no_connect(); if(has_totag()) {
# no connect for requests within dialog set_forward_no_connect();
}
This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
On 15.09.19 03:34, Anthony Joseph Messina wrote:
I'm going to keep testing against the issue I originally reported, and probably wait until after 5.3 is released. My issue may also be related to a combination of TCPOPS keepalive not keeping the proper connection open for
UAC -> Kamailio/LCR -> TLS Gateway
The connection that's kept open to the TLS Gateway is the original forward of the INVITE
<Kamailio IP>:<ephemeral port> -> <TLS Gateway>:<Port 5061>
The subsequent in-dialog connections (such as BYE from the UAC to the TLS Gateway) don't use the original TLS connection so they are prevented from re- connecting to the TLS Gateway.
Again, I have to do more testing to clear up the root issue on my end.
Also, for a more compact config, would the following achieve the same thing...
route[NATMANAGE] { #!ifdef WITH_NAT if(is_request()) { if(has_totag()) { if(check_route_param("nat=yes")) { setbflag(FLB_NATB); ### Add the command here.... set_forward_no_connect(); } } }
...
You have to differentiate between calls with one side behind nat and the other one on a pubic IP that is like a server/gateway and can accept new connections, even for requests within dialog.
My initial change to the default config file was done in the perspective that the respective configuration is routing between local users, where is not common for a user device to register, then close the connection, because it was using a ephemeral port anyhow.
Cheers, Daniel
On Wednesday, September 11, 2019 1:21:34 AM CDT Daniel-Constantin Mierla wrote:
It no longer looks like an issue related to set_reply_no_connect() or set_forward_no_connect() added by the commit you referenced. Those were added to prevent attempting to connect to devices behind the nat (in that case the device has to maintain the connection, otherwise nobody can connect back to it) as well as prevent someone in the wild sending a request then closing the connection, without waiting for the reply, which is typically routed back to via, commonly with an ephemeral port.
The follow up commit I did it in master recently is no longer setting the flag for requests within dialog, but I understand you have connection problems for requests within dialog, am I right?
Cheers, Daniel
On 10.09.19 01:08, Anthony Joseph Messina wrote:
I still ran into some trouble when one side was NAT'd.
Am I correct in thinking that it would be undesirable to maintain a TCP/TLS connection to an upstream TLS gateway that is using the well-known port 5061?
I was thinking this may be a case for TCP_REUSEPORT and force_send_socket, but that seems a little complex seeing as I can just let Kamailio reconnect (when necessary) rather than preventing the outbound TLS from connecting when it would otherwise succeed.
I'll try and work through more detailed configuration scenarios. -A
On Monday, September 9, 2019 2:12:07 AM CDT Daniel-Constantin Mierla
wrote:
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] {
# no connect for sending replies set_reply_no_connect(); if(has_totag()) {
# no connect for requests within dialog set_forward_no_connect();
}
This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Ok, Daniel. You were correct in pointing out my improper placement of the set_forward_no_connect().
I was trying to find a way to insert the command in the most efficient place. I have been testing with various nat/no-nat situations and think the following will do what it should and avoid the additional calls to "if(is_request())" and "if(has_totag())":
route[NATMANAGE] { #!ifdef WITH_NAT if(is_request()) { if(has_totag()) { if(check_route_param("nat=yes")) { setbflag(FLB_NATB); }
##----->INSERT HERE: no connect message in a dialog involving NAT traversal if(isbflagset(FLB_NATB)) { set_forward_no_connect(); } } } if(!(isflagset(FLT_NATS) || isbflagset(FLB_NATB))) return;
# We are sending everything to route(RTPENGINE_MANAGE) separately # for more specific handling and bridging, not only based on NAT # route(RTPENGINE_MANAGE);
if(is_request()) { if(!has_totag()) { if(t_is_branch_route()) { add_rr_param(";nat=yes"); } } } if(is_reply()) { if(isbflagset(FLB_NATB)) { if(is_first_hop()) { # Skip for GRUU if(@contact.uri!="" && !is_gruu(@contact.uri)) set_contact_alias(); } } }
# if(isbflagset(FLB_NATB)) { # if(is_request()) { # if(has_totag()) { # set_forward_no_connect(); # } # } # } #!endif return; }
On Wednesday, September 18, 2019 1:50:46 AM CDT Daniel-Constantin Mierla wrote:
On 15.09.19 03:34, Anthony Joseph Messina wrote:
I'm going to keep testing against the issue I originally reported, and probably wait until after 5.3 is released. My issue may also be related to a combination of TCPOPS keepalive not keeping the proper connection open for
UAC -> Kamailio/LCR -> TLS Gateway
The connection that's kept open to the TLS Gateway is the original forward of the INVITE
<Kamailio IP>:<ephemeral port> -> <TLS Gateway>:<Port 5061>
The subsequent in-dialog connections (such as BYE from the UAC to the TLS Gateway) don't use the original TLS connection so they are prevented from re- connecting to the TLS Gateway.
Again, I have to do more testing to clear up the root issue on my end.
Also, for a more compact config, would the following achieve the same thing...
route[NATMANAGE] { #!ifdef WITH_NAT
if(is_request()) { if(has_totag()) { if(check_route_param("nat=yes")) { setbflag(FLB_NATB);### Add the command here....
set_forward_no_connect(); } } }...
You have to differentiate between calls with one side behind nat and the other one on a pubic IP that is like a server/gateway and can accept new connections, even for requests within dialog.
My initial change to the default config file was done in the perspective that the respective configuration is routing between local users, where is not common for a user device to register, then close the connection, because it was using a ephemeral port anyhow.
Cheers, Daniel
On Wednesday, September 11, 2019 1:21:34 AM CDT Daniel-Constantin Mierla
wrote:
It no longer looks like an issue related to set_reply_no_connect() or set_forward_no_connect() added by the commit you referenced. Those were added to prevent attempting to connect to devices behind the nat (in that case the device has to maintain the connection, otherwise nobody can connect back to it) as well as prevent someone in the wild sending a request then closing the connection, without waiting for the reply, which is typically routed back to via, commonly with an ephemeral port.
The follow up commit I did it in master recently is no longer setting the flag for requests within dialog, but I understand you have connection problems for requests within dialog, am I right?
Cheers, Daniel
On 10.09.19 01:08, Anthony Joseph Messina wrote:
I still ran into some trouble when one side was NAT'd.
Am I correct in thinking that it would be undesirable to maintain a TCP/TLS connection to an upstream TLS gateway that is using the well-known port 5061?
I was thinking this may be a case for TCP_REUSEPORT and force_send_socket, but that seems a little complex seeing as I can just let Kamailio reconnect (when necessary) rather than preventing the outbound TLS from connecting when it would otherwise succeed.
I'll try and work through more detailed configuration scenarios. -A
On Monday, September 9, 2019 2:12:07 AM CDT Daniel-Constantin Mierla
wrote:
Hello,
I relaxed that condition to not connect on forwarding only for initial requests going though nat. Can you test with latest master and see how is going for your use case?
Cheers, Daniel
On 09.09.19 02:00, Anthony Joseph Messina wrote:
In preparation for the 5.3 release, I've been testing the following configuration change for TCP/TLS connections:
https://github.com/kamailio/kamailio/commit/ 8bba208fe6ae7ccb4c92362b8c33f1530b9f56da
route[REQINIT] {
# no connect for sending replies set_reply_no_connect(); if(has_totag()) {
# no connect for requests within dialog set_forward_no_connect();
}
This change creates issues when a UAC TLS INVITE routes to an upstream gateway using TLS to port 5061 (via the LCR module). Kamailio sends the initial outbound TLS connection from a local ephemeral port. The TCPOPS tcp_keepalive_enable function issues keepalives from the local ephemeral port to the gateway port 5061:
https://kamailio.org/docs/modules/stable/modules/ tcpops#tcpops.f.tcp_keepalive_enable
Even so, the TLS connection eventually times out, after which in-dialog requests from the UAC are no longer able to reach the upstream gateway.
ERROR: tm [../../core/forward.h:293]: msg_send_buffer(): tcp_send failed WARNING: tm [t_fwd.c:1570]: t_send_branch(): sending request on branch 0 failed ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL)
I figure I must be doing something wrong with my TCPOPS here. Is a TLS connection to an upstream gateway supposed to be maintained throughout the duration of a call?
-
Anthony Joseph Messina -
Daniel-Constantin Mierla