### Description Kamailio is using the wrong source port when relaying INVITE packets with t_relay(). Another interesting thing is that while doing this, Kamailio reports to Homer via HEPv2 that it sent the packet from port 4242, but a tcpdump on the server shows this to be false so maybe homer was just looking at the R-URI port? There is a somewhat random element to this, so it is very difficult to reproduce. I will provide the scenario which led to this problem.
#### Reproduction This is the code block which is responsible for relaying INVITEs to phone from our PBX servers, it is also the only instance of port 5062 being inserted into a SIP message. ``` if(is_method("INVITE")) { if(lookup("location", "sip:$rU@location")) { remove_hf("Contact"); append_hf("Contact: sip:$sel(cfg_get.global.publicIP):5062\r\n"); //A vendor's SIPIS server doesn't work on port 4242, does changing this port break SIP compliance? rtpproxy_manage("cow"); t_relay(); exit; } } ```
- Kamailio listens for UDP+TCP on ports 5060,5062, and 4242. - Phone A registers to tcp:KamailioIP:4242 from tcp:CustomerIP:10042 - Phone B registers to tcp:KamailioIP:4242 from tcp:CustomerIP:11042 - These phones have been re-registering from and to the same ports for a long time without problems with the same REGISTER callID (same transaction so nothing about the connection has changed). - At this point we have TCP connections `tcp:CustomerIP:10042 -> tcp:KamailioIP:4242` and `tcp:CustomerIP:11042 -> tcp:KamailioIP:4242` - Kamailio sends INVITEs from tcp:KamailioIP:4242 to the address of the registered phone with a contact header that tells the phones to send responses to tcp:KamailioIP:5062 (as seen in the above code block). This causes a new TCP connection to be made from the customer's router to Kamailio, just for this transaction. - All is well until phone A sends a response and the customer's router, by pure chance, decides to use port 11042 for the new TCP connection. `tcp:CustomerIP:11042 -> tcp:KamailioIP:5062` - From this point on Kamailio begins to route INVITEs, that were meant for phone B, to phone A using this TCP connection `tcp:CustomerIP:11042 -> tcp:KamailioIP:5062` (source port 5062!) rather than the connection that phone B is actually registered to `tcp:CustomerIP:11042 -> tcp:KamailioIP:4242`. - The customer's router sends packets from this TCP socket to phone A because the connection identified by the Kamailio source port had been created by phone A. - Kamailio is still routing other types of packets to phone B from port 4242 like NOTIFY in response to a subscription and OPTIONs keep alives, but INVITEs (which are an entirely new SIP transaction) are being sent from the wrong source port.
### Possible ~Solutions~ Problems
I have the default tcp_accept_aliases=0, so ;alias in a Via header is not a problem. We are also not using force_send_socket. I've tried to figure out how Kamailio works, and I am wondering if this function might be adding the new TCP connection from phone A to port 5062 as an alias of the already existing TCP connection from phone B to port 4242. https://github.com/kamailio/kamailio/blob/cb7810c939da9c8f4385b530539487528a...
I also found these comments interesting. https://github.com/kamailio/kamailio/blob/52111974b4571e0562e8e731df80f48dbc... https://github.com/kamailio/kamailio/blob/52111974b4571e0562e8e731df80f48dbc...
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.0.5 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 04:38:13 Feb 2 2018 with gcc 4.8.5 ```
* **Operating System**:
``` Linux dallas-sip1-int 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux ```
Closed #1532.
If you refer that the local port of a tcp connection initiated by kamailio is sort of random number, not the port on which kamailio listens, then this is how the TCP stack works. Homer likely takes the internal socket used to receive the packet or the one specific for the interface to send out.
In tcp, it is not possible (or better said, rather impossible) to force local socket for a tcp connection, you can search more on google about, for example a discussion at: * https://superuser.com/questions/1118735/how-are-source-ports-determined-and-...
I am going to close this issue -- if you want to discuss more on this topic, the right place is sr-users@lists.kamailio.org mailing list.
If I misunderstood what you reported and you still think it is an issue of kamailio, reopen and try to describe from another angle, like what it is happening for a specific case and what you expected to happen.
First of all what this article:
- https://superuser.com/questions/1118735/how-are- source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port https://superuser.com/questions/1118735/how-are-source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port
Fails to mention very simple way to force local port number on outgoing tcp connections: the bind(...) call Second why don't kamalio do not re-use an existing connection to a given destination is also interesting question
2018-06-20 14:01 GMT+02:00 Daniel-Constantin Mierla < notifications@github.com>:
If you refer that the local port of a tcp connection initiated by kamailio is sort of random number, not the port on which kamailio listens, then this is how the TCP stack works. Homer likely takes the internal socket used to receive the packet or the one specific for the interface to send out.
In tcp, it is not possible (or better said, rather impossible) to force local socket for a tcp connection, you can search more on google about, for example a discussion at:
source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port https://superuser.com/questions/1118735/how-are-source-ports-determined-and-how-can-i-force-it-to-use-a-specific-port
I am going to close this issue -- if you want to discuss more on this topic, the right place is sr-users@lists.kamailio.org mailing list.
If I misunderstood what you reported and you still think it is an issue of kamailio, reopen and try to describe from another angle, like what it is happening for a specific case and what you expected to happen.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/1532#issuecomment-398724607, or mute the thread https://github.com/notifications/unsubscribe-auth/AF36ZTnPeEFpbsX3WC7CisFNUaLYTXLCks5t-jmzgaJpZM4UAV3q .
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
By default, Kamailio is reusing the tcp connection when possible.
Again, about the local port for tcp connections, read more about tcp protocol and how a connection is identified (some references about ephemeral ports are also there) -- practically it is the tcp stack implementation/operating system kernel that does the selection of the local port, impossible to ensure it from user space.
The ephemeral ports you mention are used ONLY if you do not do BIND call on socket for outgoing connection. If before doing CONNECT you do BIND, the local port number will be one specified in BIND. Following small program demonstrates it.
#!/usr/bin/python import socket import threading
READER_ADDR = ("127.0.0.1", 23008) WRITER1_ADDR = ("127.0.0.1", 23001) WRITER2_ADDR = ("127.0.0.1", 23002)
LISTEN_SOCK = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) LISTEN_SOCK.bind(READER_ADDR) LISTEN_SOCK.listen(3)
DONE = False
class wait_conn(threading.Thread): def run(self): while not DONE: s, a = LISTEN_SOCK.accept() print("Incomming connection from: ", s.getpeername()) s.close()
t = wait_conn() t.start()
w1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) w1.bind(WRITER1_ADDR) w1.connect(READER_ADDR) w1.close()
w2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) w2.bind(WRITER2_ADDR) w2.connect(READER_ADDR) w2.close()
DONE = True
t.join(1.0) LISTEN_SOCK.close()
2018-06-20 15:19 GMT+02:00 Daniel-Constantin Mierla < notifications@github.com>:
By default, Kamailio is reusing the tcp connection when possible.
Again, about the local port for tcp connections, read more about tcp protocol and how a connection is identified (some references about ephemeral ports are also there) -- practically it is the tcp stack implementation/operating system kernel that does the selection of the local port, impossible to ensure it from user space.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/1532#issuecomment-398745383, or mute the thread https://github.com/notifications/unsubscribe-auth/AF36ZSFtK7ZUdmBxJE8DCetWIJGGXa_Lks5t-kvggaJpZM4UAV3q .
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Let me rephrase, Kamailio is using the wrong local TCP socket as the source. It is my understanding that a TCP connection is defined by 4 things: source IP, source port, destination IP, and destination port. A device which has registered to Kamailio over TCP establishes a connection which should be used for all requests from the server. The 'location' table used by the registrar module has a 'socket' column which shows the local socket that the remote device registered to. This is the socket transport:address:port which I would expect Kamailio to use for all INVITEs to the remote device.
The problem we were having was that Kamailio started sending INVITEs from the wrong local socket, then the customer's router couldn't figure out to translate NAT for that socket. This was recreated when the remote device received an INVITE which had a contact header which differed from the sending socket, and caused the device to create a new TCP connection for it's response. For some reason, Kamailio saw this new connection and started sending calls (initial INVITEs in a completely new transaction) to it instead of the actual TCP connection that was used for registration. Another possibly important factor is that the "new" socket created by the remote router for NAT was similar to another device's registration socket. It had the same remote source, so maybe Kamailio matched this with an existing registration for a different device? Device A: tcp:CustomerIP:10042 -> tcp:KamailioIP:4242 Device B: tcp:CustomerIP:11042 -> tcp:KamailioIP:4242 Device A new connection for a 200 OK response: tcp:CustomerIP:11042 -> tcp:KamailioIP:5062
I am not using any functions which update the registration for the device with this new TCP connection, and interestingly other kinds of packets like NOTIFY in response to a subscription and OPTIONs keep alives continue to use the registration socket. I think it is somehow internal to Kamailio.
Maybe you can attach a pcap with the traffic for this case, from the REGISTER sent to Kamailio to the INVITE exposing the issue.
There are some aspects related to SIP that you confuse or maybe you didn't explain it properly above. The Contact header in the INVITE is not the socket of Kamailio because Kamailio is not generating the INVITE. If the device is behind the NAT, the sip server cannot connect to it, unless the nat router is a port forwarding firewall.
Kamailio is reusing the tcp connection whenever the destination address is matching an active one. The pcap with the sip traffic should clarify what happens.
Obviously the difference is that kamailio does bind() and listen() on the socket (ip:port) first -- maybe that should have said to be clear about what case we talk here. So try with socket(), bind(), listen() and then connect().
Once there is listen() on a socket, read events for it mean accept() should be done. The kernel won't create an outgoing connection from a listen socket, but use ephemeral ports.
Well i don't want to be nit picking however, i believe you're mistaken here. You can reuse port on which you are listening for outgoing connections too. You need to use* setsockopt(... SO_REUSEPORT ).*.. here goes the demo:
#!/usr/bin/python3 import socket import threading
READER1_ADDR = ("127.0.0.1", 23008) READER2_ADDR = ("127.0.0.1", 23009)
class wait_conn(threading.Thread): def __init__(self, addr, name):
super(wait_conn,self).__init__()
self.name = name self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) self.sock.bind(addr) self.sock.listen(3)
def run(self): s, a = self.sock.accept() print(self.name, ": Incomming connection from: ", s.getpeername()) s.close() self.sock.close()
t1 = wait_conn(READER1_ADDR, "r1") t1.start() t2 = wait_conn(READER2_ADDR, "r2") t2.start()
w1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) w1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1) w1.bind(READER1_ADDR) w1.connect(READER2_ADDR) w1.close()
t1.join(1.0) t2.join(1.0)
2018-06-21 21:27 GMT+02:00 Daniel-Constantin Mierla < notifications@github.com>:
Obviously the difference is that kamailio does bind() and listen() on the socket (ip:port) first -- maybe that should have said to be clear about what case we talk here. So try with socket(), bind(), listen() and then connect().
Once there is listen() on a socket, read events for it mean accept() should be done. The kernel won't create an outgoing connection from a listen socket, but use ephemeral ports.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kamailio/kamailio/issues/1532#issuecomment-399216138, or mute the thread https://github.com/notifications/unsubscribe-auth/AF36ZceX6KXDz8U4aUKDqzbkS-kEvCLiks5t-_OKgaJpZM4UAV3q .
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Yikes, I don't have pcaps of this issue anymore and it is difficult to reproduce because it involves a random collision of port numbers on the customer's network with NAT involved. On top of that I seem to be having trouble getting my issue across.
In this case I have two connections created by phones with the same destination but different sources, then a third which has a the same source but different destination. Device A: tcp:CustomerIP:10042 -> `tcp:KamailioIP:4242` Device B: `tcp:CustomerIP:11042` -> `tcp:KamailioIP:4242` Device A new connection for a 200 OK response: `tcp:CustomerIP:11042` -> tcp:KamailioIP:5062
Kamailio sees this as: Device A: `tcp:KamailioIP:4242` -> tcp:CustomerIP:10042 Device B: `tcp:KamailioIP:4242` -> `tcp:CustomerIP:11042` Device A receives a SIP message with a contact telling them to respond to Kamailio on 5062, which causes it to create a new TCP connection which in turn confuses Kamailio. Device A tcp:KamailioIP:5062 -> `tcp:CustomerIP:11042`
Kamailio is reusing the tcp connection whenever the destination address is matching an active one.
As seen above, Kamailio thinks Device A has two connections with the same destination. This invite is not in response to any packet, it is being relayed from my PBX as the first packet of a new transaction and I am using lookup() and then t_relay() to set the Request-URI and then relay the packet to that destination. If t_relay uses the R-URI destination to find the TCP socket it should send from, how does it choose between these two sockets? The problem I am seeing is that it incorrectly chooses to send the INVITE from the `socket' that has a source port of 5062 because the newest connection it has become aware of for that destination is the socket with 5062.
I hope I've been able to clearly describe what, I think, is happening here. If not, we currently have a mitigation strategy to avoid sending a packet with a contact that makes phones want to create another connection to Kamailio.
It is very difficult to recreate because it involved the random element of a customer's router using the same originating TCP port for two connections, which is technically correct for TCP. I am seeing an INVITE meant for Device A go to Device B because there are TCP connections for each which share the same tcp:IP:port destination. I would expect that Kamailio should try to use the TCP socket (defined by the 4 numbers srcIP, srcPort, dstIP, destPort) which was used by device A for it's registration, not the new one which had been created by device.
Vadim Lebedev - you are chasing the wrong direction here. If you look at the docs, there is a global parameter tcp_reuse_port for this purpose, setting SO_REUSEPORT if available in the OS. However, the discussion here is in the context of a collision of ports as mentioned again by @Forty-Tw0 in the comment:
* https://github.com/kamailio/kamailio/issues/1532#issuecomment-399254671
So to conclude:
* kamailio is reusing the connection from REGISTER to send back traffic to the device, it is no other way if the device is behind a non-port forwarding NAT (e.g., symmetric NAT) * if it doesn't, then the pcap should help to see if the INVITE created some other connection or uses a different Contact address so the REGISTER connection is not matching anymore
The reason I closed here is because this seems like a specific use case issue that can be discussed on sr-users once we see pcaps, rather than an issue in the code. Based on conclusion there, we can chase futher the code or config options. We do not use issue tracker as a discussion forum to decide if it is a bug or not, the issue here have to be opened when it is clear a bug in the code (or a feature request).
@Forty-Tw0 - as suggested, let's discuss this use case scenario on sr-users mailing list. Again, there are many aspects that might not use a clear terminology, like `Device A receives a SIP message with a contact telling them to respond to Kamailio on 5062, which causes it to create a new TCP connection` -- what to respond on 5062? The SIP response or follow up requests? These are either Via or Record-Route or Contact headers ... so a lot to clarify and can be just config things that needs to be tuned, sr-users is the right place to discuss and get to a conclusion of how it can be fixed.
-
Daniel-Constantin Mierla -
Jared Hull
-
Vadim Lebedev