For proper clarification -- so if the port is not present, then the match is made on IP and it may select a socket that has that ip but at different port. Right?
That's indeed something to be fixed, however, i think that when protocol is tls (and maybe wss), then the 5061 port should be tried.