Hi,
I noticed a strange behavior on some of our proxy servers, all running Kamailio 5.3.8. After running for some time (weeks), our monitoring system sporadically starts reporting errors. The check connects via tls and registers to an Asterisk behind the proxy server. When this happens, the Kamailio log shows the following line:
ERROR: tls [tls_util.h:42]: tls_err_ret(): TLS accept:error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error
When restarting Kamailio, the problem goes away only to come back after some weeks uptime again.
On one host, I tried to find something using kamcmd, and I don't know why but I also issued "tls.reload". And from that point, the monitoring system has not reported the system as faulty anymore. I repeated the same thing on other hosts when the problem occured there, all with the same result. "tls.reload" helps. But from the documentation, I don't know why.
Does anybody have an explanation for it?
Regards, Sebastian
Hello Sebastian,
on a first look to the code the tls.reload does similar operations as done during normal server startup, like - load configuration - fixing domains - check sockets
If the error only happens sporadic and, on some servers, it is probably either an error that only occurs in specific circumstances unrelated to kamailio, or some internal corruption topic in the module/server.
Do you see it also on e.g., test systems without any real load? Is there a difference between the systems in kind of load, and this maybe also causes some difference when the error occurs?
Cheers,
Henning
Hi Henning,
unfortunately, I don't have a host without traffic showing the same behavior. Our dev hosts usually don't run long enough. (And they usually don't get monitored.)
The "sporadically" meant, that it can take sometimes up to one week until it occurs on the same host again. And yes, some hosts have a bit more traffic than others, I suppose that's why it occurs earlier on some hosts, later on others.
I guess we have to deploy updates more often. ;)
Regards, Sebastian
----- Ursprüngliche Mail ----- Von: "Henning Westerholt" hw@skalatan.de An: "sr-users" sr-users@lists.kamailio.org CC: "Sebastian Damm" sdamm@pascom.net Gesendet: Dienstag, 24. August 2021 14:21:31 Betreff: RE: What does "tls.reload" actually do?
Hello Sebastian,
on a first look to the code the tls.reload does similar operations as done during normal server startup, like - load configuration - fixing domains - check sockets
If the error only happens sporadic and, on some servers, it is probably either an error that only occurs in specific circumstances unrelated to kamailio, or some internal corruption topic in the module/server.
Do you see it also on e.g., test systems without any real load? Is there a difference between the systems in kind of load, and this maybe also causes some difference when the error occurs?
Cheers,
Henning
Hello,
does it happen only for connections done by the monitoring system? Or also for the connections tried from the usual sip phones?
What is the operating system and libssl version?
Cheers, Daniel
On 30.08.21 11:57, Sebastian Damm wrote:
Hi,
I suppose, it happens for real connections, too. But since it's so sporadically, I guess, clients just retry and then it works.
The operating system is an Ubuntu 18.04 (getting replaced by Ubuntu 20.04 soon), thus it's running with libssl 1.1.1.
Regards, Sebastian
----- Ursprüngliche Mail ----- Von: "miconda" miconda@gmail.com An: "sr-users" sr-users@lists.kamailio.org, "Sebastian Damm" sdamm@pascom.net Gesendet: Montag, 30. August 2021 13:28:04 Betreff: Re: [SR-Users] What does "tls.reload" actually do?
Hello,
does it happen only for connections done by the monitoring system? Or also for the connections tried from the usual sip phones?
What is the operating system and libssl version?
Cheers, Daniel
On 30.08.21 11:57, Sebastian Damm wrote:
For the archives:
If you have a configuration file for your tls connections (not kamailio.cfg modparams) I believe the TLS module will reopen connections at tls.reload. If you update the certificates the new ones will be active after reload. This does not happen if you use modparams. Meaning if you use letsencrypt, your hook to reload with new certs is tls.reload. This propably means that open connections will be closed.
I don’t know if connections are affected if you use modparams. /O
Actually the active tls connections are not closed (and thus not re-opened) on tls.reload. It should use the new tls.cfg and corresponding certs only for the new connections. Old connections should not affected by reload.
Cheers, Daniel
On 30.08.21 13:57, Olle E. Johansson wrote:
Hello,
ubuntu 18.04 is the worse distro to work with when having to use libssl. One of their upgrades introduced mixed use of libssl 1.0 and 1.1, with 1.1 being one of the early releases in 1.1.x series. First thing that broke (or was reported) was related to mysql tls connections and getting stuck stuck after a while, when using the original mysql server (not mariadb). I dug in the code of libmysqlclient and libssl for a while on ubuntu 18.04, but couldn't sort it out, they are pretty huge code base and upgrading to 20.04 seemed to solve it.
But it may worth upgrading to "ubuntu-ng" aka debian :-) -- to my knowledge, debian buster (10) is known to be reliable with tls, debian 11 is at its very beginning, so not much experience with it yet.
Cheers, Daniel
On 30.08.21 13:39, Sebastian Damm wrote:
-
Daniel-Constantin Mierla -
Henning Westerholt -
Olle E. Johansson -
Sebastian Damm