Hi,
(X-posted to sr-dev as this is getting into the nitty gritty)
As a short-term workaround for this, I've been playing with the preloaded library approach to hijack the pthread mutex calls and force them to provide process-shared mutexes. AFAICT this seems to be working and only has the minuscule performance impact of using slower process-shared mutexes in all instances, even when they aren't required.
The code for the preloaded library itself is very short and simple: https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498
As a more complete patch that integrates it into the build system (probably badly): https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17
Finally it requires adding it to the startup script, for example in a systemd service file as:
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
(that's with a hard coded path which isn't optimal of course).
I don't consider this a proper fix, but only a hacky workaround, but it might be a solution for the very near future. Throwing it out there in case other people have been working on similar approaches, and/or maybe have some comments about this.
Cheers
On 01/04/2019 04.52, Daniel-Constantin Mierla wrote:
Hello,
an update on this issue -- I spent a bit of time looking at libssl/libcrypto library and the problem can be the type of mutexes they use now internally starting with v1.1, respectively the pthread mutex. They are not process shared and kamailio is a multi-process application, working with the same tls connection from multiple processes.
Today I wrote to openssl mailing list, waiting now to see if I get any hints from there.
Cheers, Daniel
On 01.04.19 10:33, Kristijan Vrban wrote:
Hi Andrew,
yes, with openssl 1.0.2 Kamailio is now up and running since five days. Looks good so far.
Kristijan
Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk apogrebennyk@sipwise.com:
On 3/26/19 3:52 PM, Kristijan Vrban wrote:
Just curious, did you get to compile with OpenSSL 1.0 and test?
Just compiled with OpenSSL 1.0 . Gone test now.
Kristijan, any new occurrences since you have recompiled kamailio with openssl 1.0?
Regards, Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
thanks for working on this and providing the workaround solution with preloaded symbols.
I would suggest that you push it to master, to make it easier to deploy and test. There is a change I would do: install the openssl_mutex_shared via Makefile from tls module, not from the main Makefile. The ctl module does the same for kamcmd, practically is about setting a var there -- ctl has:
MOD_INSTALL_UTILS=../../../utils/kamcmd
Another one (I can look into it as well if proves not to be straightforward): place this inside the tls module, like: src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module needs the path to the folder.
If we then get couple of people testing and validating the use of process shared mutex, then we can pursue with openssl devs to add an option for it.
Cheers, Daniel
On 11.04.19 17:14, Richard Fuchs wrote:
Hi,
(X-posted to sr-dev as this is getting into the nitty gritty)
As a short-term workaround for this, I've been playing with the preloaded library approach to hijack the pthread mutex calls and force them to provide process-shared mutexes. AFAICT this seems to be working and only has the minuscule performance impact of using slower process-shared mutexes in all instances, even when they aren't required.
The code for the preloaded library itself is very short and simple: https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498
As a more complete patch that integrates it into the build system (probably badly): https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17
Finally it requires adding it to the startup script, for example in a systemd service file as:
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
(that's with a hard coded path which isn't optimal of course).
I don't consider this a proper fix, but only a hacky workaround, but it might be a solution for the very near future. Throwing it out there in case other people have been working on similar approaches, and/or maybe have some comments about this.
Cheers
On 01/04/2019 04.52, Daniel-Constantin Mierla wrote:
Hello,
an update on this issue -- I spent a bit of time looking at libssl/libcrypto library and the problem can be the type of mutexes they use now internally starting with v1.1, respectively the pthread mutex. They are not process shared and kamailio is a multi-process application, working with the same tls connection from multiple processes.
Today I wrote to openssl mailing list, waiting now to see if I get any hints from there.
Cheers, Daniel
On 01.04.19 10:33, Kristijan Vrban wrote:
Hi Andrew,
yes, with openssl 1.0.2 Kamailio is now up and running since five days. Looks good so far.
Kristijan
Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk apogrebennyk@sipwise.com:
On 3/26/19 3:52 PM, Kristijan Vrban wrote:
Just curious, did you get to compile with OpenSSL 1.0 and test?
Just compiled with OpenSSL 1.0 . Gone test now.
Kristijan, any new occurrences since you have recompiled kamailio with openssl 1.0?
Regards, Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Hi,
Thanks for the suggestion, that seems to work just fine. I've pushed it into a private branch `rfuchs/openssl-locking-fix` for now, as I'm pretty sure that this would break compilation on non-pthread systems. Ideally it should be built conditionally only for OpenSSL < 1.1. But anyway, feel free to play with it.
https://github.com/kamailio/kamailio/tree/rfuchs/openssl-locking-fix https://github.com/kamailio/kamailio/commit/efdc141ecb5ff72e3224e47deaaa79fe...
Cheers
On 11/04/2019 13.03, Daniel-Constantin Mierla wrote:
Hello,
thanks for working on this and providing the workaround solution with preloaded symbols.
I would suggest that you push it to master, to make it easier to deploy and test. There is a change I would do: install the openssl_mutex_shared via Makefile from tls module, not from the main Makefile. The ctl module does the same for kamcmd, practically is about setting a var there -- ctl has:
MOD_INSTALL_UTILS=../../../utils/kamcmd
Another one (I can look into it as well if proves not to be straightforward): place this inside the tls module, like: src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module needs the path to the folder.
If we then get couple of people testing and validating the use of process shared mutex, then we can pursue with openssl devs to add an option for it.
Cheers, Daniel
On 11.04.19 17:14, Richard Fuchs wrote:
Hi,
(X-posted to sr-dev as this is getting into the nitty gritty)
As a short-term workaround for this, I've been playing with the preloaded library approach to hijack the pthread mutex calls and force them to provide process-shared mutexes. AFAICT this seems to be working and only has the minuscule performance impact of using slower process-shared mutexes in all instances, even when they aren't required.
The code for the preloaded library itself is very short and simple: https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498
As a more complete patch that integrates it into the build system (probably badly): https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17
Finally it requires adding it to the startup script, for example in a systemd service file as:
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
(that's with a hard coded path which isn't optimal of course).
I don't consider this a proper fix, but only a hacky workaround, but it might be a solution for the very near future. Throwing it out there in case other people have been working on similar approaches, and/or maybe have some comments about this.
Cheers
On 01/04/2019 04.52, Daniel-Constantin Mierla wrote:
Hello,
an update on this issue -- I spent a bit of time looking at libssl/libcrypto library and the problem can be the type of mutexes they use now internally starting with v1.1, respectively the pthread mutex. They are not process shared and kamailio is a multi-process application, working with the same tls connection from multiple processes.
Today I wrote to openssl mailing list, waiting now to see if I get any hints from there.
Cheers, Daniel
On 01.04.19 10:33, Kristijan Vrban wrote:
Hi Andrew,
yes, with openssl 1.0.2 Kamailio is now up and running since five days. Looks good so far.
Kristijan
Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk apogrebennyk@sipwise.com:
On 3/26/19 3:52 PM, Kristijan Vrban wrote:
> Just curious, did you get to compile with OpenSSL 1.0 and test? Just compiled with OpenSSL 1.0 . Gone test now.
Kristijan, any new occurrences since you have recompiled kamailio with openssl 1.0?
Regards, Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Thanks Richard,
I pushed a README.md inside openssl_mutex_shared folder with some details about how to use it and a short note to the README of tls module to refer to it for libssl v1.1.
I think all Unix-like systems have now pthread library, the issue that can be with some old *BSD-es is the lack of implementation for PTHREAD_PROCESS_SHARED option, but I think it would compile even there.
I am going to merge to master, ask the community to do some tests and see the results.
Cheers, Daniel
On 11.04.19 20:46, Richard Fuchs wrote:
Hi,
Thanks for the suggestion, that seems to work just fine. I've pushed it into a private branch `rfuchs/openssl-locking-fix` for now, as I'm pretty sure that this would break compilation on non-pthread systems. Ideally it should be built conditionally only for OpenSSL < 1.1. But anyway, feel free to play with it.
https://github.com/kamailio/kamailio/tree/rfuchs/openssl-locking-fix https://github.com/kamailio/kamailio/commit/efdc141ecb5ff72e3224e47deaaa79fe...
Cheers
On 11/04/2019 13.03, Daniel-Constantin Mierla wrote:
Hello,
thanks for working on this and providing the workaround solution with preloaded symbols.
I would suggest that you push it to master, to make it easier to deploy and test. There is a change I would do: install the openssl_mutex_shared via Makefile from tls module, not from the main Makefile. The ctl module does the same for kamcmd, practically is about setting a var there -- ctl has:
MOD_INSTALL_UTILS=../../../utils/kamcmd
Another one (I can look into it as well if proves not to be straightforward): place this inside the tls module, like: src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module needs the path to the folder.
If we then get couple of people testing and validating the use of process shared mutex, then we can pursue with openssl devs to add an option for it.
Cheers, Daniel
On 11.04.19 17:14, Richard Fuchs wrote:
Hi,
(X-posted to sr-dev as this is getting into the nitty gritty)
As a short-term workaround for this, I've been playing with the preloaded library approach to hijack the pthread mutex calls and force them to provide process-shared mutexes. AFAICT this seems to be working and only has the minuscule performance impact of using slower process-shared mutexes in all instances, even when they aren't required.
The code for the preloaded library itself is very short and simple: https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498
As a more complete patch that integrates it into the build system (probably badly): https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17
Finally it requires adding it to the startup script, for example in a systemd service file as:
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
(that's with a hard coded path which isn't optimal of course).
I don't consider this a proper fix, but only a hacky workaround, but it might be a solution for the very near future. Throwing it out there in case other people have been working on similar approaches, and/or maybe have some comments about this.
Cheers
On 01/04/2019 04.52, Daniel-Constantin Mierla wrote:
Hello,
an update on this issue -- I spent a bit of time looking at libssl/libcrypto library and the problem can be the type of mutexes they use now internally starting with v1.1, respectively the pthread mutex. They are not process shared and kamailio is a multi-process application, working with the same tls connection from multiple processes.
Today I wrote to openssl mailing list, waiting now to see if I get any hints from there.
Cheers, Daniel
On 01.04.19 10:33, Kristijan Vrban wrote:
Hi Andrew,
yes, with openssl 1.0.2 Kamailio is now up and running since five days. Looks good so far.
Kristijan
Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk apogrebennyk@sipwise.com:
On 3/26/19 3:52 PM, Kristijan Vrban wrote: >> Just curious, did you get to compile with OpenSSL 1.0 and test? > Just compiled with OpenSSL 1.0 . Gone test now. Kristijan, any new occurrences since you have recompiled kamailio with openssl 1.0?
Regards, Andrew
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
To let everyone know that the branch was merged into git master:
* https://github.com/kamailio/kamailio/tree/master/src/modules/tls/utils/opens...
Who can test it, please do it and report the results. Some details about installation and usage are in the README.md file from that folder.
Cheers, Daniel
On 12.04.19 09:23, Daniel-Constantin Mierla wrote:
Thanks Richard,
I pushed a README.md inside openssl_mutex_shared folder with some details about how to use it and a short note to the README of tls module to refer to it for libssl v1.1.
I think all Unix-like systems have now pthread library, the issue that can be with some old *BSD-es is the lack of implementation for PTHREAD_PROCESS_SHARED option, but I think it would compile even there.
I am going to merge to master, ask the community to do some tests and see the results.
Cheers, Daniel
On 11.04.19 20:46, Richard Fuchs wrote:
Hi,
Thanks for the suggestion, that seems to work just fine. I've pushed it into a private branch `rfuchs/openssl-locking-fix` for now, as I'm pretty sure that this would break compilation on non-pthread systems. Ideally it should be built conditionally only for OpenSSL < 1.1. But anyway, feel free to play with it.
https://github.com/kamailio/kamailio/tree/rfuchs/openssl-locking-fix https://github.com/kamailio/kamailio/commit/efdc141ecb5ff72e3224e47deaaa79fe...
Cheers
On 11/04/2019 13.03, Daniel-Constantin Mierla wrote:
Hello,
thanks for working on this and providing the workaround solution with preloaded symbols.
I would suggest that you push it to master, to make it easier to deploy and test. There is a change I would do: install the openssl_mutex_shared via Makefile from tls module, not from the main Makefile. The ctl module does the same for kamcmd, practically is about setting a var there -- ctl has:
MOD_INSTALL_UTILS=../../../utils/kamcmd
Another one (I can look into it as well if proves not to be straightforward): place this inside the tls module, like: src/modules/tls/utils/openssl_mutex_shared -- the Makefile of the module needs the path to the folder.
If we then get couple of people testing and validating the use of process shared mutex, then we can pursue with openssl devs to add an option for it.
Cheers, Daniel
On 11.04.19 17:14, Richard Fuchs wrote:
Hi,
(X-posted to sr-dev as this is getting into the nitty gritty)
As a short-term workaround for this, I've been playing with the preloaded library approach to hijack the pthread mutex calls and force them to provide process-shared mutexes. AFAICT this seems to be working and only has the minuscule performance impact of using slower process-shared mutexes in all instances, even when they aren't required.
The code for the preloaded library itself is very short and simple: https://gist.github.com/rfuchs/1bb7348b6acbe37e557d94c2f69a1498
As a more complete patch that integrates it into the build system (probably badly): https://gist.github.com/rfuchs/b240ffe87938a45e6f2a4cf53fe29f17
Finally it requires adding it to the startup script, for example in a systemd service file as:
Environment='LD_PRELOAD=/usr/lib/x86_64-linux-gnu/kamailio/openssl_mutex_shared/openssl_mutex_shared.so'
(that's with a hard coded path which isn't optimal of course).
I don't consider this a proper fix, but only a hacky workaround, but it might be a solution for the very near future. Throwing it out there in case other people have been working on similar approaches, and/or maybe have some comments about this.
Cheers
On 01/04/2019 04.52, Daniel-Constantin Mierla wrote:
Hello,
an update on this issue -- I spent a bit of time looking at libssl/libcrypto library and the problem can be the type of mutexes they use now internally starting with v1.1, respectively the pthread mutex. They are not process shared and kamailio is a multi-process application, working with the same tls connection from multiple processes.
Today I wrote to openssl mailing list, waiting now to see if I get any hints from there.
Cheers, Daniel
On 01.04.19 10:33, Kristijan Vrban wrote:
Hi Andrew,
yes, with openssl 1.0.2 Kamailio is now up and running since five days. Looks good so far.
Kristijan
Am Do., 28. März 2019 um 11:09 Uhr schrieb Andrew Pogrebennyk apogrebennyk@sipwise.com: > On 3/26/19 3:52 PM, Kristijan Vrban wrote: >>> Just curious, did you get to compile with OpenSSL 1.0 and test? >> Just compiled with OpenSSL 1.0 . Gone test now. > Kristijan, > any new occurrences since you have recompiled kamailio with openssl > 1.0? > > Regards, > Andrew _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Kamailio (SER) - Development Mailing List sr-dev@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
-
Daniel-Constantin Mierla -
Richard Fuchs