### Description
When using HSM keys (via OpenSSL engine) the engine and private keys are loaded in the child processes since PKCS#11 modules rarely survive `fork()`.
With OpenSSL 1.1.1 and the call to `OPENSSL_init_ssl()` in `tls_init.c` the engine linked-list is now initialized in the master process. Subsequently the technique used in `tls_init.c:660 CONF_modules_load_file()` will fail as each child process manipulates the linked-list. Even if the engine linked-list manipulation itself is protected by global locks; traversing the linked-list will encounter invalid memory locations as the nodes are insert by other child processes.
This issue is filed as a marker; I will be providing patches to change the way HSM private keys are loaded in the child process to avoid linked-list corruption.
### Troubleshooting
- use soft (PEM) keys — no issue as OpenSSL engine is not relevant in this context
#### Reproduction
- configure TLS with OpenSSL engine key(e.g pkcs11 engine with SoftHSM keys)
### Additional Information
Root cause: the function `CONF_modules_load_file()` will configure OpenSSL engines from a OpenSSL configuration file but it also adds each engine to the global linked list. Unfortunately there is no OpenSSL function to reset the linked-list in the child process. It seems to be initialized once at start-up: the linked-list head/tail pointers are static variables and their initialization is protected by OpenSSL's `RUN_ONCE` mechanism, so it doesn't seem it supported mechanism to have them reinitialized after `fork()`.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2839
tls_init.c calls OPENSSL_init_ssl(); this initializes the
global engine linked-list and this cannot be reset in the child.
To avoid linked-list corruption we manually instantiate
the engine object required for loading private keys instead of
relying on CONF_modules_load_file().
Updates to doc/.
Addresses #2839
<!-- Kamailio Pull Request Template -->
<!--
IMPORTANT:
- for detailed contributing guidelines, read:
https://github.com/kamailio/kamailio/blob/master/.github/CONTRIBUTING.md
- pull requests must be done to master branch, unless they are backports
of fixes from master branch to a stable branch
- backports to stable branches must be done with 'git cherry-pick -x ...'
- code is contributed under BSD for core and main components (tm, sl, auth, tls)
- code is contributed GPLv2 or a compatible license for the other components
- GPL code is contributed with OpenSSL licensing exception
-->
#### Pre-Submission Checklist
<!-- Go over all points below, and after creating the PR, tick all the checkboxes that apply -->
<!-- All points should be verified, otherwise, read the CONTRIBUTING guidelines from above-->
<!-- If you're unsure about any of these, don't hesitate to ask on sr-dev mailing list -->
- [X] Commit message has the format required by CONTRIBUTING guide
- [X] Commits are split per component (core, individual modules, libs, utils, ...)
- [X] Each component has a single commit (if not, squash them into one commit)
- [X] No commits to README files for modules (changes must be done to docbook files
in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change
- [X] Small bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds new functionality)
- [ ] Breaking change (fix or feature that would change existing functionality)
#### Checklist:
<!-- Go over all points below, and after creating the PR, tick the checkboxes that apply -->
- [X] PR should be backported to stable branches
- [X] Tested changes locally
- [X] Related to issue #2839
#### Description
The call to `OPENSSL_init_ssl()` in `tls_init.c` results in the creation of the engine linked-list in the parent.
This affects per-child engine private keys as there is no api to reinitialize the engine linked-list in the child.
This PR removes the call to `CONF_modules_load_file()` which causes linked-list corruption and replaces
the initialization of engine private keys in the child with with other api calls which do not manipulate global
objects.
You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/2840
-- Commit Summary --
* tls: fix OpenSSL engine in child processes
-- File Changes --
M src/modules/tls/doc/hsm_howto.xml (15)
M src/modules/tls/tls_mod.c (93)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/2840.patchhttps://github.com/kamailio/kamailio/pull/2840.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/pull/2840
<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* http://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
### Troubleshooting
#### Reproduction
kamailio -c -x tlsf -f misc/examples/pkg/kamailio-basic.cfg
#### Debugging Data
```
core was generated by `kamailio -c -x tlsf -f misc/examples/pkg/kamailio-basic.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000555e79902495 in str_hash_add (ht=0x555e79d6e0f0 <main_rt+16>, e=0x7f01c0521a48) at core/str_hash.h:85
85 clist_insert(&ht->table[h], e, next, prev);
(gdb) bt full
#0 0x0000555e79902495 in str_hash_add (ht=0x555e79d6e0f0 <main_rt+16>, e=0x7f01c0521a48) at core/str_hash.h:85
h = 6
#1 0x0000555e79902e46 in route_add (rt=0x555e79d6e0e0 <main_rt>, name=0x555e79bd87be "0", i=0) at core/route.c:134
e = 0x7f01c0521a48
__func__ = "route_add"
#2 0x0000555e799035f3 in init_rlist (r_name=0x555e79bd87c0 "main", rt=0x555e79d6e0e0 <main_rt>, n_entries=2, hash_size=8) at core/route.c:160
__func__ = "init_rlist"
#3 0x0000555e79903647 in init_routes () at core/route.c:172
No locals.
#4 0x0000555e79776eaa in main (argc=6, argv=0x7ffec006df78) at main.c:2408
cfg_stream = 0x0
c = -1
r = 0
tmp = 0x0
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x555e79ba60a8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 0
rfd = 0
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x0
p = 0x0
st = {st_dev = 0, st_ino = 0, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 0, st_blksize = 0, st_blocks = 0, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 0,
tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __unused = {0, 0, 0}}
tbuf = '\000' <repeats 2336 times>...
option_index = 0
long_options = {{name = 0x555e79ba85a6 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x555e79ba3504 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x555e79ba85ab "alias", has_arg = 1, flag = 0x0,
val = 1024}, {name = 0x555e79ba85b1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x555e79ba85b7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x555e79ba85c0 "substdefs", has_arg = 1, flag = 0x0,
val = 1027}, {name = 0x555e79ba85ca "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x555e79ba85d4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x555e79ba85df "modparam", has_arg = 1,
flag = 0x0, val = 1030}, {name = 0x555e79ba85e8 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x555e79ba85f3 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x555e79ba85f9 "cfg-print", has_arg = 0,
flag = 0x0, val = 1033}, {name = 0x555e79ba8603 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x555e79ba860a "defenv", has_arg = 1, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__func__ = "main"
(gdb) info locals
h = 6
(gdb) list
80 struct str_hash_entry* e)
81 {
82 int h;
83
84 h=get_hash1_raw(e->key.s, e->key.len) % ht->size;
85 clist_insert(&ht->table[h], e, next, prev);
86 }
87
88
89
(gdb)
```
### Possible Solutions
works after reverting
- ddafeebfaf1537c684eee22be228867563dd02ac
- 2d7aee506c617f2d258719562f8debf2b4ba087e
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
bash-5.0# kamailio -v
version: kamailio 5.5.2 (x86_64/linux) 55e232-dirty
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 55e232 -dirty
compiled on 15:08:48 Sep 5 2021 with gcc 9.3.0
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Alpine Docker Image 3.12
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2843
Module: kamailio
Branch: 5.5
Commit: 185d8b24c411a38b5b2c2632abdbf90b3796fbf6
URL: https://github.com/kamailio/kamailio/commit/185d8b24c411a38b5b2c2632abdbf90…
Author: Daniel-Constantin Mierla <miconda(a)gmail.com>
Committer: Daniel-Constantin Mierla <miconda(a)gmail.com>
Date: 2021-09-06T10:31:26+02:00
Revert "core/mem: removed tlsf_malloc_bits.h"
This reverts commit 2d7aee506c617f2d258719562f8debf2b4ba087e.
(cherry picked from commit 90f4bd591aba7c6fbe234df1f3868edf4ff0a9f1)
---
Added: src/core/mem/tlsf_malloc_bits.h
---
Diff: https://github.com/kamailio/kamailio/commit/185d8b24c411a38b5b2c2632abdbf90…
Patch: https://github.com/kamailio/kamailio/commit/185d8b24c411a38b5b2c2632abdbf90…
---
diff --git a/src/core/mem/tlsf_malloc_bits.h b/src/core/mem/tlsf_malloc_bits.h
new file mode 100644
index 0000000000..29c783d4a4
--- /dev/null
+++ b/src/core/mem/tlsf_malloc_bits.h
@@ -0,0 +1,119 @@
+#ifndef INCLUDED_tlsfbits
+#define INCLUDED_tlsfbits
+
+#if defined(__cplusplus)
+#define tlsf_decl inline
+#else
+#define tlsf_decl static
+#endif
+
+/*
+** Architecture-specific bit manipulation routines.
+**
+** TLSF achieves O(1) cost for malloc and free operations by limiting
+** the search for a free block to a free list of guaranteed size
+** adequate to fulfill the request, combined with efficient free list
+** queries using bitmasks and architecture-specific bit-manipulation
+** routines.
+**
+** Most modern processors provide instructions to count leading zeroes
+** in a word, find the lowest and highest set bit, etc. These
+** specific implementations will be used when available, falling back
+** to a reasonably efficient generic implementation.
+**
+** NOTE: TLSF spec relies on ffs/fls returning value 0..31.
+** ffs/fls return 1-32 by default, returning 0 for error.
+*/
+
+/*
+** Detect whether or not we are building for a 32- or 64-bit (LP/LLP)
+** architecture. There is no reliable portable method at compile-time.
+*/
+#if defined (__alpha__) || defined (__ia64__) || defined (__x86_64__) \
+ || defined (_WIN64) || defined (__LP64__) || defined (__LLP64__)
+#define TLSF_64BIT
+#endif
+
+/*
+** gcc 3.4 and above have builtin support, specialized for architecture.
+** Some compilers masquerade as gcc; patchlevel test filters them out.
+**
+** Note: clang is compatible with GCC builtins and will also define those macros
+*/
+#if defined (__GNUC__) && (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4)) \
+ && defined (__GNUC_PATCHLEVEL__)
+
+tlsf_decl int tlsf_ffs(unsigned int word)
+{
+ return __builtin_ffs(word) - 1;
+}
+
+tlsf_decl int tlsf_fls(unsigned int word)
+{
+ const int bit = word ? 32 - __builtin_clz(word) : 0;
+ return bit - 1;
+}
+
+#if defined (TLSF_64BIT)
+tlsf_decl int tlsf_fls_sizet(size_t size)
+{
+ const int bit = size ? 64 - __builtin_clzl(size) : 0;
+ return bit - 1;
+}
+#endif
+#else
+/* Fall back to generic implementation. */
+
+tlsf_decl int tlsf_fls_generic(unsigned int word)
+{
+ int bit = 32;
+
+ if (!word) bit -= 1;
+ if (!(word & 0xffff0000)) { word <<= 16; bit -= 16; }
+ if (!(word & 0xff000000)) { word <<= 8; bit -= 8; }
+ if (!(word & 0xf0000000)) { word <<= 4; bit -= 4; }
+ if (!(word & 0xc0000000)) { word <<= 2; bit -= 2; }
+ if (!(word & 0x80000000)) { word <<= 1; bit -= 1; }
+
+ return bit;
+}
+
+/* Implement ffs in terms of fls. */
+tlsf_decl int tlsf_ffs(unsigned int word)
+{
+ return tlsf_fls_generic(word & (~word + 1)) - 1;
+}
+
+tlsf_decl int tlsf_fls(unsigned int word)
+{
+ return tlsf_fls_generic(word) - 1;
+}
+
+#if defined (TLSF_64BIT)
+tlsf_decl int tlsf_fls_sizet(size_t size)
+{
+ int high = (int)(size >> 32);
+ int bits = 0;
+ if (high)
+ {
+ bits = 32 + tlsf_fls(high);
+ }
+ else
+ {
+ bits = tlsf_fls((int)size & 0xffffffff);
+
+ }
+ return bits;
+}
+#endif /* defined (TLSF_64BIT) */
+
+#endif /* GNUC */
+
+
+#if !defined (TLSF_64BIT)
+#define tlsf_fls_sizet tlsf_fls
+#endif
+
+#undef tlsf_decl
+
+#endif
Module: kamailio
Branch: master
Commit: 90f4bd591aba7c6fbe234df1f3868edf4ff0a9f1
URL: https://github.com/kamailio/kamailio/commit/90f4bd591aba7c6fbe234df1f3868ed…
Author: Daniel-Constantin Mierla <miconda(a)gmail.com>
Committer: Daniel-Constantin Mierla <miconda(a)gmail.com>
Date: 2021-09-06T10:29:50+02:00
Revert "core/mem: removed tlsf_malloc_bits.h"
This reverts commit 2d7aee506c617f2d258719562f8debf2b4ba087e.
---
Added: src/core/mem/tlsf_malloc_bits.h
---
Diff: https://github.com/kamailio/kamailio/commit/90f4bd591aba7c6fbe234df1f3868ed…
Patch: https://github.com/kamailio/kamailio/commit/90f4bd591aba7c6fbe234df1f3868ed…
---
diff --git a/src/core/mem/tlsf_malloc_bits.h b/src/core/mem/tlsf_malloc_bits.h
new file mode 100644
index 0000000000..29c783d4a4
--- /dev/null
+++ b/src/core/mem/tlsf_malloc_bits.h
@@ -0,0 +1,119 @@
+#ifndef INCLUDED_tlsfbits
+#define INCLUDED_tlsfbits
+
+#if defined(__cplusplus)
+#define tlsf_decl inline
+#else
+#define tlsf_decl static
+#endif
+
+/*
+** Architecture-specific bit manipulation routines.
+**
+** TLSF achieves O(1) cost for malloc and free operations by limiting
+** the search for a free block to a free list of guaranteed size
+** adequate to fulfill the request, combined with efficient free list
+** queries using bitmasks and architecture-specific bit-manipulation
+** routines.
+**
+** Most modern processors provide instructions to count leading zeroes
+** in a word, find the lowest and highest set bit, etc. These
+** specific implementations will be used when available, falling back
+** to a reasonably efficient generic implementation.
+**
+** NOTE: TLSF spec relies on ffs/fls returning value 0..31.
+** ffs/fls return 1-32 by default, returning 0 for error.
+*/
+
+/*
+** Detect whether or not we are building for a 32- or 64-bit (LP/LLP)
+** architecture. There is no reliable portable method at compile-time.
+*/
+#if defined (__alpha__) || defined (__ia64__) || defined (__x86_64__) \
+ || defined (_WIN64) || defined (__LP64__) || defined (__LLP64__)
+#define TLSF_64BIT
+#endif
+
+/*
+** gcc 3.4 and above have builtin support, specialized for architecture.
+** Some compilers masquerade as gcc; patchlevel test filters them out.
+**
+** Note: clang is compatible with GCC builtins and will also define those macros
+*/
+#if defined (__GNUC__) && (__GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 4)) \
+ && defined (__GNUC_PATCHLEVEL__)
+
+tlsf_decl int tlsf_ffs(unsigned int word)
+{
+ return __builtin_ffs(word) - 1;
+}
+
+tlsf_decl int tlsf_fls(unsigned int word)
+{
+ const int bit = word ? 32 - __builtin_clz(word) : 0;
+ return bit - 1;
+}
+
+#if defined (TLSF_64BIT)
+tlsf_decl int tlsf_fls_sizet(size_t size)
+{
+ const int bit = size ? 64 - __builtin_clzl(size) : 0;
+ return bit - 1;
+}
+#endif
+#else
+/* Fall back to generic implementation. */
+
+tlsf_decl int tlsf_fls_generic(unsigned int word)
+{
+ int bit = 32;
+
+ if (!word) bit -= 1;
+ if (!(word & 0xffff0000)) { word <<= 16; bit -= 16; }
+ if (!(word & 0xff000000)) { word <<= 8; bit -= 8; }
+ if (!(word & 0xf0000000)) { word <<= 4; bit -= 4; }
+ if (!(word & 0xc0000000)) { word <<= 2; bit -= 2; }
+ if (!(word & 0x80000000)) { word <<= 1; bit -= 1; }
+
+ return bit;
+}
+
+/* Implement ffs in terms of fls. */
+tlsf_decl int tlsf_ffs(unsigned int word)
+{
+ return tlsf_fls_generic(word & (~word + 1)) - 1;
+}
+
+tlsf_decl int tlsf_fls(unsigned int word)
+{
+ return tlsf_fls_generic(word) - 1;
+}
+
+#if defined (TLSF_64BIT)
+tlsf_decl int tlsf_fls_sizet(size_t size)
+{
+ int high = (int)(size >> 32);
+ int bits = 0;
+ if (high)
+ {
+ bits = 32 + tlsf_fls(high);
+ }
+ else
+ {
+ bits = tlsf_fls((int)size & 0xffffffff);
+
+ }
+ return bits;
+}
+#endif /* defined (TLSF_64BIT) */
+
+#endif /* GNUC */
+
+
+#if !defined (TLSF_64BIT)
+#define tlsf_fls_sizet tlsf_fls
+#endif
+
+#undef tlsf_decl
+
+#endif