Before: -1 yielded null and -2 the last element of a header
<!-- Kamailio Pull Request Template -->
<!--
IMPORTANT:
- for detailed contributing guidelines, read:
https://github.com/kamailio/kamailio/blob/master/.github/CONTRIBUTING.md
- pull requests must be done to master branch, unless they are backports
of fixes from master branch to a stable branch
- backports to stable branches must be done with 'git cherry-pick -x ...'
- code is contributed under BSD for core and main components (tm, sl, auth, tls)
- code is contributed GPLv2 or a compatible license for the other components
- GPL code is contributed with OpenSSL licensing exception
-->
#### Pre-Submission Checklist
<!-- Go over all points below, and after creating the PR, tick all the checkboxes that apply -->
<!-- All points should be verified, otherwise, read the CONTRIBUTING guidelines from above-->
<!-- If you're unsure about any of these, don't hesitate to ask on sr-dev mailing list -->
- [x] Commit message has the format required by CONTRIBUTING guide
- [x] Commits are split per component (core, individual modules, libs, utils, ...)
- [x] Each component has a single commit (if not, squash them into one commit)
- [x] No commits to README files for modules (changes must be done to docbook files
in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change
- [x] Small bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds new functionality)
- [ ] Breaking change (fix or feature that would change existing functionality)
#### Checklist:
<!-- Go over all points below, and after creating the PR, tick the checkboxes that apply -->
- [x] PR should be backported to stable branches
- [x] Tested changes locally
- [x] Related to issue #3653
#### Description
<!-- Describe your changes in detail -->
This PR fixes a bug referenced in #3653.
Before fix `$(hfl(HEADER)[-1]` where `HEADER='Via' | Contact | Route | Record-Route ` produced `null` instead of the last element of header.
You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/3697
-- Commit Summary --
* pv_core: Fix negative index bug for hfl()
-- File Changes --
M src/modules/pv/pv_core.c (6)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/3697.patchhttps://github.com/kamailio/kamailio/pull/3697.diff
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/pull/3697
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/pull/3697(a)github.com>
### Description
References: #3635
Early initialization of tls in OpenSSL 3 in rank 0 results in the use of shared state (pointers to the same shared memory location). Note: this is not related to shared memory allocation contention as this protected by a (multi-process)futex. Under heavy traffic the workers will corrupt each others state (race condition). This is only visible under heavy loading and is the reason for the intermittent appearance in #3635.
Ping @miconda
https://github.com/kamailio/kamailio/commit/1a9b0b63617afebcee2aecb3b2240d7…
Since qm/fm are already protected by a multi-process futex this commit is redundant (it puts a pthread mutex around the futex). I have been able to reproduce OpenSSL 3 crashes with heavy loading with this commit.
Example of shared object is the error state (SSL_get_error). It should be per worker but
```
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f1d7f9c77b0), called from tls: tls_init.c: ser_free(535), first free tls: tls_init.c: ser_free(535) - ignoring
```
shows that multiple workers are accessing the same object (in OpenSSL this is `ERR_STATE *`).
### Troubleshooting
N/A
#### Reproduction
* create heaving loading of TLS clients
* observe shm logging errors
#### Debugging Data
Dump `ERR_STATE *` from two different processes: observe that these are identical meaning both workers are using the same struct.
```
# !!!!IMPORTANT!!!! 2 == OpenSSL thread local key for ERR_STATE *
(gdb) p err_thread_local
$3 = 2
# this is worker 1, process_no 7
Reading symbols from /usr/local/kamailio/lib64/kamailio/modules/debugger.so...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f1df054e80a in epoll_wait (epfd=5, events=0x7f1db0504990, maxevents=1, timeout=2000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30 return SYSCALL_CANCEL (epoll_wait, epfd, events, maxevents, timeout);
(gdb) p pthread_getspecific(2)
$1 = (void *) 0x7f1d700a65a0
# this is worker 2, process_no 8, rank 4
Reading symbols from /usr/local/kamailio/lib64/kamailio/modules/stun.so...
Reading symbols from /usr/local/kamailio/lib64/kamailio/modules/debugger.so...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--c
0x00007f1df054e80a in epoll_wait (epfd=5, events=0x7f1db0504990, maxevents=1, timeout=2000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30 return SYSCALL_CANCEL (epoll_wait, epfd, events, maxevents, timeout);
(gdb) p pthread_getspecific(2)
$2 = (void *) 0x7f1d700a65a0
```
#### Log Messages
```
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f1d7f9c77b0), called from tls: tls_init.c: ser_free(535), first free tls: tls_init.c: ser_free(535) - ignoring
```
#### SIP Traffic
N/A
### Possible Solutions
- avoid doing TLS initialization in rank 0; this will require cooperation from all modules that use OpenSSL 3 themselves. If an OpenSSL-using module initializes state before `fork()` then all workers will reuse global state
### Additional Information
- OpenSSL 3 has initialize-once and initialize-once-per-thread states. An example of this is `ERR_STATE *` - this is of the type initialize-once-per-thread.
- when kamailio does OpenSSL initialization in rank 0, the workers inherit all "global" objects. If these objects are in shared memory then the worker processes will contend for the same state leading to corruption
- due to the design of the OpenSSL 3 alot of this state (static variables, one time initialization) cannot be reinitialized after `fork()`. "initialize-once-per-thread" can be reinitialized if the child were to spawn threads.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3695
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3695(a)github.com>
#### Pre-Submission Checklist
- [X] Commit message has the format required by CONTRIBUTING guide
- [X ] Commits are split per component (core, individual modules, libs, utils, ...)
- [X] Each component has a single commit (if not, squash them into one commit)
- [X] No commits to README files for modules (changes must be done to docbook files
in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change
- [ ] Small bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds new functionality)
- [X] Breaking change (fix or feature that would change existing functionality)
#### Checklist:
<!-- Go over all points below, and after creating the PR, tick the checkboxes that apply -->
- [ ] PR should be backported to stable branches
- [x] Tested changes locally
- [X] Related to issue #3695 , #3635
#### Description
- changes to avoid setting `ERR_STATE *` in rank 0 main thread - all config is done in a transient worker thread
- `tls_fix_domains_cfg()` is performed in `PROC_SIPINIT` instead of `PROC_INIT` (could also be done in worker thread)
- changes to a module `outbound.so`: OpenSSL config is done in a transient worker thread
You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/3696
-- Commit Summary --
* tls: POC for OpenSSL 3
* outbound: OpenSSL 3 POC
* tls: skip OPENSSL_init_ssl
-- File Changes --
M src/modules/outbound/outbound_mod.c (38)
M src/modules/tls/tls_init.c (89)
M src/modules/tls/tls_mod.c (6)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/3696.patchhttps://github.com/kamailio/kamailio/pull/3696.diff
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/pull/3696
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/pull/3696(a)github.com>