<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-users.lists.kamailio…
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-dev.lists.kamailio.o…
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
<!--
Explain what you did, what you expected to happen, and what actually happened.
-->
Hello. I use uac_req_send to send registration to the asterisk pool. If some asterisk did not authorize and re-sent the 401 code, the module causes a kamailio crash
### Troubleshooting
ds_select("BACKENDS","0");
while(ds_set_dst()) {
xlog("L_DBG",">>> BACKEND du=$du\n");
$uac_req(ruri)="sip:" + $(du{uri.host});
$uac_req(furi)="sip:" + $avp(username) + "@" + $(du{uri.host});
$uac_req(turi)=$uac_req(furi);
uac_req_send();
ds_next_dst();
}
#### Reproduction
<!--
If the issue can be reproduced, describe how it can be done.
-->
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
```
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00005557272803b9 in timer_list_expire (t=1180105352, h=0x7f73b30a3bb8, slow_l=0x7f73b30a73a0, slow_mark=2907) at core/timer.c:846
846 core/timer.c: No such file or directory.
(gdb) bt full
#0 0x00005557272803b9 in timer_list_expire (t=1180105352, h=0x7f73b30a3bb8, slow_l=0x7f73b30a73a0, slow_mark=2907) at core/timer.c:846
tl = 0x7f73b3a49a88
ret = 0
#1 0x00005557272809cf in timer_handler () at core/timer.c:922
saved_ticks = 1180105352
run_slow_timer = 0
i = 859
__func__ = "timer_handler"
#2 0x0000555727280f53 in timer_main () at core/timer.c:961
No locals.
#3 0x0000555726f77736 in main_loop () at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:1831
i = 6
pid = 0
si = 0x0
si_desc = "udp receiver child=5 sock=10.153.5.40:5060\000\000\020\000\000\000\003\000\000\000!\000\000\000\000\262\270\224RE&\035(\247K'WU\000\000WqA'WU\000\000\000\000\000\000\000\000\000\000c\367@'WU\000\000!\000\000\000\000\000\000\000\260\325\022\371s\177\000\000 \236.\220\375\177\000\000ݕ\024'WU\000"
nrprocs = 6
woneinit = 1
__func__ = "main_loop"
#4 0x0000555726f83d11 in main (argc=12, argv=0x7ffd902ea398) at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:3078
cfg_stream = 0x555727e12380
c = -1
r = 0
tmp = 0x7ffd902ebe36 ""
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x55572744e898 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4270110361
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffd902ea398
p = 0x7ffd902ea310 ""
st = {st_dev = 25, st_ino = 5514, st_nlink = 2, st_mode = 16888, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1690276408, tv_nsec = 971865637}, st_mtim = {tv_sec = 1690276408,
tv_nsec = 971865637}, st_ctim = {tv_sec = 1690276408, tv_nsec = 971865637}, __glibc_reserved = {0, 0, 0}}
tbuf = '\000' <repeats 80 times>, "\377\000\000\000\377\000\000\000\000\377\000\000\000\000\000\000", '/' <repeats 16 times>, "\230\r", '\000' <repeats 14 times>, "`", '\000' <repeats 15 times>, "\001", '\000' <repeats 144 times>...
option_index = 12
long_options = {{name = 0x55572740a2e9 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55572740b84a "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55572740f6e4 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55572740a2ee "subst",
has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55572740a2f4 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55572740a2fd "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55572740a307 "server-id", has_arg = 1, flag = 0x0, val = 1028},
{name = 0x55572740a311 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55572740a31c "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55572740a325 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55572740b967 "debug",
has_arg = 1, flag = 0x0, val = 1032}, {name = 0x55572740a330 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x55572740a33a "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__func__ = "main"
(gdb) info locals
tl = 0x7f73b3a49a88
ret = 0
(gdb) list
841 in core/timer.c
(gdb)
```
```
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f73f575d4d0 in free_hash_table () at /build/kamailio-5.6.4+ubuntu22.04/src/modules/tm/h_table.c:464
464 /build/kamailio-5.6.4+ubuntu22.04/src/modules/tm/h_table.c: No such file or directory.
(gdb) bt full
#0 0x00007f73f575d4d0 in free_hash_table () at /build/kamailio-5.6.4+ubuntu22.04/src/modules/tm/h_table.c:464
p_cell = 0x0
tmp_cell = 0x0
i = 64459
__func__ = "free_hash_table"
#1 0x00007f73f5793e46 in tm_shutdown () at /build/kamailio-5.6.4+ubuntu22.04/src/modules/tm/t_funcs.c:88
__func__ = "tm_shutdown"
#2 0x00005557271db17b in destroy_modules () at core/sr_module.c:842
t = 0x7f73f68a65a0
foo = 0x7f73f68a59e0
__func__ = "destroy_modules"
#3 0x0000555726f64029 in cleanup (show_status=1) at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:561
memlog = 0
__func__ = "cleanup"
#4 0x0000555726f66044 in shutdown_children (sig=15, show_status=1) at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:704
__func__ = "shutdown_children"
#5 0x0000555726f697dc in handle_sigs () at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:802
chld = 0
chld_status = 139
any_chld_stopped = 1
memlog = 0
__func__ = "handle_sigs"
#6 0x0000555726f78f42 in main_loop () at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:1900
i = 6
pid = 37846
si = 0x0
si_desc = "udp receiver child=5 sock=10.153.5.40:5060\000\000\020\000\000\000\003\000\000\000!\000\000\000\000\262\270\224RE&\035(\247K'WU\000\000WqA'WU\000\000\000\000\000\000\000\000\000\000c\367@'WU\000\000!\000\000\000\000\000\000\000\260\325\022\371s\177\000\000 \236.\220\375\177\000\000ݕ\024'WU\000"
nrprocs = 6
woneinit = 1
__func__ = "main_loop"
#7 0x0000555726f83d11 in main (argc=12, argv=0x7ffd902ea398) at /build/kamailio-5.6.4+ubuntu22.04/src/main.c:3078
cfg_stream = 0x555727e12380
c = -1
r = 0
tmp = 0x7ffd902ebe36 ""
tmp_len = 0
port = 0
proto = 0
ahost = 0x0
aport = 0
options = 0x55572744e898 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 4270110361
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x7ffd902ea398
p = 0x7ffd902ea310 ""
st = {st_dev = 25, st_ino = 5514, st_nlink = 2, st_mode = 16888, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1690276408, tv_nsec = 971865637}, st_mtim = {tv_sec = 1690276408,
tv_nsec = 971865637}, st_ctim = {tv_sec = 1690276408, tv_nsec = 971865637}, __glibc_reserved = {0, 0, 0}}
tbuf = '\000' <repeats 80 times>, "\377\000\000\000\377\000\000\000\000\377\000\000\000\000\000\000", '/' <repeats 16 times>, "\230\r", '\000' <repeats 14 times>, "`", '\000' <repeats 15 times>, "\001", '\000' <repeats 144 times>...
option_index = 12
long_options = {{name = 0x55572740a2e9 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55572740b84a "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55572740f6e4 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55572740a2ee "subst",
has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55572740a2f4 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x55572740a2fd "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55572740a307 "server-id", has_arg = 1, flag = 0x0, val = 1028},
{name = 0x55572740a311 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55572740a31c "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55572740a325 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x55572740b967 "debug",
has_arg = 1, flag = 0x0, val = 1032}, {name = 0x55572740a330 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x55572740a33a "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
__func__ = "main"
(gdb) info locals
p_cell = 0x0
tmp_cell = 0x0
i = 64459
__func__ = "free_hash_table"
(gdb) list
459 in /build/kamailio-5.6.4+ubuntu22.04/src/modules/tm/h_table.c
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
CRITICAL: <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f8a6c440050), called from uac: uac_send.c: uac_send_tm_callback(860), first free uac: uac_send.c: uac_send_info_clone(110) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:123]: qm_debug_check_frag(): BUG: qm: fragm. 0x7f74379f46b0 (address 0x7f74379f46e8) beginning overwritten (0)! Memory allocator was called from uac: uac_send.c:860. Fragment marked by (null):0. Exec from core/mem/q_malloc.c:511.
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
proto:UDP 2023-07-25T10:00:44.452898Z 10.153.5.40:5060 ---> 192.168.50.107:5060
REGISTER sip:192.168.50.107 SIP/2.0
Via: SIP/2.0/UDP 10.153.5.40;branch=z9hG4bK6a0b.834627a6000000000000000000000000.0
To: <sip:3408@192.168.50.107>
From: <sip:3408@192.168.50.107>;tag=b2f228866a24b161d346cfb256d45132-a0527658
CSeq: 10 REGISTER
Call-ID: 2332840947916a3e-39417(a)10.153.5.40
Max-Forwards: 70
Content-Length: 0
Contact: <sip:3408@10.153.5.40:5060>
Expires: 150
User-Agent: PortSIP UC Client Android - v11.8.1
proto:UDP 2023-07-25T10:00:44.464436Z 192.168.50.107:5060 ---> 10.153.5.40:5060
SIP/2.0 401 Unauthorized
Via: SIP/2.0/UDP 10.153.5.40;rport=5060;received=10.153.5.40;branch=z9hG4bK6a0b.834627a6000000000000000000000000.0
Call-ID: 2332840947916a3e-39417(a)10.153.5.40
From: <sip:3408@192.168.50.107>;tag=b2f228866a24b161d346cfb256d45132-a0527658
To: <sip:3408@192.168.50.107>;tag=z9hG4bK6a0b.834627a6000000000000000000000000.0
CSeq: 10 REGISTER
WWW-Authenticate: Digest realm="vpp",nonce="1690279244/67fc12dbaf6a76fa34181a4bd504de00",opaque="1680d45809efac86",algorithm=md5,qop="auth"
Server: Asterisk
Content-Length: 0
proto:UDP 2023-07-25T10:00:44.466447Z 10.153.5.40:5060 ---> 192.168.50.107:5060
REGISTER sip:192.168.50.107 SIP/2.0
Via: SIP/2.0/UDP 10.153.5.40;branch=z9hG4bK7a0b.5673b4d4000000000000000000000000.0
To: <sip:3408@192.168.50.107>
From: <sip:3408@192.168.50.107>;tag=b2f228866a24b161d346cfb256d45132-a0527658
CSeq: 11 REGISTER
Call-ID: 2332840947916a3e-39417(a)10.153.5.40
Max-Forwards: 70
Content-Length: 0
Contact: <sip:3408@10.153.5.40:5060>
Expires: 150
Authorization: Digest username="3408", realm="vpp", nonce="1690279244/67fc12dbaf6a76fa34181a4bd504de00", uri="sip:192.168.50.107", opaque="1680d45809efac86", qop=auth, nc=00000001, cnonce="1935694403", response="38bc75e831e1267ce943bcc50f76390a", algorithm=MD5
User-Agent: PortSIP UC Client Android - v11.8.1
proto:UDP 2023-07-25T10:00:44.477675Z 192.168.50.107:5060 ---> 10.153.5.40:5060
SIP/2.0 401 Unauthorized
Via: SIP/2.0/UDP 10.153.5.40;rport=5060;received=10.153.5.40;branch=z9hG4bK7a0b.5673b4d4000000000000000000000000.0
Call-ID: 2332840947916a3e-39417(a)10.153.5.40
From: <sip:3408@192.168.50.107>;tag=b2f228866a24b161d346cfb256d45132-a0527658
To: <sip:3408@192.168.50.107>;tag=z9hG4bK7a0b.5673b4d4000000000000000000000000.0
CSeq: 11 REGISTER
WWW-Authenticate: Digest realm="vpp",nonce="1690279244/67fc12dbaf6a76fa34181a4bd504de00",opaque="5adc30b867f02d18",algorithm=md5,qop="auth"
Server: Asterisk
Content-Length: 0
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.6.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 11.3.0
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3522
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3522(a)github.com>
Now there is a call scenario where freeswitch ->kamailio ->rtpengine ->webrtc (jsSIP) voice and video cannot be displayed. Is there a problem with my handling and how can I solve it?
1、Freeswitch INVITE to kamailio without video code
2、kamailio request to rtpengine(rtpengine_offer("record-call=off force trust-address replace-origin replace-session-connection UDP/TLS/RTP/SAVPF"))and call to jsSIP
3、jsSIP response with "SIP/2.0 200 OK" to kamailio without video code
4、kamailio request to(rtpengine_answer("record-call=off replace-origin replace-session-connection DTLS=passive ICE=remove RTP/SAVPF"))and response to Freeswitch
5、jsSIP send re-INVITE to kamailio with video code
6、kamailio request to(rtpengine_offer("record-call=off replace-origin replace-session-connection DTLS=passive ICE=remove SDES-off UDP/TLS/RTP/SAVPF"))and response to Freeswitch
7、Freeswitch response with "SIP/2.0 200 OK" to kamailio with video code
8、kamailio request to(rtpengine_answer("record-call=off replace-origin replace-session-connection rtcp-mux-accept rtcp-mux-offer DTLS=passive SDES-off UDP/TLS/RTP/SAVPF"))and response to jsSIP
9、Freeswitch send re-INVITE to kamailio with video code
10、kamailio request to(rtpengine_offer("record-call=off replace-origin replace-session-connection rtcp-mux-accept rtcp-mux-offer DTLS=passive SDES-off UDP/TLS/RTP/SAVPF"))and response to jsSIP
11、jsSIP response with "SIP/2.0 200 OK" to kamailio with video code
12、kamailio request to(rtpengine_answer("record-call=off replace-origin replace-session-connection rtcp-mux-accept rtcp-mux-offer DTLS=passive SDES-off UDP/TLS/RTP/SAVPF"))and response to Freeswitch
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3840
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3840(a)github.com>
Module: kamailio
Branch: master
Commit: 77de86909ac8c156335e1e789bb3067932f9eff5
URL: https://github.com/kamailio/kamailio/commit/77de86909ac8c156335e1e789bb3067…
Author: S-P Chan <shihping.chan(a)gmail.com>
Committer: S-P Chan <shihping.chan(a)gmail.com>
Date: 2024-05-04T19:48:21+08:00
tls: document handling of thread-locals
---
Added: src/modules/tls/OpenSSL3-README.md
---
Diff: https://github.com/kamailio/kamailio/commit/77de86909ac8c156335e1e789bb3067…
Patch: https://github.com/kamailio/kamailio/commit/77de86909ac8c156335e1e789bb3067…
---
diff --git a/src/modules/tls/OpenSSL3-README.md b/src/modules/tls/OpenSSL3-README.md
new file mode 100644
index 00000000000..6b6a7814129
--- /dev/null
+++ b/src/modules/tls/OpenSSL3-README.md
@@ -0,0 +1,55 @@
+# OpenSSL 3 Developer Notes
+
+## Background
+
+OpenSSL since 1.1.1 uses thread-local storage. The OpenSSL internal API
+is `CRYPTO_THREAD_set_local()` and it is implemented on Linux using `pthread_setspecific()`.
+
+In a new thread, the value of a thread-local variable is 0x0; the first access of this
+variable by OpenSSL will allocate new dynamic memory. The implication for Kamailio is that
+if such variables are initialized in rank 0, then all workers will reuse the same memory
+location as Kamailio uses shared memory for OpenSSL.
+
+## OpenSSL 1.1.1 shmmem Corruption
+
+In OpenSSL 1.1.1 there are three variables that impact Kamailio: `private_drbg`, `public_drbg`, and
+`err_thread_local`. The first two variables are circumvented by an early call to `RAND_set_rand_method()`.
+
+The third variable, i.e., `err_thread_local` is reused by all workers and leads to shmmem corruption
+particularly with other users of OpenSSL such as libcurl, and db modules with TLS.
+
+Historically, since 2019, this was a low-impact bug due to use of static variables in the OpenSSL 1.1.1
+implementation.
+
+
+## OpenSSL 3 shmmem Corruption
+
+In OpenSSL 3 there is one variable that impacts Kamailio: `err_thread_local`.
+
+OpenSSL 3 uses more dynamic memory to handle the error stack and shmmem corruption is easily
+reproducible, even without libcurl or db modules.
+
+## Resolution
+
+This resolution uses non-portable internal knowledge of pthreads on Linux: that `pthread_key_t`
+is a small integer, and that it is incremented when a new thread-local key is requested
+
+OpenSSL 3 uses 6 thread-locals, and OpenSSL 1.1.1 uses 4 thread-locals.
+
+The first attempt (5.8.0/5.8.1) to resolve this issue uses the following technique:
+* `tls_threads_mode = 1`: for each function that might initialize OpenSSL, run it in
+ a temporary thread; this leaves the thread-local variables in rank 0, main thread at their
+ default value of 0x0
+* `tls_threads_mode = 2`: add an at-fork handler to set thread-local variables to 0x0.
+ The implementation will set thread-local keys from 0-15 to have value 0x0.<br>
+
+Limitation: the limitation of this method is some libraries like libpython cannot be initialized
+other than in the primary thread and they will initialize thread-locals.
+
+The revised method makes a few OpenSSL function calls so
+that OpenSSL will initialize all required thread-locals, and the tls.so sets a high-water mark.
+It is assumed that all `pthread_key_t` values at the high-water mark or greater are set
+by non-OpenSSL libraries. During fork, tls.so will clear all thread-locals up to the high-water
+mark.
+
+