### Description
When a TLS connection is closed and handle_lost_tcp=1 the entry is deleted for usrloc. This deletion is not synced via DMQ even though usrloc_delete=1.
There is also an issue on the same server if a re-registration happens right after before the timeout was supposed to expire then $var(sv_res) = 2 even though the usrloc table is empty. So it seems like the way handle_lost_tcp deletes is not deleting fully in registrar.
usrloc db_mode is 0
### Troubleshooting
#### Reproduction
Setup 2 servers with DMQ and close the TCP connection.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 5.6.1 (x86_64/linux) d8f98b
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: d8f98b
compiled on 19:05:04 Aug 16 2022 with gcc 8.3.0
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
Debian 10
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3479
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3479(a)github.com>
itpanda2024 created an issue (kamailio/kamailio#4467)
<!--
Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. Please use this template only for bug reports.
If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-users.lists.kamailio…
If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list:
* https://lists.kamailio.org/mailman3/postorius/lists/sr-dev.lists.kamailio.o…
Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue.
Note that an issue report may be closed automatically after about 2 months
if there is no interest from developers or community users on pursuing it, being
considered expired. In such case, it can be reopened by writing a comment that includes
the token `/notexpired`. About two weeks before considered expired, the issue is
marked with the label `stale`, trying to notify the submitter and everyone else
that might be interested in it. To remove the label `stale`, write a comment that
includes the token `/notstale`. Also, any comment postpone the `expire` timeline,
being considered that there is interest in pursuing the issue.
If there is no content to be filled in a section, the entire section can be removed.
You can delete the comments from the template sections when filling.
You can delete next line and everything above before submitting (it is a comment).
-->
### Description
<!--
We use db_cluster module with configuration below :
loadmodule "db_mysql.so"
modparam("db_mysql", "timeout_interval", 2)
loadmodule "db_cluster.so"
modparam("db_mysql", "timeout_interval", 2)
modparam("db_cluster", "connection","cfig_db1=>mysql://kamailio:kamailiorw@localhost/kamailio")
modparam("db_cluster", "connection","cfig_db2=>mysql://kamailio_user:Kam2025Db@172.20.22.12/kamailio")
modparam("db_cluster", "cluster", "cfig_db=>cfig_db1=9s8r;cfig_db2=9s8r")
modparam("db_cluster", "inactive_interval", 180)
modparam("db_cluster", "max_query_length", 5)
But we stop db1 . Kamailio fail ,not change to connect db2 and kamailio service stoped.
-->
### Troubleshooting
#### Reproduction
<!--
Nov 11 09:43:56 kam-02 kamailio[249517]: 391(249517) INFO: db_cluster [db_cluster_mod.c:192]: dbcl_rpc_list_connections(): read connection [cfig_db2]
Nov 11 09:43:56 kam-02 kamailio[249517]: 391(249517) INFO: db_cluster [db_cluster_mod.c:192]: dbcl_rpc_list_connections(): read connection [cfig_db1]
Nov 11 09:44:12 kam-02 kamailio[249509]: 387(249509) ERROR: db_mysql [km_dbase.c:129]: db_mysql_submit_query_impl(): driver error on query: Can't connect to local server through socket '/var/lib/mysql/mysql.sock' (2) (2002)
Nov 11 09:44:12 kam-02 kamailio[249509]: 387(249509) ERROR: <core> [db_query.c:335]: db_do_delete(): error while submitting query
Nov 11 09:44:12 kam-02 kernel: kamailio[249509]: segfault at 0 ip 00007f02cb456312 sp 00007fff12a9a3f0 error 4 in db_cluster.so[12312,7f02cb446000+2f000] likely on CPU 3 (core 0, socket 6)
Nov 11 09:44:12 kam-02 kernel: Code: 84 89 c7 e8 c0 fd fe ff 48 8b 4d a0 8b 45 c0 48 63 f0 8b 45 c4 48 63 d0 48 89 d0 48 c1 e0 03 48 29 d0 48 01 f0 48 8b 44 c1 18 <4c> 8b 38 48 8b 4d a0 8b 45 c0 48 63 f0 8b 45 c4 48 63 d0 48 89 d0
Nov 11 09:44:12 kam-02 systemd-coredump[249541]: Process 249509 (kamailio) of user 993 terminated abnormally with signal 11/SEGV, processing...
Nov 11 09:44:12 kam-02 systemd[1]: Started systemd-coredump(a)2-249541-0.service - Process Core Dump (PID 249541/UID 0).
Nov 11 09:44:13 kam-02 systemd-coredump[249542]: Process 249509 (kamailio) of user 993 dumped core.#012#012Module libuuid.so.1 from rpm util-linux-2.40.2-10.el10.x86_64#012Module libpcre2-8.so.0 from rpm pcre2-10.44-1.0.1.el10.3.x86_64#012Module libcrypt.so.2 from rpm libxcrypt-4.4.36-10.el10.x86_64#012Module libselinux.so.1 from rpm libselinux-3.8-2.el10_0.x86_64#012Module libbrotlicommon.so.1 from rpm brotli-1.1.0-6.el10.x86_64#012Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-27.el10.x86_64#012Module libevent-2.1.so.7 from rpm libevent-2.1.12-16.el10.x86_64#012Module libkeyutils.so.1 from rpm keyutils-1.6.3-5.el10.x86_64#012Module libkrb5support.so.0 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64#012Module libcom_err.so.2 from rpm e2fsprogs-1.47.1-3.el10.x86_64#012Module libk5crypto.so.3 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64#012Module libkrb5.so.3 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64#012Module libunistring.so.5 from rpm libunistring-1.1-10.el10.x86_64#012Module libz.so.1 from rpm zlib-ng-2.2.3-1.el10.x86_64#012Module libbrotlidec.so.1 from rpm brotli-1.1.0-6.el10.x86_64#012Module libgssapi_krb5.so.2 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64#012Module libcrypto.so.3 from rpm openssl-3.2.2-16.0.1.el10_0.4.x86_64#012Module libssl.so.3 from rpm openssl-3.2.2-16.0.1.el10_0.4.x86_64#012Module libpsl.so.5 from rpm libpsl-0.21.5-6.el10.x86_64#012Module libssh.so.4 from rpm libssh-0.11.1-4.el10_0.x86_64#012Module libidn2.so.0 from rpm libidn2-2.3.7-3.el10.x86_64#012Module libnghttp2.so.14 from rpm nghttp2-1.64.0-2.el10.x86_64#012Module libcurl.so.4 from rpm curl-8.9.1-5.el10.x86_64#012Module libcap.so.2 from rpm libcap-2.69-7.el10.x86_64#012Module libnss_myhostname.so.2 from rpm systemd-257-9.0.1.el10_0.1.x86_64#012Stack trace of thread 249509:#012#0 0x00007f02cb456312 db_cluster_delete (db_cluster.so + 0x12312)#012#1 0x00007f02cb3b461d tps_db_clean_branches (topos.so + 0x2e61d)#012#2 0x00007f02cb3cae2f tps_storage_clean (topos.so + 0x44e2f)#012#3 0x00000000006dd64f sr_wtimer_exec (/usr/local/sbin/kamailio + 0x2dd64f)#012#4 0x00000000006dc71d fork_sync_timer (/usr/local/sbin/kamailio + 0x2dc71d)#012#5 0x00000000006dd9c3 sr_wtimer_start (/usr/local/sbin/kamailio + 0x2dd9c3)#012#6 0x000000000042eea0 main_loop (/usr/local/sbin/kamailio + 0x2eea0)#012#7 0x000000000043902c main (/usr/local/sbin/kamailio + 0x3902c)#012#8 0x00007f02f04f230e __libc_start_call_main (libc.so.6 + 0x2a30e)#012#9 0x00007f02f04f23c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a3c9)#012#10 0x000000000041dc65 _start (/usr/local/sbin/kamailio + 0x1dc65)#012ELF object binary architecture: AMD x86-64
-->
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
Module libuuid.so.1 from rpm util-linux-2.40.2-10.el10.x86_64
Module libpcre2-8.so.0 from rpm pcre2-10.44-1.0.1.el10.3.x86_64
Module libcrypt.so.2 from rpm libxcrypt-4.4.36-10.el10.x86_64
Module libselinux.so.1 from rpm libselinux-3.8-2.el10_0.x86_64
Module libbrotlicommon.so.1 from rpm brotli-1.1.0-6.el10.x86_64
Module libsasl2.so.3 from rpm cyrus-sasl-2.1.28-27.el10.x86_64
Module libevent-2.1.so.7 from rpm libevent-2.1.12-16.el10.x86_64
Module libkeyutils.so.1 from rpm keyutils-1.6.3-5.el10.x86_64
Module libkrb5support.so.0 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64
Module libcom_err.so.2 from rpm e2fsprogs-1.47.1-3.el10.x86_64
Module libk5crypto.so.3 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64
Module libkrb5.so.3 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64
Module libunistring.so.5 from rpm libunistring-1.1-10.el10.x86_64
Module libz.so.1 from rpm zlib-ng-2.2.3-1.el10.x86_64
Module libbrotlidec.so.1 from rpm brotli-1.1.0-6.el10.x86_64
Module libgssapi_krb5.so.2 from rpm krb5-1.21.3-8.0.1.el10_0.x86_64
Module libcrypto.so.3 from rpm openssl-3.2.2-16.0.1.el10_0.4.x86_64
Module libssl.so.3 from rpm openssl-3.2.2-16.0.1.el10_0.4.x86_64
Module libpsl.so.5 from rpm libpsl-0.21.5-6.el10.x86_64
Module libssh.so.4 from rpm libssh-0.11.1-4.el10_0.x86_64
Module libidn2.so.0 from rpm libidn2-2.3.7-3.el10.x86_64
Module libnghttp2.so.14 from rpm nghttp2-1.64.0-2.el10.x86_64
Module libcurl.so.4 from rpm curl-8.9.1-5.el10.x86_64
Module libcap.so.2 from rpm libcap-2.69-7.el10.x86_64
Module libnss_myhostname.so.2 from rpm systemd-257-9.0.1.el10_0.1.x86_64
Stack trace of thread 250950:
#0 0x00007feac2913312 db_cluster_delete (db_cluster.so + 0x12312)
#1 0x00007feac287161d tps_db_clean_branches (topos.so + 0x2e61d)
#2 0x00007feac2887e2f tps_storage_clean (topos.so + 0x44e2f)
#3 0x00000000006dd64f sr_wtimer_exec (/usr/local/sbin/kamailio + 0x2dd64f)
#4 0x00000000006dc71d fork_sync_timer (/usr/local/sbin/kamailio + 0x2dc71d)
#5 0x00000000006dd9c3 sr_wtimer_start (/usr/local/sbin/kamailio + 0x2dd9c3)
#6 0x000000000042eea0 main_loop (/usr/local/sbin/kamailio + 0x2eea0)
#7 0x000000000043902c main (/usr/local/sbin/kamailio + 0x3902c)
#8 0x00007feae79ac30e __libc_start_call_main (libc.so.6 + 0x2a30e)
#9 0x00007feae79ac3c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a3c9)
#10 0x000000000041dc65 _start (/usr/local/sbin/kamailio + 0x1dc65)
ELF object binary architecture: AMD x86-64
-->
```
(paste your debugging data here)
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your log messages here)
```
#### SIP Traffic
<!--
If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
(paste your sip traffic here)
```
### Possible Solutions
<!--
If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix.
-->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
(paste your output here)
```
* **Operating System**:
<!--
Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...;
Kernel details (output of `lsb_release -a` and `uname -a`)
-->
```
(paste your output here)
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4467
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4467(a)github.com>
fulhade created an issue (kamailio/kamailio#4431)
Testing 6.0.3 noticed that log is filled with INFO log messages from title (every 5s). No errors in logs, usual stuff except info on tcpcon
```
Oct 12 00:40:24 kamailio[4350]: INFO: <core> [core/tcp_main.c:3124]: tcpconn_1st_send(): quick connect for 0x7f78f8c79b10 sock 38
Oct 12 22:42:26 kamailio[4355]: INFO: <core> [core/tcp_main.c:3124]: tcpconn_1st_send(): quick connect for 0x7f78f8c79b10 sock 32
...
Oct 13 07:42:12 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
Oct 13 07:42:17 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
Oct 13 07:42:22 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
Oct 13 07:42:27 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
Oct 13 07:42:32 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
Oct 13 07:42:37 kamailio[4355]: INFO: http_async_client [http_multi.c:88]: event_cb(): Cell for handler 0x7f78f8c79b10 not found in table
```
Using docker image built on trixie: https://pastebin.com/0wZ0X9wb
Not sure how to reproduce, it just appear on testing machine after couple of days.
Unfortunately we used 5.6.3 before this, so not sure when did it break for us. http_async_client usage:
```
$http_req(hdr) = "Content-Type: application/json";
$http_req(hdr) = "Expect: ";
$http_req(body) = $dlg_var(http_body);
http_async_query("${URL}/RoutingService/route", "ROUTING_RESPONSE");
```
Memory looks fine, but one of four CPU core spike to 96%.
<img width="1119" height="955" alt="Image" src="https://github.com/user-attachments/assets/ffff8680-84ad-4866-9892-a6e6eefd…" />
core.tcp_info, mod.stats all shm, mod.stats all pkg https://pastebin.com/QLZ8mZdD
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4431
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4431(a)github.com>
Ozzyboshi created an issue (kamailio/kamailio#4503)
Hello,
on my Kamailio installation I am experiencing a significant memory leak in SHM.
Here are the details of my system:
```
version: kamailio 6.0.3 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
NO_SIG_DEBUG, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE,
Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX,
FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS: 1024
MAX_RECV_BUFFER_SIZE: 262144
MAX_SEND_BUFFER_SIZE: 262144
MAX_URI_SIZE: 1024
BUF_SIZE: 65535
DEFAULT PKG_SIZE: 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select
compiled with gcc 14.2.0
```
The memory leak appears only when the presence feature is enabled.
When presence is active, Kamailio starts running dialog_publish(), whose code is here:
https://github.com/kamailio/kamailio/blob/9dc160d1d2bdf0542d3d9d8ae090bb135…
This function does not send the PUBLISH directly: it calls pua_send_publish(), which is a function pointer referring to the send_publish() implementation in the pua module.
Then send_publish() eventually calls set_uac_req() and tmb.t_request():
https://github.com/kamailio/kamailio/blob/9dc160d1d2bdf0542d3d9d8ae090bb135…
Digging further, tmb.t_request() maps to request() in the TM module, which calls t_uac_with_ids() and then t_uac_prepare().
Now comes the suspicious part:
If I comment out the call to t_uac_prepare(), the memory leak disappears.
This doesn’t necessarily mean the bug is inside t_uac_prepare(), but it’s a strong hint.
t_uac_prepare() allocates a new struct cell and returns it:
https://github.com/kamailio/kamailio/blob/9dc160d1d2bdf0542d3d9d8ae090bb135…
My concern is: is this cell always freed?
The matching cleanup function is free_cell(), used only here:
https://github.com/kamailio/kamailio/blob/9dc160d1d2bdf0542d3d9d8ae090bb135…
From what I can tell, free_cell() is called only if all these conditions are true:
- dst_cell == 0
- is_ack == 1
- dst_req == 0
-
In my situation no ACK is involved (Kamailio is a proxy that sends PUBLISH and immediately gets a 200 OK).
Therefore, is_ack is always false meaning the free_cell() cleanup logic is skipped entirely.
I tried forcing free_cell() unconditionally, but it leads to crashes, so clearly other parts of the code still rely on this structure.
Does the current free_cell() logic look correct to you?
Is it expected that struct cell allocated by t_uac_prepare() remains unfreed in cases where PUBLISH → 200 OK occurs without ACK?
Any guidance on how to proceed or where else to look would be greatly appreciated.
Thanks
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4503
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4503(a)github.com>
<!-- Kamailio Pull Request Template -->
<!--
IMPORTANT:
- for detailed contributing guidelines, read:
https://github.com/kamailio/kamailio/blob/master/.github/CONTRIBUTING.md
- pull requests must be done to master branch, unless they are backports
of fixes from master branch to a stable branch
- backports to stable branches must be done with 'git cherry-pick -x ...'
- code is contributed under BSD for core and main components (tm, sl, auth, tls)
- code is contributed GPLv2 or a compatible license for the other components
- GPL code is contributed with OpenSSL licensing exception
-->
#### Pre-Submission Checklist
<!-- Go over all points below, and after creating the PR, tick all the checkboxes that apply -->
<!-- All points should be verified, otherwise, read the CONTRIBUTING guidelines from above-->
<!-- If you're unsure about any of these, don't hesitate to ask on sr-dev mailing list -->
- [x] Commit message has the format required by CONTRIBUTING guide
- [x] Commits are split per component (core, individual modules, libs, utils, ...)
- [x] Each component has a single commit (if not, squash them into one commit)
- [x] No commits to README files for modules (changes must be done to docbook files
in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change
- [ ] Small bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds new functionality)
- [ ] Breaking change (fix or feature that would change existing functionality)
#### Checklist:
<!-- Go over all points below, and after creating the PR, tick the checkboxes that apply -->
- [ ] PR should be backported to stable branches
- [x] Tested changes locally
- [ ] Related to issue #XXXX (replace XXXX with an open issue number)
#### Description
<!-- Describe your changes in detail -->
In this PR we introduce a new mode and two new parameters for accomplishing the same thing for user part also in domain part.
Docs incoming if no significant changes are needed.
You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/4246
-- Commit Summary --
* topos: Add new contact_mode=3 and 2 new modparam for contact host domains
* topos: Refactor contact_mode handling
* topos: Append port and protocol param
-- File Changes --
M src/modules/topos/topos_mod.c (13)
M src/modules/topos/tps_storage.c (428)
M src/modules/topos/tps_storage.h (2)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/4246.patchhttps://github.com/kamailio/kamailio/pull/4246.diff
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/pull/4246
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/pull/4246(a)github.com>
mkl262 created an issue (kamailio/kamailio#4416)
Hello,
Im asking to add the JWT Module to the default list of modules that are being built and uploaded to [deb.kamailio.org](https://deb.kamailio.org/).
Will editing the CMAKE scripts will be enough (in which case I can open a PR for it), or does it require modifying the Jenkins pipelines?
Thanks,
Michael.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4416
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4416(a)github.com>
### Description
The Kamailio 5.4.x dialog profiles functionality can lead to dead-lock on certain high-load scenarios.
The Kamailio dialog profiles are used to track parallel channels for about 200 outgoing PSTN carrier interconnections. During high traffic times (like several thousands parallel calls) the Kamailio server will frequently (e.g. hourly) goes into an end-less loop while executing get_profile_size in the configuration script. This causes the locking for the dialog profiles never be released and Kamailio will stop serving traffic. Internal monitoring tools and RPC commands stay working, as long as they do not touch the dialog functionality.
A similar (dedicated) Kamailio setup is used for tracking parallel channels for customers. Here the dead-lock is not observed that frequently, but aparentely also some crashes happens in a much longer time interval.
### Troubleshooting
After analysis of the back-traces with GDB the get_profile_size() function was removed from the configuration script. After this change the crash did not happened anymore for several days.
#### Reproduction
Issue could not be reproduced so far.
#### Debugging Data
##### bt 1 (some data removed)
(gdb) bt
\# 0 0x00007f57cf3b00da in get_profile_size (profile=0x7f50ccbc7e80, value=0x7ffd9928f300) at dlg_profile.c:859
n = 364
i = 12
ph = 0x7f50d3e4b7d0
\# 1 0x00007f57cf419c67 in w_get_profile_size_helper (msg=0x7f57d699d418, profile=0x7f50ccbc7e80, value=0x7ffd9928f300, spd=0x7f57d6916960) at dialog.c:941
\# 2 0x00007f57cf41a459 in w_get_profile_size3 (msg=0x7f57d699d418, profile=0x7f50ccbc7e80, value=0x7f57d6935118, result=0x7f57d6916960) at dialog.c:982
\# 3 0x0000000000463fea in do_action (h=0x7ffd99293610, a=0x7f57d6936488, msg=0x7f57d699d418) at core/action.c:1094
\# 4 0x00000000004711ee in run_actions (h=0x7ffd99293610, a=0x7f57d6936488, msg=0x7f57d699d418) at core/action.c:1581
\# 5 0x000000000046058b in do_action (h=0x7ffd99293610, a=0x7f57d690fda8, msg=0x7f57d699d418) at core/action.c:700
The first back-trace was taking from a running process with gdb. The counter in f0 does not increased that much during this time, probably due the overflow of the loop counter.
##### bt2 (analysis with data structure with gdb scripts)
Here the loop counter in f0 showed a really high value. Expected size of dialog profiles hash table:
(gdb) p profile->entries[3]
$4 = {first = 0x7f9bfd4aad98, content = 2068}
(gdb) p profile->entries[7]
$3 = {first = 0x7f9c12079f70, content = 784}
(gdb) p profile->entries[12]
$6 = {first = 0x7f9c02be5d50, content = 7600}
(gdb) p profile->entries[14]
$2 = {first = 0x7f9bff636de8, content = 6764}
hash table bucket 14 shows a lot of corruption and the loop never ends (carrier names and IPs replaced). The list for hash bucket 7 got linked to the list for hash bucket 14:
counter 6755: prev 0x7f9c0b9dcde0 - current 0x7f9c02e5b378 - next 0x7f9c0a5f9ba0 - value carrier1-XX.XX - hash 14
counter 6756: prev 0x7f9c02e5b378 - current 0x7f9c0a5f9ba0 - next 0x7f9c0860b968 - value carrier1-XX.XX▒▒▒▒ - hash 14
counter 6757: prev 0x7f9c0a5f9ba0 - current 0x7f9c0860b968 - next 0x7f9bfe3f3a78 - value carrier1-XX.XX▒▒▒▒ - hash 14
counter 6758: prev 0x7f9c0860b968 - current 0x7f9bfe3f3a78 - next 0x7f9c10d977f0 - value carrier1-XX.XX - hash 14
counter 6759: prev 0x7f9bfe3f3a78 - current 0x7f9c10d977f0 - next 0x7f9c0ae198b0 - value carrier2-XX.XX▒▒▒▒ - hash 7
counter 6760: prev 0x7f9c10d977f0 - current 0x7f9c0ae198b0 - next 0x7f9c12079f70 - value carrier3-XX.XX - hash 7
counter 6761: prev 0x7f9c0ae198b0 - current 0x7f9c12079f70 - next 0x7f9c011f2540 - value-carrier2-XX.XX▒▒▒▒ - hash 7
counter 6762: prev 0x7f9c12079f70 - current 0x7f9c011f2540 - next 0x7f9bfff886f0 - value carrier2-XX.XX▒▒▒▒ - hash 7
counter 6763: prev 0x7f9c011f2540 - current 0x7f9bfff886f0 - next 0x7f9c05db00a8 - value carrier3-XX.XX= - hash 7
[...]
counter 28270: prev 0x7f9c019d06e8 - current 0x7f9bfaf18290 - next 0x7f9c12c90680 - value carrier2-XX.XX▒▒▒▒ - hash 7
counter 28271: prev 0x7f9bfaf18290 - current 0x7f9c12c90680 - next 0x7f9c086a2b58 - value-carrier2-XX.XX▒▒▒▒ - hash 7
counter 28272: prev 0x7f9c12c90680 - current 0x7f9c086a2b58 - next 0x7f9c0b4f09e8 - value carrier2-XX.XX▒▒▒▒ - hash 7
[...]
hash table bucket 7 is still consistent regarding the loop, but already shows initial sign of corruption. There is one item of the list for hash bucket 14 visible:
counter 780: prev 0x7f9c0db57ac8 - current 0x7f9c02225700 - next 0x7f9bfbf7db08 - value carrier2-XX.XX▒▒▒▒ - hash 7
counter 781: prev 0x7f9c02225700 - current 0x7f9bfbf7db08 - next 0x7f9c10d977f0 - value carrier1-XX.XX- hash 14
counter 782: prev 0x7f9bfe3f3a78 - current 0x7f9c10d977f0 - next 0x7f9c0ae198b0 - value carrier2-XX.XX▒▒▒▒ - hash 7
counter 783: prev 0x7f9c10d977f0 - current 0x7f9c0ae198b0 - next 0x7f9c12079f70 - value carrier3-XX.XX - hash 7
total size of hash table is 784
#### Log Messages
No special log messages observed.
#### SIP Traffic
SIP traffic looked ok during analysis of the core dumps.
### Possible Solutions
* adding additional safe-guards for the get_profile_size function to not access data from other hash buckets
* stopping the loop counter after some threshold
* finding and fixing the source of the internal data corruption (obviously)
* refactoring the dialog modules to use another approach for storing the dialog profile information
### Additional Information
* **Kamailio version**:
Kamailio 5.4.7, compiled from git repository
* **Operating System**:
CentOS 7.9
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2923
### Description
We are using Kamailio 5.7.4 on Debian 12 (from http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for our clients. The instance terminates SIP/TLS (with Cliencertificates) and forwards the SIP Traffic to internal systems.
After some days we are getting errors like this
`tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7318616 threshold1=8912896`
First we thought Kamailio just doesnt have enough memory, so we doubled it..
But after some days the Logmessage (and Userissues) occured again.
So we monitored the shmmem statistics and found that used and max_used are constantly growing til it reaches the limit.
As i mentioned we are using client-certificates and so we are also using the CRL feature.
We do have a systemd-timer which fetches the CRL every hour and runs 'kamcmd tls.reload' when finished.
Our tls.cfg looks like this:
```
[server:default]
method = TLSv1.2+
private_key = /etc/letsencrypt/live/hostname.de/privkey.pem
certificate = /etc/letsencrypt/live/hostname.de/fullchain.pem
ca_list = /etc/kamailio/ca_list.pem
ca_path = /etc/kamailio/ca_list.pem
crl = /etc/kamailio/combined.crl.pem
verify_certificate = yes
require_certificate = yes
[client:default]
verify_certificate = yes
require_certificate = yes
```
After testing a bit we found that every time tls.reload is executed Kamailio consumes a bit more memory which eventually leads to all the memory being consumed which leads to issues for our users.
See following example:
```
[0][root@edgar-dev:~]# while true ; do /usr/sbin/kamcmd tls.reload ; /usr/sbin/kamcmd core.shmmem ; sleep 1 ; done
Ok. TLS configuration reloaded.
{
total: 268435456
free: 223001520
used: 41352552
real_used: 45433936
max_used: 45445968
fragments: 73
}
Ok. TLS configuration reloaded.
{
total: 268435456
free: 222377960
used: 41975592
real_used: 46057496
max_used: 46069232
fragments: 78
}
Ok. TLS configuration reloaded.
{
total: 268435456
free: 221748664
used: 42604992
real_used: 46686792
max_used: 46698080
fragments: 77
}
Ok. TLS configuration reloaded.
{
total: 268435456
free: 221110832
used: 43242408
real_used: 47324624
max_used: 47335608
fragments: 81
}
^C
[130][root@edgar-dev:~]#
```
### Troubleshooting
#### Reproduction
Everytime tls.reload is called the memory consumptions grows..
#### Debugging Data
<!--
If you got a core dump, use gdb to extract troubleshooting data - full backtrace,
local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile
bt full
info locals
list
If you are familiar with gdb, feel free to attach more of what you consider to
be relevant.
-->
```
If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data!
```
#### Log Messages
<!--
Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site).
-->
```
If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data!
```
#### SIP Traffic
SIP doesnt seem to be relevant here
### Possible Solutions
Calling tls.reload less often or restart kamailio before memory is consumed ;)
### Additional Information
```
version: kamailio 5.7.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 12.2.0
```
* **Operating System**:
```
* Debian GNU/Linux 12 (bookworm)
* Linux edgar-dev 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3823
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3823(a)github.com>
qasimloay created an issue (kamailio/kamailio#4495)
The Problem
When I make a call and the call ends with BYE message containing Reason: outofcredit, the system returns wrong duration: 71582788 seconds instead of the actual call time.
This is causing major billing problems with our OCS.
S-CSCF Debug Logs
Here are the debug logs (debug level) from S-CSCF when this happens:
[scscf logs with debug (ims_charging_bug) .log](https://github.com/user-attachments/files/23698499/scscf.logs.with.debug.ims_charging_bug.log)
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4495
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4495(a)github.com>