[kamailio/kamailio] Intermittent crashes (from secsipid?) (Issue #4284) - sr-dev

13 Jun 2025


      whosgonna created an issue (kamailio/kamailio#4284)
### Description
Intermittent Kamailio crashes (a few times a week).
### Troubleshooting
Core dump created
#### Reproduction
From the backtrace it looks like the failure occurs in one of the secsipid commands.  The Identity headers/values we're passing are from sources we can't control. In some cases I know that we are receiving tokens where the data can be malformated - missing `alg=` or `ppt` parameters, having certificates that cannot be downloaded due to bad links. I can recreate/mimic these types of malformation though, and it does not cause a crash.
#### Debugging Data
For privacy reasons, I can't share the full backtrace, however I'm able to print a mildly redacted short backtrace (removing phone numbers and IP addresses).  I can povide portions of the full backtrace as needed as well:
```
root@e416aa3557fa:/dump# gdb /usr/sbin/kamailio ./2025-06-12_16.17.17.dmp
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
https://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
    http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 12]
[New LWP 58]
[New LWP 88]
[New LWP 59]
[New LWP 57]
[New LWP 106]
[New LWP 305]
[New LWP 46]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `/sbin/kamailio -dDDeE -u kamailio -g kamailio -e -d -m 1024 -M 48 -u kamailio -'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (threadid=281473563709472, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0xffffabc75020 (LWP 12))]
(gdb) bt
#0  __pthread_kill_implementation (threadid=281473563709472, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44
#1  0x0000ffffaba70ab4 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x0000ffffaba2a72c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x0000ffffaba1747c in __GI_abort () at ./stdlib/abort.c:79
#4  0x0000ffffaba64aac in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffffabb46d28 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#5  0x0000ffffaba7aebc in malloc_printerr (str=str@entry=0xffffabb42098 "double free or corruption (!prev)") at ./malloc/malloc.c:5660
#6  0x0000ffffaba7cee8 in _int_free (av=0xffffabb90af0 <main_arena>, p=p@entry=0xaaab18d4c3e0, have_lock=<optimized out>, have_lock@entry=0) at ./malloc/malloc.c:4587
#7  0x0000ffffaba7f77c in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3385
#8  0x0000ffffa6c90a0c in w_secsipid_get_url (msg=0xffffa8d511e0, purl=0xffffa8c43680 "", povar=0xffffa8c4dad0 "\004") at secsipid_mod.c:961
#9  0x0000aaaadc47d710 in do_action (h=0xfffffe56d390, a=0xffffa8c4d800, msg=0xffffa8d511e0) at core/action.c:1133
#10 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe56d390, a=0xffffa8c4d800, msg=0xffffa8d511e0) at core/action.c:1620
#11 0x0000aaaadc48db44 in run_actions_safe (h=0xfffffe56e9d0, a=0xffffa8c4d800, msg=0xffffa8d511e0) at core/action.c:1683
#12 0x0000aaaadc69a7e0 in rval_get_long (h=0xfffffe56e9d0, msg=0xffffa8d511e0, i=0xfffffe56d8c8, rv=0xffffa8c4eac8, cache=0x0) at core/rvalue.c:973
#13 0x0000aaaadc6a00f0 in rval_expr_eval_long (h=0xfffffe56e9d0, msg=0xffffa8d511e0, res=0xfffffe56d8c8, rve=0xffffa8c4eac0) at core/rvalue.c:1852
#14 0x0000aaaadc6a0118 in rval_expr_eval_long (h=0xfffffe56e9d0, msg=0xffffa8d511e0, res=0xfffffe56de00, rve=0xffffa8c4f200) at core/rvalue.c:1862
#15 0x0000aaaadc47d090 in do_action (h=0xfffffe56e9d0, a=0xffffa8c50660, msg=0xffffa8d511e0) at core/action.c:1099
#16 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe56e9d0, a=0xffffa8c4c4e0, msg=0xffffa8d511e0) at core/action.c:1620
#17 0x0000aaaadc479400 in do_action (h=0xfffffe56e9d0, a=0xffffa8c41060, msg=0xffffa8d511e0) at core/action.c:711
#18 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe56e9d0, a=0xffffa8c41060, msg=0xffffa8d511e0) at core/action.c:1620
#19 0x0000aaaadc48db44 in run_actions_safe (h=0xfffffe570870, a=0xffffa8c41060, msg=0xffffa8d511e0) at core/action.c:1683
#20 0x0000aaaadc69a7e0 in rval_get_long (h=0xfffffe570870, msg=0xffffa8d511e0, i=0xfffffe56ef08, rv=0xffffa8c411c8, cache=0x0) at core/rvalue.c:973
#21 0x0000aaaadc6a00f0 in rval_expr_eval_long (h=0xfffffe570870, msg=0xffffa8d511e0, res=0xfffffe56ef08, rve=0xffffa8c411c0) at core/rvalue.c:1852
#22 0x0000aaaadc6a0118 in rval_expr_eval_long (h=0xfffffe570870, msg=0xffffa8d511e0, res=0xfffffe56f440, rve=0xffffa8c41900) at core/rvalue.c:1862
#23 0x0000aaaadc47d090 in do_action (h=0xfffffe570870, a=0xffffa8c428e0, msg=0xffffa8d511e0) at core/action.c:1099
#24 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe570870, a=0xffffa8c428e0, msg=0xffffa8d511e0) at core/action.c:1620
#25 0x0000aaaadc47d5e8 in do_action (h=0xfffffe570870, a=0xffffa8c42a40, msg=0xffffa8d511e0) at core/action.c:1119
#26 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe570870, a=0xffffa8c362f0, msg=0xffffa8d511e0) at core/action.c:1620
#27 0x0000aaaadc479400 in do_action (h=0xfffffe570870, a=0xffffa8c325b0, msg=0xffffa8d511e0) at core/action.c:711
#28 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe570870, a=0xffffa8c325b0, msg=0xffffa8d511e0) at core/action.c:1620
#29 0x0000aaaadc48db44 in run_actions_safe (h=0xfffffe572f78, a=0xffffa8c325b0, msg=0xffffa8d511e0) at core/action.c:1683
#30 0x0000aaaadc69a7e0 in rval_get_long (h=0xfffffe572f78, msg=0xffffa8d511e0, i=0xfffffe570da8, rv=0xffffa8c32718, cache=0x0) at core/rvalue.c:973
#31 0x0000aaaadc6a00f0 in rval_expr_eval_long (h=0xfffffe572f78, msg=0xffffa8d511e0, res=0xfffffe570da8, rve=0xffffa8c32710) at core/rvalue.c:1852
#32 0x0000aaaadc6a0118 in rval_expr_eval_long (h=0xfffffe572f78, msg=0xffffa8d511e0, res=0xfffffe5712e0, rve=0xffffa8c32e50) at core/rvalue.c:1862
#33 0x0000aaaadc47d090 in do_action (h=0xfffffe572f78, a=0xffffa8c33cb0, msg=0xffffa8d511e0) at core/action.c:1099
#34 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe572f78, a=0xffffa8c313a0, msg=0xffffa8d511e0) at core/action.c:1620
#35 0x0000aaaadc4895fc in do_action (h=0xfffffe572f78, a=0xffffa8c35fb0, msg=0xffffa8d511e0) at core/action.c:1401
#36 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe572f78, a=0xffffa8c2e8e0, msg=0xffffa8d511e0) at core/action.c:1620
#37 0x0000aaaadc479400 in do_action (h=0xfffffe572f78, a=0xffffa8bb8070, msg=0xffffa8d511e0) at core/action.c:711
#38 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe572f78, a=0xffffa8bb7980, msg=0xffffa8d511e0) at core/action.c:1620
#39 0x0000aaaadc47d5a8 in do_action (h=0xfffffe572f78, a=0xffffa8bb81d0, msg=0xffffa8d511e0) at core/action.c:1115
#40 0x0000aaaadc48d3c4 in run_actions (h=0xfffffe572f78, a=0xffffa8bb0710, msg=0xffffa8d511e0) at core/action.c:1620
#41 0x0000aaaadc48dbf4 in run_top_route (a=0xffffa8bb0710, msg=0xffffa8d511e0, c=0x0) at core/action.c:1703
#42 0x0000aaaadc63fff8 in receive_msg (
    buf=0xaaaadcba84b8 <buf> "INVITE sip:15555555555@sti-cnam SIP/2.0\r\nRecord-Route: sip:1.2.3.40;lr=on;ftag=gK0e581301;dlgcor=3df.ea65\r\nVia: SIP/2.0/UDP 1.2.3.40:5060;branch=z9hG4bK7dd5.1fd1b25d525136f68cd8dba517eaf"..., len=2363,
    rcv_info=0xfffffe573790) at core/receive.c:520
#43 0x0000aaaadc7dabe0 in udp_rcv_loop () at core/udp_server.c:770
#44 0x0000aaaadc4626a8 in main_loop () at main.c:1895
#45 0x0000aaaadc471e88 in main (argc=16, argv=0xfffffe573ef8) at main.c:3406
(gdb) info locals
tid = 12
ret = 0
pd = 0xffffabc75020
old_mask = {__val = {281474948845152}}
ret = <optimized out>
(gdb) list
39      in ./nptl/pthread_kill.c
```
From the full backtrace at #8:
```
#8  0x0000ffffa6c90a0c in w_secsipid_get_url (msg=0xffffa8d511e0, purl=0xffffa8c43680 "", povar=0xffffa8c4dad0 "\004") at secsipid_mod.c:961
        ret = 0
        ovar = 0x1d4
        val = {rs = {s = 0x0, len = -27866320}, ri = 187650816818084, flags = -27866184}
        surl = {s = 0xffffa8a99a30 "https://t-mobile-sticr.fosrvt.com/8814f66226a3d07edcffb9cfb333c423bfe5dd0f76...", len = 102}
        __func__ = "w_secsipid_get_url"
```
In this case, the certificate URL is valid and retrievable.
#### Log Messages
I'm not sure how to correlate the dump file to the PID in the log - especially because Kamailio is running a container so the logged PIDs are from the container perspective, not the host perspective.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 6.0.1 (aarch64/linux) fce50d
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: fce50d
compiled on 13:26:51 May 12 2025 with gcc 12.2.0
```
* **Operating System**:
Container:
```
root@e416aa3557fa:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```
Host:
```
[bkaufman@cr-us-west-or-01~]$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023.6.20250303"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/amazon-linux-2023/"
DOCUMENTATION_URL="https://docs.aws.amazon.com/linux/"
SUPPORT_URL="https://aws.amazon.com/premiumsupport/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
VENDOR_NAME="AWS"
VENDOR_URL="https://aws.amazon.com/"
SUPPORT_END="2029-06-30"
[bkaufman@cr-us-west-or-01~]$ uname -a
Linux cr-us-west-or-01.i.prod.q.fl.gg 6.1.129-138.220.amzn2023.aarch64 #1 SMP Tue Feb 25 22:18:13 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
```
-- 
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4284
You are receiving this because you are subscribed to this thread.

Message ID: kamailio/kamailio/issues/4284@github.com