jmordica created an issue (kamailio/kamailio#4179)
### Description
Kamailio crash involving rtpengine
### Troubleshooting
#### Reproduction
<!-- If the issue can be reproduced, describe how it can be done. -->
#### Debugging Data
<!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location.
gdb /path/to/kamailio /path/to/corefile bt full info locals list
If you are familiar with gdb, feel free to attach more of what you consider to be relevant. -->
``` GNU gdb (Debian 13.1-3) 13.1 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: https://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/.
For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio... Reading symbols from /usr/lib/debug/.build-id/20/dc8be8c0aa02eba62b84db3c283789f068fb79.debug...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing [New LWP 37] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `kamailio -DD -E -m 8000 -M 512 -f /etc/kamailio/kamailio.cfg -P /var/run/kamail'. Program terminated with signal SIGABRT, Aborted. #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 44 ./nptl/pthread_kill.c: No such file or directory. (gdb) bt full #0 __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at ./nptl/pthread_kill.c:44 tid = <optimized out> ret = 0 pd = <optimized out> old_mask = {__val = {0}} ret = <optimized out> #1 0x00007f0a2dd31f1f in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78 No locals. #2 0x00007f0a2dce2fb2 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 ret = <optimized out> #3 0x00007f0a2dccd472 in __GI_abort () at ./stdlib/abort.c:79 save_stage = 1 act = {__sigaction_handler = {sa_handler = 0x20, sa_sigaction = 0x20}, sa_mask = {__val = {1880844493789993498, 1880844493789993498, 1880844493789993498, 0, 0, 0, 0, 2314885530818453536, 2314885530818453536, 2314885530818453536, 2314885530818453536, 0, 0, 0, 0, 94814674008008}}, sa_flags = 365707264, sa_restorer = 0x563bc3029320 <log_prefix_buf>} #4 0x0000563bc2e27414 in qm_debug_check_frag (qm=qm@entry=0x7f0815cc4000, f=f@entry=0x7f0834533950, file=file@entry=0x7f0a0a93420f "rtpengine: rtpengine_hash.c", line=line@entry=528, eline=eline@entry=546, efile=0x563bc2f6abcc "core/mem/q_malloc.c") at core/mem/q_malloc.c:132 p = <optimized out> __func__ = "qm_debug_check_frag" #5 0x0000563bc2e282b9 in qm_free (qmp=<optimized out>, p=0x7f0834533990, file=0x7f0a0a93420f "rtpengine: rtpengine_hash.c", func=0x7f0a0a935040 <__func__.2> "rtpengine_hash_table_free_entry", line=528, mname=0x7f0a0a92a000 "rtpengine") at core/mem/q_malloc.c:546 qm = <optimized out> f = 0x7f0834533950 size = <optimized out> next = <optimized out> prev = <optimized out> __func__ = "qm_free" #6 0x0000563bc2e2d51f in qm_shm_free (qmp=<optimized out>, p=0x7f0834533990, file=0x7f0a0a93420f "rtpengine: rtpengine_hash.c", func=0x7f0a0a935040 <__func__.2> "rtpengine_hash_table_free_entry", line=528, mname=0x7f0a0a92a000 "rtpengine") at core/mem/q_malloc.c:1532 No locals. #7 0x00007f0a0a922adb in rtpengine_hash_table_free_entry (entry=0x7f08263a3f80) at ./src/modules/rtpengine/rtpengine_hash.c:528 __func__ = "rtpengine_hash_table_free_entry" --Type <RET> for more, q to quit, c to continue without paging--info locals #8 0x00007f0a0a927dd4 in rtpengine_hash_table_lookup (callid=..., viabranch=..., op=op@entry=OP_OFFER) at ./src/modules/rtpengine/rtpengine_hash.c:428 entry = 0x7f08263a3f80 last_entry = 0x7f08174e0450 hash_index = 87 node = <optimized out> __func__ = "rtpengine_hash_table_lookup" #9 0x00007f0a0a90ec2b in select_rtpp_node_old (do_test=1, op=OP_OFFER, viabranch=..., callid=...) at ./src/modules/rtpengine/rtpengine.c:3810 node = 0x0 node = <optimized out> __func__ = "select_rtpp_node_old" __llevel = <optimized out> __kld = <optimized out> __llevel = <optimized out> __kld = <optimized out> #10 select_rtpp_node (callid=..., viabranch=..., queried_nodes_list=0x7f0a0de84430, queried_nodes=queried_nodes@entry=0, op=OP_OFFER, do_test=1) at ./src/modules/rtpengine/rtpengine.c:3864 node = 0x0 __func__ = "select_rtpp_node" #11 0x00007f0a0a91228a in rtpp_function_call (bencbuf=bencbuf@entry=0x7ffeb1d5bea0, msg=msg@entry=0x7f0a0de94350, op=<optimized out>, flags_str=<optimized out>, body_out=body_out@entry=0x7ffeb1d5be80, cl_field=cl_field@entry=0x7ffeb1d5be90) at ./src/modules/rtpengine/rtpengine.c:3178 ng_flags = {via = -1, to = 1, packetize = 0, transport = 256, directional = 0, dict = 0x7f0a0de98b78, flags = 0x7f0a0de98d08, direction = 0x7f0a0de98e68, replace = 0x7f0a0de98ec8, rtcp_mux = 0x7f0a0de98f28, sdes = 0x7f0a0de98f88, t38 = 0x7f0a0de99098, received_from = 0x7f0a0de98e08, codec = 0x7f0a0de990f8, codec_strip = 0x0, codec_offer = 0x0, codec_transcode = 0x0, codec_mask = 0x0, codec_set = 0x0, codec_except = 0x0, codec_accept = 0x0, codec_consume = 0x0, call_id = { s = 0x563bc30e195f <buf+287> "5783b150-6ad4-4e55-9555-decf4048c8fb\r\nCSeq: 23471 INVITE\r\nAllow: OPTIONS, REGISTER, SUBSCRIBE, NOTIFY, PUBLISH, INVITE, ACK, BYE, CANCEL, UPDATE, MESSAGE, INFO, REFER\r\nk: replaces, histinfo\r\nX-UNIQUEI"..., len = 36}, from_tag = { s = 0x563bc30e18fb <buf+187> "47d9d572-1f09-46bf-b966-16988c17cf56\r\nt: sip:02-5333@10.142.0.56\r\nm: sip:vxm@10.52.7.2:5081\r\ni: 5783b150-6ad4-4e55-9555-decf4048c8fb\r\nCSeq: 23471 INVITE\r\nAllow: OPTIONS, REGISTER, SUBSCRIBE, NOTIF"..., len = 36}, to_tag = {s = 0x0, len = 0}} item = <optimized out> resp = <optimized out> viabranch = {s = 0x7f0a0dd15720 "z9hG4bKPje1c3efee-2490-4035-9e2e-b124ec630b820", len = 46} body = { s = 0x563bc30e1b33 <buf+755> "v=0\r\no=- 1936793669 1936793669 IN IP4 10.52.7.2\r\ns=VXM\r\nc=IN IP4 10.52.7.2\r\nt=0 0\r\nm=audio 18690 RTP/AVP 0 101\r\na=rtpmap:101 telephone-event/8000\r\na=fmtp:101 0-16\r\na=ptime:20\r\na=maxptime:140\r\na=sendre"..., len = 204} --Type <RET> for more, q to quit, c to continue without paging--list error = <optimized out> ret = 1 queried_nodes = 0 cont_type = <optimized out> node = <optimized out> cp = <optimized out> pv_val = {rs = {s = 0x7f083a53d3c0 "p", len = -1025336010}, ri = 47244640256, flags = 1} md5 = "\004\220\361\302;V\000\000\260r\366\302;V", '\000' <repeats 11 times>, "\345\323A\270\253\205\301" branch_buf = "\210\033\016\303;V\000\000\177W\315\302;V\000\000\216\033\016\303;V\000\000\000\345\323A\270\253\205\301\224\033\016\303;V\000\000\000\345\323A\270\253\205\301\220\201\351\r\n\177\000\000\000\345" result = <optimized out> t = <optimized out> branch_idx = <optimized out> __func__ = "rtpp_function_call" select_node = <optimized out> #12 0x00007f0a0a917346 in rtpp_function_call_ok (bencbuf=bencbuf@entry=0x7ffeb1d5bea0, msg=msg@entry=0x7f0a0de94350, op=<optimized out>, flags_str=<optimized out>, body=body@entry=0x7ffeb1d5be80, cl_field=cl_field@entry=0x7ffeb1d5be90) at ./src/modules/rtpengine/rtpengine.c:3371 ret = <optimized out> __func__ = "rtpp_function_call_ok" #13 0x00007f0a0a919708 in rtpengine_offer_answer (msg=0x7f0a0de94350, flags=<optimized out>, op=<optimized out>, more=0) at ./src/modules/rtpengine/rtpengine.c:4495 bencbuf = {pieces = 0x7f0a0de9a270, free_list = 0x0, error = 0} dict = <optimized out> body = {s = 0x7f0a0de94350 "\233@\006", len = 1104405760} newbody = <optimized out> anchor = <optimized out> pv_val = {rs = {s = 0x0, len = 177318755}, ri = 0, flags = 4} cur_body = {s = 0x0, len = 0} cl_field = {s = 0x0, len = 0} cl_repl = {s = 0x0, len = 0} __func__ = "rtpengine_offer_answer" error_free = <optimized out> #14 0x0000563bc2c77195 in sr_kemi_exec_func (ket=ket@entry=0x7f0a0a93d5c8 <sr_kemi_rtpengine_exports+72>, msg=<optimized out>, pno=pno@entry=1, vps=<optimized out>) at core/kemiexec.c:84 ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` CRITICAL: {1 23471 INVITE 5783b150-6ad4-4e55-9555-decf4048c8fb} <core> [core/mem/q_malloc.c:148]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten(c0c00a0d, abcdefed)[0x7f0834533950:0x7f0834533990]! Memory allocator was called from rtpengine: rtpengine_hash.c:528. Fragment marked by rtpengine: ../../core/parser/../ut.h:891. Exec from core/mem/q_malloc.c:546.
CRITICAL: {1 23471 INVITE 5783b150-6ad4-4e55-9555-decf4048c8fb} <core> [core/mem/q_malloc.c:156]: qm_debug_check_frag(): BUG: qm: prev. fragm. tail overwritten [0x7f08345335b0:0x7f08345335f0] - fragment marked by core: core/msg_translator.c:2305 ```
#### SIP Traffic
<!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` (paste your sip traffic here) ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.8.5 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 12.2.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm Linux gke-us-east1-external-sip-4b67daea-x3rj 5.15.0-1020-gke #25-Ubuntu SMP Thu Oct 27 05:38:18 UTC 2022 x86_64 GNU/Linux ```