[sr-dev] [kamailio/kamailio] Application crash due to intensive "dispather.add" using (Issue #3424)

### Description I want to add addresses to dispatcher.list dynamically ``` modparam("dispatcher", "db_url", DBURL) modparam("dispatcher", "table_name", "dispatcher") modparam("dispatcher", "ds_ping_interval", 10) modparam("dispatcher", "ds_probing_mode", 1) modparam("dispatcher", "ds_inactive_threshold", 1) modparam("dispatcher", "ds_probing_threshold", 1) modparam("dispatcher", "reload_delta", 0) modparam("dispatcher", "ds_timer_mode", 1) ... if(!ds_is_from_list("1","0","$var(contact)")) { jsonrpc_exec( '{ "jsonrpc": "2.0", "method": "dispatcher.add", "params": [1, "$var(contact)"], "id": 1 }' ); } ``` this scheme works with low load(one or two UAs) but it doesn't work with dozens of requests, kamailio crashes ### Troubleshooting ``` Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: CRITICAL: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f2f964578b8), called from dispatcher: dispatch.c: reindex_dests(790), first free dispatcher: dispatch.c: ds_avl_destroy(4013) - ignoring Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: INFO: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: {1 16898 REGISTER eebaeca9-c642-465f-bc09-67d1448ba6b7} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri [] Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 - Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:774]: handle_sigs(): child process 6825 exited by a signal 11 Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:777]: handle_sigs(): core was not generated Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: <core> [main.c:854]: sig_usr(): signal 15 received Apr 22 22:18:10 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Main process exited, code=exited, status=1/FAILURE Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Failed with result 'exit-code'. Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Consumed 1.304s CPU time. ``` #### Reproduction use code from Description section on request route #### Debugging Data ``` (gdb) bt full #0 0x000055945e0ccd20 in ?? () No symbol table info available. #1 0x000055945e0cd6ee in qm_malloc () No symbol table info available. #2 0x000055945e0da9c9 in qm_shm_malloc () No symbol table info available. #3 0x00007f325a05e921 in pack_dest () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #4 0x00007f325a060ad4 in add_dest2list () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #5 0x00007f325a087cf9 in ds_filter_dest_cb () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #6 0x00007f325a05affb in ds_iter_set () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #7 0x00007f325a088706 in ds_remove_dst () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #8 0x00007f325a0af7a8 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so No symbol table info available. #9 0x00007f325e7e82f5 in jsonrpc_exec_ex () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so No symbol table info available. #10 0x00007f325e7e875e in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so No symbol table info available. #11 0x000055945ddce3f3 in do_action () No symbol table info available. #12 0x000055945dddd391 in run_actions () No symbol table info available. #13 0x000055945ddce2d3 in do_action () No symbol table info available. #14 0x000055945dddd391 in run_actions () No symbol table info available. #15 0x000055945ddce2d3 in do_action () No symbol table info available. #16 0x000055945dddd391 in run_actions () No symbol table info available. #17 0x000055945ddca244 in do_action () No symbol table info available. --Type <RET> for more, q to quit, c to continue without paging-- #18 0x000055945dddd391 in run_actions () No symbol table info available. #19 0x000055945ddce2d3 in do_action () No symbol table info available. #20 0x000055945dddd391 in run_actions () No symbol table info available. #21 0x000055945ddddc0b in run_top_route () No symbol table info available. #22 0x000055945df3b1b2 in receive_msg () No symbol table info available. #23 0x000055945e09ea5d in udp_rcv_loop () No symbol table info available. #24 0x000055945ddb709b in main_loop () No symbol table info available. #25 0x000055945ddc420c in main () No symbol table info available. ``` ```(gdb) info locals No symbol table info available.``` ```(gdb) list 1 ../sysdeps/unix/sysv/linux/dl-vdso-setup.c: No such file or directory.``` #### SIP Traffic [kamailio.txt](https://github.com/kamailio/kamailio/files/11302151/kamailio.… ### Possible Solutions no idea ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` root@kamailio:[/etc]: kamailio -v version: kamailio 5.6.3 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 10.2.1 ``` * **Operating System**: ``` root@kamailio:[/etc]: uname -a Linux kamailio 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/3424 You are receiving this because you are subscribed to this thread. Message ID: &lt;kamailio/kamailio/issues/3424(a)github.com&gt;