## Description

kamailio dispatcher cache is inaccessible and ds_select_domain is failing. all new calls are rejected.

``` # kamctl kamcmd dispatcher.list error: 500 - no destination sets ``` ## Troubleshooting

This is happened is production after a dispatcher reload and we are unable to reproduced it again. we restarted kamailio service and that fixed the issue.

## Reproduction

I think this happened after we did a 10 simultaneous RPC command to reload the dispatcher cache.

## Log Messages

``` Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=XX.XX.XX.7:5060 hdr(Via)=<SIP/2.0/TCP XX.XX.XX.62:1900> Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: INFO: <script>: [XX.XX.XX.62:1900]: Successful authentication Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: http_rpc detected Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Host)=XX.XX.XX.7:5060 hdr(Via)=<SIP/2.0/TCP XX.XX.XX.54:3314> Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: /http_rpc/dispatcher/dispatcher.reload?arg= hdr(Authorization)=<yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> = <yyyyyyyzzzzzzzzzzxxxxxzzzzzzzwww> Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: INFO: <script>: [XX.XX.XX.54:3314]: Successful authentication Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69c949b0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ae3840), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f696c5458), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69ce8610), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698e3cd0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f697498c8), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662058), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699b1990), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663320), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69a4f418), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663928), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699125a0), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698b4c30), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69acbee8), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69716f28), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69978d18), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69662d38), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f698c4e70), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69b13130), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f699027a0), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69663f10), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69664400), called from dispatcher: dispatch.c: ds_avl_destroy(3032), first free dispatcher: dispatch.c: ds_avl_destroy(3032) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69667bd0), called from dispatcher: dispatch.c: ds_avl_destroy(3037), first free dispatcher: dispatch.c: ds_avl_destroy(3037) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69661a50), called from dispatcher: dispatch.c: ds_avl_destroy(3038), first free dispatcher: dispatch.c: ds_avl_destroy(3038) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32246]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f69857670), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting Apr 26 10:28:54 kamailio-gateway-01 /usr/sbin/kamailio[32245]: CRITICAL: <core> [core/mem/q_malloc.c:500]: qm_free(): BUG: freeing already freed pointer (0x7f6f6987f500), called from dispatcher: dispatch.c: reindex_dests(648), first free dispatcher: dispatch.c: reindex_dests(648) - aborting Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32182]: INFO: <script>: ===== redis sip code insert: SIP Code: 183 * time: 1556274535 * Rebtel-CID: TJI4V81TAP24NCN0DHHJG54NKG@YY.YY.YY.107 * Call-Id: 4b6b9685-c797-4e13-8cf5-cd6f7d23a39f Apr 26 10:28:55 kamailio-gateway-01 /usr/sbin/kamailio[32191]: INFO: <script>: ===== redis sip code insert: SIP Code: 181 * time: 1556274535 * Rebtel-CID: UBQ1HL47T51HH51BESVJIOE66K@YY.YY.YY.107 * Call-Id: d74592af-d240-481f-92bc-6b6f1010887c Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274536 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32187]: INFO: <script>: ===== redis sip code insert: SIP Code: 487 * time: 1556274523 * Rebtel-CID: 12-12-73EB33B1-5CC2DD5B000379CB-51039700 * Call-Id: b8078da6-7dff-4689-a28b-76268200a01a Apr 26 10:28:56 kamailio-gateway-01 /usr/sbin/kamailio[32206]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32210]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32207]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32195]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32209]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets Apr 26 10:28:57 kamailio-gateway-01 /usr/sbin/kamailio[32202]: ERROR: dispatcher [dispatch.c:1808]: ds_select_dst_limit(): no destination sets ```

## Possible Solutions Restarting Kamailio

## Additional Information Kamailio Version - output of kamailio -v

``` version: kamailio 5.1.3 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 13:05:41 Apr 24 2018 with gcc 4.8.5 ```

## Operating System:

CentOS Linux release 7.4.1708 (Core)