[kamailio/kamailio] carrierroute: cr.dump_routes prints error or hangs after over 100 prefixes (#1304)
### Description Try to fill carrierroute with routing data and test result with `kamcmd`, but the rpc command never gives a result.
### Troubleshooting
#### Reproduction
``` /* ----- ctl params -------------------------------------------------------- */ modparam("ctl", "binrpc", "unix:/var/run/kamailio/kamailio_ctl") modparam("ctl", "mode", 0660)
/* ----- carrierroute params ----------------------------------------------- */ modparam("carrierroute", "config_source", "db") modparam("carrierroute", "db_url", "EXTERN_READ_DBURL") modparam("carrierroute", "fetch_rows", 2000) modparam("carrierroute", "db_load_description", 1) modparam("carrierroute", "match_mode", 10) modparam("carrierroute", "avoid_failed_destinations", 1) ```
Kamailio is running with the following parameters: ``` /usr/local/sbin/kamailio -u kamailio -g kamailio -m 512 -M 20 -n 16 ```
Fill carrierroute table with 110 Routes, do a routes reload and than try to get the routes dumped.
`kamcmd cr.reload_routes` and after that issue `kamcmd cr.dump_routes`, which hangs and don't return a result.
#### Log Messages
``` nothing regarding carrierroute in the log ```
### Possible Solutions
<!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. -->
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.1.0-dev8 (x86_64/freebsd) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, select, kqueue. id: unknown compiled on 20:35:54 Nov 1 2017 with clang 4.0 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) -->
``` FreeBSD 11.1-RELEASE ```
It could be a deadlock ... do the following.
* start kamailio * do `kamctl ps` and grab the pid of the process ""ctl handler" * reproduce the issue
Then use gdb to attach to the pid:
``` gdb /usr/local/sbin/kamailio _CTLPID_ bt full ```
Paste the output here.
``` (gdb) bt full #0 0x0000000800f97e3a in _kevent () from /lib/libc.so.7 No symbol table info available. #1 0x00000008046c4ca2 in pthread_suspend_all_np () from /lib/libthr.so.3 No symbol table info available. #2 0x000000080567c944 in io_wait_loop_kqueue (h=0x8058be9f0, t=10, repeat=0) at io_wait.h:1094 n = 0 r = 0 tspec = {tv_sec = 10, tv_nsec = 0} fm = (struct fd_map *) 0x801982738 orig_changes = 0 apply_changes = 0 revents = 1 #3 0x0000000805676d19 in io_listen_loop (fd_no=1, cs_lst=0x801648a00) at io_listener.c:293 max_fd_no = 289 poll_err = 0x0 poll_method = 6 cs = (struct ctrl_socket *) 0x0 type = 2 #4 0x0000000805656411 in mod_child (rank=0) at ctl.c:337 rpc_handler = 1 pid = 0 cs = (struct ctrl_socket *) 0x0 #5 0x0000000000505cdd in init_mod_child (m=0x801859fe8, rank=0) at core/sr_module.c:938 No locals. #6 0x0000000000505727 in init_mod_child (m=0x80185a648, rank=0) at core/sr_module.c:935 No locals. #7 0x0000000000505727 in init_mod_child (m=0x80185c0d0, rank=0) at core/sr_module.c:935 No locals. #8 0x0000000000505727 in init_mod_child (m=0x80185c898, rank=0) at core/sr_module.c:935 No locals. #9 0x0000000000505727 in init_mod_child (m=0x80185d098, rank=0) at core/sr_module.c:935 No locals. #10 0x0000000000505727 in init_mod_child (m=0x80185d7f0, rank=0) at core/sr_module.c:935 No locals. #11 0x0000000000505727 in init_mod_child (m=0x80185f600, rank=0) at core/sr_module.c:935 No locals. #12 0x0000000000505727 in init_mod_child (m=0x801860410, rank=0) at core/sr_module.c:935 No locals. #13 0x0000000000505727 in init_mod_child (m=0x801860a48, rank=0) at core/sr_module.c:935 No locals. #14 0x0000000000505727 in init_mod_child (m=0x801860f60, rank=0) at core/sr_module.c:935 No locals. #15 0x0000000000505727 in init_mod_child (m=0x8018613d0, rank=0) at core/sr_module.c:935 No locals. #16 0x0000000000505727 in init_mod_child (m=0x8018620c0, rank=0) at core/sr_module.c:935 No locals. #17 0x0000000000505727 in init_mod_child (m=0x801862488, rank=0) at core/sr_module.c:935 No locals. #18 0x0000000000505727 in init_mod_child (m=0x801862e48, rank=0) at core/sr_module.c:935 No locals. #19 0x0000000000505727 in init_mod_child (m=0x80186a6b0, rank=0) at core/sr_module.c:935 No locals. #20 0x0000000000505727 in init_mod_child (m=0x80186b530, rank=0) at core/sr_module.c:935 No locals. #21 0x0000000000505727 in init_mod_child (m=0x80186c0b0, rank=0) at core/sr_module.c:935 No locals. #22 0x0000000000505727 in init_mod_child (m=0x80186c998, rank=0) at core/sr_module.c:935 No locals. #23 0x0000000000505727 in init_mod_child (m=0x80186ce90, rank=0) at core/sr_module.c:935 No locals. #24 0x0000000000505727 in init_mod_child (m=0x80186d530, rank=0) at core/sr_module.c:935 No locals. #25 0x0000000000505727 in init_mod_child (m=0x80186dac0, rank=0) at core/sr_module.c:935 No locals. #26 0x0000000000505727 in init_mod_child (m=0x80186e338, rank=0) at core/sr_module.c:935 No locals. #27 0x0000000000505727 in init_mod_child (m=0x80186ed38, rank=0) at core/sr_module.c:935 No locals. #28 0x0000000000505727 in init_mod_child (m=0x80186f4f8, rank=0) at core/sr_module.c:935 No locals. #29 0x0000000000505727 in init_mod_child (m=0x80186f9d8, rank=0) at core/sr_module.c:935 No locals. #30 0x00000000005056d8 in init_child (rank=0) at core/sr_module.c:964 No locals. #31 0x000000000042f286 in main_loop () at main.c:1708 i = 16 pid = 19525 si = (struct socket_info *) 0x0 si_desc = 0x7fffffffd020 "udp receiver child=15 sock=192.168.10.120:5060" nrprocs = 16 woneinit = 1 #32 0x000000000043b1ff in main (argc=15, argv=0x7fffffffeb50) at main.c:2646 cfg_stream = (FILE *) 0x80120aca0 c = -1 r = 0 tmp = 0x7fffffffee1a "" tmp_len = 32767 port = -5392 proto = 0 options = 0x97dfe7 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1687602241 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = (struct name_lst *) 0xd1e64309666ca822 p = 0xf <Error reading address 0xf: Bad address> st = {st_dev = 1153570505, st_ino = 52988, st_mode = 16877, st_nlink = 2, st_uid = 1001, st_gid = 1001, st_rdev = 4294967295, st_atim = {tv_sec = 1509564024, tv_nsec = 794875000}, st_mtim = {tv_sec = 1510082134, tv_nsec = 719802000}, st_ctim = {tv_sec = 1510082134, tv_nsec = 719802000}, st_size = 2, st_blocks = 1, st_blksize = 4096, st_flags = 2048, st_gen = 0, st_lspare = 0, st_birthtim = {tv_sec = 1509564024, tv_nsec = 794875000}} ```
When the data is growing the command gives an error. (281 entries in carrierroute table) ``` kamcmd> cr.reload_routes kamcmd> cr.dump_routes error: 500 - Failed to add data to response ```
And this in the logs: ``` Nov 7 22:01:44 <daemon.err> ic1 kamailio[88580]: ERROR: carrierroute [cr_fifo.c:1230]: cr_rpc_dump_routes(): dump tree recursor failure at count 6/1 ```
It doesn't hang. The backtrace shows that it is waiting on the socket for new rpc commands.
The kamcmd error suggests that the module doesn't have enough space in its response buffer to write all the records -- see the buffer related params in the ctl readme, likely you need to increase their values.
As alternative, you can use jsonrpcs module with kamctl and do:
``` kamctl rpc cr.reload_routes kamctl rpc cr.dump_routes ``` jsonrpcs should not use a static internal buffer and should work out of the box.
Thank you for the answer, with increased memory the error messages are gone but no reply will be send, like its happening with a lower number of entries.
``` modparam("ctl", "binrpc_buffer_size", 2048) modparam("ctl", "binrpc_max_body_size", 128) modparam("ctl", "binrpc_struct_max_body_size", 64) ```
kamctl works but the system has no bash installed, thats why I preferred kamcmd over kamctl ``` sh /usr/local/sbin/kamctl rpc cr.dump_routes ```
Maybe `kamcli` is an alternative to look at if you have Python there:
* https://github.com/kamailio/kamcli
Anyhow, to get the kamcmd sorted out, run with `debug=3` inside kamailio.cfg and look at log messages, it should reveal what happens there.
Closed #1304.
Reopen if still an issue after playing with debug=3 and larger buffers (no follow up after quite some time).
-
Daniel-Constantin Mierla -
Rick