### Description
One of our proxy instances died after one of the rtpengines "disappeared". I don't know if there is a real connection between those two events, the only thing I know the small timespan between them.
### Troubleshooting
#### Reproduction
It was only on one of our proxy servers, and it doesn't happen all the time.
#### Debugging Data
gdb output:
``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 31359]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000663091 in destroy_avp_list_unsafe (list=0x7efe49050bd0) at core/usr_avp.c:625 625 core/usr_avp.c: No such file or directory. (gdb) bt #0 0x0000000000663091 in destroy_avp_list_unsafe (list=0x7efe49050bd0) at core/usr_avp.c:625 #1 0x00007efe6b4daadc in free_cell_helper (dead_cell=0x7efe49050a48, silent=0, fname=0x7efe6b5db100 "timer.c", fline=654) at h_table.c:242 #2 0x00007efe6b587703 in wait_handler (ti=1289862248, wait_tl=0x7efe49050ac8, data=0x7efe49050a48) at timer.c:654 #3 0x0000000000650af0 in timer_list_expire (t=1289862248, h=0x7efe42c476c8, slow_l=0x7efe42c47e20, slow_mark=92) at core/timer.c:874 #4 0x0000000000650f50 in timer_handler () at core/timer.c:939 #5 0x00000000006513fe in timer_main () at core/timer.c:978 #6 0x0000000000427e7b in main_loop () at main.c:1698 #7 0x000000000042eb71 in main (argc=17, argv=0x7fff90152e78) at main.c:2645 (gdb) bt full #0 0x0000000000663091 in destroy_avp_list_unsafe (list=0x7efe49050bd0) at core/usr_avp.c:625 avp = 0xec83485354415541 foo = 0xec83485354415541 __FUNCTION__ = "destroy_avp_list_unsafe" #1 0x00007efe6b4daadc in free_cell_helper (dead_cell=0x7efe49050a48, silent=0, fname=0x7efe6b5db100 "timer.c", fline=654) at h_table.c:242 b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x16b4d8226 cbs = 0x0 cbs_tmp = 0x7fff90152860 __FUNCTION__ = "free_cell_helper" #2 0x00007efe6b587703 in wait_handler (ti=1289862248, wait_tl=0x7efe49050ac8, data=0x7efe49050a48) at timer.c:654 p_cell = 0x7efe49050a48 ret = 32767 #3 0x0000000000650af0 in timer_list_expire (t=1289862248, h=0x7efe42c476c8, slow_l=0x7efe42c47e20, slow_mark=92) at core/timer.c:874 tl = 0x7efe49050ac8 ret = 0 #4 0x0000000000650f50 in timer_handler () at core/timer.c:939 saved_ticks = 1289862248 run_slow_timer = 0 i = 92 __FUNCTION__ = "timer_handler" #5 0x00000000006513fe in timer_main () at core/timer.c:978 No locals. #6 0x0000000000427e7b in main_loop () at main.c:1698 i = 32 pid = 0 si = 0x0 si_desc = "udp receiver child=31 sock=172.20.40.6:5060\000\376~\000\000\200+\025\220\377\177\000\000\000E\004\204\252\235\005\320\200+\025\220\377\177\000\000\374e#V\000\000\000\000\060\261A\000\000\000\000\000p.\025\220\377\177", '\000' <repeats 18 times>, "`+\025\220\377\177\000\000\354uV\000\000\000\000" nrprocs = 32 woneinit = 1 __FUNCTION__ = "main_loop" #7 0x000000000042eb71 in main (argc=17, argv=0x7fff90152e78) at main.c:2645 cfg_stream = 0x28e6010 c = -1 r = 0 tmp = 0x7fff90154f9b "" tmp_len = 1878702080 port = 32510 proto = 4221504 options = 0x761e08 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1275650845 rfd = 6 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7efe70963520 p = 0x0 st = {st_dev = 14, st_ino = 25628, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 160, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1532943957, tv_nsec = 228188274}, st_mtim = { tv_sec = 1533222622, tv_nsec = 138276459}, st_ctim = {tv_sec = 1533222624, tv_nsec = 666345913}, __unused = {0, 0, 0}} __FUNCTION__ = "main" (gdb) info locals avp = 0xec83485354415541 foo = 0xec83485354415541 __FUNCTION__ = "destroy_avp_list_unsafe" (gdb) list 620 in core/usr_avp.c ```
As there are two core dumps from the same time from different PIDs, here's the other gdb output:
``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 31322]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x00007efe6b4d9635 in free_cell_helper (dead_cell=0x7efe485bf778, silent=1, fname=0x7efe6b5b8119 "h_table.c", fline=448) at h_table.c:171 171 h_table.c: No such file or directory. (gdb) bt #0 0x00007efe6b4d9635 in free_cell_helper (dead_cell=0x7efe485bf778, silent=1, fname=0x7efe6b5b8119 "h_table.c", fline=448) at h_table.c:171 #1 0x00007efe6b4db2c7 in free_hash_table () at h_table.c:448 #2 0x00007efe6b507caf in tm_shutdown () at t_funcs.c:89 #3 0x00000000005d522e in destroy_modules () at core/sr_module.c:832 #4 0x000000000041baa3 in cleanup (show_status=1) at main.c:528 #5 0x000000000041d132 in shutdown_children (sig=15, show_status=1) at main.c:670 #6 0x000000000041fd54 in handle_sigs () at main.c:775 #7 0x0000000000428d5b in main_loop () at main.c:1759 #8 0x000000000042eb71 in main (argc=17, argv=0x7fff90152e78) at main.c:2645 (gdb) bt full #0 0x00007efe6b4d9635 in free_cell_helper (dead_cell=0x7efe485bf778, silent=1, fname=0x7efe6b5b8119 "h_table.c", fline=448) at h_table.c:171 b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x6442c024c0 cbs = 0x30363035362e cbs_tmp = 0x7efe4826bd48 __FUNCTION__ = "free_cell_helper" #1 0x00007efe6b4db2c7 in free_hash_table () at h_table.c:448 p_cell = 0x7efe485bf778 tmp_cell = 0x7efe42c9d5e0 i = 8123 __FUNCTION__ = "free_hash_table" #2 0x00007efe6b507caf in tm_shutdown () at t_funcs.c:89 __FUNCTION__ = "tm_shutdown" #3 0x00000000005d522e in destroy_modules () at core/sr_module.c:832 t = 0x7efe6dff9428 foo = 0x7efe6dff8db0 __FUNCTION__ = "destroy_modules" #4 0x000000000041baa3 in cleanup (show_status=1) at main.c:528 memlog = 0 __FUNCTION__ = "cleanup" #5 0x000000000041d132 in shutdown_children (sig=15, show_status=1) at main.c:670 __FUNCTION__ = "shutdown_children" #6 0x000000000041fd54 in handle_sigs () at main.c:775 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = 8 __FUNCTION__ = "handle_sigs" #7 0x0000000000428d5b in main_loop () at main.c:1759 i = 32 pid = 31412 si = 0x0 si_desc = "udp receiver child=31 sock=172.20.40.6:5060\000\376~\000\000\200+\025\220\377\177\000\000\000E\004\204\252\235\005\320\200+\025\220\377\177\000\000\374e#V\000\000\000\000\060\261A\000\000\000\000\000p.\025\220\377\177", '\000' <repeats 18 times>, "`+\025\220\377\177\000\000\354uV\000\000\000\000" nrprocs = 32 woneinit = 1 __FUNCTION__ = "main_loop" #8 0x000000000042eb71 in main (argc=17, argv=0x7fff90152e78) at main.c:2645 cfg_stream = 0x28e6010 c = -1 r = 0 tmp = 0x7fff90154f9b "" tmp_len = 1878702080 port = 32510 proto = 4221504 options = 0x761e08 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1275650845 rfd = 6 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7efe70963520 p = 0x0 st = {st_dev = 14, st_ino = 25628, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 160, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1532943957, tv_nsec = 228188274}, st_mtim = { tv_sec = 1533222622, tv_nsec = 138276459}, st_ctim = {tv_sec = 1533222624, tv_nsec = 666345913}, __unused = {0, 0, 0}} __FUNCTION__ = "main" (gdb) info locals b = 0x0 i = 1 rpl = 0x0 tt = 0x0 foo = 0x6442c024c0 cbs = 0x30363035362e cbs_tmp = 0x7efe4826bd48 __FUNCTION__ = "free_cell_helper" (gdb) list 166 in h_table.c ```
A few minutes later, we had another crash, again with two core dumps.
1: ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 34380]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2dea9d1998 in t_forward_nonack (t=0x7f2dc717ab98, p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0) at t_fwd.c:1721 1721 t_fwd.c: No such file or directory. (gdb) bt full #0 0x00007f2dea9d1998 in t_forward_nonack (t=0x7f2dc717ab98, p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0) at t_fwd.c:1721 branch_ret = 0 lowest_ret = -1 current_uri = {s = 0x0, len = 0} added_branches = 1 first_branch = 0 i = 1711599296 q = -1 t_invite = 0x7ffe66048cb0 success_branch = 32766 try_new = 1 lock_replies = -358549307 dst_uri = {s = 0x0, len = 0} path = {s = 0x0, len = 0} instance = {s = 0x0, len = 0} ruid = {s = 0x0, len = 0} location_ua = {s = 0x0, len = 0} si = 0x0 backup_bflags = 16 bflags = 0 __FUNCTION__ = "t_forward_nonack" #1 0x00007f2dea9bb0fb in t_relay_to (p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0, replicate=0) at t_funcs.c:331 ret = 0 new_tran = -10 t = 0x7f2dc717ab98 dst = {send_sock = 0x7a4d80, to = {s = {sa_family = 61120, sa_data = "\004f\340\000\000\000\244\323z\000\000\000\000"}, sin = {sin_family = 61120, sin_port = 26116, sin_addr = {s_addr = 224}, sin_zero = "\244\323z\000\000\000\000"}, sin6 = {sin6_family = 61120, sin6_port = 26116, sin6_flowinfo = 224, sin6_addr = {__in6_u = {__u6_addr8 = "\244\323z\000\000\000\000\000\021Nz\000\000\000\000", __u6_addr16 = {54180, 122, 0, 0, 19985, 122, 0, 0}, __u6_addr32 = { 8049572, 0, 8015377, 0}}}, sin6_scope_id = 944}}, id = 0, proto = 16 '\020', send_flags = {f = 60740, blst_imask = 32557}} port = 32766 host = {s = 0x0, len = 1482919684} comp = 0 __FUNCTION__ = "t_relay_to" #2 0x00007f2deaa475ad in _w_t_relay_to (p_msg=0x7f2ded9f17a0, proxy=0x0, force_proto=0) at tm.c:1555 t = 0x57d382 res = 1711575656 __FUNCTION__ = "_w_t_relay_to" #3 0x00007f2deaa48832 in w_t_relay (p_msg=0x7f2ded9f17a0, _foo=0x0, _bar=0x0) at tm.c:1755 No locals. #4 0x0000000000437599 in do_action (h=0x7ffe660497e0, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1067 ret = -5 v = 778400865 dst = {send_sock = 0x7ffe660496ec, to = {s = {sa_family = 38704, sa_data = "\004f\376\177\000\000\000\000\000\000\001\000\000"}, sin = {sin_family = 38704, sin_port = 26116, sin_addr = {s_addr = 32766}, sin_zero = "\000\000\000\000\001\000\000"}, sin6 = {sin6_family = 38704, sin6_port = 26116, sin6_flowinfo = 32766, sin6_addr = {__in6_u = {__u6_addr8 = "\000\000\000\000\001\000\000\000\001\000\000\000\001\000\000", __u6_addr16 = {0, 0, 1, 0, 1, 0, 1, 0}, __u6_addr32 = {0, 1, 1, 1}}}, sin6_scope_id = 0}}, id = 0, proto = -96 '\240', send_flags = {f = 26116, blst_imask = 32766}} tmp = 0x0 new_uri = 0x7f2ded771524 "presenced.1587246.transfer.routing-enum.sipgate.net" end = 0x7f2ded5c6d08 "\002" crt = 0x7ffe660492d0 "" cmd = 0x7f2ded4aaa08 len = 1953719668 user = 1702047332 uri = {user = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, passwd = {s = 0x0, len = 0}, host = {s = 0x7ffe66049510 "\300\226\004f\376\177", len = 5785753}, port = { s = 0x7f2ded5c6d08 "\002", len = 0}, params = {s = 0x3 <Address 0x3 out of bounds>, len = -308405224}, sip_params = {s = 0x0, len = 1711576640}, headers = {s = 0x7ffe6604974c "\377\377\377\377", len = 1711576652}, port_no = 38200, proto = 26116, type = 32766, flags = 1711576528, transport = {s = 0x1007ffe660493a8 <Address 0x1007ffe660493a8 out of bounds>, len = -276933127}, ttl = {s = 0x7ffe00000000 <Address 0x7ffe00000000 out of bounds>, len = 3}, user_param = {s = 0x7ffe66049570 "_sip._udp.sgtestd.service.sipgate.net.netzquadrat.net", len = 1}, maddr = {s = 0x7ffe6604950f "", len = 8192}, method = {s = 0x7ffe66049530 "", len = 1711576432}, lr = { s = 0x7ffe66049570 "_sip._udp.sgtestd.service.sipgate.net.netzquadrat.net", len = 1026}, r2 = {s = 0x37 <Address 0x37 out of bounds>, len = 10973856}, gr = {s = 0x1 <Address 0x1 out of bounds>, len = -276932218}, transport_val = {s = 0x755f047069735f04 <Address 0x755f047069735f04 out of bounds>, len = 1929867364}, ttl_val = {s = 0x6976726573076474 <Address 0x6976726573076474 out of bounds>, len = 1929864547}, user_param_val = { s = 0x6e0b74656e036574 <Address 0x6e0b74656e036574 out of bounds>, len = 1903850597}, maddr_val = {s = 0x74656e037461 <Address 0x74656e037461 out of bounds>, len = -280413204}, method_val = {s = 0x7f2ded5c6510 "\a", len = -276936262}, lr_val = {s = 0x7f2def7d4e40 "", len = 1}, r2_val = {s = 0x7f2d00000003 <Address 0x7f2d00000003 out of bounds>, len = 0}, gr_val = {s = 0x20 <Address 0x20 out of bounds>, len = 1711576928}} next_hop = {user = {s = 0x7ffe6604986c "", len = 1711576940}, passwd = {s = 0x7ffe66049658 "\350\226\004f\376\177", len = 1711576816}, host = {s = 0x1007ffe660494c8 <Address 0x1007ffe660494c8 out of bounds>, len = 1711577864}, port = {s = 0x7ffe660496e8 "\217", len = 38201672}, params = {s = 0x7f2def7d4e40 "", len = 1}, sip_params = {s = 0x7ffe6604976c "\asgtestd\aser\004\217cX", len = 256}, headers = {s = 0x7ffe66049650 "\320\226\004f\376\177", len = 1711576816}, port_no = 38176, proto = 26116, type = 32766, flags = 1712850917, transport = {s = 0x7ffe660496c0 "\332 c[", len = 1711576768}, ttl = {s = 0x7ffe66049540 "", len = 164420000}, user_param = { s = 0x7f2def7d7300 "\001", len = -276991232}, maddr = {s = 0x7f2def7d7300 "\001", len = 1}, method = {s = 0x0, len = -276941688}, lr = {s = 0x7ffe660499d0 "0\261A", len = 1711589200}, r2 = { s = 0x64755f2e7069735f <Address 0x64755f2e7069735f out of bounds>, len = 1735601776}, gr = {s = 0x6369767265732e64 <Address 0x6369767265732e64 out of bounds>, len = 1769156197}, transport_val = { s = 0x656e2e74656e2e65 <Address 0x656e2e74656e2e65 out of bounds>, len = 1970371188}, ttl_val = {s = 0x74656e2e74 <Address 0x74656e2e74 out of bounds>, len = -270498136}, user_param_val = { s = 0x7ffe0000008f <Address 0x7ffe0000008f out of bounds>, len = 1533223131}, maddr_val = {s = 0x2d6604c7a0 <Address 0x2d6604c7a0 out of bounds>, len = 45}, method_val = {s = 0x166049660 <Address 0x166049660 out of bounds>, len = 1711576928}, lr_val = {s = 0x1 <Address 0x1 out of bounds>, len = 1711576768}, r2_val = {s = 0x66049640 <Address 0x66049640 out of bounds>, len = 0}, gr_val = {s = 0x0, len = 1}} u = 0x57c9e1 port = 11877 dst_host = 0x0 i = 1667855986 flags = 7628142 avp = 0x0 st = {flags = 0, id = 32, name = {n = 1711576884, s = {s = 0x7ffe66049734 "-\177", len = -308340832}, re = 0x7ffe66049734}, avp = 0x58638f04} sct = 0x94dce0809c190b00 sjt = 0x7ffe00000000 rve = 0x0 mct = 0x58638f04 rv = 0x0 rv1 = 0x0 c1 = {cache_type = 1711579824, val_type = 32766, c = {avp_val = {n = 1711576804, s = {s = 0x7ffe660496e4 "\020", len = 1711576784}, re = 0x7ffe660496e4}, pval = {rs = {s = 0x7ffe660496e4 "\020", len = 1711576784}, ri = 1711576808, flags = 32766}}, i2s = "\230\226\004f\376\177\000\000\300\356\004f\001\000\000\000\350\226\004f\376\177"} s = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296} srevp = {0x0, 0x0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #5 0x000000000044445a in run_actions (h=0x7ffe660497e0, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded734a20 ret = -1 mod = 0x7467730700000000 ms = 0 __FUNCTION__ = "run_actions" #6 0x0000000000444b3b in run_actions_safe (h=0x7ffe6604e2f0, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1633 ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {1482919684, -1089664629675265400, 4305200, 140730610020032, 0, 0, -1089664629721402744, 1088876375176245896}, __mask_was_saved = 0, __saved_mask = { __val = {71, 38070316, 140730609998288, 140730609998288, 1026, 55, 10982048, 0, 139835268290950, 8457483506077097732, 8315180274669809764, 7599387201281418356, 0, 0, 4294967296, 10973600}}}}} ret = 32557 ign_on_break = 0 #7 0x000000000057ebdd in rval_get_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, i=0x7ffe66049cd4, rv=0x7f2ded733a88, cache=0x0) at core/rvalue.c:912 r_avp = 0x7f2def7e9324 avp_val = {n = 0, s = {s = 0x0, len = 1711579239}, re = 0x0} pval = {rs = {s = 0xa772a0 "\226\361\205\203", len = 10982048}, ri = 1, flags = 0} tmp = {s = 0x7ffe6604a020 "C_\356\351-\177", len = 1711579239} s = 0x0 r = -276917473 ret = 0 destroy_pval = 0 __FUNCTION__ = "rval_get_int" #8 0x0000000000583264 in rval_expr_eval_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, res=0x7ffe66049cd4, rve=0x7f2ded733a80) at core/rvalue.c:1910 i1 = 32557 i2 = -280424894 ret = -1 c1 = {cache_type = RV_CACHE_AVP, val_type = RV_INT, c = {avp_val = {n = -312806152, s = {s = 0x7f2ded5af4f8 "\002", len = 1711578160}, re = 0x7f2ded5af4f8}, pval = {rs = {s = 0x7f2ded5af4f8 "\002", len = 1711578160}, ri = -1676080384, flags = -1797463936}}, i2s = "\000\000\000\000\000\000\000\000\004\217cX#\240\004f\000\000\000\000\000"} c2 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, pval = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}}, i2s = " \000\000\000\000\000\000\000\370\364Z\355-\177\000\000\000\000\000\000(\240"} rv1 = 0x1 rv2 = 0x1 __FUNCTION__ = "rval_expr_eval_int" #9 0x00000000005836c3 in rval_expr_eval_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, res=0x7ffe66049fd4, rve=0x7f2ded735068) at core/rvalue.c:1918 i1 = 32766 i2 = 0 ret = -1 c1 = {cache_type = 3983625136, val_type = 32557, c = {avp_val = {n = -308340832, s = {s = 0x7f2ded9f17a0 "-\003", len = 176}, re = 0x7f2ded9f17a0}, pval = {rs = {s = 0x7f2ded9f17a0 "-\003", len = 176}, ri = 8365360, flags = 0}}, i2s = "\003\000\000\000\000\000\000\000\260Kq\355-\177\000\000\000\000\000\000\000"} c2 = {cache_type = 16, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 1711578704}, re = 0x0}, pval = {rs = {s = 0x0, len = 1711578704}, ri = 1482919684, flags = 0}}, i2s = "0\261A\000\000\000\000\000\300\356\004f\376\177\000\000\000\000\000\000\000"} rv1 = 0xffffffff rv2 = 0x7f2ded718348 __FUNCTION__ = "rval_expr_eval_int" #10 0x000000000043703d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded738438, msg=0x7f2ded9f17a0) at core/action.c:1043 ret = -5 v = 32766 dst = {send_sock = 0x0, to = {s = {sa_family = 43208, sa_data = "\004f\000\000\000\000\030\235M\355-\177\000"}, sin = {sin_family = 43208, sin_port = 26116, sin_addr = {s_addr = 0}, sin_zero = "\030\235M\355-\177\000"}, sin6 = { sin6_family = 43208, sin6_port = 26116, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\030\235M\355-\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {40216, 60749, 32557, 0, 0, 0, 0, 0}, __u6_addr32 = { 3981286680, 32557, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 1}} tmp = 0x58638f04 <Address 0x58638f04 out of bounds> new_uri = 0x3000000010 <Address 0x3000000010 out of bounds> end = 0x7ffe6604a430 "\310aZ" crt = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u" cmd = 0x7f2ded4b8b58 len = 0 user = 1711579232 uri = {user = {s = 0x2010 <Address 0x2010 out of bounds>, len = 0}, passwd = {s = 0x7f2ded9ba638 "\001", len = 1711580536}, host = {s = 0x7f2ded9f17a0 "-\003", len = -966039344}, port = {s = 0x0, len = 0}, params = { s = 0x7f2de97ab7a0 "2457624e1@sipgate.de", len = 20}, sip_params = {s = 0x7f2ded6e82e0 "\002", len = 24}, headers = {s = 0x7f2dc66b6ca1 "sip:217.10.79.9;lr;received=sip:35.197.222.14:5060>", len = 50}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 0, transport = {s = 0x7f2dc66b6ca0 "sip:217.10.79.9;lr;received=sip:35.197.222.14:5060", len = 52}, ttl = {s = 0x7f2ded6e82e4 "", len = 9}, user_param = {s = 0x0, len = 0}, maddr = { s = 0x7f2ded6e82ee "", len = 10}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 0}, ttl_val = { s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 0, transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 1}, user_param = {s = 0x7f2ded4d9d30 "", len = -308340832}, maddr = {s = 0x7f2ded9b9818 "\a", len = 0}, method = { s = 0x46604a518 <Address 0x46604a518 out of bounds>, len = -445341169}, lr = {s = 0x7f2ded4d9d7f "useronline)", len = 10}, r2 = {s = 0x1900000000 <Address 0x1900000000 out of bounds>, len = 1711581300}, gr = { s = 0x6604a8c8005a61c8 <Address 0x6604a8c8005a61c8 out of bounds>, len = -1676080384}, transport_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, ttl_val = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604a650 "", len = 5782197}, method_val = { s = 0x6604a4c0 <Address 0x6604a4c0 out of bounds>, len = -445340088}, lr_val = {s = 0x7f2ded9b9f28 "\001", len = -308576824}, r2_val = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, gr_val = { s = 0x1 <Address 0x1 out of bounds>, len = 100}} u = 0x7ffe6604eec0 port = 0 dst_host = 0x52a5fc i = 0 flags = 5078790 avp = 0x0 st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 1711579808}, re = 0x0}, avp = 0x57d918} sct = 0x0 sjt = 0xffffffa8 rve = 0x7f2ded735068 mct = 0x0 rv = 0x7ffe6604a1f0 rv1 = 0x0 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -308568280, s = {s = 0x7f2ded9b9f28 "\001", len = 1}, re = 0x7f2ded9b9f28}, pval = {rs = {s = 0x7f2ded9b9f28 "\001", len = 1}, ri = -1676080384, flags = -1797463936}}, i2s = "\030\034\236\355-\177\000\000\004\217cX\000\000\000\000\060\261A\000\000"} s = {s = 0x7f2dc230f130 "p\360\060\302-\177", len = -308340832} srevp = {0x7ddd14, 0x100000000} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #11 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded734508, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded738438 ret = 1 mod = 0x7f2dc717ab98 ms = 0 __FUNCTION__ = "run_actions" #12 0x0000000000433d5d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded9bfb50, msg=0x7f2ded9f17a0) at core/action.c:691 ret = -5 v = 0 dst = {send_sock = 0x246f270, to = {s = {sa_family = 45072, sa_data = "D\355-\177\000\000\320\255\004f\376\177\000"}, sin = {sin_family = 45072, sin_port = 60740, sin_addr = {s_addr = 32557}, sin_zero = "\320\255\004f\376\177\000"}, sin6 = {sin6_family = 45072, sin6_port = 60740, sin6_flowinfo = 32557, sin6_addr = {__in6_u = {__u6_addr8 = "\320\255\004f\376\177\000\000i\222\035\344-\177\000", __u6_addr16 = {44496, 26116, 32766, 0, 37481, 58397, 32557, 0}, __u6_addr32 = {1711582672, 32766, 3827143273, 32557}}}, sin6_scope_id = 3983434840}}, id = 32557, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} tmp = 0x7f2d00000000 <Address 0x7f2d00000000 out of bounds> new_uri = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u" end = 0x7ffe6604eec0 "\021" crt = 0x0 cmd = 0x7f2ded4aadc0 len = 32557 user = 2 uri = {user = {s = 0x7f2dc230f130 "p\360\060\302-\177", len = -308340832}, passwd = {s = 0x0, len = -314265584}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 1482919684}, params = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x7ffe6604b270 "`\263\004f\376\177", len = 4421153}, port_no = 173, proto = 0, type = ERROR_URI_T, flags = 3944734720, transport = {s = 0x7ffe6604ab10 "P\253\004f\376\177", len = 4872016}, ttl = {s = 0x0, len = 8255817}, user_param = {s = 0x0, len = 6780192}, maddr = {s = 0x7ffe6604ab50 "", len = -314265584}, method = {s = 0x7f2deb1fd310 "core", len = -314265584}, lr = {s = 0x7ffe6604ab90 "\300\253\004f\376\177", len = -467824023}, r2 = {s = 0x7f2def7d7620 "\f\234<\002", len = 0}, gr = { s = 0x2000000 <Address 0x2000000 out of bounds>, len = 27858832}, transport_val = {s = 0x491de8 "", len = 5695600}, ttl_val = {s = 0x5a61c8 "", len = 8}, user_param_val = {s = 0xac <Address 0xac out of bounds>, len = -335884528}, maddr_val = {s = 0x7ffe6604abc0 "\300\254\004f\376\177", len = 4872016}, method_val = {s = 0x0, len = 8255817}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x7ffe6604acc0 "\220\337\231\355-\177", len = 6761026}, gr_val = {s = 0x7f2debfabb30 "db_mysql", len = -314265584}} next_hop = {user = {s = 0x1fe0 <Address 0x1fe0 out of bounds>, len = -311532544}, passwd = {s = 0x40041b130 <Address 0x40041b130 out of bounds>, len = -350235888}, host = {s = 0x0, len = 0}, port = { s = 0x7ffe6604acc0 "\220\337\231\355-\177", len = -350313618}, params = {s = 0x1fe0 <Address 0x1fe0 out of bounds>, len = -314265584}, sip_params = {s = 0x7f2debfabb30 "db_mysql", len = -467824023}, headers = { s = 0x7f2debfacf10 "db_mysql_get_columns", len = -335888704}, port_no = 16, proto = 0, type = ERROR_URI_T, flags = 3980701712, transport = {s = 0x200491db8 <Address 0x200491db8 out of bounds>, len = 5695552}, ttl = { s = 0x1005a61c8 <Address 0x1005a61c8 out of bounds>, len = -311497368}, user_param = {s = 0x7f2ded6eed68 "\020", len = -314265584}, maddr = {s = 0x7ffe6604acd0 "p", len = -311519352}, method = { s = 0x80041b130 <Address 0x80041b130 out of bounds>, len = -335889616}, lr = {s = 0x0, len = -308340832}, r2 = {s = 0x7f2ded99df90 "\374\f", len = 1711584160}, gr = {s = 0x70 <Address 0x70 out of bounds>, len = 2}, transport_val = {s = 0x1 <Address 0x1 out of bounds>, len = 5695568}, ttl_val = {s = 0x60 <Address 0x60 out of bounds>, len = -311809520}, user_param_val = {s = 0x246f270 "", len = 4305200}, maddr_val = {s = 0x7ffe6604eec0 "\021", len = 96}, method_val = {s = 0x8ef7d4e40 <Address 0x8ef7d4e40 out of bounds>, len = -335889616}, lr_val = {s = 0x7ffe6604eec0 "\021", len = 0}, r2_val = {s = 0x7ffe6604ade0 "0\261A", len = 6780192}, gr_val = { s = 0x7ffe6604ad90 "", len = -314265584}} u = 0x7ffe00000000 port = 0 dst_host = 0x58638f04 i = 10 flags = -380160831 avp = 0x0 st = {flags = 38204512, id = 0, name = {n = 197, s = {s = 0xc5 <Address 0xc5 out of bounds>, len = 38204496}, re = 0xc5}, avp = 0x0} sct = 0x0 sjt = 0x7ffe6604aa20 rve = 0x7f2ded9bad40 mct = 0x7f2de957155a rv = 0x0 rv1 = 0x0 c1 = {cache_type = 33554432, val_type = RV_NONE, c = {avp_val = {n = 27858872, s = {s = 0x1a917b8 <Address 0x1a917b8 out of bounds>, len = 4791744}, re = 0x1a917b8}, pval = {rs = {s = 0x1a917b8 <Address 0x1a917b8 out of bounds>, len = 4791744}, ri = 5695560, flags = 0}}, i2s = "\310aZ\000\000\000\000\000\b\000\000\000\000\000\000\000\257\000\000\000\000"} s = {s = 0x2010 <Address 0x2010 out of bounds>, len = 0} srevp = {0x7f2dc230f130, 0x7f2ded9f17a0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #13 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded9b7bd0, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded9bfb50 ret = 1 mod = 0x673ba6 ms = 0 __FUNCTION__ = "run_actions" #14 0x0000000000433d5d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded99e2a8, msg=0x7f2ded9f17a0) at core/action.c:691 ret = -5 v = 0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = "\000\000\000\000\000\000\060^\220\355-\177\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "0^\220\355-\177\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "0^\220\355-\177\000\000\070}\000\343-\177\000", __u6_addr16 = {24112, 60816, 32557, 0, 32056, 58112, 32557, 0}, __u6_addr32 = {3985661488, 32557, 3808460088, 32557}}}, sin6_scope_id = 3808424056}}, id = 32557, proto = 88 'X', send_flags = {f = 58111, blst_imask = 32557}} tmp = 0x94dce0809c190b00 <Address 0x94dce0809c190b00 out of bounds> new_uri = 0x7ffe6604b360 "\200\264\004f\376\177" end = 0x44445a "\211E\254H\307\005\350\206a" crt = 0x9 <Address 0x9 out of bounds> cmd = 0x7ffe6604b310 len = 32766 user = 1711583760 uri = {user = {s = 0x0, len = 1711584128}, passwd = {s = 0x7ffe00000000 <Address 0x7ffe00000000 out of bounds>, len = 5779786}, host = {s = 0x0, len = 0}, port = {s = 0x7f2ded99b590 "\a", len = 1482919684}, params = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x7ffe6604b480 "\260\265\004f\376\177", len = 4475707}, port_no = 47488, proto = 125, type = ERROR_URI_T, flags = 3986626464, transport = {s = 0x7f2ded99df90 "\374\f", len = 1711596272}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 1}, maddr = {s = 0x58638f04 <Address 0x58638f04 out of bounds>, len = -1369050488}, method = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, lr = {s = 0x0, len = 0}, r2 = {s = 0xf0e0bbcaada5f288 <Address 0xf0e0bbcaada5f288 out of bounds>, len = 1253175944}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x7ffe6604b440 "\200\264\004f\376\177", len = 6789099}, ttl_val = {s = 0x7db980 "core", len = -309305704}, user_param_val = {s = 0x7ddc98 "destroy_avp", len = 8239493}, maddr_val = {s = 0x7f2dc7176e60 "_", len = -1039486976}, method_val = {s = 0x7ffe6604b480 "\260\265\004f\376\177", len = 6696862}, lr_val = {s = 0x7ffe00000006 <Address 0x7ffe00000006 out of bounds>, len = -954765728}, r2_val = {s = 0x7ffe00000001 <Address 0x7ffe00000001 out of bounds>, len = -309305704}, gr_val = {s = 0x7f2dc7176e60 "_", len = 0}} next_hop = {user = {s = 0x7ffe6604b5b0 "\220\267\004f\376\177", len = 5762013}, passwd = {s = 0x7ffe6604b560 "\340\325g\355-\177", len = -309305704}, host = {s = 0x6 <Address 0x6 out of bounds>, len = 0}, port = { s = 0x7f2ded99bc98 "!", len = 1711584948}, params = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, sip_params = {s = 0x7ffe00000006 <Address 0x7ffe00000006 out of bounds>, len = -954765872}, headers = {s = 0x7ffe6604b5a0 "", len = 0}, port_no = 58720, proto = 60825, type = 32557, flags = 1711585716, transport = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, ttl = {s = 0x7f2ded4bc420 "-\250\035\344-\177", len = 0}, user_param = { s = 0x7f2ded905e48 "", len = -308340832}, maddr = {s = 0x0, len = 0}, method = {s = 0x7f2ded914750 "\001", len = 1711584816}, lr = {s = 0x0, len = -308340832}, r2 = {s = 0x7f2ded67d5e0 "\006\006", len = 1711586368}, gr = { s = 0x0, len = -1676080384}, transport_val = {s = 0xfffffffa <Address 0xfffffffa out of bounds>, len = 1482919684}, ttl_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604b790 "@\277\004f\376\177", len = 5780068}, method_val = {s = 0x7ffe6604b5e0 "\240\027\237\355-\177", len = -359105917}, lr_val = {s = 0x7f2ded91c788 "\a", len = -309213624}, r2_val = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, gr_val = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 0}} u = 0x58638f04 port = 0 dst_host = 0x0 i = 50 flags = 1711583792 avp = 0x41b130 st = {flags = 1711584112, id = 9, name = {n = -308340832, s = {s = 0x7f2ded9f17a0 "-\003", len = -308682864}, re = 0x7f2ded9f17a0}, avp = 0x7ffe6604b3a0} sct = 0x52a5fc sjt = 0x41b130 rve = 0x0 mct = 0x7ffe6604eec0 rv = 0x5a00000101 rv1 = 0x7ffe6604eec0 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -309213304, s = {s = 0x7f2ded91c788 "\a", len = 0}, re = 0x7f2ded91c788}, pval = {rs = {s = 0x7f2ded91c788 "\a", len = 0}, ri = -1676080384, flags = -1797463936}}, i2s = "`*\312\352-\177\000\000\004\217cX\000\000\000\000\060\261A\000\000"} s = {s = 0x0, len = 0} srevp = {0x7ffe6604b360, 0x7ffe6604b380} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #15 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded99e2a8, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded99e2a8 ret = -1 mod = 0x94dce0809c190b00 ms = 0 __FUNCTION__ = "run_actions" #16 0x0000000000437505 in do_action (h=0x7ffe6604e2f0, a=0x7f2ded9a1620, msg=0x7f2ded9f17a0) at core/action.c:1058 ret = 1 v = 1 dst = {send_sock = 0x7ffe6604bf00, to = {s = {sa_family = 36612, sa_data = "cX\000\000\000\000\020\322g\355-\177\000"}, sin = {sin_family = 36612, sin_port = 22627, sin_addr = {s_addr = 0}, sin_zero = "\020\322g\355-\177\000"}, sin6 = {sin6_family = 36612, sin6_port = 22627, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\020\322g\355-\177\000\000\300\356\004f\376\177\000", __u6_addr16 = {53776, 60775, 32557, 0, 61120, 26116, 32766, 0}, __u6_addr32 = {3983004176, 32557, 1711599296, 32766}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} tmp = 0x94dce0809c190b00 <Address 0x94dce0809c190b00 out of bounds> new_uri = 0x7ffe6604bc00 " \275\004f\376\177" end = 0x44445a "\211E\254H\307\005\350\206a" crt = 0x7db980 "core" cmd = 0x4f00000000 len = 32766 user = 0 uri = {user = {s = 0x0, len = 1482919684}, passwd = {s = 0x0, len = -1039452992}, host = {s = 0x0, len = 0}, port = {s = 0x7ffe6604bc20 "\340\325g\355-\177", len = 1482919684}, params = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x7ffe6604bd20 "P\276\004f\376\177", len = 4475707}, port_no = 46, proto = 0, type = 6258477, flags = 3986626464, transport = {s = 0x7f2ded67d5e0 "\006\006", len = 1711596272}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = -6}, maddr = {s = 0x58638f04 <Address 0x58638f04 out of bounds>, len = -1331301752}, method = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, lr = {s = 0x0, len = 0}, r2 = {s = 0xf0e0bbcabee5f288 <Address 0xf0e0bbcabee5f288 out of bounds>, len = 1253175944}, gr = {s = 0x7f2d00000000 <Address 0x7f2d00000000 out of bounds>, len = -954765712}, transport_val = {s = 0x7ffe6604bd40 "0\261A", len = -313764040}, ttl_val = {s = 0x10 <Address 0x10 out of bounds>, len = 1482919684}, user_param_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x7ffe6604bd30 "\200\275\004f\376\177", len = 6688887}, lr_val = {s = 0x0, len = 1482919684}, r2_val = {s = 0x7f2ded67d277 "sendpush_call)", len = 13}, gr_val = {s = 0x111ed67d228 <Address 0x111ed67d228 out of bounds>, len = -954749664}} next_hop = {user = {s = 0x7ffe6604be50 "0\300\004f\376\177", len = 5762013}, passwd = {s = 0x7ffe6604bd80 "@\276\004f\376\177", len = 6689138}, host = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 0}, port = {s = 0x58638f04 <Address 0x58638f04 out of bounds>, len = -311963017}, params = {s = 0xd <Address 0xd out of bounds>, len = 1711599296}, sip_params = {s = 0x1000000000 <Address 0x1000000000 out of bounds>, len = -954749664}, headers = {s = 0x7ffe6604be40 "", len = 0}, port_no = 57368, proto = 60775, type = 32557, flags = 1711587924, transport = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, ttl = {s = 0x7ffe6604bed0 "", len = 0}, user_param = {s = 0x7f2ded67d228 "", len = -308340832}, maddr = {s = 0x0, len = 0}, method = {s = 0x7f2ded9f17a0 "-\003", len = -954750056}, lr = {s = 0x7f2ded67d277 "sendpush_call)", len = 13}, r2 = {s = 0x0, len = 1482919684}, gr = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, transport_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, ttl_val = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604c030 "\340\307\004f\376\177", len = 5780068}, method_val = { s = 0x17ffe6604bf20 <Address 0x17ffe6604bf20 out of bounds>, len = 0}, lr_val = {s = 0x7f2ded67c610 "\001", len = -311964392}, r2_val = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, gr_val = {s = 0x0, len = -311495504}} u = 0x58638f04 port = 0 dst_host = 0x0 i = 0 flags = 1711585888 avp = 0x41b130 st = {flags = 1711586256, id = 1, name = {n = -308340832, s = {s = 0x7f2ded9f17a0 "-\003", len = -311962144}, re = 0x7f2ded9f17a0}, avp = 0x7ffe6604bc40} sct = 0x7f2dc7170808 sjt = 0x7ffe6604bb70 rve = 0x7f2ded99e558 mct = 0x181004daff1 rv = 0x7ffe6604bbd0 rv1 = 0x7ffe6604eec0 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -311966192, s = {s = 0x7f2ded67c610 "\001", len = 0}, re = 0x7f2ded67c610}, pval = {rs = {s = 0x7f2ded67c610 "\001", len = 0}, ri = -1676080384, flags = -1797463936}}, i2s = "0\261A\000\000\000\000\000\004\217cX\000\000\000\000\060\261A\000\000"} s = {s = 0x7db980 "core", len = 6694789} srevp = {0x7ddd14, 0x7db985} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #17 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded99dd98, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded9a1620 ret = 1 mod = 0x94dce0809c190b00 ms = 0 __FUNCTION__ = "run_actions" #18 0x0000000000433d5d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded67ee90, msg=0x7f2ded9f17a0) at core/action.c:691 ret = -5 v = 0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = "\000\000\000\000\000\000\f\000\000\000\003\000\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\f\000\000\000\003\000\000"}, sin6 = { sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\f\000\000\000\003\000\000\000\360\350F\002\000\000\000", __u6_addr16 = {12, 0, 3, 0, 59632, 582, 0, 0}, __u6_addr32 = {12, 3, 38201584, 0}}}, sin6_scope_id = 22}}, id = 0, proto = 64 '@', send_flags = {f = 61309, blst_imask = 32557}} tmp = 0x7f2def513ac3 "\211\302\203= K," new_uri = 0x7ad3d1 "rval_destroy" end = 0x7a4e11 "core: core/rvalue.c" crt = 0x0 cmd = 0x7f2ded4aa500 len = 32766 user = 10 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x7f2ded665f08 "\004", len = 1711589364}, host = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, port = {s = 0x7ffe6604c4a0 "", len = 0}, params = {s = 0x7db980 "core", len = 1482919684}, sip_params = {s = 0x0, len = 1711599296}, headers = {s = 0x0, len = 0}, port_no = 50448, proto = 26116, type = 32766, flags = 5779072, transport = {s = 0x0, len = -312969928}, ttl = { s = 0x2000000000 <Address 0x2000000000 out of bounds>, len = -1676080384}, user_param = {s = 0x7f2ded9f17a0 "-\003", len = 1482919684}, maddr = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, method = {s = 0x0, len = 0}, lr = {s = 0x7ffe6604c6f0 "\320\310\004f\376\177", len = 5781532}, r2 = {s = 0x7f2ded587538 "\002", len = 0}, gr = {s = 0x7f2dc7170d40 "", len = 1711589024}, transport_val = { s = 0x7f2ded4f02e0 "", len = -308340832}, ttl_val = {s = 0x7ffe6604c590 " \306\004f\376\177", len = -314265584}, user_param_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, maddr_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 6780192}, method_val = {s = 0x7ffe6604c5c0 "", len = -314265584}, lr_val = {s = 0x7ffe6604c620 "", len = -314265584}, r2_val = { s = 0x7ffe6604c600 "\240\027\237\355-\177", len = -467824023}, gr_val = {s = 0x5a61c8 "", len = 0}} next_hop = {user = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 27861712}, passwd = {s = 0x491410 "", len = 5692720}, host = {s = 0x5a61c8 "", len = 8}, port = {s = 0x7f2ded66a9f8 "\017", len = 1711589364}, params = { s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 28000, proto = 60760, type = 32557, flags = 3981997368, transport = {s = 0x7723d0 "core", len = -279900536}, ttl = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 1}, user_param = {s = 0x4913d8 "\213", len = 5692664}, maddr = {s = 0x5a61c8 "", len = 1711589168}, method = {s = 0xbc <Address 0xbc out of bounds>, len = 1711576608}, lr = {s = 0x7ffe66049610 "\354\226\004f\376\177", len = -308340832}, r2 = {s = 0x400000001 <Address 0x400000001 out of bounds>, len = 1711590132}, gr = {s = 0x7ffe6604a660 "\200\251\004f\376\177", len = -1676080384}, transport_val = {s = 0x7ffe00000002 <Address 0x7ffe00000002 out of bounds>, len = 1482919684}, ttl_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604c8d0 "\200\320\004f\376\177", len = 5781532}, method_val = {s = 0x16 <Address 0x16 out of bounds>, len = 1482919684}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = { s = 0x3ff000000000000 <Address 0x3ff000000000000 out of bounds>, len = 1}} u = 0x0 port = 0 dst_host = 0x8100000000 i = 43 flags = 1711588096 avp = 0x0 st = {flags = 3986626464, id = 32557, name = {n = 1711596272, s = {s = 0x7ffe6604e2f0 "\b", len = 1711588528}, re = 0x7ffe6604e2f0}, avp = 0x57c9e1} sct = 0x7f2ded44b010 sjt = 0x0 rve = 0x7f2ded67e010 mct = 0x581c4a rv = 0x0 rv1 = 0xb024ef c1 = {cache_type = 1711589488, val_type = 32766, c = {avp_val = {n = 38199328, s = {s = 0x246e020 "0", len = 0}, re = 0x246e020}, pval = {rs = {s = 0x246e020 "0", len = 0}, ri = 0, flags = 0}}, i2s = "0\351F\002\000\000\000\000\iL\357-\177\000\000\000\000\000\000\000"} s = {s = 0x7ffe6604c510 "\360\306\004f\376\177", len = 5762013} srevp = {0x7f2ded44b010, 0x7f2ded6e8298} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #19 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded670188, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded67ee90 ret = 1 mod = 0x94dce0809c190b00 ms = 0 __FUNCTION__ = "run_actions" #20 0x000000000043755b in do_action (h=0x7ffe6604e2f0, a=0x7f2ded67efc0, msg=0x7f2ded9f17a0) at core/action.c:1062 ret = 1 v = 0 dst = {send_sock = 0x0, to = {s = {sa_family = 54176, sa_data = "\004f\376\177\000\000\000\210S\355-\177\000"}, sin = {sin_family = 54176, sin_port = 26116, sin_addr = {s_addr = 32766}, sin_zero = "\000\210S\355-\177\000"}, sin6 = { sin6_family = 54176, sin6_port = 26116, sin6_flowinfo = 32766, sin6_addr = {__in6_u = {__u6_addr8 = "\000\210S\355-\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {34816, 60755, 32557, 0, 0, 0, 0, 0}, __u6_addr32 = { 3981674496, 32557, 0, 0}}}, sin6_scope_id = 3983451176}}, id = 32557, proto = -32 '\340', send_flags = {f = 60782, blst_imask = 32557}} tmp = 0x94dce0809c190b00 <Address 0x94dce0809c190b00 out of bounds> new_uri = 0x7ffe6604cd40 "`\316\004f\376\177" end = 0x44445a "\211E\254H\307\005\350\206a" crt = 0x7db980 "core" cmd = 0x7f2ded4aad18 len = 0 user = 9 uri = {user = {s = 0x7723d0 "core", len = 0}, passwd = {s = 0x7f2d00000000 <Address 0x7f2d00000000 out of bounds>, len = 1711591572}, host = {s = 0x0, len = 1711596272}, port = {s = 0x7ffe6604cd60 "x\X\355-\177", len = 1482919684}, params = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x7ffe6604ce60 "\220\317\004f\376\177", len = 4475707}, port_no = 56, proto = 0, type = 3014656, flags = 3986626464, transport = {s = 0x7f2ded585c78 ".\003", len = 1711596272}, ttl = {s = 0x76604ce70 <Address 0x76604ce70 out of bounds>, len = 0}, user_param = {s = 0x0, len = -1}, maddr = { s = 0x58638f04 <Address 0x58638f04 out of bounds>, len = -1676080384}, method = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, lr = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, r2 = {s = 0x0, len = 0}, gr = {s = 0x7ffe6604ce60 "\220\317\004f\376\177", len = 5663788}, transport_val = { s = 0x8 <Address 0x8 out of bounds>, len = -954766616}, ttl_val = {s = 0x100773258 <Address 0x100773258 out of bounds>, len = 1482919684}, user_param_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = -308340832}, maddr_val = {s = 0x7f2ded664e70 "\002", len = 1711596272}, method_val = {s = 0x16604ce70 <Address 0x16604ce70 out of bounds>, len = 6688887}, lr_val = {s = 0x7f2ded539108 "", len = 1482919684}, r2_val = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, gr_val = {s = 0x0, len = 0}} next_hop = {user = {s = 0x7ffe6604cf90 "p\321\004f\376\177", len = 5761911}, passwd = {s = 0x7ffe6604cec0 "\200\317\004f\376\177", len = 6689138}, host = {s = 0x100000008 <Address 0x100000008 out of bounds>, len = -313290488}, port = {s = 0x7f2ded54a118 "\020", len = 1711591572}, params = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, sip_params = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 11608256}, headers = {s = 0x7ffe6604cf80 "", len = 0}, port_no = 20208, proto = 60774, type = 32557, flags = 1711592340, transport = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, ttl = {s = 0x7ffe6604d010 "\b\221S\355-\177", len = 0}, user_param = {s = 0x7f2ded538818 "", len = -308340832}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x7f2ded538868 "src_addr))", len = 8}, r2 = {s = 0x7f2ded539108 "", len = 0}, gr = {s = 0x7f2ded6e8b58 "\250\003", len = -1676080384}, transport_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, ttl_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = { s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604d170 " \331\004f\376\177", len = 5780068}, method_val = {s = 0x5000000011 <Address 0x5000000011 out of bounds>, len = 0}, lr_val = {s = 0x7f2ded538a00 "\002", len = -313292608}, r2_val = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, gr_val = {s = 0x0, len = 1711591520}} u = 0x58638f04 port = 0 dst_host = 0x0 i = 34 flags = 0 avp = 0x41b130 st = {flags = 3986626464, id = 32557, name = {n = -308340832, s = {s = 0x7f2ded9f17a0 "-\003", len = -312976264}, re = 0x7f2ded9f17a0}, avp = 0x7ffe6604cd80} sct = 0x7ffe6604eec0 sjt = 0x0 rve = 0x7f2ded66c830 mct = 0x0 rv = 0x7ffe6604cd10 rv1 = 0x7ffe6604eec0 c1 = {cache_type = 3981676808, val_type = 32557, c = {avp_val = {n = -313292288, s = {s = 0x7f2ded538a00 "\002", len = 0}, re = 0x7f2ded538a00}, pval = {rs = {s = 0x7f2ded538a00 "\002", len = 0}, ri = -1676080384, flags = -1797463936}}, i2s = "0\261A\000\000\000\000\000\004\217cX\000\000\000\000\060\261A\000\000"} s = {s = 0x7ffe6604cdb0 "0\261A", len = 5761911} srevp = {0x7ddd14, 0x7db985} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #21 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded6688a8, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded67efc0 ret = 1 mod = 0x94dce0809c190b00 ms = 0 __FUNCTION__ = "run_actions" #22 0x0000000000437505 in do_action (h=0x7ffe6604e2f0, a=0x7f2ded67f0f0, msg=0x7f2ded9f17a0) at core/action.c:1058 ret = 1 v = 1 dst = {send_sock = 0x0, to = {s = {sa_family = 2498, sa_data = "\300\357-\177\000\000P\326M\355-\177\000"}, sin = {sin_family = 2498, sin_port = 61376, sin_addr = {s_addr = 32557}, sin_zero = "P\326M\355-\177\000"}, sin6 = { sin6_family = 2498, sin6_port = 61376, sin6_flowinfo = 32557, sin6_addr = {__in6_u = {__u6_addr8 = "P\326M\355-\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {54864, 60749, 32557, 0, 0, 0, 0, 0}, __u6_addr32 = { 3981301328, 32557, 0, 0}}}, sin6_scope_id = 4022357936}}, id = 32557, proto = -96 '\240', send_flags = {f = 61253, blst_imask = 32557}} tmp = 0x94dce0809c190b00 <Address 0x94dce0809c190b00 out of bounds> new_uri = 0x7ffe6604d5e0 "" end = 0x44445a "\211E\254H\307\005\350\206a" crt = 0x7db980 "core" cmd = 0x41b130 len = 32766 user = 0 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x7f2d00000000 <Address 0x7f2d00000000 out of bounds>, len = 1711593780}, host = {s = 0x0, len = 1711596272}, port = {s = 0x7ffe6604d600 "\240tS\355-\177", len = 1482919684}, params = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x7ffe6604d700 "0\330\004f\376\177", len = 4475707}, port_no = 54864, proto = 26116, type = 6127614, flags = 3986626464, transport = {s = 0x7f2ded5374a0 "\025\002", len = 1711596272}, ttl = {s = 0x1900000000 <Address 0x1900000000 out of bounds>, len = 0}, user_param = {s = 0x0, len = 1}, maddr = { s = 0x58638f04 <Address 0x58638f04 out of bounds>, len = 1667625608}, method = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, lr = {s = 0x0, len = 0}, r2 = { s = 0xf0e0bbca62a5f288 <Address 0xf0e0bbca62a5f288 out of bounds>, len = 1253175944}, gr = {s = 0x7ffe00000000 <Address 0x7ffe00000000 out of bounds>, len = 5663788}, transport_val = {s = 0x0, len = -954771808}, ttl_val = { s = 0x0, len = 1482919684}, user_param_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = -308340832}, maddr_val = {s = 0x7f2ded5187f0 "\002", len = 1711596272}, method_val = { s = 0x16604d710 <Address 0x16604d710 out of bounds>, len = 6688887}, lr_val = {s = 0x7f2ded4dd3f0 "", len = 1482919684}, r2_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, gr_val = {s = 0x0, len = 0}} next_hop = {user = {s = 0x7ffe6604d830 "\020\332\004f\376\177", len = 5762013}, passwd = {s = 0x7ffe6604d7e0 "(\340M\355-\177", len = -313665776}, host = {s = 0x7ffe6604d8a8 "\240UE\357-\177", len = -313666576}, port = { s = 0x7f2ded534f58 "\001", len = 1711593780}, params = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, sip_params = {s = 0x100000000 <Address 0x100000000 out of bounds>, len = 0}, headers = {s = 0x7ffe6604d820 "", len = 0}, port_no = 30168, proto = 60755, type = 32557, flags = 1711594548, transport = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, ttl = {s = 0x7ffe6604d8b0 "(\340M\355-\177", len = 0}, user_param = {s = 0x7f2ded4dcaf8 "", len = -308340832}, maddr = {s = 0x0, len = 0}, method = {s = 0x7f2ded4dd668 "\001", len = -308340832}, lr = {s = 0x7f2d00000001 <Address 0x7f2d00000001 out of bounds>, len = 17}, r2 = {s = 0x7f2ded4de028 "", len = 0}, gr = { s = 0x7ffe6604d960 "`H:\002", len = -1676080384}, transport_val = {s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1482919684}, ttl_val = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x7ffe6604da10 "\300\341\004f\376\177", len = 5780068}, method_val = { s = 0x7ffe00000000 <Address 0x7ffe00000000 out of bounds>, len = -1}, lr_val = {s = 0x7f2ded4dd920 "\002", len = -313665568}, r2_val = {s = 0x7f2ded9f17a0 "-\003", len = 1711596272}, gr_val = {s = 0x0, len = 0}} u = 0x58638f04 port = 0 dst_host = 0x0 i = 0 flags = 1711592512 avp = 0x41b130 st = {flags = 3986626464, id = 32557, name = {n = -308340832, s = {s = 0x7f2ded9f17a0 "-\003", len = -313297760}, re = 0x7f2ded9f17a0}, avp = 0x7ffe6604d620} sct = 0x7ffe6604eec0 sjt = 0x0 rve = 0x7f2ded664ee8 mct = 0x0 rv = 0x7ffe6604d5b0 rv1 = 0x7ffe6604eec0 c1 = {cache_type = 3981303848, val_type = 32557, c = {avp_val = {n = -313665248, s = {s = 0x7f2ded4dd920 "\002", len = 0}, re = 0x7f2ded4dd920}, pval = {rs = {s = 0x7f2ded4dd920 "\002", len = 0}, ri = -1676080384, flags = -1797463936}}, i2s = "\300\356\004f\376\177\000\000\004\217cX\000\000\000\000\060\261A\000\000"} s = {s = 0x7ffe6604d650 "", len = 5762013} srevp = {0x7ddd14, 0x0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #23 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded5385f0, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded67f0f0 ret = 1 mod = 0x94dce0809c190b00 ms = 0 __FUNCTION__ = "run_actions" #24 0x0000000000437505 in do_action (h=0x7ffe6604e2f0, a=0x7f2ded6876b8, msg=0x7f2ded9f17a0) at core/action.c:1058 ret = 1 v = 1 dst = {send_sock = 0x7ffe6604e300, to = {s = {sa_family = 22080, sa_data = "i\000\000\000\000\000p\341\004f\376\177\000"}, sin = {sin_family = 22080, sin_port = 105, sin_addr = {s_addr = 0}, sin_zero = "p\341\004f\376\177\000"}, sin6 = {sin6_family = 22080, sin6_port = 105, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "p\341\004f\376\177\000\000\020\260D\355-\177\000", __u6_addr16 = {57712, 26116, 32766, 0, 45072, 60740, 32557, 0}, __u6_addr32 = {1711595888, 32766, 3980701712, 32557}}}, sin6_scope_id = 8302096}}, id = 0, proto = 32 ' ', send_flags = {f = 103, blst_imask = 0}} tmp = 0x60540000f150 <Address 0x60540000f150 out of bounds> new_uri = 0x40041b130 <Address 0x40041b130 out of bounds> end = 0x413c40310 <Address 0x413c40310 out of bounds> crt = 0x0 cmd = 0xb02177 len = 0 user = 33554432 uri = {user = {s = 0x7ffe6604df30 "\310aZ", len = -358738281}, passwd = {s = 0x7ffe6604dfc0 "P\bI", len = 6780192}, host = {s = 0x7ffe6604de80 "", len = -314265584}, port = {s = 0xa5cd80 "\002", len = -314265584}, params = { s = 0x7ffe6604dec0 "\360\336\004f\376\177", len = -467824023}, sip_params = {s = 0x7ffe6604df60 "\360\340\004f\376\177", len = 0}, headers = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 27866048}, port_no = 800, proto = 73, type = ERROR_URI_T, flags = 3983471792, transport = {s = 0xb0269d "", len = 11542788}, ttl = {s = 0x50500f9 <Address 0x50500f9 out of bounds>, len = 0}, user_param = {s = 0x7ffe6604def0 "\360\337\004f\376\177", len = 4872016}, maddr = {s = 0x0, len = 8255817}, method = {s = 0x0, len = 0}, lr = {s = 0x7ffe6604dff0 " \340\004f\376\177", len = 6761026}, r2 = {s = 0x7eae10 "core", len = 0}, gr = { s = 0x2000000 <Address 0x2000000 out of bounds>, len = -311520168}, transport_val = {s = 0x0, len = 11543058}, ttl_val = {s = 0x5c8005a61c8 <Address 0x5c8005a61c8 out of bounds>, len = 11542741}, user_param_val = { s = 0x40041b130 <Address 0x40041b130 out of bounds>, len = 8314672}, maddr_val = {s = 0x7ffe6604dff0 " \340\004f\376\177", len = 7076483}, method_val = {s = 0x7ffe6604e0f0 " \341\004f\376\177", len = 6780192}, lr_val = { s = 0x7ffe6604dfb0 "", len = -314265584}, r2_val = {s = 0xe8 <Address 0xe8 out of bounds>, len = -314265584}, gr_val = {s = 0x7ffe6604dff0 " \340\004f\376\177", len = -467824023}} next_hop = {user = { s = 0xb02155 "Via: SIP/2.0/UDP 217.10.77.64:5060;branch=z9hG4bK081adebe\r\nMax-Forwards: 68\r\nFrom: "0xxxxxxxxxx" sip:0xxxxxxxxxx@sipgate.de;tag=as2ecab772\r\nTo: sip:0049xxxxxxxxx@sipgate.de\r\nContact: <sip:071"..., len = 0}, passwd = {s = 0x2000000 <Address 0x2000000 out of bounds>, len = 27864720}, host = {s = 0x490850 "H\213E\220H\213@ \213\225t\377\377\377\211P\020\351\313\002", len = 5689712}, port = {s = 0x5a61c8 "", len = 8}, params = { s = 0xed <Address 0xed out of bounds>, len = 8314656}, sip_params = {s = 0x7ffe6604e020 "@\017o\355-\177", len = 7090331}, headers = {s = 0x0, len = 8255817}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 0, transport = { s = 0x7f2ded6f0f40 "\001", len = -311488720}, ttl = {s = 0x7eae10 "core", len = -314265584}, user_param = {s = 0x7ffe6604e0a0 " \337~", len = 1482919684}, maddr = { s = 0x41b130 "1\355I\211\321^H\211\342H\203\344\360PTI\307\300 \344u", len = 1711599296}, method = {s = 0xb0269d "", len = 11543169}, lr = {s = 0x7ffe6604e120 "", len = 7017566}, r2 = {s = 0x7ffe6604e120 "", len = 7076483}, gr = {s = 0x7eae10 "core", len = 8314642}, transport_val = {s = 0x7edf20 "get_hdr_field", len = -311488744}, ttl_val = {s = 0xb0269d "", len = 11543159}, user_param_val = {s = 0x600000000 <Address 0x600000000 out of bounds>, len = -311516664}, maddr_val = {s = 0xb0269d "", len = 11543169}, method_val = { s = 0xb0227b "INVITE\r\nProxy-Authorization: Digest username="sgintern", realm="sipgate.de", algorithm=MD5, uri="sip:0049xxxxxxxxx@sipgate.de", nonce="W2MiBltjINqPpQ5t44/G2x0JzDBkyTCQ", response="7831ac65f3e563fa0"..., len = 11543169}, lr_val = {s = 0x7ffe6604e120 "", len = 1482919684}, r2_val = {s = 0x40041b130 <Address 0x40041b130 out of bounds>, len = 8302096}, gr_val = {s = 0x0, len = 0}} u = 0x7f2ded6ea358 port = 0 dst_host = 0xb02173 i = 0 flags = 0 avp = 0x7f2ded6ea358 st = {flags = 4305200, id = 2, name = {n = -358123536, s = {s = 0x7f2deaa777f0 "tm", len = 0}, re = 0x7f2deaa777f0}, avp = 0x0} sct = 0xb02177 sjt = 0x7ffe6604e120 rve = 0x7f2ded5375d0 mct = 0x6f0cc2 rv = 0x800000000 rv1 = 0x7ffe6604dfe4 c1 = {cache_type = 1711595920, val_type = 32766, c = {avp_val = {n = -314265584, s = {s = 0x7f2ded44b010 "\001", len = 64}, re = 0x7f2ded44b010}, pval = {rs = {s = 0x7f2ded44b010 "\001", len = 64}, ri = -314265584, flags = 32557}}, i2s = "\320\341\004f\376\177\000\000i\222\035\344-\177\000\000\260\364n\355-\177"} s = { s = 0xb021a0 "\r\nFrom: "0xxxxxxxxxx" sip:0xxxxxxxxx@sipgate.de;tag=as2ecab772\r\nTo: sip:0xxxxxxxxxxx@sipgate.de\r\nContact: sip:0xxxxxxxxx@217.10.77.64:5060\r\nCall-ID: 3c5228d7028a5f641ed1636259298615@sip"..., len = -1038288640} srevp = {0xb020c5, 0x9} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <Address 0x2 out of bounds>, str = { s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 13}, data = 0x0, attr = 0x0, select = 0x0}}, { type = MODFIXUP_ST, u = {number = 139835233294080, string = 0x7f2ded685700 "8\331h\355-\177", str = {s = 0x7f2ded685700 "8\331h\355-\177", len = 15}, data = 0x7f2ded685700, attr = 0x7f2ded685700, select = 0x7f2ded685700}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #25 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded4d1a78, msg=0x7f2ded9f17a0) at core/action.c:1565 t = 0x7f2ded6876b8 ret = 1 mod = 0x14058638f04 ms = 0 __FUNCTION__ = "run_actions" #26 0x0000000000444c24 in run_top_route (a=0x7f2ded4d1a78, msg=0x7f2ded9f17a0, c=0x0) at core/action.c:1654 ctx = {rec_lev = 8, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1482919684, -1089664633334795640, 4305200, 140730610020032, 0, 0, -1089664633380932984, 1088876375176245896}, __mask_was_saved = 0, __saved_mask = { __val = {139835236882336, 140730610017232, 5894335, 140730610017152, 140730611271653, 139835236882344, 11543171, 4294967296, 139835236882336, 4611686026705805832, 0, 140730610017232, 1482919684, 4305200, 140730610020032, 0}}}}} p = 0x7ffe6604e2f0 ret = 0 sfbk = 0 #27 0x0000000000540776 in receive_msg ( buf=0xb020a0 "INVITE sip:xxxxxxxx@sipgate.de SIP/2.0\r\nRecord-Route: sip:217.10.68.137;lr;ftag=as2ecab772\r\nVia: SIP/2.0/UDP 217.10.68.137;branch=z9hG4bK48ae.ef188a601621368ec42a8f6fd1d0cf9c.0\r\nVia: SIP/2.0/UDP 21"..., len=1533, rcv_info=0x7ffe6604e770) at core/receive.c:277 msg = 0x7f2ded9f17a0 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {1482919684, -1089664633334795640, 4305200, 140730610020032, 0, 0, -1089664633380932984, 1088876375176245896}, __mask_was_saved = 0, __saved_mask = { __val = {1, 139834505774096, 139835233425928, 50195, 1024, 6006564416, 6648504, 140730610017856, 6647679, 32, 139834505774096, 140730610018848, 139834505774096, 139835233425920, 139835233425920, 140730610017920}}}}} bctx = 0x7f2defe14520 ret = 1 stats_on = 0 tvb = {tv_sec = 140730610017680, tv_usec = 140730610017704} tve = {tv_sec = 139835264582552, tv_usec = 139835274727424} tz = {tz_minuteswest = 7931840, tz_dsttime = 0} diff = 0 inb = {s = 0xb020a0 "INVITE sip:xxxxxxx@sipgate.de SIP/2.0\r\nRecord-Route: sip:217.10.68.137;lr;ftag=as2ecab772\r\nVia: SIP/2.0/UDP 217.10.68.137;branch=z9hG4bK48ae.ef188a601621368ec42a8f6fd1d0cf9c.0\r\nVia: SIP/2.0/UDP 21"..., len = 1533} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x7f2defc009c2} keng = 0x0 evp = {data = 0x7ffe6604e520, rcv = 0x7ffe6604e770, dst = 0x0} __FUNCTION__ = "receive_msg" #28 0x000000000065f2ae in udp_rcv_loop () at core/udp_server.c:554 len = 1533 buf = "INVITE sip:xxxxxxxx@sipgate.de SIP/2.0\r\nRecord-Route: sip:217.10.68.137;lr;ftag=as2ecab772\r\nVia: SIP/2.0/UDP 217.10.68.137;branch=z9hG4bK48ae.ef188a601621368ec42a8f6fd1d0cf9c.0\r\nVia: SIP/2.0/UDP 21"... tmp = 0x7f2defe14520 "\200D\341\357-\177" from = 0x7f2ded6a5a00 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {2302937817, 1}, addr32 = {2302937817, 0, 1, 0}, addr16 = {2777, 35140, 0, 0, 1, 0, 0, 0}, addr = "\331\nD\211\000\000\000\000\001\000\000\000\000\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {103290028, 0}, addr32 = {103290028, 0, 0, 0}, addr16 = {5292, 1576, 0, 0, 0, 0, 0, 0}, addr = "\254\024(\006", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\331\nD\211\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 2302937817}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 2302937817, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f2ded4a87e0, proto = 1 '\001'} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\020\353\004f\376\177\000\000\322tK\357-\177\000\000\001\200\255\373\v\000\000\000\310$L\355-\177\000\000\020\353\004f\376\177\000\000\020\353\004f\376\177\000\000\020\353\004f\376\177\000\000\064Pt\302-\177\000\000\060\261A\000\000\000\000\000\300\356\004f\376\177", '\000' <repeats 18 times>, " \351\004f\376\177\000\000!S]", '\000' <repeats 21 times>, "\001\000\000\000\000\000\000\000\377\377\377\377\345[\325F\344L\261\000\000\000\000\000\344J\261\000\000\000\000\000\340F\261\000\000\000\000\000\004\217cX\000\000\000\000\060\261A\000\000\000\000\000\300\356\004f\376\177", '\000' <repeats 18 times>, "\020\351\004f\376\177\000\000_\376p\000\000\000\000\000\000\000\000\000\001\000\000\000\340F\261\000\v\000\000\000@\351\004f\v\000\000\000\000(L\355-\177\000\000\306\a,y\263\346\325\220\233\020(2\331\226{3W\213\370\357\021\317\232x4Pt\302-\177\000\000"... i = 4266208 j = 0 l = 0 __FUNCTION__ = "udp_rcv_loop" #29 0x000000000042743d in main_loop () at main.c:1626 i = 10 pid = 0 si = 0x7f2ded4a87e0 si_desc = "udp receiver child=10 sock=172.20.40.6:5060\000-\177\000\000\320\353\004f\376\177\000\000\000\v\031\234\200\340\334\224\320\353\004f\376\177\000\000\004\217cX\000\000\000\000\060\261A\000\000\000\000\000\300\356\004f\376\177", '\000' <repeats 18 times>"\260, \353\004f\376\177\000\000\354uV\000\000\000\000" nrprocs = 32 woneinit = 1 __FUNCTION__ = "main_loop" #30 0x000000000042eb71 in main (argc=17, argv=0x7ffe6604eec8) at main.c:2645 cfg_stream = 0x2286010 c = -1 r = 0 tmp = 0x7ffe66050e78 "" tmp_len = -280639488 port = 32557 proto = 4221504 options = 0x761e08 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3117719630 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7f2defe14520 p = 0x0 st = {st_dev = 14, st_ino = 25628, st_nlink = 2, st_mode = 16877, st_uid = 110, st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 180, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1532943957, tv_nsec = 228188274}, st_mtim = { tv_sec = 1533222745, tv_nsec = 73653908}, st_ctim = {tv_sec = 1533223122, tv_nsec = 208015059}, __unused = {0, 0, 0}} __FUNCTION__ = "main" (gdb) info locals branch_ret = 0 lowest_ret = -1 current_uri = {s = 0x0, len = 0} added_branches = 1 first_branch = 0 i = 1711599296 q = -1 t_invite = 0x7ffe66048cb0 success_branch = 32766 try_new = 1 lock_replies = -358549307 dst_uri = {s = 0x0, len = 0} path = {s = 0x0, len = 0} instance = {s = 0x0, len = 0} ruid = {s = 0x0, len = 0} location_ua = {s = 0x0, len = 0} si = 0x0 backup_bflags = 16 bflags = 0 __FUNCTION__ = "t_forward_nonack" (gdb) list 1716 in t_fwd.c ```
2 (bt only output, w/o full): ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 34381]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2dea9d1998 in t_forward_nonack (t=0x7f2dc7310660, p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0) at t_fwd.c:1721 1721 t_fwd.c: No such file or directory. (gdb) bt #0 0x00007f2dea9d1998 in t_forward_nonack (t=0x7f2dc7310660, p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0) at t_fwd.c:1721 #1 0x00007f2dea9bb0fb in t_relay_to (p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0, replicate=0) at t_funcs.c:331 #2 0x00007f2deaa475ad in _w_t_relay_to (p_msg=0x7f2ded9f17a0, proxy=0x0, force_proto=0) at tm.c:1555 #3 0x00007f2deaa48832 in w_t_relay (p_msg=0x7f2ded9f17a0, _foo=0x0, _bar=0x0) at tm.c:1755 #4 0x0000000000437599 in do_action (h=0x7ffe6604c300, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1067 #5 0x000000000044445a in run_actions (h=0x7ffe6604c300, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1565 #6 0x0000000000444b3b in run_actions_safe (h=0x7ffe6604e2f0, a=0x7f2ded734a20, msg=0x7f2ded9f17a0) at core/action.c:1633 #7 0x000000000057ebdd in rval_get_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, i=0x7ffe6604c7f4, rv=0x7f2ded733a88, cache=0x0) at core/rvalue.c:912 #8 0x0000000000583264 in rval_expr_eval_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, res=0x7ffe6604c7f4, rve=0x7f2ded733a80) at core/rvalue.c:1910 #9 0x00000000005836c3 in rval_expr_eval_int (h=0x7ffe6604e2f0, msg=0x7f2ded9f17a0, res=0x7ffe6604caf4, rve=0x7f2ded735068) at core/rvalue.c:1918 #10 0x000000000043703d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded738438, msg=0x7f2ded9f17a0) at core/action.c:1043 #11 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded734508, msg=0x7f2ded9f17a0) at core/action.c:1565 #12 0x0000000000433d5d in do_action (h=0x7ffe6604e2f0, a=0x7f2ded680170, msg=0x7f2ded9f17a0) at core/action.c:691 #13 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded5385f0, msg=0x7f2ded9f17a0) at core/action.c:1565 #14 0x0000000000437505 in do_action (h=0x7ffe6604e2f0, a=0x7f2ded6876b8, msg=0x7f2ded9f17a0) at core/action.c:1058 #15 0x000000000044445a in run_actions (h=0x7ffe6604e2f0, a=0x7f2ded4d1a78, msg=0x7f2ded9f17a0) at core/action.c:1565 #16 0x0000000000444c24 in run_top_route (a=0x7f2ded4d1a78, msg=0x7f2ded9f17a0, c=0x0) at core/action.c:1654 #17 0x0000000000540776 in receive_msg ( buf=0xb020a0 "INVITE sip:xxxxxxxxx@sipconnect.sipgate.de SIP/2.0\r\nRecord-Route: sip:217.10.68.137;lr;ftag=as5bb6b6b9\r\nVia: SIP/2.0/UDP 217.10.68.137;branch=z9hG4bK2201.40c6aa6542cc3bfaaad4e5c14b8d6738.0\r\nVia: SIP"..., len=1594, rcv_info=0x7ffe6604e770) at core/receive.c:277 #18 0x000000000065f2ae in udp_rcv_loop () at core/udp_server.c:554 #19 0x000000000042743d in main_loop () at main.c:1626 #20 0x000000000042eb71 in main (argc=17, argv=0x7ffe6604eec8) at main.c:2645 (gdb) info locals branch_ret = 0 lowest_ret = -1 current_uri = {s = 0x0, len = 0} added_branches = 1 first_branch = 0 i = 1711599296 q = -1 t_invite = 0x7ffe6604b7d0 success_branch = 32766 try_new = 1 lock_replies = -358549307 dst_uri = {s = 0x0, len = 0} path = {s = 0x0, len = 0} instance = {s = 0x0, len = 0} ruid = {s = 0x0, len = 0} location_ua = {s = 0x0, len = 0} si = 0x0 backup_bflags = 16 bflags = 0 __FUNCTION__ = "t_forward_nonack" (gdb) list 1716 in t_fwd.c ```
#### Log Messages
In the log file the only thing I could see is that it stopped handling SIP packets a few seconds before the crashes. And before the second crash I could see that it logged the unavailability of the rtpengine.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.1.4 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.7.2 ```
* **Operating System**:
This all happens on a Debian wheezy.
``` Linux sprotte.srv.sipgate.net 3.2.0-5-amd64 #1 SMP Debian 3.2.96-3 x86_64 GNU/Linux ```
Can you read anything out of those crashes?
Can you get the output for:
* first core file
``` frame 1 p *dead_cell ``` * the core for frame 0 being 0x00007f2dea9d1998 in t_forward_nonack ...
``` list p *t ```
First core file:
``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 31359]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x0000000000663091 in destroy_avp_list_unsafe (list=0x7efe49050bd0) at core/usr_avp.c:625 625 core/usr_avp.c: No such file or directory. (gdb) frame 1 #1 0x00007efe6b4daadc in free_cell_helper (dead_cell=0x7efe49050a48, silent=0, fname=0x7efe6b5db100 "timer.c", fline=654) at h_table.c:242 242 h_table.c: No such file or directory. (gdb) p *dead_cell $1 = {next_c = 0x0, prev_c = 0x0, hash_index = 30681, label = 705812179, flags = 32, nr_of_outgoings = 1, ref_count = {val = 0}, from = { s = 0x7efe48f3796d "From: sip:1234567@sipgate.de;tag=7fcea877-80e3-4069-b\r\nCall-ID: DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-State: Active\r\nRoute: sip:217.10.79.9;lr;ftag=3951995175\r\nConte"..., len = 57}, callid = { s = 0x7efe48f379a6 "Call-ID: DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-State: Active\r\nRoute: sip:217.10.79.9;lr;ftag=3951995175\r\nContent-Type: application/simple-message-summary\r\nVia: SIP/2.0"..., len = 41}, cseq_n = { s = 0x7efe48f3792d "CSeq: 7950 NOTIFY\r\nTo: sip:1234567@sipgate.de;tag=3951995175\r\nFrom: sip:1234567@sipgate.de;tag=7fcea877-80e3-4069-b\r\nCall-ID: DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-S"..., len = 10}, to = { s = 0x7efe48f37940 "To: sip:1234567@sipgate.de;tag=3951995175\r\nFrom: sip:1234567@sipgate.de;tag=7fcea877-80e3-4069-b\r\nCall-ID: DF246541ADA4EDA2@84.149.15.231\r\nEvent: message-summary\r\nSubscription-State: Active\r\nRoute"..., len = 45}, method = { s = 0x7efe48f378e0 "NOTIFY sip:1442863@84.149.15.231;uniq=29A7ECC65F8884BD635517C89AFD8 SIP/2.0\r\nCSeq: 7950 NOTIFY\r\nTo: sip:1442863@sipgate.de;tag=3951995175\r\nFrom: sip:1442863@sipgate.de;tag=7fcea877-80e3-4069-b\r\nCa"..., len = 6}, tmcb_hl = {first = 0x0, reg_types = 0}, wait_timer = {next = 0x0, prev = 0x0, expire = 1289862248, initial_timeout = 320, data = 0x7efe49050a48, f = 0x7efe6b5875fa <wait_handler>, flags = 513, slow_idx = 0}, uas = {request = 0x7efe48f371c8, end_request = 0x7efe48f38070 "\210", response = {activ_type = 481, flags = 128, t_active = 0 '\000', branch = 0, buffer_len = 323, buffer = 0x7efe49077660 "SIP/2.0 481 Call Leg/Transaction Does Not Exist\r\nVia: SIP/2.0/UDP 217.10.76.144:5060\r\nFrom: sip:1442863@sipgate.de;tag=7fcea877-80e3-4069-b\r\nTo: sip:1442863@sipgate.de;tag=3951995175\r\nCall-ID: DF2"..., my_T = 0x7efe49050a48, timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x0, f = 0x7efe6b5870b2 <retr_buf_handler>, flags = 0, slow_idx = 0}, dst = { send_sock = 0x7efe6dff77e0, to = {s = {sa_family = 2, sa_data = "\023\304\331\nL\220\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 2420902617}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 2420902617, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 1 '\001', send_flags = {f = 0, blst_imask = 0}}, retr_expire = 0, fr_expire = 0}, local_totag = { s = 0x0, len = 0}, cancel_reas = 0x0, status = 481}, uac = 0x7efe49050c50, async_backup = {backup_route = 0, backup_branch = 0, blind_uac = 0, ruri_new = 0}, fwded_totags = 0x0, uri_avps_from = 0x7efe49030900, uri_avps_to = 0x0, user_avps_from = 0x0, user_avps_to = 0x0, domain_avps_from = 0x0, domain_avps_to = 0x0, xavps_list = 0x0, reply_mutex = {val = 0}, reply_locker_pid = { val = 0}, reply_rec_lock_level = 0, fr_timeout = 320, fr_inv_timeout = 2000, rt_t1_timeout_ms = 500, rt_t2_timeout_ms = 4000, end_of_life = 1289862439, relayed_reply_branch = 0, on_failure = 0, on_branch_failure = 0, on_reply = 0, on_branch = 0, on_branch_delayed = 0, md5 = 0x7efe49050c30 "f17e9566ecae744152b7aed3c5ba0273"} ```
The other core: ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/bf/0f481eacd5661e7adec0c358237730d825ea2a.debug...done. done. [New LWP 34381]
warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio_sip_proxy.cfg -P /var/run/kamailio'. Program terminated with signal 11, Segmentation fault. #0 0x00007f2dea9d1998 in t_forward_nonack (t=0x7f2dc7310660, p_msg=0x7f2ded9f17a0, proxy=0x0, proto=0) at t_fwd.c:1721 1721 t_fwd.c: No such file or directory. (gdb) list 1716 in t_fwd.c (gdb) p *t $1 = {next_c = 0x7f2dc212e0c0, prev_c = 0x7f2dc212e0c0, hash_index = 3986, label = 1504273493, flags = 2, nr_of_outgoings = 1, ref_count = {val = 1}, from = { s = 0x7f2dc71e8dbb "From: sip:dispatcher@localhost;tag=3d02c524d32baf26b7c31e3ac94b01dc-7e61\r\nCSeq: 10 OPTIONS\r\nCall-ID: 1afe16d6638c333a-34402@172.20.40.6\r\nMax-Forwards: 70\r\nContent-Length: 0\r\n\r\n", len = 76}, callid = {s = 0x7f2dc71e8e19 "Call-ID: 1afe16d6638c333a-34402@172.20.40.6\r\nMax-Forwards: 70\r\nContent-Length: 0\r\n\r\n", len = 45}, cseq_n = { s = 0x7f2dc71e8e07 "CSeq: 10 OPTIONS\r\nCall-ID: 1afe16d6638c333a-34402@172.20.40.6\r\nMax-Forwards: 70\r\nContent-Length: 0\r\n\r\n", len = 8}, to = { s = 0x7f2dc71e8da4 "To: sip:172.20.40.5\r\nFrom: sip:dispatcher@localhost;tag=3d02c524d32baf26b7c31e3ac94b01dc-7e61\r\nCSeq: 10 OPTIONS\r\nCall-ID: 1afe16d6638c333a-34402@172.20.40.6\r\nMax-Forwards: 70\r\nContent-Length: 0\r\n\r"..., len = 23}, method = { s = 0x7f2dc71e8d30 "OPTIONS sip:172.20.40.5 SIP/2.0\r\nVia: SIP/2.0/UDP 172.20.40.6;branch=z9hG4bK29f.8a425ca5", '0' <repeats 24 times>, ".0\r\nTo: sip:172.20.40.5\r\nFrom: sip:dispatcher@localhost;tag=3d02c524d32baf26b7c31e3a"..., len = 7}, tmcb_hl = {first = 0x7f2dc72fe838, reg_types = 1024}, wait_timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x7f2dc7310660, f = 0x7f2deaa385fa <wait_handler>, flags = 1, slow_idx = 0}, uas = {request = 0x0, end_request = 0x0, response = {activ_type = 0, flags = 0, t_active = 0 '\000', branch = 0, buffer_len = 0, buffer = 0x0, my_T = 0x7f2dc7310660, timer = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x0, f = 0x7f2deaa380b2 <retr_buf_handler>, flags = 0, slow_idx = 0}, dst = {send_sock = 0x0, to = {s = { sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}}, retr_expire = 0, fr_expire = 0}, local_totag = {s = 0x0, len = 0}, cancel_reas = 0x0, status = 0}, uac = 0x7f2dc7310868, async_backup = { backup_route = 0, backup_branch = 0, blind_uac = 0, ruri_new = 0}, fwded_totags = 0x0, uri_avps_from = 0x0, uri_avps_to = 0x0, user_avps_from = 0x0, user_avps_to = 0x0, domain_avps_from = 0x0, domain_avps_to = 0x0, xavps_list = 0x0, reply_mutex = {val = 0}, reply_locker_pid = {val = 0}, reply_rec_lock_level = 0, fr_timeout = 320, fr_inv_timeout = 2000, rt_t1_timeout_ms = 500, rt_t2_timeout_ms = 4000, end_of_life = 1323165532, relayed_reply_branch = -1, on_failure = 1, on_branch_failure = 0, on_reply = 2, on_branch = 0, on_branch_delayed = 0, md5 = 0x7f2dc7310848 "8a425ca5", '0' <repeats 24 times>} ```
For *The other core* is looks a bit strange. The first backtrace shows and INVITE, but in the last comment is an OPTIONS.
It looks like the OPTIONS gets to a t_relay() (or something similar), although the request is generated locally, so this should not happen. Do you have any tm event route that does t_relay()?
I pushed a commit with a safety check, but this situation was not supposed to happen and may still have side effects even after this safety check.
No, we don't have an tm event routes at all. The only event routes are:
``` event_route[xhttp:request] { event_route[dispatcher:dst-down] { event_route[dispatcher:dst-up] { ```
I had a coredump with version 5.1.4+0~20180725075519.79+trusty ( bf6468f7a4a54600e04dc6fd8aab537bde5ee464 ) maybe it's related:
``` (gdb) bt #0 0x00007f1d7c438814 in rtpengine_manage (msg=0x7f1d87bcc858, flags=0x7f1d87bfc9b0 "direction=public direction=public replace-origin replace-session-connection trust-address") at rtpengine.c:3247 #1 0x00007f1d7c4389a8 in rtpengine_manage_wrap (msg=0x7f1d87bcc858, d=0x7f1d87bfc9b0, more=0) at rtpengine.c:3270 #2 0x00007f1d7c436ab4 in rtpengine_rtpp_set_wrap (msg=0x7f1d87bcc858, func=0x7f1d7c438982 <rtpengine_manage_wrap>, data=0x7f1d87bfc9b0, direction=1) at rtpengine.c:3074 #3 0x00007f1d7c438cfe in rtpengine_manage1_f (msg=0x7f1d87bcc858, str1=0x7f1d87fa7fd0 "\260f\372\207\035\177", str2=0x0) at rtpengine.c:3286 #4 0x0000000000434807 in do_action (h=0x7ffc50479920, a=0x7f1d87fa7668, msg=0x7f1d87bcc858) at core/action.c:1073 #5 0x0000000000441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87fa7668, msg=0x7f1d87bcc858) at core/action.c:1565 #6 0x00000000004346ee in do_action (h=0x7ffc50479920, a=0x7f1d87fa8b60, msg=0x7f1d87bcc858) at core/action.c:1058 #7 0x0000000000441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87fa4920, msg=0x7f1d87bcc858) at core/action.c:1565 #8 0x0000000000430f48 in do_action (h=0x7ffc50479920, a=0x7f1d87f83ad8, msg=0x7f1d87bcc858) at core/action.c:691 #9 0x0000000000441630 in run_actions (h=0x7ffc50479920, a=0x7f1d87f837b8, msg=0x7f1d87bcc858) at core/action.c:1565 #10 0x0000000000441df9 in run_top_route (a=0x7f1d87f837b8, msg=0x7f1d87bcc858, c=0x7ffc50479920) at core/action.c:1654 #11 0x00007f1d814b8b0e in prepare_new_uac (t=0x7f1d734543f8, i_req=0x7f1d87bcc858, branch=0, uri=0x7ffc504798c0, path=0x7ffc504798e0, next_hop=0x7ffc504798d0, fsocket=0x0, snd_flags=..., fproto=0, flags=0, instance=0x7ffc504798f0, ruid=0x7ffc50479900, location_ua=0x7ffc50479910) at t_fwd.c:356 #12 0x00007f1d814bd457 in add_uac (t=0x7f1d734543f8, request=0x7f1d87bcc858, uri=0x7f1d87bccad0, next_hop=0x7f1d87bccae0, path=0x7f1d87bcce90, proxy=0x0, fsocket=0x0, snd_flags=..., proto=0, flags=0, instance=0x7f1d87bccea0, ruid=0x7f1d87bcceb8, location_ua=0x7f1d87bccec8) at t_fwd.c:800 #13 0x00007f1d814c5758 in t_forward_nonack (t=0x7f1d734543f8, p_msg=0x7f1d87bcc858, proxy=0x0, proto=0) at t_fwd.c:1680 #14 0x00007f1d814af0ac in t_relay_to (p_msg=0x7f1d87bcc858, proxy=0x0, proto=0, replicate=0) at t_funcs.c:331 #15 0x00007f1d8153c3e8 in _w_t_relay_to (p_msg=0x7f1d87bcc858, proxy=0x0, force_proto=0) at tm.c:1555 #16 0x00007f1d8153d66d in w_t_relay (p_msg=0x7f1d87bcc858, _foo=0x0, _bar=0x0) at tm.c:1755 #17 0x000000000043477f in do_action (h=0x7ffc5047a700, a=0x7f1d87f9b358, msg=0x7f1d87bcc858) at core/action.c:1067 #18 0x0000000000441630 in run_actions (h=0x7ffc5047a700, a=0x7f1d87f9b358, msg=0x7f1d87bcc858) at core/action.c:1565 #19 0x0000000000441d10 in run_actions_safe (h=0x7ffc5047d670, a=0x7f1d87f9b358, msg=0x7f1d87bcc858) at core/action.c:1633 #20 0x000000000057c66b in rval_get_int (h=0x7ffc5047d670, msg=0x7f1d87bcc858, i=0x7ffc5047aaf4, rv=0x7f1d87f9b618, cache=0x0) at core/rvalue.c:912 #21 0x0000000000580cf6 in rval_expr_eval_int (h=0x7ffc5047d670, msg=0x7f1d87bcc858, res=0x7ffc5047aaf4, rve=0x7f1d87f9b610) at core/rvalue.c:1910 #22 0x0000000000581158 in rval_expr_eval_int (h=0x7ffc5047d670, msg=0x7f1d87bcc858, res=0x7ffc5047ac64, rve=0x7f1d87f9bd40) at core/rvalue.c:1918 #23 0x000000000043422a in do_action (h=0x7ffc5047d670, a=0x7f1d87f9e8c0, msg=0x7f1d87bcc858) at core/action.c:1043 #24 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87f983f8, msg=0x7f1d87bcc858) at core/action.c:1565 #25 0x0000000000430f48 in do_action (h=0x7ffc5047d670, a=0x7f1d87d5bfd0, msg=0x7f1d87bcc858) at core/action.c:691 #26 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87d5bb60, msg=0x7f1d87bcc858) at core/action.c:1565 #27 0x00000000004346ee in do_action (h=0x7ffc5047d670, a=0x7f1d87d5eb00, msg=0x7f1d87bcc858) at core/action.c:1058 #28 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87d49d78, msg=0x7f1d87bcc858) at core/action.c:1565 #29 0x0000000000430f48 in do_action (h=0x7ffc5047d670, a=0x7f1d87e07e80, msg=0x7f1d87bcc858) at core/action.c:691 #30 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87e07e80, msg=0x7f1d87bcc858) at core/action.c:1565 #31 0x000000000043e47a in do_action (h=0x7ffc5047d670, a=0x7f1d87e08748, msg=0x7f1d87bcc858) at core/action.c:1314 #32 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87e062e0, msg=0x7f1d87bcc858) at core/action.c:1565 #33 0x00000000004346ee in do_action (h=0x7ffc5047d670, a=0x7f1d87e08898, msg=0x7f1d87bcc858) at core/action.c:1058 #34 0x0000000000441630 in run_actions (h=0x7ffc5047d670, a=0x7f1d87dff690, msg=0x7f1d87bcc858) at core/action.c:1565 #35 0x0000000000441df9 in run_top_route (a=0x7f1d87dff690, msg=0x7f1d87bcc858, c=0x0) at core/action.c:1654 #36 0x000000000053e22f in receive_msg (buf=0xb010c0 <buf> "INVITE sip:00442075344466@kamailio-leader.query.consul:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 172.18.231.7:5080;branch=z9hG4bK47b18559\r\nMax-Forwards: 69\r\nFrom: "1001-FoehnQA" <sip:1001@sip.staging.voxivo.com:"..., len=1193, rcv_info=0x7ffc5047dab0) at core/receive.c:277 #37 0x000000000065cf85 in udp_rcv_loop () at core/udp_server.c:554 #38 0x0000000000424697 in main_loop () at main.c:1626 #39 0x000000000042bdbf in main (argc=13, argv=0x7ffc5047e1b8) at main.c:2645 (gdb) f 0 #0 0x00007f1d7c438814 in rtpengine_manage (msg=0x7f1d87bcc858, flags=0x7f1d87bfc9b0 "direction=public direction=public replace-origin replace-session-connection trust-address") at rtpengine.c:3247 3247 tmb.t_gett()->uas.request->msg_flags |= FL_SDP_BODY; (gdb) p tmb $2 = {register_tmcb = 0x7f1d814c805d <register_tmcb>, t_relay_to_udp = 0x7f1d8153c720 <w_t_relay_to_udp>, t_relay_to_tcp = 0x7f1d8153c77a <w_t_relay_to_tcp>, t_relay = 0x7f1d8153d643 <w_t_relay>, t_replicate = 0x7f1d8153c924 <t_replicate_uri>, t_newtran = 0x7f1d8153b46f <w_t_newtran>, t_reply = 0x7f1d81540bec <w_t_reply_wrp>, t_reply_with_body = 0x7f1d81513119 <t_reply_with_body>, t_is_local = 0x7f1d814dbc08 <t_is_local>, t_get_trans_ident = 0x7f1d814da499 <t_get_trans_ident>, t_lookup_ident = 0x7f1d814db22f <t_lookup_ident>, t_addblind = 0x7f1d814bc006 <add_blind_uac>, t_reply_unsafe = 0x0, t_reply_trans = 0x7f1d8150c5ed <t_reply>, t_forward_nonack = 0x7f1d81539d32 <w_t_forward_nonack>, t_request_within = 0x7f1d8155a159 <req_within>, t_request_outside = 0x7f1d8155a805 <req_outside>, t_request = 0x7f1d8155ac63 <request>, new_dlg_uac = 0x7f1d81471739 <new_dlg_uac>, dlg_response_uac = 0x7f1d814753b0 <dlg_response_uac>, new_dlg_uas = 0x7f1d814772d3 <new_dlg_uas>, update_dlg_uas = 0x7f1d8147830b <update_dlg_uas>, dlg_request_uas = 0x7f1d814792d2 <dlg_request_uas>, set_dlg_target = 0x7f1d8147a6fb <set_dlg_target>, free_dlg = 0x7f1d81479ff6 <free_dlg>, print_dlg = 0x7f1d8147a2eb <print_dlg>, t_gett = 0x7f1d814cae7e <get_t>, t_gett_branch = 0x7f1d814caed0 <get_t_branch>, t_sett = 0x7f1d814cae8b <set_t>, calculate_hooks = 0x7f1d8147171f <w_calculate_hooks>, t_uac = 0x7f1d8155823e <t_uac>, t_uac_with_ids = 0x7f1d81558262 <t_uac_with_ids>, t_release = 0x7f1d8153b0c8 <w_t_release>, t_unref = 0x7f1d814d937f <t_unref>, run_failure_handlers = 0x7f1d81506416 <run_failure_handlers>, run_branch_failure_handlers = 0x7f1d8150761a <run_branch_failure_handlers>, cancel_uacs = 0x7f1d81491516 <cancel_uacs>, cancel_all_uacs = 0x7f1d814915f8 <cancel_all_uacs>, prepare_request_within = 0x7f1d815566e1 <prepare_req_within>, send_prepared_request = 0x7f1d8155821a <send_prepared_request>, route_mode = 0x0, reserved1 = 0x0, reserved2 = 0x0, ack_local_uac = 0x7f1d81558563 <ack_local_uac>, t_get_canceled_ident = 0x7f1d814daae7 <t_get_canceled_ident>, t_suspend = 0x7f1d8152051b <t_suspend>, t_continue = 0x7f1d81527cd0 <t_continue>, t_continue_cb = 0x7f1d81527d00 <t_continue_cb>, t_cancel_suspend = 0x7f1d81527d38 <t_cancel_suspend>, t_get_reply_totag = 0x7f1d814ffb46 <t_get_reply_totag>, t_get_picked_branch = 0x7f1d814ffba4 <t_get_picked_branch>, t_lookup_callid = 0x7f1d814dc23d <t_lookup_callid>, generate_callid = 0x7f1d8146fbb7 <generate_callid>, generate_fromtag = 0x7f1d81550506 <generate_fromtag>, t_lookup_request = 0x7f1d814cd4f6 <t_lookup_request>, t_lookup_original = 0x7f1d814cff4c <t_lookupOriginalT>, t_check = 0x7f1d814d6015 <t_check>, unref_cell = 0x7f1d814b1dec <unref_cell>, prepare_to_cancel = 0x7f1d81491475 <prepare_to_cancel>, get_stats = 0x7f1d8151f18d <tm_get_stats>, get_table = 0x7f1d8147be67 <tm_get_table>, dlg_add_extra = 0x7f1d81472236 <dlg_add_extra>, t_cancel_uac = 0x7f1d81493b75 <t_uac_cancel>, tm_ctx_get = 0x7f1d814dfe1e <tm_ctx_get>, t_append_branches = 0x7f1d8148d6a1 <t_append_branches>, t_load_contacts = 0x7f1d81518c0c <t_load_contacts>, t_next_contacts = 0x7f1d8151cbf4 <t_next_contacts>, set_fr = 0x7f1d814de09e <t_set_fr>} (gdb) p tmb.t_gett $3 = (tgett_f) 0x7f1d814cae7e <get_t> ```
``` Aug 20 07:45:42 kamailio-2105-0 kamailio[3936]: ALERT: <script>: No more next_dispatch to try - ci='211ec253163c6fef53c346f00d6826fe@sip.staging.voxivo.com' Aug 20 07:45:42 kamailio-2105-0 kamailio[3941]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3934]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tm [t_lookup.c:1487]: t_unref(): script writer didn't release transaction Aug 20 07:45:42 kamailio-2105-0 kamailio[3936]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3938]: INFO: <script>: Dialog failed event profile='outbound_c' - ci='123' Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3959]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3934]: ERROR: tm [t_reply.c:480]: _reply_light(): can't generate 487 reply when a final 487 was sent out Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: ERROR: tm [t_reply.c:480]: _reply_light(): can't generate 487 reply when a final 487 was sent out Aug 20 07:45:42 kamailio-2105-0 kamailio[3936]: ERROR: tm [t_reply.c:480]: _reply_light(): can't generate 487 reply when a final 487 was sent out Aug 20 07:45:42 kamailio-2105-0 kamailio[3938]: WARNING: tm [h_table.c:134]: free_cell_helper(): removed cell 0x7f1d73470300 is still linked in hash table (t_lookup.c:1497) Aug 20 07:45:42 kamailio-2105-0 kamailio[3938]: WARNING: tm [h_table.c:139]: free_cell_helper(): cell 0x7f1d73470300 is still linked in wait timer (t_lookup.c:1497) - skip freeing now Aug 20 07:45:42 kamailio-2105-0 kamailio[3934]: ERROR: tm [t_lookup.c:1482]: t_unref(): generation of a delayed stateful reply failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3933]: ERROR: sl [sl_funcs.c:362]: sl_reply_error(): stateless error reply used: I'm terribly sorry, server error occurred (1/SL) Aug 20 07:45:42 kamailio-2105-0 kamailio[3941]: INFO: <script>: Dialog failed event profile='outbound_c' - ci='123' Aug 20 07:45:42 kamailio-2105-0 kamailio[3939]: INFO: <script>: Dialog failed event profile='outbound_c' - ci='123' Aug 20 07:45:42 kamailio-2105-0 kamailio[3939]: WARNING: <script>: CDR generated by kamailio - ci='123' Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: ERROR: tm [t_lookup.c:1482]: t_unref(): generation of a delayed stateful reply failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3933]: ERROR: tm [t_reply.c:480]: _reply_light(): can't generate 487 reply when a final 487 was sent out Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: CRITICAL: <core> [core/mem/q_malloc.c:504]: qm_free(): BUG: freeing already freed pointer (0x7f1d73463808), called from tm: h_table.c: free_cell_helper(155), first free tm: h_table.c: free_cell_helper(155) - aborting Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tm [h_table.c:134]: free_cell_helper(): removed cell 0x7f1d7348c188 is still linked in hash table (t_lookup.c:1497) Aug 20 07:45:42 kamailio-2105-0 kamailio[3933]: ERROR: tm [t_lookup.c:1482]: t_unref(): generation of a delayed stateful reply failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: CRITICAL: <core> [core/mem/q_malloc.c:504]: qm_free(): BUG: freeing already freed pointer (0x7f1d7348d938), called from tm: h_table.c: free_cell_helper(174), first free tm: h_table.c: free_cell_helper(174) - aborting Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tm [h_table.c:139]: free_cell_helper(): cell 0x7f1d7348c188 is still linked in wait timer (t_lookup.c:1497) - skip freeing now Aug 20 07:45:42 kamailio-2105-0 kamailio[3936]: ERROR: tm [t_lookup.c:1482]: t_unref(): generation of a delayed stateful reply failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: CRITICAL: <core> [core/mem/q_malloc.c:504]: qm_free(): BUG: freeing already freed pointer (0x7f1d73468550), called from tm: h_table.c: free_cell_helper(174), first free tm: h_table.c: free_cell_helper(174) - aborting Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: CRITICAL: <core> [core/mem/q_malloc.c:504]: qm_free(): BUG: freeing already freed pointer (0x7f1d7345ef80), called from tm: h_table.c: free_cell_helper(174), first free tm: h_table.c: free_cell_helper(174) - aborting Aug 20 07:45:42 kamailio-2105-0 kamailio[3940]: CRITICAL: <core> [core/mem/q_malloc.c:504]: qm_free(): BUG: freeing already freed pointer (0x7f1d734a0028), called from tm: h_table.c: free_cell_helper(251), first free tm: h_table.c: free_cell_helper(251) - aborting Aug 20 07:45:42 kamailio-2105-0 kamailio[3941]: WARNING: tm [h_table.c:134]: free_cell_helper(): removed cell 0x7f1d734797d8 is still linked in hash table (t_lookup.c:1497) Aug 20 07:45:42 kamailio-2105-0 kamailio[3941]: WARNING: tm [h_table.c:139]: free_cell_helper(): cell 0x7f1d734797d8 is still linked in wait timer (t_lookup.c:1497) - skip freeing now Aug 20 07:45:42 kamailio-2105-0 kamailio[3935]: INFO: <script>: from NAT address rm='ACK' ci='1514b435553213ee49274dae7c18953b@sip.staging.voxivo.com' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: evapi: Message Received event='{"Event":"CGR_AUTH_REPLY","TransactionIndex":9573,"TransactionLabel":1002983302,"MaxSessionTime":10800,"Suppliers":"","ResourceAllocated":true,"AllocationMessage":"","Error":""}' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tm [t_suspend.c:202]: t_continue_helper(): transaction is not suspended [9573:1002983302] Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tmx [tmx_mod.c:760]: w_t_continue(): resuming the processing of transaction [9573:1002983302] failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: evapi: Message Received event='{"Event":"CGR_AUTH_REPLY","TransactionIndex":13293,"TransactionLabel":887324556,"MaxSessionTime":10800,"Suppliers":"","ResourceAllocated":true,"AllocationMessage":"","Error":""}' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: ERROR: tm [t_suspend.c:197]: t_continue_helper(): transaction not found Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tmx [tmx_mod.c:760]: w_t_continue(): resuming the processing of transaction [13293:887324556] failed Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: evapi: Message Received event='{"Event":"CGR_AUTH_REPLY","TransactionIndex":2151,"TransactionLabel":1857937848,"MaxSessionTime":10800,"Suppliers":"","ResourceAllocated":true,"AllocationMessage":"","Error":""}' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: evapi reply on CGR_DIALOG_TIMEOUT with ci='1514b435553213ee49274dae7c18953b@sip.staging.voxivo.com' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: CGR_AUTH_REPLY ci='1514b435553213ee49274dae7c18953b@sip.staging.voxivo.com' MaxSessionTime='10800' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: CRITICAL: dialog [dlg_timer.c:200]: update_dlg_timer(): Trying to update a bogus dlg tl=0x7f1d73465288 tl->next=(nil) tl->prev=(nil) Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: ERROR: dialog [dlg_hash.c:1174]: update_dlg_timeout(): failed to update dialog lifetime Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: ALERT: <script>: CGR_AUTH_REPLY set dialog didn't work for ci='1514b435553213ee49274dae7c18953b@sip.staging.voxivo.com' and user='sip:1001@sip.staging.voxivo.com:5080' timeout='10800' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: INFO: <script>: evapi: Message Received event='{"Event":"CGR_AUTH_REPLY","TransactionIndex":6415,"TransactionLabel":1188410926,"MaxSessionTime":10800,"Suppliers":"","ResourceAllocated":true,"AllocationMessage":"","Error":""}' Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: ERROR: tm [t_suspend.c:197]: t_continue_helper(): transaction not found Aug 20 07:45:42 kamailio-2105-0 kamailio[3967]: WARNING: tmx [tmx_mod.c:760]: w_t_continue(): resuming the processing of transaction [6415:1188410926] failed Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:746]: handle_sigs(): child process 3930 exited by a signal 11 Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:749]: handle_sigs(): core was generated Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:746]: handle_sigs(): child process 3931 exited by a signal 11 Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:749]: handle_sigs(): core was generated Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:746]: handle_sigs(): child process 3937 exited by a signal 11 Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: ALERT: <core> [main.c:749]: handle_sigs(): core was generated Aug 20 07:45:43 kamailio-2105-0 kamailio[3910]: INFO: <core> [main.c:771]: handle_sigs(): terminating due to SIGCHLD Aug 20 07:45:43 kamailio-2105-0 kamailio[3978]: INFO: <core> [main.c:826]: sig_usr(): signal 15 received ```
But the log related to the msg in the core: ``` Aug 20 07:40:40 kamailio-2105-0 kamailio[3929]: INFO: <script>: RELAY_BRANCH rm='INVITE' - ci='3560dd53013c79fe7233fc1f6e83be62@sip.staging.voxivo.com' Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:A.B.C.20:7222 Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.9:7222 found, support for it enabled Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.4:7222 found, support for it enabled Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.7:7222 found, support for it enabled Aug 20 07:40:42 kamailio-2105-0 kamailio[3934]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.14:7222 found, support for it enabled Aug 20 07:40:42 kamailio-2105-0 kamailio[3936]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:127.0.0.1:7222 found, support for it enabled Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:A.B.C.20:7222 Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.9:7222 found, support for it enabled Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.4:7222 found, support for it enabled Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.7:7222 found, support for it enabled Aug 20 07:40:47 kamailio-2105-0 kamailio[3936]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.14:7222 found, support for it enabled Aug 20 07:40:47 kamailio-2105-0 kamailio[3931]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:127.0.0.1:7222 found, support for it enabled Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:A.B.C.20:7222 Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.9:7222 found, support for it enabled Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.4:7222 found, support for it enabled Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.7:7222 found, support for it enabled Aug 20 07:40:52 kamailio-2105-0 kamailio[3931]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.14:7222 found, support for it enabled Aug 20 07:40:52 kamailio-2105-0 kamailio[3933]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:127.0.0.1:7222 found, support for it enabled Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:A.B.C.20:7222 Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.9:7222 found, support for it enabled Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.C.4:7222 found, support for it enabled Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.7:7222 found, support for it enabled Aug 20 07:40:57 kamailio-2105-0 kamailio[3933]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:A.B.D.14:7222 found, support for it enabled Aug 20 07:40:57 kamailio-2105-0 kamailio[3937]: INFO: rtpengine [rtpengine.c:2423]: rtpp_test(): rtp proxy udp:127.0.0.1:7222 found, support for it enabled Aug 20 07:40:59 kamailio-2105-0 kamailio[3932]: INFO: <script>: from NAT address rm='CANCEL' ci='3560dd53013c79fe7233fc1f6e83be62@sip.staging.voxivo.com' Aug 20 07:40:59 kamailio-2105-0 kamailio[3932]: INFO: <script>: RELAY: method='CANCEL' ip='A.B.C.7:5080' du='<null>' ru='sip:00442075344466@kamailio-leader.query.consul:5060' fu='sip:1001@sip.staging.voxivo.com:5080' ci='3560dd53013c79fe7233fc1f6e83be62@sip.staging.voxivo.com' Aug 20 07:40:59 kamailio-2105-0 kamailio[3932]: INFO: <script>: Dialog failed event profile='outbound_c' - ci='123' Aug 20 07:40:59 kamailio-2105-0 kamailio[3932]: WARNING: <script>: CDR generated by kamailio - ci='123' ```
Hi,
I can see same issue on 5.1.4 too
``` version: kamailio 5.1.4 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 5.3.1
```
but without crash. It just stops processing sip messages (happened to me couple of times after **kamcmd rtpengine.reload** ) As in my case it's running on top of docker I've implemented simple sipp heathcheck that restarts container.
Hi,
i also can see the same issue on 5.1.4.
This seems to be more tm issue rather then rtpengengine module. I think it may happen if for any reason RTPEngine dissapears between offer in invite and ack
``` Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:rtpengine_ip:22222 Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2749]: select_rtpp_node(): rtpengine failed to select new for calllen=30 callid=eldnybgg6e_87665 Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE eldnybgg6e_87665 src_ip:3010@udp]--> ERROR: rtpengine [rtpengine.c:2239]: rtpp_function_call(): no available proxies Nov 28 10:08:11 ip-10-23-247-205 kernel: [586868.258562] show_signal_msg: 30 callbacks suppressed Nov 28 10:08:11 ip-10-23-247-205 kernel: [586868.258566] kamailio[14702]: segfault at 24 ip 00007efca782242e sp 00007fffbef66410 error 4 in tm.so[7efca775c000+138000]
```
@marcinkowalczyk it sounds like it can be reproduced, which will make troubleshooting easier.
But to understand properly, what do you mean by "RTPEngine disappears between offer in invite and ack". Is the SIP ACK, which is after 200ok response? Can you narrow a bit more when it happens or provide the steps to reproduce?
The backtrace you posted points to sending out the INVITE.
Hi,
From kam logs looks crash was related to ACK for CANCEL message when proxy offered in INVITE did not reply back
``` Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: New request - M=INVITE R=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com F=sip:+311111111@10.23.248.40 T=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com IP=push_server:3010 DST=0.0.0.0:5060 ID=domain.example.com_87665 Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: Generated MOS A label for domain.example.com_87665 - cc7ff795-c757-4b47-8ac2-26d1efb044cf Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: Generated MOS B label for domain.example.com_87665 - 4e5bc057-5403-4df9-9dd1-d89d0e27fd91 Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: RTPEngineID proxy.example.com set from header to 7 ID=domain.example.com_87665 Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: <script>: Sending push notification push_server:3010 sip:374349d9254ccb003a5159ea30aca801@proxy.example.com notification: new_call Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> WARNING: topos [tps_storage.c:406]: tps_storage_record(): no local address - do record routing for all initial requests Nov 28 10:07:51 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: SIP > SIP Nov 28 10:07:53 ip-10-23-247-205 /usr/sbin/kamailio[14701]: [CANCEL domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: New request - M=CANCEL R=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com F=sip:+311111111@10.23.248.40 T=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com IP=push_server:3010 DST=0.0.0.0:5060 ID=domain.example.com_87665 Nov 28 10:07:53 ip-10-23-247-205 /usr/sbin/kamailio[14701]: [CANCEL domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: Stop mediaproxy for current branch using first Via - R=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com ID=domain.example.com_87665 Nov 28 10:07:53 ip-10-23-247-205 /usr/sbin/kamailio[14701]: [CANCEL domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2273]: rtpp_function_call(): proxy replied with error: Unknown call-id Nov 28 10:07:53 ip-10-23-247-205 /usr/sbin/kamailio[14704]: [ACK domain.example.com_87665 push_server:3010@udp]--> INFO: <script>: New request - M=ACK R=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com F=sip:+311111111@10.23.248.40 T=sip:374349d9254ccb003a5159ea30aca801@proxy.example.com IP=push_server:3010 DST=0.0.0.0:5060 ID=domain.example.com_87665 Nov 28 10:08:01 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:rtpengine_ip:22222 Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2538]: send_rtpp_command(): timeout waiting reply from RTP proxy udp:rtpengine_ip:22222 Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2412]: rtpp_test(): proxy did not respond to ping Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2749]: select_rtpp_node(): rtpengine failed to select new for calllen=30 callid=domain.example.com_87665 Nov 28 10:08:11 ip-10-23-247-205 /usr/sbin/kamailio[14702]: [INVITE domain.example.com_87665 push_server:3010@udp]--> ERROR: rtpengine [rtpengine.c:2239]: rtpp_function_call(): no available proxies Nov 28 10:08:11 ip-10-23-247-205 kernel: [586868.258566] kamailio[14702]: segfault at 24 ip 00007efca782242e sp 00007fffbef66410 error 4 in tm.so[7efca775c000+138000] Nov 28 10:19:42 ip-10-23-247-205 /usr/sbin/kamailio[10594]: WARNING: tm [tm.c:509]: fixup_routes(): t_on_reply("INV_RESUME"): empty/non existing route Nov 28 10:52:20 ip-10-23-247-205 /usr/sbin/kamailio[10596]: [CANCEL domain.example.com_87732 push_server:3010@udp]--> ERROR: tm [t_reply.c:480]: _reply_light(): can't generate 487 reply when a final 200 was sent out
```
Hi,
Seems that 5.1.6 is afected too, but I don't have coredump from 5.1.6 (yet)
Anyone able to test with the commit referenced above (I just pushed the patch to master)? Might have been an issue caused by a delayed processing (eg., waiting for rtp engine command execution timeout) that also induced some response retransmissions. I want to be sure it doesn't have side effects in some way, so be careful, watching closely if you push it to production.
Hi will try to test it tomorrow and check how it goes
This should be fixed in master and 5.2 branches (commit 814d5cc1f4f5b1e4b95737108dffc1e7d7bd566f).
Closed #1613.