It seems that kamailio received two BYE messages at the same time and dlg was gone?
#### Debugging Data
``` # gdb `which kamailio` core.kamailio.sig11.19255 GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ac/30bd0b9cd954ea6a966ccc9d55affb6291beb5.debug...done. done. [New LWP 19255] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/proxy/kamailio.cfg -P /var/run/kamailio/kam'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f8c1d999565 in remove_dialog_timer () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so (gdb) (gdb) bt full #0 0x00007f8c1d999565 in remove_dialog_timer () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so No symbol table info available. #1 0x00007f8c1d97b4e5 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so No symbol table info available. #2 0x00007f8c2551728d in run_rr_callbacks (req=0x7f8c28bea120, rr_param=0x7ffc7e4bee40) at rr_cb.c:96 l_param = { s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber""..., len = 147} cbp = 0x7f8c290558c0 __FUNCTION__ = "run_rr_callbacks" #3 0x00007f8c255098a2 in after_loose (_m=0x7f8c28bea120, preloaded=2) at loose.c:928 puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = { s = 0x9aab5a <buf+410> "Y.Y.Y.Y:5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest userna"..., len = 10}, port = { s = 0x9aab65 <buf+421> "5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="3562741"..., len = 4}, params = { s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber""..., len = 147}, sip_params = { s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber""..., len = 147}, headers = {s = 0x0, len = 0}, port_no = 5062, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = { s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber""..., len = 5}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = { s = 0x9aab6d <buf+429> "on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber",re"..., len = 2}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} rt = 0x0 status = 0 uri = { s = 0x9aab56 <buf+406> "sip:Y.Y.Y.Y:5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest us"..., len = 167} use_ob = 0 rparams = { s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username="anumber""..., len = 147} __FUNCTION__ = "after_loose" #4 0x0000000000446650 in do_action (h=0x7ffc7e4bf560, a=0x7f8c27f994c8, msg=0x7f8c28bea120) at action.c:1060 ret = -5 v = 24865014 dst = {send_sock = 0x17b6aab, to = {s = {sa_family = 52142, sa_data = "B\000\000\000\000\000\253j{\001\000\000\000"}, sin = {sin_family = 52142, sin_port = 66, sin_addr = {s_addr = 0}, sin_zero = "\253j{\001\000\000\000"}, sin6 = {sin6_family = 52142, sin6_port = 66, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\253j{\001\000\000\000\000\205RD\000\000\000\000", __u6_addr16 = {27307, 379, 0, 0, 21125, 68, 0, 0}, __u6_addr32 = {24865451, 0, 4477573, 0}}}, sin6_scope_id = 2118906304}}, id = 32764, proto = -16 '\360', send_flags = {f = 241 '\361', blst_imask = 75 'K'}} end = 0x7ffc7e4bf560 "\001" uri = {user = {s = 0x7ffc7e4c0830 "\003", len = 938514842}, passwd = {s = 0xa00000000 <error: Cannot access memory at address 0xa00000000>, len = 22844112}, host = { s = 0x100000003 <error: Cannot access memory at address 0x100000003>, len = 681881128}, port = {s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, params = { s = 0x200000001 <error: Cannot access memory at address 0x200000001>, len = 2118906304}, sip_params = {s = 0x15c91c0 "", len = -1642256896}, headers = { s = 0x9aa9c8 <buf+8> "Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: sip:10.64.0.14;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:84.255.4.7:5060\r\nRecord-Route: <sip:84.255.4.7;r2=on;lr=on;ftag=922229c699;ngcplb=y"..., len = 671147880}, port_no = 65535, proto = 65535, type = ERROR_URI_T, flags = (unknown: 683581728), transport = {s = 0x7ffc7e4c0830 "\003", len = 2118906688}, ttl = { s = 0x17b6aea "{\001", len = 4377518}, user_param = {s = 0x7f8c27f6ae20 "\001", len = 671137416}, maddr = {s = 0x7f8c28bea120 "+-\002", len = 671137528}, method = { s = 0x7e4bf280 <error: Cannot access memory at address 0x7e4bf280>, len = 623674360}, lr = {s = 0x7ffc7e4c0670 "\277\001", len = 2118906724}, r2 = {s = 0x7f8c00000000 "", len = 0}, gr = { s = 0x7ffc7e4bf4c4 "\214\177", len = -1642256896}, transport_val = {s = 0xd0de87bdb0391519 <error: Cannot access memory at address 0xd0de87bdb0391519>, len = 671137536}, ttl_val = {s = 0x0, len = 671137416}, user_param_val = {s = 0x7f8c28bea120 "+-\002", len = 671137528}, maddr_val = {s = 0x7ffc7e4bf280 "", len = 4589069}, method_val = {s = 0x0, len = 519504614}, lr_val = { s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250335224}, r2_val = {s = 0xe9a188 <error: Cannot access memory at address 0xe9a188>, len = 18100232}, gr_val = { s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}} next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x1800000000 <error: Cannot access memory at address 0x1800000000>, len = 6243476}, host = { s = 0xe98ed8 <error: Cannot access memory at address 0xe98ed8>, len = -1642256896}, port = {s = 0x6afc59 <__FUNCTION__.8262> "rval_new_empty", len = 671154536}, params = { s = 0x7ffc7e4c0830 "\003", len = 683581728}, sip_params = {s = 0x7f8c2800e768 "\017", len = 683581728}, headers = {s = 0x17b6aea "{\001", len = 4479757}, port_no = 0, proto = 4096, type = ERROR_URI_T, flags = (unknown: 936), transport = {s = 0x19 <error: Cannot access memory at address 0x19>, len = 683581728}, ttl = {s = 0x7ffc7e4c0830 "\003", len = 2118908048}, ---Type <return> to continue, or q <return> to quit--- user_param = {s = 0x7f8c287c7320 "\002", len = 4349792}, maddr = {s = 0x7ffc7e4bf350 "\224\363K~\374\177", len = 4349855}, method = {s = 0x7f8c00000000 "", len = 4362395}, lr = { s = 0x7ffc7e4bf394 "6", len = 1}, r2 = {s = 0x0, len = -1642256896}, gr = {s = 0x1141e58 <error: Cannot access memory at address 0x1141e58>, len = -1642256896}, transport_val = { s = 0x1d <error: Cannot access memory at address 0x1d>, len = 0}, ttl_val = {s = 0x3600000064 <error: Cannot access memory at address 0x3600000064>, len = 532}, user_param_val = {s = 0x0, len = 1}, maddr_val = {s = 0x7f8c28bea120 "+-\002", len = 939161967}, method_val = {s = 0x0, len = 5665286}, lr_val = {s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250337280}, r2_val = {s = 0xe99938 <error: Cannot access memory at address 0xe99938>, len = -1642256896}, gr_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 497}} u = 0x7ffc7e4bf560 flags = 0 st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x5f4494 <fm_malloc+3044>} sct = 0x7ffc7e4bf560 sjt = 0x7ffc7e4bf560 mct = 0x17b6aea rv = 0x7f8c28a9d728 rv1 = 0x7f8c28bea120 c1 = {cache_type = 681970264, val_type = RV_NONE, c = {avp_val = {n = 683581728, s = {s = 0x7f8c28bea120 "+-\002", len = 0}, re = 0x7f8c28bea120}, pval = {rs = {s = 0x7f8c28bea120 "+-\002", len = 0}, ri = 939035407, flags = 32652}}, i2s = '\000' <repeats 16 times>, "\270\361K~\374\177"} s = {s = 0x1ffffffff <error: Cannot access memory at address 0x1ffffffff>, len = 683581728} srevp = {0x7ffc7e4c0670, 0x7ffc7e4bf214} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #5 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, h@entry=0x7ffc7e4bf560, a=0x6, msg=0x1) at action.c:1549 ret = 2118912048 mod = 0x7f8c27f994c8 __FUNCTION__ = "run_actions" #6 0x0000000000451ae7 in run_actions_safe (h=0x7ffc7e4c0830, a=<optimized out>, msg=<optimized out>) at action.c:1614 ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {140722427393376, -3396127869322586855, 140239955730720, 140722427398192, 140722427394192, 24865514, -3396127820052097767, 3397538908824868121}, __mask_was_saved = 0, __saved_mask = {__val = {5665521, 140722427393848, 140239955730720, 140722427393848, 7568605033954878976, 140722427393812, 497, 140239942814231, 140239942814231, 140722427393744, 0, 99, 5666971, 15310176, 18099168, 18453488}}}}} ret = <optimized out> ign_on_break = 0 #7 0x0000000000426c70 in rval_get_int (h=0x7ffc7e4c0830, msg=0x7f8b5fb23500, i=0x7ffc7e4bf890, rv=0x0, cache=0x9d6460 <tcp_main_ltimer+112672>) at rvalue.c:912 r_avp = 0x0 avp_val = {n = 2118907856, s = {s = 0x7ffc7e4bf7d0 "", len = 5668051}, re = 0x7ffc7e4bf7d0} pval = {rs = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 683540480}, ri = 1, flags = 32764} tmp = {s = 0x7ffc00000001 <error: Cannot access memory at address 0x7ffc00000001>, len = 670665240} s = 0x7f8c28bea120 r = 0 ret = 0 destroy_pval = 2118912048 __FUNCTION__ = "rval_get_int" #8 0x000000000042c9fc in rval_expr_eval_int (h=0x7ffc7e4c0830, msg=0x7f8c28bea120, res=0x7ffc7e4bf890, rve=0x7f8c27f995f8) at rvalue.c:1910 i1 = 32652 i2 = 1 ret = -1 c1 = {cache_type = 670665240, val_type = 32652, c = {avp_val = {n = 31, s = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 670994688}, re = 0x6909162d0000001f}, pval = {rs = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 670994688}, ri = -1642256896, flags = 1762203181}}, i2s = "\000\000\000\000\004\000\000\000ȋ\371'\214\177\000\000\000\000\000\000\000"} c2 = {cache_type = 683581728, val_type = 32652, c = {avp_val = {n = 670665664, s = {s = 0x7f8c27f98bc0 "\001", len = 670665672}, re = 0x7f8c27f98bc0}, pval = {rs = {s = 0x7f8c27f98bc0 "\001", len = 670665672}, ri = 4589069, flags = 0}}, i2s = "\304\023\000\000\001\000\000\000\260\211\371'\214\177\000\000\000\000\000\000\000"} rv1 = 0x7f8c252c7f66 <pv_set_avp+1270> rv2 = 0x10000017b6aea __FUNCTION__ = "rval_expr_eval_int" #9 0x0000000000446b27 in do_action (h=0x7ffc7e4c0830, a=0x7f8c27fe6830, msg=0x7f8c28bea120) at action.c:1030 ret = -5 ---Type <return> to continue, or q <return> to quit--- v = 0 dst = {send_sock = 0x7f8c28bea120, to = {s = {sa_family = 7, sa_data = "\000\000\000\000\000\000\260\376K~\374\177\000"}, sin = {sin_family = 7, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\260\376K~\374\177\000"}, sin6 = {sin6_family = 7, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\260\376K~\374\177\000\000 \241\276(\214\177\000", __u6_addr16 = {65200, 32331, 32764, 0, 41248, 10430, 32652, 0}, __u6_addr32 = {2118909616, 32764, 683581728, 32652}}}, sin6_scope_id = 24865514}}, id = 0, proto = 91 '[', send_flags = { f = 119 'w', blst_imask = 87 'W'}} end = 0x7f8c27f995f8 "\001" uri = {user = {s = 0x7ffc7e4bfeb0 " \241\276(\214\177", len = 683581728}, passwd = {s = 0x7ffc7e4bfeb0 " \241\276(\214\177", len = 683581728}, host = {s = 0x17b6aea "{\001", len = 4482640}, port = {s = 0x7f8c2818fe58 "\220\001", len = 687376112}, params = {s = 0x7ffc7e4c0830 "\003", len = -5}, sip_params = {s = 0x7ffc7e4bfab0 "", len = 683581728}, headers = { s = 0x1666830 "\220m\177\001", len = 2118909840}, port_no = 13540, proto = 14338, type = 32652, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 939668708), transport = {s = 0x0, len = 938508137}, ttl = {s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 4349792}, user_param = {s = 0x7ffc7e4bfb20 "", len = 4349855}, maddr = { s = 0x7f8c287c7320 "\002", len = 4362395}, method = {s = 0x7f8b5fb95036 "udp", len = 3}, lr = {s = 0x7f8c37f0999a <vfprintf+22490> "\200\275(\373\377\377", len = 2118909824}, r2 = { s = 0x7f8c287c7320 "\002", len = 540189421}, gr = {s = 0x7f8c20547260 <log_flag> "\377\377\377\377\001", len = 4349855}, transport_val = {s = 0x7f8c285b4f00 "\002", len = 4362395}, ttl_val = { s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = -5}, user_param_val = {s = 0x4b37 <error: Cannot access memory at address 0x4b37>, len = 10}, maddr_val = { s = 0x10000020 <error: Cannot access memory at address 0x10000020>, len = 938689182}, method_val = {s = 0x1666830 "\220m\177\001", len = 687260760}, lr_val = {s = 0x7ffc7e4c0000 "", len = 683581728}, r2_val = {s = 0x7ffc7e4c0000 "", len = 683581728}, gr_val = {s = 0x17b6aea "{\001", len = 4482640}} next_hop = {user = {s = 0x7ffc7e4bfc00 "Y\374j", len = -1642256896}, passwd = {s = 0x0, len = 687213120}, host = {s = 0x7ffc7e4c0830 "\003", len = -5}, port = {s = 0x7ffc7e4bfc10 "", len = 683581728}, params = {s = 0x17b6aea "{\001", len = 4477573}, sip_params = {s = 0x7ffc7e4bfc10 "", len = 2118908992}, headers = {s = 0x7f8c28f60a40 "\334\001", len = 683581728}, port_no = 0, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = 0}, ttl = { s = 0x7ffc7e4c0020 " \241\276(\214\177", len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 938514842}, method = {s = 0x7f8c37f0999a <vfprintf+22490> "\200\275(\373\377\377", len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 519504614}, gr = { s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250336336}, transport_val = {s = 0xe99d30 <error: Cannot access memory at address 0xe99d30>, len = 18099120}, ttl_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, user_param_val = {s = 0x19b <error: Cannot access memory at address 0x19b>, len = -1642256896}, maddr_val = {s = 0x1142bb0 <error: Cannot access memory at address 0x1142bb0>, len = 6243476}, method_val = {s = 0x7ffc7e4c0830 "\003", len = 6978301}, lr_val = { s = 0x6afc59 <__FUNCTION__.8262> "rval_new_empty", len = -1642256896}, r2_val = {s = 0x0, len = 687224008}, gr_val = {s = 0x7ffc7e4c0830 "\003", len = -5}} u = 0x7f8c27f995f8 flags = 0 st = {flags = 2118908928, id = 32764, name = {n = -1642256896, s = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 0}, re = 0x6909162d9e1d2600}, avp = 0x7f8c28f60a40} sct = 0x7f8c27f995f8 sjt = 0x7f8c27f995f8 mct = 0x17b6aea rv = 0x0 rv1 = 0x7f8c28bea120 c1 = {cache_type = 687185704, val_type = 32652, c = {avp_val = {n = -1642256896, s = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 683581728}, re = 0x6909162d9e1d2600}, pval = {rs = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 683581728}, ri = 542603232, flags = 32652}}, i2s = "\360\223\031\001F\000\000\000\000&\035\236-\026\ti\243\001\000\000\000"} s = {s = 0x186b4f57011424d8 <error: Cannot access memory at address 0x186b4f57011424d8>, len = -1642256896} srevp = {0x9aad1a <buf+858>, 0x5e7} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #10 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, a=0x6, msg=0x1) at action.c:1549 ret = 1 mod = 0x7f8c27fe6830 __FUNCTION__ = "run_actions" #11 0x0000000000446d40 in do_action (h=0x7ffc7e4c0830, a=0x7f8c288d33a8, msg=0x7f8c28bea120) at action.c:678 ret = -5 v = 0 dst = {send_sock = 0x17b6aab, to = {s = {sa_family = 21125, sa_data = "D\000\000\000\000\000\300y\020(\214\177\000"}, sin = {sin_family = 21125, sin_port = 68, sin_addr = {s_addr = 0}, sin_zero = "\300y\020(\214\177\000"}, sin6 = {sin6_family = 21125, sin6_port = 68, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "\300y\020(\214\177\000\000/\000\000\000\000\000\000", __u6_addr16 = {31168, 10256, 32652, 0, 47, 0, 0, 0}, __u6_addr32 = {672168384, 32652, 47, 0}}}, sin6_scope_id = 980445555}}, id = 32652, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}} end = 0x6 <error: Cannot access memory at address 0x6> uri = {user = {s = 0x7ffc7e4c0064 "", len = 6404846}, passwd = {s = 0x7ffc7e4c0830 "\003", len = 2118910388}, host = {s = 0x17b6aea "{\001", len = -1642256896}, port = { s = 0x7f8c28bea120 "+-\002", len = 687260760}, params = {s = 0x7ffc7e4c0830 "\003", len = -5}, sip_params = {s = 0x7ffc7e4c0000 "", len = 683581728}, headers = {s = 0x17b6aea "{\001", ---Type <return> to continue, or q <return> to quit--- len = 4477573}, port_no = 0, proto = 32332, type = 32764, flags = (URI_USER_NORMALIZE | unknown: 4212331800), transport = {s = 0x7f8c28f6c458 ")\002", len = 683581728}, ttl = { s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 683581728}, user_param = {s = 0x7ffc7e4c0088 "", len = 2118910056}, maddr = {s = 0x7ffc7e4c0064 "", len = 5800526}, method = {s = 0x7f8c28f68630 "\001", len = 683581728}, lr = {s = 0x7ffc7e4c0830 "\003", len = 519504614}, r2 = {s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250336336}, gr = {s = 0xe99d30 <error: Cannot access memory at address 0xe99d30>, len = 18099120}, transport_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, ttl_val = {s = 0x19b <error: Cannot access memory at address 0x19b>, len = -1642256896}, user_param_val = {s = 0x7f8c287c7320 "\002", len = 6233684}, maddr_val = { s = 0x7ffc7e4c0080 "", len = 679244504}, method_val = {s = 0x7f8c38263620 "", len = 0}, lr_val = {s = 0x7ffc7e4c0830 "\003", len = 683581728}, r2_val = {s = 0x7ffc7e4c0830 "\003", len = 2118910768}, gr_val = {s = 0x17b6aea "{\001", len = 4528871}} next_hop = {user = {s = 0x0, len = 1}, passwd = {s = 0x7ffc7e4c0000 "", len = 1094259993}, host = {s = 0x7f8c28bea120 "+-\002", len = 2118912048}, port = {s = 0x7ffc7e4c0330 "\001", len = 24865514}, params = {s = 0xd0de87bdacb91519 <error: Cannot access memory at address 0xd0de87bdacb91519>, len = -82635495}, sip_params = {s = 0x0, len = 544780589}, headers = { s = 0x7e4c0168 <error: Cannot access memory at address 0x7e4c0168>, len = 687244856}, port_no = 36328, proto = 10486, type = 32652, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 939139724), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x7ffc7e4c0140 "h\345\230 \214\177", len = 2118910240}, maddr = {s = 0x7f8c3a706973 <error: Cannot access memory at address 0x7f8c3a706973>, len = 683581728}, method = {s = 0x1816f60 "\340hm\001", len = 0}, lr = { s = 0x40 <error: Cannot access memory at address 0x40>, len = -1642256896}, r2 = {s = 0x300000039 <error: Cannot access memory at address 0x300000039>, len = 2118910768}, gr = { s = 0x7ffc7e4c0830 "\003", len = 4353136}, transport_val = {s = 0x1000000000000 <error: Cannot access memory at address 0x1000000000000>, len = 7200}, ttl_val = {s = 0x1541c40 "CEST", len = 623629966}, user_param_val = {s = 0x7f8b5fc10115 "5060", len = 10498376}, maddr_val = {s = 0x9a9790 <_km_log_func> "\340jA", len = 0}, method_val = { s = 0x7f8c2098e568 <_xlog_prefix> "\242\263x \214\177", len = -1642256896}, lr_val = {s = 0x416ae0 syslog@plt "\377%\222\310X", len = 687262272}, r2_val = { s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 683581728}, gr_val = {s = 0x7ffc7e4c0830 "\003", len = 2118910768}} u = 0x6 flags = 32652 st = {flags = 0, id = 0, name = {n = 1, s = {s = 0x7f8c00000001 "", len = 2118909952}, re = 0x7f8c00000001}, avp = 0xd0de87b241391519} sct = 0x6 sjt = 0x6 mct = 0x17b6aea rv = 0x7f8c28f79ba8 rv1 = 0x7f8c28bea120 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 5060, s = {s = 0x29000013c4 <error: Cannot access memory at address 0x29000013c4>, len = 2118912048}, re = 0x29000013c4}, pval = {rs = {s = 0x29000013c4 <error: Cannot access memory at address 0x29000013c4>, len = 2118912048}, ri = 672168304, flags = 32652}}, i2s = "\270y\020(\214\177\000\000\210\000L~\374\177\000\000\210\000L~\374\177"} s = {s = 0x7f8c287c766e "4", len = 3} srevp = {0x7ffc7e4c0670, 0x7f8c28bea120} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #12 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, a=0x6, msg=0x1) at action.c:1549 ret = 2118912048 mod = 0x7f8c28f6c808 __FUNCTION__ = "run_actions" #13 0x0000000000446b78 in do_action (h=0x7ffc7e4c0830, a=0x7f8c28f6ee18, msg=0x7f8c28bea120) at action.c:1049 ret = 1 v = 1 dst = {send_sock = 0x7f8c28bea120, to = {s = {sa_family = 51646, sa_data = "\334%\214\177\000\000\264\000\000\000\374\177\000"}, sin = {sin_family = 51646, sin_port = 9692, sin_addr = { s_addr = 32652}, sin_zero = "\264\000\000\000\374\177\000"}, sin6 = {sin6_family = 51646, sin6_port = 9692, sin6_flowinfo = 32652, sin6_addr = {__in6_u = { __u6_addr8 = "\264\000\000\000\374\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {180, 0, 32764, 0, 0, 0, 0, 0}, __u6_addr32 = {180, 32764, 0, 0}}}, sin6_scope_id = 674766208}}, id = 32652, proto = 32 ' ', send_flags = {f = 161 '\241', blst_imask = 190 '\276'}} end = 0x7f8c28f6ca40 "\001" uri = {user = {s = 0x7ffc7e4c05dc "", len = 635512766}, passwd = {s = 0x7f8b5fdb1890 "\224*\002", len = 683581728}, host = {s = 0x5fcdb5d8 <error: Cannot access memory at address 0x5fcdb5d8>, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x4881 <error: Cannot access memory at address 0x4881>, len = -1642256896}, headers = { s = 0x6 <error: Cannot access memory at address 0x6>, len = 634972939}, port_no = 180, proto = 0, type = 32651, flags = (unknown: 2118911600), transport = {s = 0x0, len = 681887176}, ttl = { s = 0x7f8c28bea120 "+-\002", len = 1610866240}, user_param = {s = 0x7f8b6003dc38 "\240\377h_\213\177", len = 4529039}, maddr = {s = 0x0, len = 0}, method = { s = 0x4 <error: Cannot access memory at address 0x4>, len = 635512766}, lr = {s = 0x0, len = 683581728}, r2 = {s = 0x7f8b5ff2fff8 "", len = 200}, gr = {s = 0x0, len = 683581728}, transport_val = {s = 0x7f8c28837df0 <incomplete sequence \350>, len = 638037828}, ttl_val = {s = 0x7ffc7e4c05dc "", len = -1642256896}, user_param_val = { s = 0x7f8c260671f0 <tm_cfg> "h\371\220_\213\177", len = 634977214}, maddr_val = {s = 0x7f8b00000004 <error: Cannot access memory at address 0x7f8b00000004>, len = 1609884936}, method_val = { s = 0x4 <error: Cannot access memory at address 0x4>, len = 683581728}, lr_val = {s = 0xc8 <error: Cannot access memory at address 0xc8>, len = 1609885456}, r2_val = {s = 0x7f8b5ff4e508 "", len = 634984442}, gr_val = {s = 0x0, len = -1642256896}} next_hop = {user = {s = 0x7f8c28bea120 "+-\002", len = 2118911456}, passwd = {s = 0x5 <error: Cannot access memory at address 0x5>, len = 683581728}, host = { ---Type <return> to continue, or q <return> to quit--- s = 0xb4 <error: Cannot access memory at address 0xb4>, len = 1610866240}, port = {s = 0x7f8b6003dc38 "\240\377h_\213\177", len = 635245418}, params = { s = 0x7ffc000000b7 <error: Cannot access memory at address 0x7ffc000000b7>, len = 10493856}, sip_params = {s = 0xa01fa8 <def_list+8> "", len = 10493872}, headers = { s = 0xa01fb8 <def_list+24> "", len = 10493896}, port_no = 7448, proto = 160, type = ERROR_URI_T, flags = (URI_SIP_USER_PHONE | unknown: 519504612), transport = { s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 519504614}, ttl = {s = 0x0, len = 250281984}, user_param = {s = 0x0, len = 519504614}, maddr = { s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250338176}, method = {s = 0xe98c70 <error: Cannot access memory at address 0xe98c70>, len = 18097280}, lr = { s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, r2 = {s = 0x1dc <error: Cannot access memory at address 0x1dc>, len = 519504614}, gr = { s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250341208}, transport_val = {s = 0xe989e0 <error: Cannot access memory at address 0xe989e0>, len = 18094248}, ttl_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, user_param_val = {s = 0x1bf <error: Cannot access memory at address 0x1bf>, len = -1642256896}, maddr_val = {s = 0x11418a8 <error: Cannot access memory at address 0x11418a8>, len = 6243476}, method_val = {s = 0x7f8c28a4c5c8 "\231\034", len = 6978301}, lr_val = { s = 0x71c670 <__FUNCTION__.7459> "parse_via", len = 1610865720}, r2_val = {s = 0xd0de87b257b91519 <error: Cannot access memory at address 0xd0de87b257b91519>, len = -82635495}, gr_val = { s = 0x0, len = -1642256896}} u = 0x7f8c28f6ca40 flags = 32651 st = {flags = 683581728, id = 32652, name = {n = 2118911456, s = {s = 0x7ffc7e4c05e0 "", len = 5}, re = 0x7ffc7e4c05e0}, avp = 0x7f8c28bea120} sct = 0x7f8c28f6ca40 sjt = 0x7f8c28f6ca40 mct = 0x17b6aea rv = 0x1 rv1 = 0x7f8c28bea120 c1 = {cache_type = 180, val_type = RV_NONE, c = {avp_val = {n = 1610865912, s = {s = 0x7f8b6003dcf8 "\264", len = 10482636}, re = 0x7f8b6003dcf8}, pval = {rs = {s = 0x7f8b6003dcf8 "\264", len = 10482636}, ri = 5, flags = 1}}, i2s = "\340\005L~\374\177\000\000\070\334\003`\213\177\000\000\240\377h_\213\177"} s = {s = 0xa48980 <_pkg_root> "\344\016p", len = 808857653} srevp = {0x7f8c27ede430, 0x7f8b6003de40} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #14 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, h@entry=0x7ffc7e4c0830, a=0x6, a@entry=0x7f8c28f1b930, msg=0x1, msg@entry=0x7f8c28bea120) at action.c:1549 ret = 1 mod = 0x7f8c28f6ee18 __FUNCTION__ = "run_actions" #15 0x0000000000451b85 in run_top_route (a=0x7f8c28f1b930, msg=0x7f8c28bea120, c=<optimized out>) at action.c:1635 ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140722427398192, 3397538390581253401, 140239959079216, 140239955730720, 0, 0, -3396127868821367527, 3397538908824868121}, __mask_was_saved = 0, __saved_mask = {__val = {10136323, 1354, 7568605033954878976, 10136001, 6243476, 140239948066720, 140239955730720, 10136044, 10136000, 1354, 7568605033954878976, 0, 140239960257184, 1073741825, 140239907703544, 140239960247216}}}}} p = <optimized out> ret = <optimized out> sfbk = 0 #16 0x0000000000556f9e in receive_msg (buf=0x0, len=683581728, rcv_info=0x7ffc7e4c0b10) at receive.c:240 msg = 0x7f8c28bea120 ctx = {rec_lev = 23098816, run_flags = 0, last_retcode = 23098816, jmp_env = {{__jmpbuf = {0, 6, 140236580472680, 140236549023960, 140722427398688, 140239791653606, 268435456, 250568360}, __mask_was_saved = 15165008, __saved_mask = {__val = {10105736, 0, 140722427398950, 5, 1, 140239942050864, 140240218561991, 5, 0, 7063492, 140240210401128, 140239955456248, 140239942050864, 140240220669584, 140240218587877, 140240220669584}}}}} inb = { s = 0x9aa9c0 <buf> "BYE sip:Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: sip:10.64.0.14;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:84.255.4.7:5060\r\nRecord-Route: <sip:84.255.4.7;r2=on;lr=on;ftag=922229c699;"..., len = 1354} netinfo = {data = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1571855534}, dst = 0x7f8c37ec9768, rcv = 0x7f8c38895000} __FUNCTION__ = "receive_msg" #17 0x0000000000479f10 in udp_rcv_loop () at udp_server.c:495 len = 1354 buf = "BYE sip:Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: sip:10.64.0.14;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:84.255.4.7:5060\r\nRecord-Route: <sip:84.255.4.7;r2=on;lr=on;ftag=922229c699;"... from = 0x7f8c28ba70f8 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {234897418, 140236580472680}, addr32 = {234897418, 0, 1603290984, 32651}, addr16 = {16394, 3584, 0, 0, 18280, 24464, 32651, 0}, addr = "\n@\000\016\000\000\000\000hG\220_\213\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {335560714, 0}, addr32 = {335560714, 0, 0, 0}, addr16 = {16394, 5120, 0, 0, 0, 0, 0, 0}, ---Type <return> to continue, or q <return> to quit--- addr = "\n@\000\024", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5062, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n@\000\016\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 234897418}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = { sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 234897418, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f8c27ede430, proto = 1 '\001'} __FUNCTION__ = "udp_rcv_loop" #18 0x00000000004ffe39 in main_loop () at main.c:1614 si_desc = "udp receiver child=5 sock=Y.Y.Y.Y:5062\000\000\000\001\000\000\000pX\300(\214\177\000\000\000\000\000\000\000\000\000\000n\000\000\000\000\000\000\000\315f_\000\000\000\000\000\003\000\000\000\000\000\000\000\000&\035\236-\026\ti\000\000\000\000\000\000\000\000\000&\035\236-\026\ti\310\002\245\000\000\000\000\000\005\000\000\000\000\000\000" woneinit = 1 __FUNCTION__ = "main_loop" #19 0x000000000041cd6c in main (argc=0, argv=0x0) at main.c:2631 cfg_stream = 0x0 r = 0 tmp = 0x7ffc7e4c2ee4 "" tmp_len = 0 port = 0 proto = 948524496 ret = 0 seed = 1296618944 debug_save = 0 debug_flag = 4 dont_fork_cnt = 0 st = {st_dev = 15, st_ino = 122623671, st_nlink = 2, st_mode = 16877, st_uid = 106, st_gid = 108, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = { tv_sec = 1488564228, tv_nsec = 817944387}, st_mtim = {tv_sec = 1490749503, tv_nsec = 811780252}, st_ctim = {tv_sec = 1490749503, tv_nsec = 811780252}, __glibc_reserved = {0, 0, 0}} __FUNCTION__ = "main" (gdb) ```
#### Log Messages
<!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). -->
``` Mar 29 09:02:14 sp2 proxy[19254]: NOTICE: <script>: New request on proxy - M=ACK R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (10.64.0.14:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715' Mar 29 09:03:57 sp2 proxy[19255]: NOTICE: <script>: New request on proxy - M=BYE R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (10.64.0.14:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715' Mar 29 09:03:57 sp2 proxy[19257]: NOTICE: <script>: New request on proxy - M=BYE R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (10.64.0.14:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715' Mar 29 09:03:57 sp2 proxy[19257]: NOTICE: <script>: NAT-Reply - S=200 - OK M=BYE IP=X.X.X.X:5060 (Y.Y.Y.Y:5080) ID=bf8e1a679ee462a4 UA='<null>' ```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 4.4.4 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.9.2 ```
This is ``NGCP`` based on 4.4.4 with tm deep cloning edb9df8ace8ff74 backported
Any idea why frames 1 and 2 don't have the symbols? Was there a single core file, presuming the options for core files per pid was set?
Any idea why frames 1 and 2 don't have the symbols? Was there a single core file, presuming the options for core files per pid was set?
No idea, per-process core file is enabled too. ``` # cat /proc/sys/kernel/core_uses_pid 1 ```
same thing in another customer
``` #0 0x00007fae3f1f7e14 in strnlen () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007fae3f1c0b19 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007fae3f25a216 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #3 0x00007fae3f25a36f in syslog () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #4 0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, param=0x7fff717dab10) at dlg_handlers.c:1350 ```
Can you do: * frame 4 * list * info locals
The code in branch 4.4 for dlg_handlers.c:1350 is:
``` LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] " "with clid '%.*s' and tags '%.*s' '%.*s'\n", dlg, dlg->h_entry, dlg->h_id, dlg->callid.len, dlg->callid.s, dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s, dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s); ```
So no strlen() to be used there.
This is the ``NGCP`` version of 4.4.5 sources at https://github.com/sipwise/kamailio/tree/mr4.5.4 but be aware that list of patches are at https://github.com/sipwise/kamailio/blob/mr4.5.4/debian/patches/series and I changed the sensible data from the paste
In order you to see the source easily I pasted dlg_handlers.c already patched at https://paste.debian.net/926240/
``` (gdb) f 4 #4 0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, param=0x7fff717dab10) at dlg_handlers.c:1350 1350 dlg_handlers.c: No such file or directory. (gdb) info locals val = { s = 0x9aaf79 <buf+1465> "c68.0d81;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=Y3Z5enJeQmp8bVRNYGQtEwxdZHR2LHRwc1RAaX0hWEFyMw1mZg-->\r\nP-NGCP-Src-Ip: 172.17.65.99\r\nP-NGCP-Src-Port: 5060\r\nP-NGCP-Src-Proto: udp\r\nP-NG"..., len = 8} ftag = { s = 0x9aabaa <buf+490> "C03675F3E457BC60\r\nTo: sip:00000000000@sip.local:5060;tag=0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="2233"..., len = 16} old_state = 4 dir = 1 ret = 0 callid = { s = 0x9aac1a <buf+602> "3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="1111111111", realm="sip.local", nonce="WOUBrFjkp9Qj4YjEwa9jvelH7ty68fD/", uri="sip:00000000000@172.17.66.118"..., len = 28} new_state = 5 unref = 1 event = 10498440 dlg = 0x7fad74f20258 ttag = { s = 0x9aabed <buf+557> "0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="1111111111", realm="sip.local", nonce="WOUBrFjkp9Qj4YjEwa9jvelH"..., len = 34} h_id = 854147520 dlg = 0x7fad00000000 ```
So relevant code is too ``` LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] " "with clid '%.*s' and tags '%.*s' '%.*s'\n", dlg, dlg->h_entry, dlg->h_id, dlg->callid.len, dlg->callid.s, dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s, dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s); ```
So no strlen() to be used there.
but the ``strlen() `` is coming from ``syslog()``
I would say the problem is ``dlg`` is gone from memory when trying to use the string
Did you put in pastebin the wrong file (like dlg_db_handlers.c)?
All the tokens provided by kamailio in that log message are by length and pointer (%.*s), so there should be no strlen done on the pointers we set, so not sure on what syslog can do the strlen and crash.
I wonder if there is a copy/paste issue or the `dlg` is printed twice by `info locals`:
``` event = 10498440 dlg = 0x7fad74f20258 ttag = { s = 0x9aabed <buf+557> "0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="1111111111", realm="sip.local", nonce="WOUBrFjkp9Qj4YjEwa9jvelH"..., len = 34} h_id = 854147520 dlg = 0x7fad00000000 ```
After event and at the end. First looks like a valid value, second a bit more strange.
Can you get in the same frame:
``` p dlg p *dlg ```
Did you put in pastebin the wrong file (like dlg_db_handlers.c)?
Yes, I did, sorry. The right file at https://paste.debian.net/926247/
``` (gdb) f 4 #4 0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, param=0x7fff717dab10) at dlg_handlers.c:1350 1350 dlg_handlers.c: No such file or directory. (gdb) p dlg $1 = (dlg_cell_t *) 0x7fad74f20258 (gdb) p *dlg $2 = {ref = 1264713769, next = 0x7fad74e7aa78, prev = 0x7fad74f20288, h_id = 20, h_entry = 0, state = 1962017437, lifetime = 32685, init_ts = 11, start_ts = 0, end_ts = 1734963823, dflags = 1667457375, iflags = 1818321759, sflags = 1601332588, toroute = 1919251317, toroute_name = { s = 0x3633323631393631 <error: Cannot access memory at address 0x3633323631393631>, len = 1768189440}, from_rr_nb = 115, tl = {next = 0x38, prev = 0x0, timeout = 0}, callid = { s = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000>, len = 890716088}, from_uri = {s = 0x7fae3517bf90 <__FUNCTION__.15256> "rtpp_function_call", len = 890714464}, to_uri = {s = 0x981 <error: Cannot access memory at address 0x981>, len = -252645136}, req_uri = { s = 0x7fad74f20380 "SDj02te01-da0287f0f142f9dac6bda3fd889f1592-06a7f50", len = 50}, tag = {{ s = 0x7fad757740f8 "z9hG4bK41d6.cdcb6121b68837c41a117213d8d3d9a4.0", len = 46}, { s = 0x7fad74aa0688 "", len = 6742548}}, cseq = {{s = 0x0, len = 56}, {s = 0x0, len = 0}}, route_set = {{s = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000>, len = 890720288}, {s = 0x7fae3517beeb <__FUNCTION__.6323> "shm_str_dup", len = 890714464}}, contact = {{s = 0x2d3 <error: Cannot access memory at address 0x2d3>, len = -252645136}, { s = 0x30657432306a4453 <error: Cannot access memory at address 0x30657432306a4453>, len = 1633955121}}, bind_addr = {0x3966323431663066, 0x3361646236636164}, cbs = { first = 0x3531663938386466, types = 808268345}, profile_links = 0x3035, vars = 0x48} ```
be aware that this is from the second core
``` (gdb) info locals val = { s = 0x9aaf79 <buf+1465> "c68.0d81;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=Y3Z5enJeQmp8bVRNYGQtEwxdZHR2LHRwc1RAaX0hWEFyMw1mZg-->\r\nP-NGCP-Src-Ip: 172.17.65.99\r\nP-NGCP-Src-Port: 5060\r\nP-NGCP-Src-Proto: udp\r\nP-NG"..., len = 8} ftag = { s = 0x9aabaa <buf+490> "C03675F3E457BC60\r\nTo: sip:01732962557@sip.netcologne.de:5060;tag=0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="2233"..., len = 16} old_state = 4 dir = 1 ret = 0 callid = { s = 0x9aac1a <buf+602> "3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="2233965503", realm="sip.netcologne.de", nonce="WOUBrFjkp9Qj4YjEwa9jvelH7ty68fD/", uri="sip:01732962557@172.17.66.118"..., len = 28} new_state = 5 unref = 1 event = 10498440 dlg = 0x7fad74f20258 ttag = { s = 0x9aabed <buf+557> "0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username="2233965503", realm="sip.netcologne.de", nonce="WOUBrFjkp9Qj4YjEwa9jvelH"..., len = 34} h_id = 854147520 dlg = 0x7fad00000000 ```
for the relevant part of code ``` LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] " "with clid '%.*s' and tags '%.*s' '%.*s'\n", dlg, dlg->h_entry, dlg->h_id, dlg->callid.len, dlg->callid.s, dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s, dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s); ```
``` (gdb) p dlg->h_entry $3 = 0 (gdb) p dlg->h_id $4 = 20 (gdb) p dlg->callid.len $5 = 890716088 (gdb) p dlg->callid.s $6 = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000> (gdb) p dlg->tag[0].len $7 = 46 (gdb) p dlg->tag[0].s $8 = 0x7fad757740f8 "z9hG4bK41d6.cdcb6121b68837c41a117213d8d3d9a4.0" (gdb) p dlg->tag[1].len $9 = 6742548 (gdb) p dlg->tag[1].s $10 = 0x7fad74aa0688 "" ```
and we have this at the log at the time of the core, 11064 is another process. ``` Apr 5 10:19:36 sipwise1-prx03a proxy[11064]: WARNING: dialog [dlg_handlers.c:1220]: dlg_onroute(): unable to find dialog for BYE with route param 'c68.0d81' [2156:6352] ```
Is it possible that there is a time consuming operations for BYE (eg, a slow database operation -- accounting or something else) that can take more than 5 seconds?
more than 5 secs? I don't think so. @apogrebennyk can you confirm?
We have latency logging enabled but there were no abnormally long delays reported. The last lines before the crash are like this: ``` Apr 5 10:19:39 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket` ,`cflags`,`pat] Apr 5 10:19:39 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket` ,`cflags`,`pat] Apr 5 10:19:39 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:39 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:39 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:40 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:40 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:40 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:40 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:40 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:41 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:41 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:41 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:42 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:43 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:43 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:43 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:43 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:43 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:44 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:44 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:44 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat] Apr 5 10:19:44 sipwise1 proxy[11056]: ALERT: <core> [main.c:740]: handle_sigs(): child process 11061 exited by a signal 11 Apr 5 10:19:44 sipwise1 proxy[11056]: ALERT: <core> [main.c:743]: handle_sigs(): core was generated ```
If you haven't ported the patches referenced above yourself, I just did it. Can you try with latest 4.4 branch and see the results?
Closing this one, eventually fixed with the patches from the few weeks ago. If issue persists, re-open.
Closed #1059.