### Description
Same as reported in #2828 but using a commit that was supposed to have that issue corrected : ``` [root@lab002103-flip-server kamailio-master]$ git log |head -n 5 commit 0d363cf1c1f09b4920e137eac74e1593e7120531 Author: korayvt korayvt@users.noreply.github.com Date: Fri Oct 14 21:27:03 2022 +0300
rtpengine: fixed set id value in log lines (#3265) ``` ``` (gdb) bt #0 0x00007fcc6a36b4df in print_lists (dlg=0x7fcc2f0e6da8) at dlg_var.c:277 #1 0x00007fcc6a36f9fa in pv_set_dlg_variable (msg=0x7fcc6af81de0, param=0x7fcc6adb4d00, op=254, val=0x7ffc545af190) at dlg_var.c:587 #2 0x000055b8d971a148 in lval_pvar_assign (h=0x7ffc545b0530, msg=0x7fcc6af81de0, lv=0x7fcc6af54f38, rv=0x7fcc6af55be8) at core/lvalue.c:352 #3 0x000055b8d971ad64 in lval_assign (h=0x7ffc545b0530, msg=0x7fcc6af81de0, lv=0x7fcc6af54f38, rve=0x7fcc6af55be0) at core/lvalue.c:400 #4 0x000055b8d964f0d6 in do_action (h=0x7ffc545b0530, a=0x7fcc6af55228, msg=0x7fcc6af81de0) at core/action.c:1458 #5 0x000055b8d96511bf in run_actions (h=0x7ffc545b0530, a=0x7fcc6af55228, msg=0x7fcc6af81de0) at core/action.c:1584 #6 0x000055b8d96428a7 in do_action (h=0x7ffc545b0530, a=0x7fcc6af5ed40, msg=0x7fcc6af81de0) at core/action.c:1070 #7 0x000055b8d96511bf in run_actions (h=0x7ffc545b0530, a=0x7fcc6af5ed40, msg=0x7fcc6af81de0) at core/action.c:1584 #8 0x000055b8d9651975 in run_top_route (a=0x7fcc6af5ed40, msg=0x7fcc6af81de0, c=0x7ffc545b0530) at core/action.c:1669 #9 0x00007fcc6abec88d in reply_received (p_msg=0x7fcc6af81de0) at t_reply.c:2546 #10 0x000055b8d96db775 in do_forward_reply (msg=0x7fcc6af81de0, mode=0) at core/forward.c:764 #11 0x000055b8d96dd7df in forward_reply (msg=0x7fcc6af81de0) at core/forward.c:865 #12 0x000055b8d97b2071 in receive_msg ( buf=0x55b8d9c0c5a0 <buf> "SIP/2.0 100 Trying\r\nTo: sip:0311112222@192.168.2.103;tag=3208514SIPpTag01501\r\nFrom: 0300000080 sip:0300000080@the.domain.0.com;tag=3207307SIPpTag001780\r\nCall-ID: 1780-3207307@10.255.255.104\r\nCSeq:"..., len=474, rcv_info=0x7ffc545b0f90) at core/receive.c:609 #13 0x000055b8d990da93 in udp_rcv_loop () at core/udp_server.c:587 #14 0x000055b8d962bcd7 in main_loop () at main.c:1724 #15 0x000055b8d9638b77 in main (argc=8, argv=0x7ffc545b1808) at main.c:3085 ``` #### Reproduction
This cannot be reproduced at will. It happened while load testing 4 kamailio instances using commit 0d363cf1 (3 other kamailio instances are still running after 2 weeks).
#### Debugging Data
``` (gdb) bt full #0 0x00007fcc6a36b4df in print_lists (dlg=0x7fcc2f0e6da8) at dlg_var.c:277 varlist = 0xc0c0c0c0 __func__ = "print_lists" #1 0x00007fcc6a36f9fa in pv_set_dlg_variable (msg=0x7fcc6af81de0, param=0x7fcc6adb4d00, op=254, val=0x7ffc545af190) at dlg_var.c:587 dlg = 0x7fcc2f0e6da8 ret = 0 __func__ = "pv_set_dlg_variable" #2 0x000055b8d971a148 in lval_pvar_assign (h=0x7ffc545b0530, msg=0x7fcc6af81de0, lv=0x7fcc6af54f38, rv=0x7fcc6af55be8) at core/lvalue.c:352 pvar = 0x7fcc6adb4ce8 pval = {rs = { s = 0x55b8d9c0c5a8 <buf+8> "100 Trying\r\nTo: sip:0311112222@192.168.2.103;tag=3208514SIPpTag01501\r\nFrom: 0300000080 sip:0300000080@the.domain.0.com;tag=3207307SIPpTag001780\r\nCall-ID: 1780-3207307@10.255.255.104\r\nCSeq: 802 INV"..., len = 3}, ri = 100, flags = 28} r_avp = 0x7fcc6af81de0 avp_val = {n = 1415246576, s = {s = 0x7ffc545af2f0 "\260\371ZT\374\177", len = -645912953}, re = 0x7ffc545af2f0} ret = 1 v = 32764 destroy_pval = 1 __func__ = "lval_pvar_assign" #3 0x000055b8d971ad64 in lval_assign (h=0x7ffc545b0530, msg=0x7fcc6af81de0, lv=0x7fcc6af54f38, rve=0x7fcc6af55be0) at core/lvalue.c:400 rv = 0x7fcc6af55be8 ret = -1 __func__ = "lval_assign" #4 0x000055b8d964f0d6 in do_action (h=0x7ffc545b0530, a=0x7fcc6af55228, msg=0x7fcc6af81de0) at core/action.c:1458 ret = -5 v = 21944 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = "\000\000\000\000\000\000\000\000\000\000\001\000\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\001\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "\000\000\000\000\001\000\000\000\000\000\000\000\000\000\000", __u6_addr16 = {0, 0, 1, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 1, 0, 0}}}, sin6_scope_id = 0}, sas = { ss_family = 0, __ss_padding = "\000\000\000\000\000\000\000\000\000\000\001", '\000' <repeats 23 times>, "\001\000\000\000\060\254aٸU", '\000' <repeats 26 times>, "\240\370ZT\374\177\000\000\255\030eٸU\000\000\000\000\000\000\000\000\000\000\340\035\370j\314\177\000\000\030I\365j\314\177\000\000\060\005[T\374\177\000", __ss_align = 0}}, id = 1, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} tmp = 0x7ffc545af9f0 "(R\365j\314\177" new_uri = 0x7ffc545b0900 "" end = 0x7fcc6ad91368 "\004\002" crt = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D" cmd = 0x7fcc6af81de0 len = 1 user = 0 uri = {user = {s = 0x7fcc6ad5f5b0 "\004", len = 1415249188}, passwd = {s = 0x7fcc6af81de0 "\270\243[", len = 0}, host = { s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x7ffc545af790 "\240\370ZT\374\177", len = -647687745}, sip_params = { s = 0x55b8d9c0c6e7 <buf+327> "785de0.0, SIP/2.0/UDP 10.255.255.104:5015;received=192.168.2.253;rport=5015;branch=z9hG4bK-3207307-1780-18\r\nUser-Agent: sipp\r\nContent-Length: 0\r\n\r\n", len = 28}, headers = {s = 0x0, len = 0}, port_no = 1, proto = 0, type = TELS_URI_T, flags = (unknown: 0x6af81de0), transport = {s = 0x7fcc6af54918 "\225\006", len = 1415247808}, ttl = { s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x7ffc545af830 "\360\376ZT\374\177", len = -647687745}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x7ffc545af830 "\360\376ZT\374\177", len = -645911744}, ttl_val = {s = 0x0, len = 1794645472}, user_param_val = {s = 0x7fcc6aefa8c8 "\215\005", len = 1415252224}, maddr_val = {s = 0x7fcc6ad905e0 "\004", len = 1415249188}, method_val = {s = 0x7fcc6af81de0 "\270\243[", len = 1415252224}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} next_hop = {user = {s = 0x5 <error: Cannot access memory at address 0x5>, len = 0}, passwd = {s = 0x7fcc6ad8feb8 "\004", len = 1415247844}, host = {s = 0x7fcc6af81de0 "\270\243[", len = 1415252224}, port = {s = 0x0, len = 0}, params = {s = 0x7ffc545afb40 "", len = -647747129}, sip_params = {s = 0x3000000030 <error: Cannot access memory at address 0x3000000030>, len = 52}, headers = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 1794645472}, port_no = 3440, proto = 27352, type = 32716, flags = (unknown: 0x545afd00), transport = {s = 0x7ffc545af570 "9", len = -647477599}, ttl = {s = 0x1d9b674c0 <error: Cannot access memory at address 0x1d9b674c0>, len = 1}, user_param = { s = 0x7fcc6af5d818 "", len = 1794496376}, maddr = {s = 0xe7f6aa2975bf95f7 <error: Cannot access memory at address 0xe7f6aa2975bf95f7>, len = -1989306889}, method = {s = 0x0, len = 0}, lr = { s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}, transport_val = {s = 0x0, len = 0}, --Type <RET> for more, q to quit, c to continue without paging-- ttl_val = {s = 0x7ffc545af6c0 "0\370ZT\374\177", len = 1792177856}, user_param_val = {s = 0x7ffc545af570 "9", len = 1024}, maddr_val = {s = 0x7ffc545a0039 "", len = 1782196981}, method_val = { s = 0x7fcc6ad8feb0 "\001", len = 1415247844}, lr_val = {s = 0x7fcc6af81de0 "\270\243[", len = 1794099024}, r2_val = {s = 0x1545af7a0 <error: Cannot access memory at address 0x1545af7a0>, len = 1792400528}, gr_val = {s = 0x7ffc545af7e4 "\270U", len = 0}} u = 0x7ffc545afc90 port = 0 dst_host = 0x0 i = 1791557648 flags = 0 avp = 0x7fcc6aa5b1b0 <sql_str> st = {flags = 56, id = 0, name = {n = 1, s = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 1415247184}, re = 0x1}, avp = 0x55b8d97fd496 <rval_get_int+314>} sct = 0x7ffc545af990 sjt = 0x5d9ad4174 rve = 0x55b8d97fd496 <rval_get_int+314> mct = 0x1 rv = 0x0 rv1 = 0x7fcc6afc68a8 c1 = {cache_type = 1801659538, val_type = 32716, c = {avp_val = {n = 1415247217, s = {s = 0x7ffc545af571 "", len = 1801978096}, re = 0x7ffc545af571}, pval = {rs = {s = 0x7ffc545af571 "", len = 1801978096}, ri = 1415248368, flags = 32764}}, i2s = "\000\000\000\000\314\177\000\000\000\000\000\000\000\000\000\000\320^\247ٸU"} s = {s = 0x300000006f <error: Cannot access memory at address 0x300000006f>, len = 1415248832} srevp = {0xffffffff00000000, 0x5d9ad4174} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = { s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #5 0x000055b8d96511bf in run_actions (h=0x7ffc545b0530, a=0x7fcc6af55228, msg=0x7fcc6af81de0) at core/action.c:1584 t = 0x7fcc6af55228 ret = -1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = 24, tz_dsttime = 0} tdiff = 3650377775 __func__ = "run_actions" #6 0x000055b8d96428a7 in do_action (h=0x7ffc545b0530, a=0x7fcc6af5ed40, msg=0x7fcc6af81de0) at core/action.c:1070 ret = 1 v = 1 dst = {send_sock = 0x55b8d961ac30 <_start>, to = {s = {sa_family = 20696, sa_data = "\367j\314\177\000\000\325\305\300ٸU\000"}, sin = {sin_family = 20696, sin_port = 27383, sin_addr = { s_addr = 32716}, sin_zero = "\325\305\300ٸU\000"}, sin6 = {sin6_family = 20696, sin6_port = 27383, sin6_flowinfo = 32716, sin6_addr = {__in6_u = { __u6_addr8 = "\325\305\300ٸU\000\000\000\000\000\000\036\000\000", __u6_addr16 = {50645, 55744, 21944, 0, 0, 0, 30, 0}, __u6_addr32 = {3653289429, 21944, 0, 30}}}, sin6_scope_id = 1415250048}, sas = {ss_family = 20696, __ss_padding = "\367j\314\177\000\000\325\305\300ٸU\000\000\000\000\000\000\036\000\000\000\200\000[T\374\177\000\000\000\000\000\000\001\000\000\000\060\254aٸU", '\000' <repeats 26 times>, "\350\305\301ٸU\000\000(\307\301ٸU\000\000\000\000\000\000\000\000\000\000ذ\204\061\314\177\000\000\260\000[T\374\177\000\000S\277\263j\314\177\000", __ss_align = 0}}, id = 0, send_flags = { f = 1, blst_imask = 0}, proto = 32 ' ', proto_pad0 = 1 '\001', proto_pad1 = 21595} tmp = 0x7fcc2f4e2758 "`\327[" new_uri = 0x7fcc6ab9056a <run_trans_callbacks_internal+1193> "\220H\215e\330[A\A]A^A_]\303UH\211\345H\203\354`\211}\274H\211u\260H\211U\250H\211M\240D\211E\270H\213E\260H\213@xH\205\300tYH\213E\260\213\200\200" end = 0x7ffc545b01b0 " \002[T\374\177" crt = 0x7fcc6af81de0 "\270\243[" cmd = 0x55b8d9c1c720 <_xavu_list_head> len = 32716 user = 1790510585 --Type <RET> for more, q to quit, c to continue without paging-- uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x7ffc545afed0 "\340\377ZT\374\177", len = -647477599}, host = {s = 0x7ffc545aff20 "\340\377ZT\374\177", len = 0}, port = { s = 0x7fcc6aed5330 "sip:0311112222@192.168.2.10391\300\300\300\300", len = 1794496376}, params = {s = 0x55b8d9ae22e2 "core", len = 0}, sip_params = { s = 0x55b8d9ae22d0 <__func__.1> "parse_from_header", len = -642899008}, headers = {s = 0x1d000001d8 <error: Cannot access memory at address 0x1d000001d8>, len = 1794496376}, port_no = 0, proto = 0, type = 6, flags = (unknown: 0x6af823f0), transport = {s = 0x7fcc6af21028 "", len = 1794496536}, ttl = {s = 0x100000455 <error: Cannot access memory at address 0x100000455>, len = 0}, user_param = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}, maddr = {s = 0x0, len = 0}, method = { s = 0x7ffc545affe0 "\350\305\301ٸU", len = 1771925255}, lr = {s = 0x0, len = 0}, r2 = {s = 0x7ffc545aff20 "\340\377ZT\374\177", len = -644195000}, gr = { s = 0x7ffc545b0050 "\340\214z2\314\177", len = 793655224}, transport_val = {s = 0x7fcc6af21470 "\003", len = 1794645472}, ttl_val = {s = 0x7ffc545affe0 "\350\305\301ٸU", len = -644312878}, user_param_val = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}, maddr_val = {s = 0x0, len = 1794645472}, method_val = { s = 0x7ffc545aff70 "\260", len = 1781754133}, lr_val = {s = 0x7fcc6aed5330 "sip:0311112222@192.168.2.10391\300\300\300\300", len = 30}, r2_val = {s = 0x7ffc545b00b0 "\260\001[T\374\177", len = 1781619319}, gr_val = {s = 0x7ffc545affa0 "\325\305\300ٸU", len = 1794496376}} next_hop = {user = {s = 0x7fcc6af5d7e0 "h", len = 1791557648}, passwd = {s = 0x7ffc545afd00 "0\375ZT\374\177", len = 1769020622}, host = { s = 0x33a510 <error: Cannot access memory at address 0x33a510>, len = 0}, port = {s = 0x800000 <error: Cannot access memory at address 0x800000>, len = 5023560}, params = { s = 0x250098 <error: Cannot access memory at address 0x250098>, len = 3365048}, sip_params = {s = 0x33a510 <error: Cannot access memory at address 0x33a510>, len = 8}, headers = { s = 0x3 <error: Cannot access memory at address 0x3>, len = 1791557648}, port_no = 64816, proto = 21594, type = 32764, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 0xd96c6a74), transport = {s = 0x0, len = -642956940}, ttl = {s = 0x7ffc545afe20 "\320\376ZT\374\177", len = -644630586}, user_param = {s = 0x7ffc545afe20 "\320\376ZT\374\177", len = -644629391}, maddr = {s = 0x38 <error: Cannot access memory at address 0x38>, len = 1791557648}, method = { s = 0x55b8d9a75d50 "core", len = 1769020622}, lr = {s = 0x55b8d9a75ed0 <__func__.16> "insert_new_lump_after", len = -643343019}, r2 = { s = 0x38 <error: Cannot access memory at address 0x38>, len = 1791557648}, gr = {s = 0x24fff8 <error: Cannot access memory at address 0x24fff8>, len = 3364784}, transport_val = { s = 0x33a510 <error: Cannot access memory at address 0x33a510>, len = 1}, ttl_val = {s = 0x4 <error: Cannot access memory at address 0x4>, len = 1794496320}, user_param_val = { s = 0x7ffc545afde0 "", len = 1791557648}, maddr_val = {s = 0x0, len = 1}, method_val = {s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 1794496480}, lr_val = {s = 0x0, len = 1791557648}, r2_val = {s = 0x7ffc545afed0 "\340\377ZT\374\177", len = 0}, gr_val = { s = 0x55b8d961ac30 <_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\212\222D", len = 0}} u = 0x7ffc545b0220 port = 0 dst_host = 0x7fcc6ad91368 i = 0 flags = 32716 avp = 0x0 st = {flags = 1794496480, id = 32716, name = {n = 1791557648, s = {s = 0x7fcc6ac90010 "\001", len = 1415249104}, re = 0x7fcc6ac90010}, avp = 0x55b8d994582f <qm_info+46>} sct = 0x7ffc545b01e0 sjt = 0x7fcc3184af10 rve = 0x7fcc6af554b0 mct = 0x55b8d993b7c6 <qm_malloc+1394> rv = 0x7fcc6ac90010 rv1 = 0x0 c1 = {cache_type = 1782103832, val_type = 32716, c = {avp_val = {n = 1782103844, s = {s = 0x7fcc6a38bf24 "dialog: dlg_var.c", len = 20}, re = 0x7fcc6a38bf24}, pval = {rs = { s = 0x7fcc6a38bf24 "dialog: dlg_var.c", len = 20}, ri = -642897498, flags = 21944}}, i2s = "\001\000\000\000\000\000\000\000/X\224ٸU\000\000\220\374ZT\374\177"} s = {s = 0x7ffc545afcf0 "\003", len = -644306074} srevp = {0x7ffc545afc50, 0x7ffc545afff8} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = { s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #7 0x000055b8d96511bf in run_actions (h=0x7ffc545b0530, a=0x7fcc6af5ed40, msg=0x7fcc6af81de0) at core/action.c:1584 t = 0x7fcc6af5ed40 ret = -1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} tz = {tz_minuteswest = -641677624, tz_dsttime = 21944} tdiff = 3653289677 --Type <RET> for more, q to quit, c to continue without paging-- __func__ = "run_actions" #8 0x000055b8d9651975 in run_top_route (a=0x7fcc6af5ed40, msg=0x7fcc6af81de0, c=0x7ffc545b0530) at core/action.c:1669 ctx = {rec_lev = 1415251740, run_flags = 32764, last_retcode = 1794645472, jmp_env = {{__jmpbuf = {0, 25126846836, 140721723737312, 3650336710, 140515944555072, 140515944556560, 1, 140514980835552}, __mask_was_saved = -641677446, __saved_mask = {__val = {3653289779, 140721723737184, 140515940805547, 64, 140514980835552, 140721723737200, 1415251444, 3652047055, 4294967296, 140721723737216, 140515940810260, 0, 140514980835552, 1415251120, 140721723737264, 94252410043218}}}}} p = 0x7ffc545b0530 ret = 0 sfbk = 0 #9 0x00007fcc6abec88d in reply_received (p_msg=0x7fcc6af81de0) at t_reply.c:2546 msg_status = 100 last_uac_status = 0 ack = 0x1200000001 <error: Cannot access memory at address 0x1200000001> ack_len = 3652036939 branch = 0 reply_status = -644489806 onreply_route = 1 cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = -644629391}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = -644629391}}}} uac = 0x7fcc3184b128 t = 0x7fcc3184ae98 lack_dst = {send_sock = 0x7fcc6af5e240, to = {s = {sa_family = 29208, sa_data = "\365j\000\000\000\000\063\307\300ٸU\000"}, sin = {sin_family = 29208, sin_port = 27381, sin_addr = { s_addr = 0}, sin_zero = "3\307\300ٸU\000"}, sin6 = {sin6_family = 29208, sin6_port = 27381, sin6_flowinfo = 0, sin6_addr = {__in6_u = { __u6_addr8 = "3\307\300ٸU\000\000S\307\300ٸU\000", __u6_addr16 = {50995, 55744, 21944, 0, 51027, 55744, 21944, 0}, __u6_addr32 = {3653289779, 21944, 3653289811, 21944}}}, sin6_scope_id = 1415251616}, sas = {ss_family = 29208, __ss_padding = "\365j\000\000\000\000\063\307\300ٸU\000\000S\307\300ٸU\000\000\240\006[T\374\177\000\000\000\000\000\000\001\000\000\000\060\254aٸU", '\000' <repeats 26 times>, "\360\a[T\374\177\000\000\067p\225ٸU\000\000\020\245\063\000\000\000\000\000\b\000\000\000\000\000\000\000\017\000\000\000\000\000\000\000\020\000\311j\314\177\000", __ss_align = 140721723737856}}, id = 1794470080, send_flags = {f = 32716, blst_imask = 0}, proto = 122 'z', proto_pad0 = -57 '\307', proto_pad1 = -9792} backup_user_from = 0x55b8d9c1c5d0 <def_list+16> backup_user_to = 0x55b8d9c1c5d8 <def_list+24> backup_domain_from = 0x55b8d9c1c5e0 <def_list+32> backup_domain_to = 0x55b8d9c1c5e8 <def_list+40> backup_uri_from = 0x55b8d9c1c5c0 <def_list> backup_uri_to = 0x55b8d9c1c5c8 <def_list+8> backup_xavps = 0x55b8d9c1c718 <_xavp_list_head> backup_xavus = 0x55b8d9c1c720 <_xavu_list_head> backup_xavis = 0x55b8d9c1c728 <_xavi_list_head> replies_locked = 1 branch_ret = 1415252208 prev_branch = 0 failover_continue = 0 blst_503_timeout = 0 hf = 0x0 onsend_params = {req = 0x7fcc6aebe230, rpl = 0x7fcc6ac90010, param = 0x7ffc545b0680, code = 0, flags = 0, branch = 0, t_rbuf = 0x46d9c0c5f0, dst = 0x55b8d9c0c69e <buf+254>, send_buf = { s = 0x7fcc6aebe290 "", len = 1}} ctx = {rec_lev = 2, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {4294967296, -1732009906818542089, 94252409400368, 0, 0, 0, -1732009906768210441, -5512493533158861321}, __mask_was_saved = 0, __saved_mask = {__val = {94252410104439, 140515944556560, 25126846836, 140721723737760, 3650336710, 140721723737760, 94252412681329, 1, 140515944526016, 94252414380363, 510459430555, 94252414380504, 94252414381080, 240, 140515941613584, 140721723737664}}}}} bctx = 0x40 keng = 0x0 ret = 328 evname = {s = 0x7fcc6ac613cf "on_sl_reply", len = 11} __func__ = "reply_received" #10 0x000055b8d96db775 in do_forward_reply (msg=0x7fcc6af81de0, mode=0) at core/forward.c:764 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, --Type <RET> for more, q to quit, c to continue without paging-- sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 0, __ss_padding = '\000' <repeats 117 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} new_len = 21944 r = 2 ip = {af = 3651762997, len = 21944, u = {addrl = {94248762343440, 0}, addr32 = {16, 21944, 0, 0}, addr16 = {16, 0, 21944, 0, 0, 0, 0, 0}, addr = "\020\000\000\000\270U\000\000\000\000\000\000\000\000\000"}} s = 0x0 len = 0 __func__ = "do_forward_reply" #11 0x000055b8d96dd7df in forward_reply (msg=0x7fcc6af81de0) at core/forward.c:865 No locals. #12 0x000055b8d97b2071 in receive_msg ( buf=0x55b8d9c0c5a0 <buf> "SIP/2.0 100 Trying\r\nTo: sip:0311112222@192.168.2.103;tag=3208514SIPpTag01501\r\nFrom: 0300000080 sip:0300000080@the.domain.0.com;tag=3207307SIPpTag001780\r\nCall-ID: 1780-3207307@10.255.255.104\r\nCSeq:"..., len=474, rcv_info=0x7ffc545b0f90) at core/receive.c:609 msg = 0x7fcc6af81de0 ctx = {rec_lev = 538976288, run_flags = 538976288, last_retcode = 538976288, jmp_env = {{__jmpbuf = {0, 0, -256, -1, 0, 0, -256, -1}, __mask_was_saved = 0, __saved_mask = {__val = {0, 0, 0, 0, 0, 0, 0, 0, 0, 2314885530818453536, 2314885530818453536, 0, 0, 18446744073709551360, 18446744073709551615, 0}}}}} bctx = 0x0 ret = 0 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = { s = 0x55b8d9c0c5a0 <buf> "SIP/2.0 100 Trying\r\nTo: sip:0311112222@192.168.2.103;tag=3208514SIPpTag01501\r\nFrom: 0300000080 sip:0300000080@the.domain.0.com;tag=3207307SIPpTag001780\r\nCall-ID: 1780-3207307@10.255.255.104\r\nCSeq:"..., len = 474} netinfo = {data = {s = 0x0, len = 0}, bufsize = 0, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0x7ffc545b0af0, obuf = {s = 0x0, len = 0}, rcv = 0x7ffc545b0f90, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 0 __func__ = "receive_msg" #13 0x000055b8d990da93 in udp_rcv_loop () at core/udp_server.c:587 len = 474 buf = "SIP/2.0 100 Trying\r\nTo: sip:0311112222@192.168.2.103;tag=3208514SIPpTag01501\r\nFrom: 0300000080 sip:0300000080@the.domain.0.com;tag=3207307SIPpTag001780\r\nCall-ID: 1780-3207307@10.255.255.104\r\nCSeq:"... tmp = 0x12c289290 <error: Cannot access memory at address 0x12c289290> fromaddr = 0x7fcc6af74ff0 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {1745004736, 0}, addr32 = {1745004736, 0, 0, 0}, addr16 = {43200, 26626, 0, 0, 0, 0, 0, 0}, addr = "\300\250\002h", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {1728227520, 0}, addr32 = {1728227520, 0, 0, 0}, addr16 = {43200, 26370, 0, 0, 0, 0, 0, 0}, addr = "\300\250\002g", '\000' <repeats 11 times>}}, src_port = 5030, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\246\300\250\002h\000\000\000\000\000\000\000"}, sin = { sin_family = 2, sin_port = 42515, sin_addr = {s_addr = 1745004736}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 42515, sin6_flowinfo = 1745004736, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\023\246\300\250\002h", '\000' <repeats 111 times>, __ss_align = 0}}, bind_address = 0x7fcc6ad202d0, rflags = 0, proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = '\000' <repeats 88 times>, "\300\265',\314\177\000\000\060\254aٸU", '\000' <repeats 26 times>, "\220\016[T\374\177\000\000\027V\206ٸU", '\000' <repeats 38 times>, "\n\000\000\000\230\t\324j\n", '\000' <repeats 91 times>... i = -1 j = 117760 l = 10 __func__ = "udp_rcv_loop" --Type <RET> for more, q to quit, c to continue without paging-- #14 0x000055b8d962bcd7 in main_loop () at main.c:1724 i = 9 pid = 0 si = 0x7fcc6ad202d0 si_desc = "udp receiver child=9 sock=192.168.2.103:5060\000U\000\000\260\021[T\374\177\000\000\221A`i\000\000\270\000\020\022[T\374\177\000\000\000\000\000\000\000\000\000\000 \022[T\374\177\000\000\061\354[i\314\177\000\000\260\027\322j\314\177\000\000e2\i\314\177", '\000' <repeats 14 times>, "\001\000\000" nrprocs = 12 woneinit = 1 __func__ = "main_loop" #15 0x000055b8d9638b77 in main (argc=8, argv=0x7ffc545b1808) at main.c:3085 cfg_stream = 0x55b8daf852a0 c = -1 r = 0 tmp = 0x7ffc545b1d03 "" tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x55b8d9a67278 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2856674964 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0xc2 <error: Cannot access memory at address 0xc2> st = {st_dev = 22, st_ino = 1269, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1653980021, tv_nsec = 397553084}, st_mtim = {tv_sec = 1666076443, tv_nsec = 909059560}, st_ctim = {tv_sec = 1666076448, tv_nsec = 69145711}, __glibc_reserved = {0, 0, 0}} tbuf = "\020/\177k\314\177\000\000\300:Lk\001\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "(\026\200k\314\177\000\000\350ɂk\314\177\000\000\377\377\377\377", '\000' <repeats 12 times>, "@\365Jk\314\177\000\000\020\064\177k\314\177\000\000\350Ԃk\314\177\000\000\204\031\200k\314\177\000\000\060\024\200k\314\177\000\000X\222hk\314\177\000\000hɂk\314\177\000\000`\300\202k\314\177\000\000P\030[T\374\177\000\000\200тk\314\177\000\000\000\000\000\000\000\000\000\000#\006\201k\314\177\000\000\001", '\000' <repeats 23 times>, "(\026\200k\314\177\000\000`\026[T\374\177\000\000\003u\201k"... option_index = 12 long_options = {{name = 0x55b8d9a69706 "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x55b8d9a64521 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x55b8d9a6970b "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x55b8d9a69711 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x55b8d9a69717 "substdef", has_arg = 1, flag = 0x0, val = 1026}, { name = 0x55b8d9a69720 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x55b8d9a6972a "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x55b8d9a69734 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x55b8d9a6973f "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x55b8d9a69748 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x55b8d9a69753 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x55b8d9a69759 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x55b8d9a69763 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x55b8d9a6976a "all-errors", has_arg = 0, flag = 0x0, val = 1035}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __func__ = "main" (gdb) (gdb) info locals varlist = 0xc0c0c0c0 __func__ = "print_lists" (gdb) list 272 while (varlist) { 273 LM_DBG("%.*s=%.*s (flags %i)\n", 274 varlist->key.len, varlist->key.s, 275 varlist->value.len, varlist->value.s, 276 varlist->vflags); 277 varlist = varlist->next; 278 } 279 } 280 } 281 (gdb)
```
#### Log Messages
There are no relevant log messages related to this issue.
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` [root@lab002103-flip-server kamailio-master]$ /usr/local/src/git/kamailio-master/src/kamailio -v version: kamailio 5.7.0-dev1 (x86_64/linux) 0d363c flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 0d363c compiled on 15:51:55 Oct 18 2022 with gcc 10.2.1 ```
* **Operating System**:
``` [root@lab002103-flip-server kamailio-master]$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 11 (bullseye) Release: 11 Codename: bullseye
[root@lab002103-flip-server kamailio-master]$ uname -a Linux lab002103-flip-server 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux
```
Hello, thanks for the report. We did a review of the dialog module last week, and noticed that the print_list function was missing protection against concurrent access. So I deactivated it by commit a6d9505b107204ffbe6 (and also the older releases). Please give it a try in your test suite, it should not execute this function now anymore.
OK. Latest commit 8531e4f50ec33f06562bd102e334532df2474102 passed all functional tests. I have started 4 load test environments with it. Thank you.
Closed #3276 as completed.
@MayamaTakeshi If you get another crash in another function from your long term tests, just open a new bug report.
@henningw Sure, I will do this from now on. Thanks.