[kamailio/kamailio] kamailio 5.4.5 crash after dialog [dlg_hash.c:1182]: next_state_dlg(): bogus event 2 in state 5 for dlg (#2757)
### Description kamailio 5.4.5 crashes with a segfault shortly after producing messages in the log like
Jun 1 10:12:14 thissystem /usr/sbin/kamailio[20001]: CRITICAL: {2 1 INVITE 6787142-3831531134-1330894187@some.telco.domain} dialog [dlg_hash.c:1182]: next_state_dlg(): bogus event 2 in state 5 for dlg 0x7fae153d3cf0 [3973:6059] with clid '82608924-3831089984-1833452161@some.telco.domain' and tags '3831089984-684203260' ''
Every time it crashes there is a corresponding "bogus event 2 in state 5" in the log and every time it issues that message it crashes.
### Troubleshooting Installed matching kamailio-debuginfo rpm and obtained a core dump.
#### Reproduction Cannot reproduce on demand but happens frequently enough so it's just a matter of time.
#### Debugging Data (gdb) bt full #0 0x00007f9c3a5da34b in update_dialog_dbinfo_unsafe (cell=0x7f9c30413998) at dlg_db_handler.c:879 i = 17 var = 0x0 jdoc = {root = 0x0, flags = 0, buf = {s = 0x0, len = 0}, malloc_fn = 0x41b550 malloc@plt, free_fn = 0x41aeb0 free@plt} values = {{type = DB1_INT, nul = 0, free = 0, val = {int_val = 579, ll_val = 579, double_val = 2.8606400894208175e-321, time_val = 579, string_val = 0x243 <Address 0x243 out of bounds>, str_val = {s = 0x243 <Address 0x243 out of bounds>, len = 7194536}, blob_val = { s = 0x243 <Address 0x243 out of bounds>, len = 7194536}, bitmap_val = 579, uint_val = 579, ull_val = 579}}, {type = DB1_INT, nul = 0, free = 1194072, val = {int_val = 8082, ll_val = 8082, double_val = 3.9930385496889546e-320, time_val = 8082, string_val = 0x1f92 <Address 0x1f92 out of bounds>, str_val = {s = 0x1f92 <Address 0x1f92 out of bounds>, len = 8}, blob_val = { s = 0x1f92 <Address 0x1f92 out of bounds>, len = 8}, bitmap_val = 8082, uint_val = 8082, ull_val = 8082}}, {type = DB1_STR, nul = 0, free = 8504131, val = {int_val = 809581320, ll_val = 140308801207048, double_val = 6.9321758485571243e-310, time_val = 140308801207048, string_val = 0x7f9c30413b08 "331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net", str_val = { s = 0x7f9c30413b08 "331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net", len = 60}, blob_val = { s = 0x7f9c30413b08 "331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net", len = 60}, bitmap_val = 809581320, uint_val = 809581320, ull_val = 140308801207048}}, {type = DB1_STR, nul = 0, free = 1085088656, val = {int_val = 809581381, ll_val = 140308801207109, double_val = 6.9321758485601381e-310, time_val = 140308801207109, string_val = 0x7f9c30413b45 "sip:+442890793919@uk.sdin.bt.net;user=phone", str_val = { s = 0x7f9c30413b45 "sip:+442890793919@uk.sdin.bt.net;user=phone", len = 43}, blob_val = { s = 0x7f9c30413b45 "sip:+442890793919@uk.sdin.bt.net;user=phone", len = 43}, bitmap_val = 809581381, uint_val = 809581381, ull_val = 140308801207109}}, {type = DB1_STR, nul = 0, free = 7116492, val = {int_val = 826702352, ll_val = 140308818328080, double_val = 6.9321766944484975e-310, time_val = 140308818328080, string_val = 0x7f9c31467a10 "3831608037-1237727804\177", str_val = { s = 0x7f9c31467a10 "3831608037-1237727804\177", len = 21}, blob_val = {s = 0x7f9c31467a10 "3831608037-1237727804\177", len = 21}, bitmap_val = 826702352, uint_val = 826702352, ull_val = 140308818328080}}, {type = DB1_STR, nul = 0, free = 8504131, val = { int_val = 809581425, ll_val = 140308801207153, double_val = 6.932175848562312e-310, time_val = 140308801207153, string_val = 0x7f9c30413b71 "sip:+442890792214@inh.voiceflex.com;user=phone", str_val = { s = 0x7f9c30413b71 "sip:+442890792214@inh.voiceflex.com;user=phone", len = 46}, blob_val = { s = 0x7f9c30413b71 "sip:+442890792214@inh.voiceflex.com;user=phone", len = 46}, bitmap_val = 809581425, uint_val = 809581425, ull_val = 140308801207153}}, {type = DB1_STR, nul = 0, free = 7477445, val = {int_val = 809768592, ll_val = 140308801394320, double_val = 6.9321758578095904e-310, time_val = 140308801394320, string_val = 0x7f9c30441690 "as7527953a@172.16.128.200:5090", str_val = {s = 0x7f9c30441690 "as7527953a@172.16.128.200:5090", len = 10}, blob_val = { s = 0x7f9c30441690 "as7527953a@172.16.128.200:5090", len = 10}, bitmap_val = 809768592, uint_val = 809768592, ull_val = 140308801394320}}, {type = DB1_STR, nul = 0, free = 803684352, val = {int_val = 1086075056, ll_val = 140309077700784, double_val = 6.9321895091627491e-310, time_val = 140309077700784, string_val = 0x7f9c40bc30b0 "@", str_val = {s = 0x7f9c40bc30b0 "@", len = 815282368}, blob_val = {s = 0x7f9c40bc30b0 "@", len = 815282368}, bitmap_val = 1086075056, uint_val = 1086075056, ull_val = 140309077700784}}, {type = DB1_STR, nul = 0, free = 979051415, val = {int_val = 979748736, ll_val = 140308971374464, double_val = 6.932184255944553e-310, time_val = 140308971374464, string_val = 0x7f9c3a65c780 <__FUNCTION__.14488> "populate_leg_info", str_val = {s = 0x7f9c3a65c780 <__FUNCTION__.14488> "populate_leg_info", len = 979733912}, blob_val = { s = 0x7f9c3a65c780 <__FUNCTION__.14488> "populate_leg_info", len = 979733912}, bitmap_val = 979748736, uint_val = 979748736, ull_val = 140308971374464}}, {type = DB1_INT, nul = 0, free = 1085054992, val = {int_val = 1622619239, ll_val = 5917586535, double_val = 2.9236762132362393e-314, time_val = 5917586535, string_val = 0x160b73467 <Address 0x160b73467 out of bounds>, str_val = { s = 0x160b73467 <Address 0x160b73467 out of bounds>, len = -1417213536}, blob_val = { s = 0x160b73467 <Address 0x160b73467 out of bounds>, len = -1417213536}, bitmap_val = 1622619239, uint_val = 1622619239, ull_val = 5917586535}}, {type = DB1_INT, nul = 0, free = -1417213568, val = {int_val = 3, ll_val = 140720308486147, double_val = 6.9525070095187664e-310, time_val = 140720308486147, string_val = 0x7ffc00000003 <Address 0x7ffc00000003 out of bounds>, str_val = {s = 0x7ffc00000003 <Address 0x7ffc00000003 out of bounds>, len = 809580952}, blob_val = { s = 0x7ffc00000003 <Address 0x7ffc00000003 out of bounds>, len = 809580952}, bitmap_val = 3, uint_val = 3, ull_val = 140720308486147}}, {type = DB1_INT, nul = 0, free = 1086231760, val = {int_val = 1538940139, ll_val = 27308743915, double_val = 1.3492312199477687e-313, time_val = 27308743915, string_val = 0x65bba5ceb <Address 0x65bba5ceb out of bounds>, str_val = { s = 0x65bba5ceb <Address 0x65bba5ceb out of bounds>, len = 979706976}, blob_val = { s = 0x65bba5ceb <Address 0x65bba5ceb out of bounds>, len = 979706976}, bitmap_val = 1538940139, uint_val = 1538940139, ull_val = 27308743915}}, {type = DB1_STR, nul = 0, free = 1086248480, val = {int_val = 1, ll_val = 1, double_val = 4.9406564584124654e-324, time_val = 1, string_val = 0x1 <Address 0x1 out of bounds>, str_val = { s = 0x1 <Address 0x1 out of bounds>, len = 807487484}, blob_val = {s = 0x1 <Address 0x1 out of bounds>, len = 807487484}, bitmap_val = 1, uint_val = 1, ull_val = 1}}, {type = DB1_STR, nul = 0, free = 1085054992, val = {int_val = 2400, ll_val = 2400, double_val = 1.1857575500189917e-320, time_val = 2400, string_val = 0x960 <Address 0x960 out of bounds>, str_val = { s = 0x960 <Address 0x960 out of bounds>, len = 1086248656}, blob_val = {s = 0x960 <Address 0x960 out of bounds>, len = 1086248656}, bitmap_val = 2400, uint_val = 2400, ull_val = 2400}}, {type = DB1_STR, nul = 0, free = 0, val = {int_val = 8207672, ll_val = 8207672, double_val = 4.0551287675331157e-317, time_val = 8207672, string_val = 0x7d3d38 "", str_val = {s = 0x7d3d38 "", len = 30}, blob_val = { s = 0x7d3d38 "", len = 30}, bitmap_val = 8207672, uint_val = 8207672, ull_val = 8207672}}, {type = DB1_STR, nul = 0, free = 8504131, val = {int_val = -1417213408, ll_val = 140723186240032, double_val = 6.9526491894519429e-310, time_val = 140723186240032, string_val = 0x7ffcab870a20 "\360\v\207\253\374\177", str_val = {s = 0x7ffcab870a20 "\360\v\207\253\374\177", len = 979234888}, blob_val = {s = 0x7ffcab870a20 "\360\v\207\253\374\177", len = 979234888}, bitmap_val = 2877753888, uint_val = 2877753888, ull_val = 140723186240032}}, {type = DB1_STR, nul = 0, free = 0, val = {int_val = -1417213424, ll_val = 140723186240016, double_val = 6.9526491894511524e-310, time_val = 140723186240016, string_val = 0x7ffcab870a10 " :<A\234\177", str_val = { s = 0x7ffcab870a10 " :<A\234\177", len = 979084403}, blob_val = {s = 0x7ffcab870a10 " :<A\234\177", len = 979084403}, bitmap_val = 2877753872, uint_val = 2877753872, ull_val = 140723186240016}}, {type = DB1_STR, nul = 32668, free = 813201384, val = { int_val = 1, ll_val = 1, double_val = 4.9406564584124654e-324, time_val = 1, string_val = 0x1 <Address 0x1 out of bounds>, str_val = { s = 0x1 <Address 0x1 out of bounds>, len = 807487484}, blob_val = {s = 0x1 <Address 0x1 out of bounds>, len = 807487484}, bitmap_val = 1, uint_val = 1, ull_val = 1}}, {type = DB1_INT, nul = 0, free = 807487484, val = {int_val = -1417213680, ll_val = 7172720912, double_val = 3.5437949898262949e-314, time_val = 7172720912, string_val = 0x1ab870910 <Address 0x1ab870910 out of bounds>, str_val = {s = 0x1ab870910 <Address 0x1ab870910 out of bounds>, len = 807487484}, blob_val = {s = 0x1ab870910 <Address 0x1ab870910 out of bounds>, len = 807487484}, bitmap_val = 2877753616, uint_val = 2877753616, ull_val = 7172720912}}, {type = DB1_STR, nul = 32764, free = 979026405, val = {int_val = -1417213664, ll_val = 28647557408, double_val = 1.4153773952557707e-313, time_val = 28647557408, string_val = 0x6ab870920 <Address 0x6ab870920 out of bounds>, str_val = {s = 0x6ab870920 <Address 0x6ab870920 out of bounds>, len = 979706976}, blob_val = {s = 0x6ab870920 <Address 0x6ab870920 out of bounds>, len = 979706976}, bitmap_val = 2877753632, uint_val = 2877753632, ull_val = 28647557408}}, {type = DB1_STR, nul = 6, free = 0, val = {int_val = -1417213440, ll_val = 140723186240000, double_val = 6.9526491894503619e-310, time_val = 140723186240000, string_val = 0x7ffcab870a00 "8=}", str_val = {s = 0x7ffcab870a00 "8=}", len = -1417213104}, blob_val = {s = 0x7ffcab870a00 "8=}", len = -1417213104}, bitmap_val = 2877753856, uint_val = 2877753856, ull_val = 140723186240000}}, {type = DB1_STR, nul = 1, free = 829901784, val = { int_val = 1086063344, ll_val = 140309077689072, double_val = 6.9321895085840994e-310, time_val = 140309077689072, string_val = 0x7f9c40bc02f0 "m\002\005", str_val = {s = 0x7f9c40bc02f0 "m\002\005", len = 809580952}, blob_val = { s = 0x7f9c40bc02f0 "m\002\005", len = 809580952}, bitmap_val = 1086063344, uint_val = 1086063344, ull_val = 140309077689072}}, { type = DB1_INT, nul = 3, free = 3, val = {int_val = 1086063344, ll_val = 140309077689072, double_val = 6.9321895085840994e-310, time_val = 140309077689072, string_val = 0x7f9c40bc02f0 "m\002\005", str_val = {s = 0x7f9c40bc02f0 "m\002\005", len = 809580952}, blob_val = {s = 0x7f9c40bc02f0 "m\002\005", len = 809580952}, bitmap_val = 1086063344, uint_val = 1086063344, ull_val = 140309077689072}}} insert_keys = {0x7f9c3a86f8e0 <h_entry_column>, 0x7f9c3a86f8d0 <h_id_column>, 0x7f9c3a86f880 <call_id_column>, 0x7f9c3a86f890 <from_uri_column>, 0x7f9c3a86f8a0 <from_tag_column>, 0x7f9c3a86f8b0 <to_uri_column>, 0x7f9c3a86f8c0 <to_tag_column>, 0x7f9c3a86f990 <from_sock_column>, 0x7f9c3a86f980 <to_sock_column>, 0x7f9c3a86f900 <start_time_column>, 0x7f9c3a86f8f0 <state_column>, 0x7f9c3a86f910 <timeout_column>, 0x7f9c3a86f930 <from_cseq_column>, 0x7f9c3a86f920 <to_cseq_column>, 0x7f9c3a86f970 <from_contact_column>, 0x7f9c3a86f960 <to_contact_column>, 0x7f9c3a86f950 <from_route_column>, 0x7f9c3a86f940 <to_route_column>, 0x7f9c3a86f9a0 <sflags_column>, 0x7f9c3a86f9c0 <toroute_name_column>, 0x7f9c3a86f9d0 <req_uri_column>, 0x7f9c3a86f9e0 <xdata_column>, 0x7f9c3a86f9b0 <iflags_column>} __FUNCTION__ = "update_dialog_dbinfo_unsafe" #1 0x00007f9c3a5db0bf in update_dialog_dbinfo (cell=0x7f9c30413998) at dlg_db_handler.c:977 No locals. #2 0x00007f9c3a5e2d4d in dlg_onreply (t=0x7f9c31774bd8, type=1048576, param=0x7ffcab870d60) at dlg_handlers.c:546 dlg = 0x7f9c30413998 iuid = 0x7f9c3025faf0 new_state = 3 old_state = 5 unref = 0 event = 3 tag = { s = 0xad0d27 <buf.7141+391> "as7527953a\r\nCall-ID: 11481241-3831608038-1141932217@sbc-uk-l-far07b.uk.sdin.bt.net\r\nCSeq: 1 INVITE\r\nServer: Voiceflex\r\nAllow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY, INFO, PUBLISH"..., len = 10} req = 0x7f9c307877e8 rpl = 0x7f9c40bc02f0 __FUNCTION__ = "dlg_onreply" #3 0x00007f9c3d4408a4 in run_trans_callbacks_internal (cb_lst=0x7f9c31774c50, type=1048576, trans=0x7f9c31774bd8, params=0x7ffcab870d60) at t_hooks.c:258 cbp = 0x7f9c30fce308 backup_from = 0xb879b0 <def_list+16> backup_to = 0xb879b8 <def_list+24> backup_dom_from = 0xb879c0 <def_list+32> backup_dom_to = 0xb879c8 <def_list+40> backup_uri_from = 0xb879a0 <def_list> backup_uri_to = 0xb879a8 <def_list+8> backup_xavps = 0xb2c988 <_xavp_list_head> backup_xavus = 0xb2c990 <_xavu_list_head> backup_xavis = 0xb2c998 <_xavi_list_head> __FUNCTION__ = "run_trans_callbacks_internal" #4 0x00007f9c3d440abc in run_trans_callbacks_with_buf (type=1048576, rbuf=0x7f9c31774ca8, req=0x7f9c307877e8, repl=0x7f9c40bc02f0, flags=0) at t_hooks.c:303 params = {req = 0x7f9c307877e8, rpl = 0x7f9c40bc02f0, param = 0x7f9c30fce318, code = 200, flags = 0, branch = 0, t_rbuf = 0x7f9c31774ca8, dst = 0x7f9c31774cf8, send_buf = { s = 0x7f9c302db310 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 193.113.183.202:5060;branch=z9hG4bKdd6a6cbff3b1c256626bd914fd8db551\r\nRecord-Route: sip:93.95.124.34;lr;did=342.29f1\r\nFrom: sip:anonymous@uk.sdin.bt.net;user=phone;"..., len = 935}} trans = 0x7f9c31774bd8 #5 0x00007f9c3d3cc07d in relay_reply (t=0x7f9c31774bd8, p_msg=0x7f9c40bc02f0, branch=0, msg_status=200, cancel_data=0x7ffcab871250, do_put_on_wait=1) at t_reply.c:2096 relay = 0 save_clone = 0 buf = 0x7f9c40bed0f8 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 193.113.183.202:5060;branch=z9hG4bKdd6a6cbff3b1c256626bd914fd8db551\r\nRecord-Route: sip:93.95.124.34;lr;did=342.29f1\r\nFrom: sip:anonymous@uk.sdin.bt.net;user=phone;"... res_len = 935 relayed_code = 200 relayed_msg = 0x7f9c40bc02f0 reply_bak = 0x7f9c31774e20 bm = {to_tag_val = {s = 0x7ffcab870fc0 "\200\023\207\253\374\177", len = 5217506}} totag_retr = 0 reply_status = RPS_COMPLETED uas_rb = 0x7f9c31774ca8 to_tag = 0x0 reason = {s = 0xbba28 <Address 0xbba28 out of bounds>, len = 0} onsend_params = {req = 0x7f9c40becdf0, rpl = 0x7ffcab871090, param = 0x7f9c40bc02f0, code = 1086072408, flags = 32668, branch = 0, t_rbuf = 0x7ffcab870f00, dst = 0x4f4ca1 <futex_release+29>, send_buf = {s = 0x800000 "<core> [core/tcp_main.c:2999]: ", len = 803921760}} ip = {af = 1094466080, len = 32668, u = {addrl = {8504131, 140723186241472}, addr32 = {8504131, 0, 2877755328, 32764}, addr16 = {49987, 129, 0, 0, 4032, 43911, 32764, 0}, addr = "CÁ\000\000\000\000\000\300\017\207\253\374\177\000"}} __FUNCTION__ = "relay_reply" #6 0x00007f9c3d3d1751 in reply_received (p_msg=0x7f9c40bc02f0) at t_reply.c:2682 msg_status = 200 last_uac_status = 180 ack = 0x81c343 "INFO" ack_len = 2877756064 branch = 0 reply_status = 1086230768 onreply_route = 1 cancel_data = {cancel_bitmap = 0, reason = {cause = 200, u = {text = {s = 0x0, len = 30}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 30}}}} uac = 0x7f9c31774e68 t = 0x7f9c31774bd8 lack_dst = {send_sock = 0xad0fb2 <buf.7141+1042>, to = {s = {sa_family = 0, sa_data = "\000\000\000\000\000\000\070=}\000\000\000\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "8=}\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "8=}\000\000\000\000\000\036\000\000\000\000\000\000", __u6_addr16 = {15672, 125, 0, 0, 30, 0, 0, 0}, __u6_addr32 = {8207672, 0, 30, 0}}}, sin6_scope_id = 1094466080}, sas = {ss_family = 0, __ss_padding = "\000\000\000\000\000\000\070=}\000\000\000\000\000\036\000\000\000\000\000\000\000 :<A\234\177\000\000CÁ\000\000\000\000\000 \023\207\253\374\177\000\000]\261\247<\234\177\000\000`\022\271", '\000' <repeats 13 times>, "\262\017\255\000\000\000\000\000\201\r\255", '\000' <repeats 13 times>, "\360\220\276@\234\177\000\000\262\017\255\000\000\000\000\000\260\024\207\253\374\177\000", __ss_align = 140309077856352}}, id = -1417210704, send_flags = {f = 32764, blst_imask = 0}, proto = 80 'P', proto_pad0 = 43 '+', proto_pad1 = 16572} backup_user_from = 0xb879b0 <def_list+16> backup_user_to = 0xb879b8 <def_list+24> backup_domain_from = 0xb879c0 <def_list+32> backup_domain_to = 0xb879c8 <def_list+40> backup_uri_from = 0xb879a0 <def_list> backup_uri_to = 0xb879a8 <def_list+8> backup_xavps = 0xb2c988 <_xavp_list_head> backup_xavus = 0xb2c990 <_xavu_list_head> backup_xavis = 0xb2c998 <_xavi_list_head> replies_locked = 1 branch_ret = 0 prev_branch = 0 blst_503_timeout = 7583887 hf = 0x859150 onsend_params = {req = 0x7ffcab8712b0, rpl = 0x6c72b3 <qm_malloc+2479>, param = 0xad0eb0 <buf.7141+784>, code = 7564075, flags = 0, branch = 0, t_rbuf = 0x7f9c40becd80, dst = 0xad0fb2 <buf.7141+1042>, send_buf = {s = 0x0, len = 11341488}} ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {0, -4259732974664084315, 8207672, 30, 140309086091808, 8504131, -4259732974771039067, 4259056081323627685}, __mask_was_saved = 0, __saved_mask = {__val = {140309024319992, 140309076697584, 0, 8388608, 7653592, 18446744073709551615, 140309077689072, 1322664, 8, 140723186242212, 2877756072, 140307991625729, 140309077689072, 0, 21483478416, 1304}}}}} bctx = 0x7f9c40bc02f0 keng = 0x0 ret = 32764 evname = {s = 0x7f9c3d472dcc "on_sl_reply", len = 11} __FUNCTION__ = "reply_received" #7 0x000000000058c08b in do_forward_reply (msg=0x7f9c40bc02f0, mode=0) at core/forward.c:764 new_buf = 0x0 dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = { s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = { __in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 0, __ss_padding = '\000' <repeats 117 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0} new_len = 0 r = 1 ip = {af = 8192053, len = 0, u = {addrl = {0, 8207672}, addr32 = {0, 0, 8207672, 0}, addr16 = {0, 0, 0, 0, 15672, 125, 0, 0}, addr = "\000\000\000\000\000\000\000\000\070=}\000\000\000\000"}} s = 0x7d3d38 "" len = 0 __FUNCTION__ = "do_forward_reply" #8 0x000000000058dd71 in forward_reply (msg=0x7f9c40bc02f0) at core/forward.c:865 No locals. #9 0x00000000005ef38f in receive_msg ( buf=0xad0ba0 <buf.7141> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 93.95.124.34;branch=z9hG4bKa726.2c6f345ef9f158b5a47c15a263fb7d03.0;received=93.95.124.34\r\nVia: SIP/2.0/UDP 193.113.183.202:5060;branch=z9hG4bKdd6a6cbff3b1c256626bd914f"..., len=1042, rcv_info=0x7ffcab871b80) at core/receive.c:509 msg = 0x7f9c40bc02f0 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {0, -4259732975544888155, 8207672, 30, 140309086091808, 8504131, -4259732975383407451, 4259056081323627685}, __mask_was_saved = 0, __saved_mask = {__val = {0, 16797685758500035936, 2147483648, 0, 274877906944, 0, 0, 16789419411038535680, 2147483648, 219043332192, 0, 0, 0, 1065151889463, 459561500749, 8224}}}}} bctx = 0x0 ret = 1 tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = { s = 0xad0ba0 <buf.7141> "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 93.95.124.34;branch=z9hG4bKa726.2c6f345ef9f158b5a47c15a263fb7d03.0;received=93.95.124.34\r\nVia: SIP/2.0/UDP 193.113.183.202:5060;branch=z9hG4bKdd6a6cbff3b1c256626bd914f"..., len = 1042} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0x0 evp = {data = 0x7ffcab8716b0, obuf = {s = 0x0, len = 0}, rcv = 0x7ffcab871b80, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 0 __FUNCTION__ = "receive_msg" #10 0x00000000004d144c in udp_rcv_loop () at core/udp_server.c:543 len = 1042 buf = "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 93.95.124.34;branch=z9hG4bKa726.2c6f345ef9f158b5a47c15a263fb7d03.0;received=93.95.124.34\r\nVia: SIP/2.0/UDP 193.113.183.202:5060;branch=z9hG4bKdd6a6cbff3b1c256626bd914f"... tmp = 0x7f9c2feadc88 "" fromaddr = 0x7f9c40bc3000 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {1803312989, 0}, addr32 = {1803312989, 0, 0, 0}, addr16 = {24413, 27516, 0, 0, 0, 0, 0, 0}, addr = "]_|k", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {578576221, 0}, addr32 = {578576221, 0, 0, 0}, addr16 = {24413, 8828, 0, 0, 0, 0, 0, 0}, addr = "]_|"", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304]_|k\000\000\000\000\000\000\000"}, sin = { sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 1803312989}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 1803312989, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\023\304]_|k", '\000' <repeats 111 times>, __ss_align = 0}}, bind_address = 0x7f9c40b5aa70, proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = "\000\000\000\000\003\000\000\000\220\062\267@\234\177", '\000' <repeats 90 times>, "\234Hc0\234\177\000\000\070=}\000\004\000\000\00 0\220\366\200\000\003\000\000\000 :<A\234\177\000\000CÁ\000\000\000\000\000\320\032\207\253\374\177\000\000\203\204Y", '\000' <repeats 93 times>... i = -1 j = 72960 l = 1 __FUNCTION__ = "udp_rcv_loop" #11 0x0000000000429f5b in main_loop () at main.c:1711 i = 2 pid = 0 si = 0x7f9c40b5aa70 si_desc = "udp receiver child=2 sock=93.95.124.34:5060\000\234\177\000\000\233\205\177", '\000' <repeats 13 times>, "\060#\207\253\374\177\000\000\070=}\000\000\000\000\000\036\000\000\000\000\000\000\000 :<A\234\177\000\000CÁ\000\000\000\000\000\257:<A\234\177\000\000\340\271A\000\000\000\000\000\220C\274@\234\177\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #12 0x000000000043423e in main (argc=10, argv=0x7ffcab872418) at main.c:2942 cfg_stream = 0x21e8040 c = -1 r = 0 tmp = 0x7ffcab872f05 "" tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7d69b8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3932459496 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0xff00000000ff00 p = 0x0 st = {st_dev = 19, st_ino = 12063, st_nlink = 2, st_mode = 16832, st_uid = 996, st_gid = 993, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1621869721, tv_nsec = 871018353}, st_mtim = {tv_sec = 1622542335, tv_nsec = 169887496}, st_ctim = {tv_sec = 1622542335, tv_nsec = 169887496}, __unused = {0, 0, 0}} tbuf = ".N=\366\000\000\000\000\273-\334A\234\177\000\000\000\000\000\000\000\000\000\000\300\037\207\253\374\177\000\000\210\353,A\234\177\000\000\350\263-A\234\177\000\000\320 \207\253\374\177\000\000\300 \207\253\374\177\000\000.", '\000' <repeats 23 times>, "\320\n\375A\234\177\000\000\310\004\375A\234\177\000\000\274\016A\000\000\000\000\000п-A\234\177\000\000\bA@\000\000\000\000\000\000\000\000\000\001\000\000\000D\b\000\000\001", '\000' <repeats 11 times>, "\210!\207\253\374\177\000\000`!\207\253\374\177\000\000\001\000\000\000\000\000\000\000\320\n\375A\234\177\000\000\250\304\375A\234\177\000\000P\301\375A\234\177\000\000\237"... option_index = 0 long_options = {{name = 0x7d8cef "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d4194 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7d8cf4 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7d8cfa "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x7d8d00 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7d8d09 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7d8d13 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7d8d1d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, { name = 0x7d8d28 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7d8d31 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x7d8d3c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7d8d42 "atexit", has_arg = 1, flag = 0x0, val = 1034}, { name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) (gdb) list 874 cell->bind_addr[DLG_CALLER_LEG]->sock_str.len, 875 cell->bind_addr[DLG_CALLER_LEG]->sock_str.s, 876 cell->bind_addr[DLG_CALLEE_LEG]->sock_str.len, 877 cell->bind_addr[DLG_CALLEE_LEG]->sock_str.s); 878 879 SET_STR_VALUE(values+7, cell->bind_addr[DLG_CALLER_LEG]->sock_str); 880 SET_STR_VALUE(values+8, cell->bind_addr[DLG_CALLEE_LEG]->sock_str); 881 882 SET_STR_VALUE(values+12, cell->cseq[DLG_CALLER_LEG]); 883 SET_STR_VALUE(values+13, cell->cseq[DLG_CALLEE_LEG]); (gdb) info locals cfg_stream = 0x21e8040 c = -1 r = 0 tmp = 0x7ffcab872f05 "" tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7d69b8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 3932459496 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 2 n_lst = 0xff00000000ff00 p = 0x0 st = {st_dev = 19, st_ino = 12063, st_nlink = 2, st_mode = 16832, st_uid = 996, st_gid = 993, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1621869721, tv_nsec = 871018353}, st_mtim = {tv_sec = 1622542335, tv_nsec = 169887496}, st_ctim = {tv_sec = 1622542335, tv_nsec = 169887496}, __unused = {0, 0, 0}} tbuf = ".N=\366\000\000\000\000\273-\334A\234\177\000\000\000\000\000\000\000\000\000\000\300\037\207\253\374\177\000\000\210\353,A\234\177\000\000\350\263-A\234\177\000\000\320 \207\253\374\177\000\000\300 \207\253\374\177\000\000.", '\000' <repeats 23 times>, "\320\n\375A\234\177\000\000\310\004\375A\234\177\000\000\274\016A\000\000\000\000\000п-A\234\177\000\000\bA@\000\000\000\000\000\000\000\000\000\001\000\000\000D\b\000\000\001", '\000' <repeats 11 times>, "\210!\207\253\374\177\000\000`!\207\253\374\177\000\000\001\000\000\000\000\000\000\000\320\n\375A\234\177\000\000\250\304\375A\234\177\000\000P\301\375A\234\177\000\000\237"... option_index = 0 long_options = {{name = 0x7d8cef "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d4194 "version", has_arg = 0, flag = 0x0, val = 118}, { name = 0x7d8cf4 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7d8cfa "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x7d8d00 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7d8d09 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7d8d13 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7d8d1d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, { name = 0x7d8d28 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7d8d31 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, { name = 0x7d8d3c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7d8d42 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb)
(gdb) p cell->callid $1 = {s = 0x7f9c30413b08 "331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net", len = 60} (gdb) p cell->bind_addr[0] $2 = (struct socket_info *) 0x0 (gdb) p cell->bind_addr[1] $3 = (struct socket_info *) 0x7f9c40b5aa70 (gdb) p cell->cseq[0] $4 = {s = 0x7f9c30e50d78 "1%", len = 1} (gdb) p cell->contact[0] $5 = {s = 0x7f9c304b7368 "sip:+442890793919@62.239.9.26:5060F1\234\177", len = 34} (gdb) p cell->route_set[0] $6 = {s = 0x7f9c30c8c698 "", len = 0} (gdb) p cell->tag[0] $7 = {s = 0x7f9c31467a10 "3831608037-1237727804\177", len = 21} (gdb) p cell->bind_addr[1]->sock_str $8 = {s = 0x7f9c40bcd6a8 "udp:93.95.124.34:5060", len = 21}
#### Log Messages see above
#### SIP Traffic If you need this I will need to get one of our sip guys to provide it. It should all be logged in homer.
### Possible Solutions
### Additional Information version: kamailio 5.4.5 (x86_64/linux) c46342 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: c46342 compiled on 23:39:39 Apr 25 2021 with gcc 4.8.5
* **Operating System**: CentOS 7.9 Linux kamibt01.voiceflex.com 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Can you get also from gdb:
``` p *cell ```
I am interested on a few other fields to see if the cell was somehow in process of being destroyed.
Getting the pcap with all sip messages of this call would be very useful, the time difference between packets helps to figure out if some timer routines conflict somehow.
(gdb) p *cell $3 = {ref = 3, next = 0x7f9c302d0068, prev = 0x0, h_id = 8082, h_entry = 579, state = 3, lifetime = 0, init_ts = 1622619237, start_ts = 1622619239, end_ts = 0, dflags = 513, iflags = 64, sflags = 0, toroute = 0, toroute_name = {s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = {s = 0x7f9c30413b08 "331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net", len = 60}, from_uri = {s = 0x7f9c30413b45 "sip:+442890793919@uk.sdin.bt.net;user=phone", len = 43}, to_uri = {s = 0x7f9c30413b71 "sip:+442890792214@inh.voiceflex.com;user=phone", len = 46}, req_uri = { s = 0x7f9c30413ba0 "sip:+442890792214@inh.voiceflex.com;user=phone", len = 46}, tag = {{s = 0x7f9c31467a10 "3831608037-1237727804\177", len = 21}, {s = 0x7f9c30441690 "as7527953a@172.16.128.200:5090", len = 10}}, cseq = {{s = 0x7f9c30e50d78 "1%", len = 1}, { s = 0x7f9c304c3e60 "0ip:+447394671501@193.113.183.202:5060", len = 1}}, route_set = {{s = 0x7f9c30c8c698 "", len = 0}, {s = 0x7f9c309838c0 "\244\324`:\234\177", len = 0}}, contact = {{s = 0x7f9c304b7368 "sip:+442890793919@62.239.9.26:5060F1\234\177", len = 34}, { s = 0x7f9c30ad65c0 "sip:+441159790818@93.95.124.107:5060", len = 36}}, bind_addr = {0x0, 0x7f9c40b5aa70}, cbs = {first = 0x0, types = 0}, profile_links = 0x0, vars = 0x7f9c31451dc0, ka_src_counter = 0, ka_dst_counter = 0}
file:///home/trevor/Downloads/2757_scrubbed.zip
The attachment was not uploaded, is only the path to it in the comment.
And the log message `... bogus event 2 in state 5 for ...` is not directly related to the dialog causing the crash, the print of the `*cell` shows it is in state 3, not 5.
I don't seem to be able to add attachments here. I drag and drop and I either get nothing at all or it inserts a file:// link pointing to my local file. I can mail this direct to you if that would work?
The bogus state message may not be directly related but it's a one to one correspondence between that message and the crash and it occurs within < 1s of it being issued.
# zgrep -h "bogus event 2 in state 5" /var/log/kamailio.log* Jun 1 10:12:14 kamibt01 /usr/sbin/kamailio[20001]: CRITICAL: {2 1 INVITE 6787142-3831531134-1330894187@sbc-uk-l-far07b.uk.sdin.bt.net} dialog [dlg_hash.c:1182]: next_state_dlg(): bogus event 2 in state 5 for dlg 0x7fae153d3cf0 [3973:6059] with clid '82608924-3831089984-1833452161@sbc-uk-l-far07a.uk.sdin.bt.net' and tags '3831089984-684203260' '' Jun 2 07:33:59 kamibt01 /usr/sbin/kamailio[3789]: CRITICAL: {2 1 INVITE 11481241-3831608038-1141932217@sbc-uk-l-far07b.uk.sdin.bt.net} dialog [dlg_hash.c:1182]: next_state_dlg(): bogus event 2 in state 5 for dlg 0x7f9c30413998 [579:8082] with clid '331873271-3831608037-62771434@sbc-uk-l-nws01a.uk.sdin.bt.net' and tags '3831608037-1237727804' '' # grep segfault /var/log/messages Jun 1 10:12:14 kamibt01 kernel: [914047.335442] kamailio[20000]: segfault at f0 ip 00007fae1f07f34b sp 00007ffd33f4c0c0 error 4 in dialog.so[7fae1f047000+cd000] Jun 2 07:33:59 kamibt01 kernel: [990952.027386] kamailio[3788]: segfault at f0 ip 00007f9c3a5da34b sp 00007ffcab870510 error 4 in dialog.so[7f9c3a5a2000+cd000]
[root@kamibh01 ~]# zgrep -h "bogus event 2 in state 5" /var/log/kamailio.log* May 27 07:39:47 kamibh01 /usr/sbin/kamailio[11320]: CRITICAL: {2 1 INVITE 240487810-3831089987-474199942@sbc-uk-gw-th01b.uk.sdin.bt.net} dialog [dlg_hash.c:1182]: next_state_dlg(): bogus event 2 in state 5 for dlg 0x7f4887eb8420 [2002:9497] with clid '228609061-3830856817-93082544@sbc-uk-gw-th01b.uk.sdin.bt.net' and tags '3830856817-660210341' '' [root@kamibh01 ~]# zgrep -h segfault /var/log/messages* May 27 07:39:47 kamibh01 kernel: [2300614.036738] kamailio[11324]: segfault at f0 ip 00007f4891cbb34b sp 00007ffc1a7096b0 error 4 in dialog.so[7f4891c83000+cd000] [root@kamibh01 ~]#
[2757_scrubbed.zip](https://github.com/kamailio/kamailio/files/6587206/2757_scrubbed.zip)
I got the pcap via email, the call-id matches the `bt full` output, but does not match the output of `p *cell`. Maybe you used a different corefile, can you get the output from the same corefile as for `bt full`? Otherwise, I cannot correlate the data between them.
Also, the `(gdb) p cell->callid` in the first comment, after `bt full` show a different call-id.
What I noticed in the call pcap is that the processing of 200ok (INVITE) takes over 6 seconds, do you have any blocking operations done during 200ok? For example, accounting can happen at that time, you should check if database server is slow for some reason when inserting records in acc table (you can also enable latency core parameters to get logs on slow actions/database statements).
Btw, do you do dialog replication via dmq?
That's a yes to the dialog replication question. Am checking the p *cell output now. I have two core dumps so it's possible I mixed them up.
< (gdb) p *cell $1 = {ref = 3, next = 0x7fae15a524a8, prev = 0x0, h_id = 6059, h_entry = 3973, state = 3, lifetime = 0, init_ts = 1622101184, start_ts = 1622542334, end_ts = 0, dflags = 513, iflags = 64, sflags = 0, toroute = 0, toroute_name = {s = 0x0, len = 0}, from_rr_nb = 0, tl = {next = 0x0, prev = 0x0, timeout = 0}, callid = {s = 0x7fae153d3e60 "82608924-3831089984-1833452161@sbc-uk-l-far07a.uk.sdin.bt.net", len = 61}, from_uri = {s = 0x7fae153d3e9e "sip:+447849109057@uk.sdin.bt.net;user=phone", len = 43}, to_uri = {s = 0x7fae153d3eca "sip:+441274770772@inh.voiceflex.com;user=phone", len = 46}, req_uri = { s = 0x7fae153d3ef9 "sip:+441274770772@inh.voiceflex.com;user=phone", len = 46}, tag = {{s = 0x7fae15034620 "3831089984-6842032602\177", len = 20}, {s = 0x7fae14f26750 "as716a6806@172.16.128.199:5090", len = 10}}, cseq = {{s = 0x7fae1528b448 "1\036", len = 1}, { s = 0x7fae1612c480 "0ip:dialog@172.16.128.200:5090", len = 1}}, route_set = {{s = 0x7fae15126010 "\313\017", len = 0}, {s = 0x7fae14e69f50 "sip:172.16.128.199:5090", len = 0}}, contact = {{s = 0x7fae1583ea98 "sip:+447849109057@193.113.183.202:5060", len = 38}, { s = 0x7fae154471e8 "sip:+441634401166@93.95.124.77:5060\026\256\177", len = 35}}, bind_addr = {0x0, 0x7fae255ffa70}, cbs = {first = 0x0, types = 0}, profile_links = 0x0, vars = 0x7fae15f66a18, ka_src_counter = 0, ka_dst_counter = 0}
this is the p *cell from the other core dump but I do not think this is the correct one, all the other info came from the other core dump.
If there is dialog replication via dmq, can you paste here all the modparams you set for dialog and dmq modules (hide/replace sensitive data, such as passwords, ips, ... if it is the case). It can be a conflict on dialog matching due to replication.
[root@kamibt01 ~]# grep -iE "dialog|dmq" /etc/kamailio/kamailio.cfg loadmodule "dmq.so" loadmodule "dialog.so" # -- dialog params -- modparam("dialog", "enable_dmq", 1) modparam("dialog", "db_url", DBURL) modparam("dialog", "db_mode", 1) modparam("dialog", "dlg_match_mode", 1) modparam("dialog", "enable_stats", 1) modparam("dialog", "rr_param", "did") modparam("dialog", "send_bye", 1) modparam("dialog", "dlg_extra_hdrs", "Hint: inactivity timeout\r\n") modparam("dialog", "track_cseq_updates", 1) modparam("dialog", "profiles_no_value", "inbound ; outbound") modparam("dialog", "profiles_with_value", "callerin ; callerout") # ---- dmq params ---- modparam("dmq", "server_address", DMQLST) modparam("dmq", "notification_address", DMQTG) modparam("dmq", "multi_notify", 1) modparam("dmq", "num_workers", 4) modparam("dmq", "ping_interval", 90)
You have to set `h_id_start` and `h_id_step` for all servers in the replication group so the generated ids do not overlap:
* https://www.kamailio.org/docs/modules/stable/modules/dialog.html#dialog.p.h_...
Likely the issue was caused by matching a dialog with same `h_id`, which was generated by a different server and not in the state that was the corresponding dialog for the processed sip message.
Eventually you can also set `dlg_match_mode=2`.
I will also add checks for the socket str, in case of replication might be safer.
We've made the changes for h_id_start and h_id_step and bounced everything. Will monitor and hope that the problem is resolved. Thanks for all the help. Will update in a week or two with results either way.
Closed #2757.
Reopen if you encounter another case that looks like being the same issue.
-
Daniel-Constantin Mierla -
nathanronchetti
-
themsley-voiceflex