Hello,
Situation:
Use a client (MCU) to call other clients through kamailio and hang up these calls, repeatedly Suddenly some clients were found fail to register, crash on kamailio was confirmed after checking the following log:
Jan 7 03:42:24 /kamailio[1425]: CRITICAL: <core> [pass_fdc:293]: receive_fd(): EOF on 11 Jan 7 03:42:24 /kamailio[1415]: ALERT: <core> [mainc:784]: handle_sigs(): child process 1416 exited by a signal 6 Jan 7 03:42:24 /kamailio[1415]: ALERT: <core> [mainc:787]: handle_sigs(): core was generated
And here is the `bt full` output from the coredump file:
--------------------------------------------------------------------------------------------------------- #0 0x401cd8e8 in raise () from /lib/libcso6 No symbol table info available #1 0x401d171c in abort () from /lib/libcso6 No symbol table info available #2 0x401c6bf0 in __assert_fail_base () from /lib/libcso6 No symbol table info available #3 0x401c6cb8 in __assert_fail () from /lib/libcso6 No symbol table info available #4 0x400cb0a8 in pthread_mutex_lock () from /lib/libpthreadso0 No symbol table info available #5 0x40cdccdc in add_uac (t=0x419eb780, request=0x40408d18, uri=0x40408e88, next_hop=0x40408e90, path=0x40409094, proxy=0x0, fsocket=0x0, snd_flags=, proto=0, flags=0, instance=0x4040909c, ruid=0x404090a8, location_ua=0x404090b0) at t_fwdc:863 ret = 0 branch = 0 __FUNCTION__ = "add_uac" #6 0x40ce5598 in t_forward_nonack (t=0x419eb780, p_msg=0x40408d18, proxy=0x0, proto=0) at t_fwdc:1723 branch_ret = 0 lowest_ret = -1 current_uri = {s = 0x0, len = 0} added_branches = 0 first_branch = 0 i = 1 q = 0 t_invite = 0xffffffff success_branch = 1 try_new = 1 lock_replies = -1093035388 dst_uri = {s = 0x376a88 "<core>: dsetc", len = 1093296396} path = {s = 0x0, len = 0} instance = {s = 0xbed99a5c "\200\325\314@\034\342>@\204\342-@x\021\002", len = 1087391576} ruid = {s = 0x0, len = 1073991680} location_ua = {s = 0x21178 "", len = 0} si = 0x0 backup_bflags = 0 bflags = 0 __FUNCTION__ = "t_forward_nonack" #7 0x40ccd580 in t_relay_to (p_msg=0x40408d18, proxy=0x0, proto=0, replicate=0) at t_funcsc:355 ret = 0 new_tran = 1 t = 0x419eb780 dst = {send_sock = 0x40408d18, to = {s = {sa_family = 40716, sa_data = "\331\276\030\215@@\004\301\003A8\342>@"}, sin = {sin_family = 40716, sin_port = 48857, sin_addr = {s_addr = 1077972248}, sin_zero = "\004\301\003A8\342>@"}, sin6 = {sin6_family = 40716, sin6_port = 48857, sin6_flowinfo = 1077972248, sin6_addr = { __in6_u = {__u6_addr8 = "\004\301\003A8\342>@\000\000\000\000\fa*A", __u6_addr16 = {49412, 16643, 57912, 16446, 0, 0, 24844, 16682}, __u6_addr32 = { 1090765060, 1077862968, 0, 1093296396}}}, sin6_scope_id = 0}}, id = 135544, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}} port = 48857 host = {s = 0x4031c534 "\001", len = 1077972248} comp = -1 __FUNCTION__ = "t_relay_to" #8 0x40d1cb98 in _w_t_relay_to (p_msg=0x40408d18, proxy=0x0, force_proto=0) at tmc:1517 t = 0x41086c80 res = -1093035156 __FUNCTION__ = "_w_t_relay_to" #9 0x40d1e1f4 in w_t_relay (p_msg=0x40408d18, _foo=0x0, _bar=0x0) at tmc:1718 No locals #10 0x00029aa0 in do_action (h=0xbed99fc8, a=0x40320f74, msg=0x40408d18) at actionc:1088 ret = -5 v = -1093034460 dst = {send_sock = 0x4036ee58, to = {s = {sa_family = 36120, sa_data = "@@\001\000\000\000`OK\000\324\235\331\276"}, sin = {sin_family = 36120, sin_port = 16448, sin_addr = {s_addr = 1}, sin_zero = "`OK\000\324\235\331\276"}, sin6 = {sin6_family = 36120, sin6_port = 16448, sin6_flowinfo = 1, sin6_addr = {__in6_u = { __u6_addr8 = "`OK\000\324\235\331\276\000\000\000\000\000\000\000", __u6_addr16 = {20320, 75, 40404, 48857, 0, 0, 0, 0}, __u6_addr32 = {4935520, 3201932756, 0, 0}}}, sin6_scope_id = 3201932896}}, id = 1077341784, proto = 24 '\030', send_flags = {f = 141 '\215', blst_imask = 64 '@'}} tmp = 0x4003d000 "H\377\002" new_uri = 0x0 end = 0xd72a4 "" crt = 0x0 cmd = 0x402ff898 len = 0 user = 135544 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 1610612752}, port = {s = 0x8a <Address 0x8a out of bounds>, len = 2623776}, params = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = -1093034636}, sip_params = {s = 0x98f6c " ", len = 1077000232}, headers = {s = 0xbed99e78 "", len = 1077972248}, port_no = 46240, proto = 48857, type = 3727636, flags = 8388608, transport = {s = 0x6e9c68 <Address 0x6e9c68 out of bounds>, len = 967680}, ttl = {s = 0xbed99d7c "\001", len = 22}, user_param = {s = 0x4b4f4c "", len = 0}, maddr = {s = 0xbed99d5c "", len = 19}, method = {s = 0xbed99d84 "", len = 303571655}, lr = {s = 0x0, len = 135544}, r2 = {s = 0x0, len = 0}, gr = {s = 0xbed99d8c "\b\230\021", len = 1152864}, transport_val = { s = 0xbed99db0 "\001", len = 22}, ttl_val = {s = 0x4b4f4c "", len = 1}, user_param_val = {s = 0x280920 "\b0\033\345\004 \223\345\f0\033\345", len = 1}, maddr_val = {s = 0x4b4f60 "2", len = 0}, method_val = {s = 0xbed99d9c "$\341\021", len = 1153032}, lr_val = {s = 0x0, len = 1}, r2_val = {s = 0xbed99dbc "", len = 1171748}, gr_val = {s = 0x1 <Address 0x1 out of bounds>, len = -1093034400}} next_hop = {user = {s = 0xbed99c3c "", len = 1077972248}, passwd = {s = 0x4032aaac "0\003", len = -1093034880}, host = {s = 0x40408d18 "l\006", len = 1076849808}, port = {s = 0x3 <Address 0x3 out of bounds>, len = 0}, params = {s = 0x40317840 "", len = 1077972248}, sip_params = { s = 0xffffffff <Address 0xffffffff out of bounds>, len = 1076983872}, headers = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, port_no = 4472, proto = 2, type = ERROR_URI_T, flags = 0, transport = {s = 0x4003d000 "H\377\002", len = -1093033948}, ttl = {s = 0x29b18 "", len = 0}, user_param = { s = 0x40408d18 "l\006", len = 1077062316}, maddr = {s = 0xbed9b4a0 "\004", len = 0}, method = {s = 0x0, len = -1}, lr = { s = 0xbed99cac "l\234\331\276\370\233\331\276hi\003", len = 303571655}, r2 = {s = 0x0, len = 135544}, gr = {s = 0x40408d18 "l\006", len = 1077018000}, transport_val = {s = 0xbed9a0b0 "", len = 1073991680}, ttl_val = {s = 0xbed99c6c "\030\233\002", len = -1093035016}, user_param_val = {s = 0x36968 "", len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} u = 0x0 port = 4632 dst_host = 0x4032abdc i = 4104020 flags = -1093034948 avp = 0x40310d88 st = {flags = 0, id = 0, name = {n = 1610612752, s = {s = 0x60000010 <Address 0x60000010 out of bounds>, len = 567153825}, re = 0x60000010}, avp = 0x280920} sct = 0x128264 sjt = 0x0 rve = 0x4 mct = 0x40408d18 rv = 0x4101b35c rv1 = 0x40408d18 c1 = {cache_type = RV_CACHE_EMPTY, val_type = 303571655, c = {avp_val = {n = 0, s = {s = 0x0, len = 135544}, re = 0x0}, pval = {rs = {s = 0x0, len = 135544}, ri = 0, flags = 0}}, i2s = "\000\320\003@l\234\331\276\350l\003\000\003\000\000\000\377\377\377\377l\234"} s = {s = 0x36968 "", len = 0} srevp = {0xbed99b84, 0xbed99b10} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = { number = 2, string = 0x2 <Address 0x2 out of bounds>, str = {s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 3}, data = 0x0, attr = 0x0, select = 0x0}}, {type = MODFIXUP_ST, u = { number = 1077333124, string = 0x4036cc84 "X\314\066@ ", str = {s = 0x4036cc84 "X\314\066@ ", len = 3}, data = 0x4036cc84, attr = 0x4036cc84, select = 0x4036cc84}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #11 0x00036ce8 in run_actions (h=0xbed99fc8, a=0x40320f74, msg=0x40408d18) at actionc:1583 t = 0x40320f74 ret = -1 mod = 0x0 ms = 0 __FUNCTION__ = "run_actions" #12 0x00037478 in run_actions_safe (h=0xbed9b4a0, a=0x40320f74, msg=0x40408d18) at actionc:1648 ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {303571655, 0, 135544, 0, 0, 0, 1073991680, -1093034060, -1093034176, 223592, 0 <repeats 16 times>, 1610612752, 1100845520, 1416, 1100429564, 1164518012, 1077972248, 1077018000, -1093033808, 0, 1073991680, 1, 0, 0, 1077972248, -1, 0, 303571655, 303571655, 0, 135544, 0, 0, -1093033388, 226424, 0, 1077972248, 1077018000, -1093028704, 0, 0, -1, 1164092908, 303571655, 0, 135544, 0, 0, 0}, __mask_was_saved = 0, __saved_mask = {__val = {3201933468, 3201933352, 223592, 0 <repeats 16 times>, 1610612752, 3, 2623776, 3201933652, 0, 5, 3201933692, 1085865828, 7249368, 1077864140, 4849333, 4849164, 8}}}}} ret = -1 ign_on_break = 0 #13 0x0017ed68 in rval_get_int (h=0xbed9b4a0, msg=0x40408d18, i=0xbed9a390, rv=0x40321184, cache=0x0) at rvaluec:924 r_avp = 0x40408d18 avp_val = {n = 0, s = {s = 0x0, len = 4848947}, re = 0x0} pval = {rs = {s = 0x3dc238 "parse_headers", len = 4035544}, ri = 1077864140, flags = 1076748296} tmp = {s = 0x0, len = 5} s = 0x0 r = 0 ret = 0 destroy_pval = 0 __FUNCTION__ = "rval_get_int" #14 0x00183b3c in rval_expr_eval_int (h=0xbed9b4a0, msg=0x40408d18, res=0xbed9a390, rve=0x40321180) at rvaluec:1918 i1 = 5060 i2 = 1077972248 ret = -1 c1 = {cache_type = 135544, val_type = 1077018528, c = {avp_val = {n = -1093032424, s = {s = 0xbed9a618 "", len = 1077972248}, re = 0xbed9a618}, pval = {rs = { s = 0xbed9a618 "", len = 1077972248}, ri = -1093028704, flags = -1093033244}}, i2s = "\340r\346@\344\242\331\276$\211\344@\001\000\000\000\000\000\000\224"} c2 = {cache_type = 137, val_type = 4294967295, c = {avp_val = {n = 0, s = {s = 0x0, len = -1093033244}, re = 0x0}, pval = {rs = {s = 0x0, len = -1093033244}, ri = 1568104, flags = 192}}, i2s = "\000\000\000\000\b\000\000\000\220\221\235A\000\000\000\000\364@\347@\000"} rv1 = 0x0 rv2 = 0x402de008 __FUNCTION__ = "rval_expr_eval_int" #15 0x00183f00 in rval_expr_eval_int (h=0xbed9b4a0, msg=0x40408d18, res=0xbed9a618, rve=0x40321524) at rvaluec:1926 i1 = 0 i2 = -1093028704 ret = -1 c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, pval = {rs = {s = 0x0, len = 0}, ri = 0, flags = 0}}, i2s = "\000\000\000\000\000F0@\000\000\000\000\000\000\000\000\030\215@@\324\260"} c2 = {cache_type = 12, val_type = 4848849, c = {avp_val = {n = 4, s = {s = 0x4 <Address 0x4 out of bounds>, len = 303571655}, re = 0x4}, pval = {rs = { s = 0x4 <Address 0x4 out of bounds>, len = 303571655}, ri = 0, flags = 135544}}, i2s = "\000\000\000\000\000\000\000\000\000\320\003@\314\243\331\276\350l\003\000\000"} rv1 = 0x49fcc4 rv2 = 0x0 __FUNCTION__ = "rval_expr_eval_int" #16 0x00029600 in do_action (h=0xbed9b4a0, a=0x403218c8, msg=0x40408d18) at actionc:1064 ret = -5 v = 0 dst = {send_sock = 0x0, to = {s = {sa_family = 33508, sa_data = ">\000\240\325=\000\001\000\000\000\004\000\000"}, sin = {sin_family = 33508, sin_port = 62, sin_addr = {s_addr = 4052384}, sin_zero = "\001\000\000\000\004\000\000"}, sin6 = {sin6_family = 33508, sin6_port = 62, sin6_flowinfo = 4052384, sin6_addr = { __in6_u = {__u6_addr8 = "\001\000\000\000\004\000\000\000\300\247\331\276\000\000\000", __u6_addr16 = {1, 0, 4, 0, 42944, 48857, 0, 0}, __u6_addr32 = {1, 4, 3201935296, 0}}}, sin6_scope_id = 303571655}}, id = 0, proto = 199 '\307', send_flags = {f = 34 '"', blst_imask = 24 '\030'}} tmp = 0x0 new_uri = 0x3 <Address 0x3 out of bounds> end = 0xbed9a714 "<;\030" crt = 0x0 cmd = 0x40329d38 len = 0 user = 135544 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = { s = 0x0, len = 1610612752}, headers = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = -1093032500}, port_no = 36716, proto = 9, type = 7, flags = 5, transport = {s = 0x40312380 "\001", len = -1093032232}, ttl = {s = 0x40408d18 "l\006", len = -1093028704}, user_param = {s = 0xbed9a59c "h\355\027", len = 1077572256}, maddr = {s = 0x0, len = 5}, method = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, lr = {s = 0xbed9a62c "0?\030", len = 1568104}, r2 = { s = 0x6e9894 <Address 0x6e9894 out of bounds>, len = 968660}, gr = {s = 0x6 <Address 0x6 out of bounds>, len = 0}, transport_val = { s = 0x3dc7b4 "<core>: parser/parse_addr_specc", len = 303571655}, ttl_val = {s = 0x0, len = 135544}, user_param_val = {s = 0x0, len = 0}, maddr_val = { s = 0xbed9a63c "\214M)", len = 2604464}, method_val = {s = 0x3dd5a0 "parse_to_param", len = 285}, lr_val = {s = 0x40304600 "l?\002A\020\070\001A\001", len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 5}} next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 1077972248}, params = {s = 0x40329c04 "%\003", len = -1093032736}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 4294967295, transport = {s = 0x0, len = -1093032756}, ttl = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, user_param = { s = 0x21178 "", len = 0}, maddr = {s = 0x0, len = -1093032316}, method = {s = 0x37478 "\f", len = 141}, lr = {s = 0x40408d18 "l\006", len = 1077058564}, r2 = { s = 0xbed9b4a0 "\004", len = 0}, gr = {s = 0x0, len = -1}, transport_val = {s = 0x403a72a0 "\002", len = 303571655}, ttl_val = {s = 0x0, len = 135544}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 1073991680}, method_val = {s = 0xbed9a4cc "xt\003", len = -1093032872}, lr_val = {s = 0x36968 "", len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} u = 0x0 port = 4632 dst_host = 0xbed9a8e4 i = -1093028704 flags = 1088786032 avp = 0x40408d18 st = {flags = 3201934540, id = 27880, name = {n = 0, s = {s = 0x0, len = 1073991680}, re = 0x0}, avp = 0xbed9a4b4} sct = 0x0 sjt = 0x121822c7 rve = 0x40321524 mct = 0x403ef9e0 rv = 0x4003d000 rv1 = 0xbed9aa48 c1 = {cache_type = 3201935400, val_type = 331612160, c = {avp_val = {n = 1, s = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, re = 0x1}, pval = {rs = { s = 0x1 <Address 0x1 out of bounds>, len = 0}, ri = 1077972248, flags = 4848957}}, i2s = "\001\000\000\000\307"\030\022\000\000\000\000x\021\002\000\000\000\000\000\000"} s = {s = 0x40e47ce8 "", len = 0} srevp = {0x40e474f0, 0xbed9a464} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = { number = 2, string = 0x2 <Address 0x2 out of bounds>, str = {s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 3}, data = 0x0, attr = 0x0, select = 0x0}}, {type = MODFIXUP_ST, u = { number = 1077333124, string = 0x4036cc84 "X\314\066@ ", str = {s = 0x4036cc84 "X\314\066@ ", len = 3}, data = 0x4036cc84, attr = 0x4036cc84, select = 0x4036cc84}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #17 0x00036ce8 in run_actions (h=0xbed9b4a0, a=0x4031ea14, msg=0x40408d18) at actionc:1583 t = 0x403218c8 ret = 1 mod = 0x0 ms = 0 __FUNCTION__ = "run_actions" #18 0x000263ac in do_action (h=0xbed9b4a0, a=0x4032b32c, msg=0x40408d18) at actionc:712 ret = -5 v = 0 dst = {send_sock = 0x0, to = {s = {sa_family = 24, sa_data = "\000\000\002\375I\000\000\000\000\000\320\220\235A"}, sin = {sin_family = 24, sin_port = 0, sin_addr = { s_addr = 4848898}, sin_zero = "\000\000\000\000\320\220\235A"}, sin6 = {sin6_family = 24, sin6_port = 0, sin6_flowinfo = 4848898, sin6_addr = {__in6_u = { __u6_addr8 = "\000\000\000\000\320\220\235A\000\000\000\000\000\000\000", __u6_addr16 = {0, 0, 37072, 16797, 0, 0, 0, 0}, __u6_addr32 = {0, 1100845264, 0, 0}}}, sin6_scope_id = 0}}, id = 0, proto = 199 '\307', send_flags = {f = 34 '"', blst_imask = 24 '\030'}} tmp = 0xbed9abb4 "\350l\003" new_uri = 0x121822c7 <Address 0x121822c7 out of bounds> end = 0x403ebfec "\232\301>@\030" crt = 0x280920 "\b0\033\345\004 \223\345\f0\033\345" cmd = 0x40329698 len = 0 user = 1077034684 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = { s = 0x0, len = 1610612752}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = 18, flags = 0, transport = {s = 0x40326164 "\001", len = -1093031160}, ttl = {s = 0x40408d18 "l\006", len = -1093028704}, user_param = {s = 0xbed9a9cc "h\355\027", len = 1076748296}, maddr = { s = 0xbed9a9f4 "\360\253\331\276\030\215@@\240\264\331\276\004", len = 1085865828}, method = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, lr = { s = 0xbed9aa5c "<;\030", len = 1568104}, r2 = {s = 0x6e9a90 <Address 0x6e9a90 out of bounds>, len = 968152}, gr = { s = 0x116570 "\004\060\240\343\202!\240\341\002 \201\340\003\060\202\340", len = 1215756}, transport_val = {s = 0x4 <Address 0x4 out of bounds>, len = 145}, ttl_val = {s = 0x0, len = 303571655}, user_param_val = {s = 0x40328d8c "\004", len = -1093030928}, maddr_val = {s = 0x40408d18 "l\006", len = -1093028704}, method_val = {s = 0x4 <Address 0x4 out of bounds>, len = 1091096887}, lr_val = {s = 0xb <Address 0xb out of bounds>, len = 1091096872}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 5}} next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x60000010 <Address 0x60000010 out of bounds>, len = 56}, host = {s = 0x403636c0 "\n", len = 3}, port = { s = 0x2 <Address 0x2 out of bounds>, len = 1077972248}, params = {s = 0x40329600 "#\003", len = -1093031664}, sip_params = { s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 1076733136}, headers = {s = 0x12 <Address 0x12 out of bounds>, len = 0}, port_no = 0, proto = 0, type = 1077972248, flags = URI_USER_NORMALIZE, transport = {s = 0x0, len = 303571655}, ttl = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, user_param = {s = 0x21178 "", len = 0}, maddr = {s = 0x0, len = -1093031244}, method = {s = 0x37478 "\f", len = 1077972248}, lr = {s = 0x40408d18 "l\006", len = 1077057024}, r2 = {s = 0xbed9b4a0 "\004", len = 0}, gr = {s = 0x0, len = 1}, transport_val = {s = 0x40408d18 "l\006", len = 303571655}, ttl_val = {s = 0x0, len = 135544}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 1073991680}, method_val = {s = 0xbed9a8fc "xt\003", len = -1093031800}, lr_val = { s = 0x36968 "", len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} u = 0x403242fc port = 48857 dst_host = 0x40408d18 i = 2 flags = 0 avp = 0x40408d18 st = {flags = 3201935612, id = 27880, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x0} sct = 0x403636a8 sjt = 0x40408d18 rve = 0x40329d34 mct = 0x113 rv = 0x4003d000 rv1 = 0xbed9ae78 c1 = {cache_type = RV_CACHE_EMPTY, val_type = 1073991680, c = {avp_val = {n = -1093031916, s = {s = 0xbed9a814 "xt\003", len = -1093032032}, re = 0xbed9a814}, pval = { rs = {s = 0xbed9a814 "xt\003", len = -1093032032}, ri = 223592, flags = 0}}, i2s = "\000\000\000\000\307"\030\022\000\000\000\000x\021\002\000\000\000\000\000\000"} s = {s = 0x0, len = 0} srevp = {0x0, 0x21178} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = { number = 2, string = 0x2 <Address 0x2 out of bounds>, str = {s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 3}, data = 0x0, attr = 0x0, select = 0x0}}, {type = MODFIXUP_ST, u = { number = 1077333124, string = 0x4036cc84 "X\314\066@ ", str = {s = 0x4036cc84 "X\314\066@ ", len = 3}, data = 0x4036cc84, attr = 0x4036cc84, select = 0x4036cc84}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #19 0x00036ce8 in run_actions (h=0xbed9b4a0, a=0x40329ad4, msg=0x40408d18) at actionc:1583 t = 0x4032b32c ret = 1 mod = 0x0 ms = 0 __FUNCTION__ = "run_actions" #20 0x00029a2c in do_action (h=0xbed9b4a0, a=0x4032b454, msg=0x40408d18) at actionc:1079 ret = 1 v = 1 dst = {send_sock = 0x0, to = {s = {sa_family = 64960, sa_data = " @\b \000\000pD!@\340\070U"}, sin = {sin_family = 64960, sin_port = 16416, sin_addr = { s_addr = 8200}, sin_zero = "pD!@\340\070U"}, sin6 = {sin6_family = 64960, sin6_port = 16416, sin6_flowinfo = 8200, sin6_addr = {__in6_u = { __u6_addr8 = "pD!@\340\070U\000\000\240-@\000\000\000", __u6_addr16 = {17520, 16417, 14560, 85, 40960, 16429, 0, 0}, __u6_addr32 = {1075922032, 5585120, 1076731904, 0}}}, sin6_scope_id = 1}}, id = 0, proto = 21 '\025', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}} tmp = 0x4108cee0 "xlog" new_uri = 0x4109e174 "\214\240\001" end = 0x79 <Address 0x79 out of bounds> crt = 0x4 <Address 0x4 out of bounds> cmd = 0x403152a0 len = 135544 user = 0 uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = { s = 0x0, len = 1610612752}, headers = {s = 0x402da228 "\001", len = 134217728}, port_no = 0, proto = 13362, type = 1076597596, flags = 1076611795, transport = { s = 0x402da228 "\001", len = 5585624}, ttl = {s = 0x402da4d0 "", len = 132392}, user_param = {s = 0x555a60 "", len = 124320}, maddr = {s = 0x555a60 "", len = 1075903936}, method = {s = 0xbed9ae9f "A\373\037", len = 1076728908}, lr = {s = 0x1 <Address 0x1 out of bounds>, len = 1076731904}, r2 = { s = 0x1 <Address 0x1 out of bounds>, len = 0}, gr = {s = 0x3 <Address 0x3 out of bounds>, len = -1093030216}, transport_val = {s = 0x402b73cc "%H:%M:%S", len = 1076731904}, ttl_val = {s = 0x402d926c ",t+@", len = 5585496}, user_param_val = {s = 0x402da4d0 "", len = 8200}, maddr_val = { s = 0x80 <Address 0x80 out of bounds>, len = 8200}, method_val = {s = 0xbed9af60 "", len = 5593696}, lr_val = {s = 0x402da228 "\001", len = 5585112}, r2_val = { s = 0x402da4d0 "", len = 132904}, gr_val = {s = 0x553ad8 "failed\n", len = 132392}} next_hop = {user = { s = 0x419d4700 "4 CIF=1;QCIF=1\r\na=rtpmap:31 H261/90000\r\na=fmtp:31 CIF=2;QCIF=2\r\na=sendrecv\r\nm=application 30006 RTP/SAVP 100\r\na=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:iBuE6kKiNbn30g7hkU2/qRE0ROKMAJZDD/Zg48t0\r\na=encr", len = 0}, passwd = {s = 0x419d4a44 "\220\001", len = 1100826908}, host = {s = 0x0, len = 5}, port = { s = 0x419d4a9c "\001", len = 1077972248}, params = {s = 0x40315020 "\226\002", len = -1093030592}, sip_params = {s = 0x49feb5 "", len = 4849103}, headers = { s = 0x8 <Address 0x8 out of bounds>, len = 0}, port_no = 0, proto = 0, type = 1089616044, flags = 4294967295, transport = {s = 0x0, len = 0}, ttl = { s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, user_param = {s = 0x21178 "", len = 0}, maddr = {s = 0x0, len = -1093030172}, method = { s = 0x37478 "\f", len = 1100827356}, lr = {s = 0x40408d18 "l\006", len = 1076973600}, r2 = {s = 0xbed9b4a0 "\004", len = 0}, gr = {s = 0x0, len = -1}, transport_val = {s = 0x40231380 "\b0\235\345\003`\206\340\336\372\377\352\v", len = 303571655}, ttl_val = {s = 0x0, len = 135544}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 1073991680}, method_val = {s = 0xbed9ad2c "xt\003", len = -1093030728}, lr_val = {s = 0x36968 "", len = 0}, r2_val = { s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} u = 0x4109e174 port = 48857 dst_host = 0x1 i = -1093028704 flags = -1 avp = 0x40408d18 st = {flags = 3201936684, id = 27880, name = {n = 152, s = {s = 0x98 <Address 0x98 out of bounds>, len = 1100827388}, re = 0x98}, avp = 0xffffffff} sct = 0x403ece20 sjt = 0x40408d18 rve = 0x40329694 mct = 0x402d944c rv = 0x4003d000 rv1 = 0xbed9b2a8 c1 = {cache_type = 4849103, val_type = 1452104383, c = {avp_val = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, pval = {rs = {s = 0x0, len = 0}, ri = 8388608, flags = 7249432}}, i2s = "\254\070\362@\307"\030\022\000\000\000\000x\021\002\000\000\000\000\000\000"} s = {s = 0x403ff4fc "\033", len = 4849333} srevp = {0xbed9ac8c, 0xbed9ac8c} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = { number = 2, string = 0x2 <Address 0x2 out of bounds>, str = {s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 3}, data = 0x0, attr = 0x0, select = 0x0}}, {type = MODFIXUP_ST, u = { number = 1077333124, string = 0x4036cc84 "X\314\066@ ", str = {s = 0x4036cc84 "X\314\066@ ", len = 3}, data = 0x4036cc84, attr = 0x4036cc84, select = 0x4036cc84}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #21 0x00036ce8 in run_actions (h=0xbed9b4a0, a=0x403294d0, msg=0x40408d18) at actionc:1583 t = 0x4032b454 ret = 1 mod = 0x4003d000 ms = 0 __FUNCTION__ = "run_actions" #22 0x000263ac in do_action (h=0xbed9b4a0, a=0x403160fc, msg=0x40408d18) at actionc:712 ret = -5 v = 0 dst = {send_sock = 0x115f68, to = {s = {sa_family = 36108, sa_data = "\022\000\004\000\000\000\240\000\000\000\364\360\334@"}, sin = {sin_family = 36108, sin_port = 18, sin_addr = {s_addr = 4}, sin_zero = "\240\000\000\000\364\360\334@"}, sin6 = {sin6_family = 36108, sin6_port = 18, sin6_flowinfo = 4, sin6_addr = {__in6_u = {__u6_addr8 = "\240\000\000\000\364\360\334@\307"\030\022\004\263\331\276", __u6_addr16 = {160, 0, 61684, 16604, 8903, 4632, 45828, 48857}, __u6_addr32 = {160, 1088221428, 303571655, 3201938180}}}, sin6_scope_id = 1077862732}}, id = 0, proto = 0 '\000', send_flags = {f = 208 '\320', blst_imask = 3 '\003'}} tmp = 0x40e43104 "\f\220\002" new_uri = 0x121822c7 <Address 0x121822c7 out of bounds> end = 0x49fcb8 "Route: sip:101009336:5060;nat=yes;lr=on\r\nFrom: sip:1000@101009336:5060;tag=875590146\r\nTo: sip:9144@101009336:5060;tag=748310794\r\nCall-ID: 54544907\r\nCSeq: 6 INFO\r\nContact: <sip:1000@101" crt = 0x21178 "" cmd = 0x40304fe0 len = 626540 user = -1093028780 uri = {user = { s = 0x49fc9f "branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: sip:1000@101009336:5060;tag=875590146\r\nTo: sip:9144@101009336:5060;tag=748310794\r\nCall-ID: 54544907\r\nCSeq: 6 INFO", len = 0}, passwd = {s = 0x0, len = -1093029204}, host = {s = 0x40d588cc "<0\033\345\002\001", len = 2042}, port = {s = 0x7d8 <Address 0x7d8 out of bounds>, len = 1077972248}, params = { s = 0x49fc8c "09336:5062;rport;branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: sip:1000@101009336:5060;tag=875590146\r\nTo: sip:9144@101009336:5060;tag=748310794\r\nCall-ID: 545", len = 4115756}, sip_params = {s = 0x3ebab8 "<core>: parser/parse_viac", len = 32}, headers = { s = 0x402de008 "\001", len = 0}, port_no = 12660, proto = 16799, type = 98365684, flags = 4848822, transport = {s = 0x403ee130 " ", len = -1093029288}, ttl = { s = 0x4 <Address 0x4 out of bounds>, len = 4848788}, user_param = { s = 0x49fc98 ";rport;branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: sip:1000@101009336:5060;tag=875590146\r\nTo: sip:9144@101009336:5060;tag=748310794\r\nCall-ID: 54544907\r\nCSeq:", len = 4848792}, maddr = {s = 0x4 <Address 0x4 out of bounds>, len = 331743235}, method = { s = 0xbed9b374 "$b(", len = -1093029004}, lr = {s = 0x403ee14c "\350", len = 1077933164}, r2 = {s = 0xbed9b2a4 "", len = 1100951944}, gr = {s = 0xbed9b4d8 "", len = 1}, transport_val = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = -1}, ttl_val = {s = 0xbed9b2ac "\373\377\377\377\310\264\331\276\001", len = 1077933164}, user_param_val = {s = 0x49feb5 "", len = 4848763}, maddr_val = {s = 0x0, len = 84271112}, method_val = {s = 0xbed9b29c "L\341>@", len = 1085865828}, lr_val = {s = 0x419f31e0 "\310", len = 0}, r2_val = {s = 0x4 <Address 0x4 out of bounds>, len = 8388608}, gr_val = { s = 0x6ea098 <Address 0x6ea098 out of bounds>, len = 966608}} next_hop = {user = {s = 0x0, len = 135544}, passwd = {s = 0x280920 "\b0\033\345\004 \223\345\f0\033\345", len = -1093029596}, host = {s = 0xbed9b124 "", len = 1076748296}, port = {s = 0xbed9b14c "l\217\t", len = 1085865828}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 8388608}, headers = { s = 0x6ea058 <Address 0x6ea058 out of bounds>, len = 966672}, port_no = 24488, proto = 17, type = 1215756, flags = 4, transport = { s = 0x9e <Address 0x9e out of bounds>, len = 0}, ttl = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = -1093029452}, user_param = {s = 0x98f6c " ", len = 109}, maddr = {s = 0x40addfec "", len = -1093029428}, method = {s = 0x27d028 "\004", len = 4}, lr = {s = 0x3e9fa4 "core", len = 0}, r2 = { s = 0x3 <Address 0x3 out of bounds>, len = -1093029340}, gr = {s = 0x2e14d0 "", len = 0}, transport_val = {s = 0x0, len = -1093029428}, ttl_val = { s = 0xbed9b1cc "\314\210\325@\372\a", len = 1088115668}, user_param_val = {s = 0x40db0528 "tm: t_replyc", len = 8}, maddr_val = {s = 0x0, len = 4111032}, method_val = {s = 0x121822c7 <Address 0x121822c7 out of bounds>, len = 0}, lr_val = {s = 0x21178 "", len = 0}, r2_val = {s = 0x0, len = -1093029282}, gr_val = { s = 0xbed9b25f "\005\234\262\331\276d\377\270@\340\061\237A", len = 4849333}} u = 0x157 port = 61 dst_host = 0x8 i = 4 flags = 626540 avp = 0xbed9b44c st = {flags = 4, id = 141, name = {n = 343, s = {s = 0x157 <Address 0x157 out of bounds>, len = 1088221428}, re = 0x157}, avp = 0xbed9b15c} sct = 0x1d sjt = 0x40408d18 rve = 0x4031529c mct = 0x49fddf rv = 0x128d0c rv1 = 0x0 c1 = {cache_type = 3201937612, val_type = 3201937612, c = {avp_val = {n = 1076748296, s = {s = 0x402de008 "\001", len = -1093029644}, re = 0x402de008}, pval = {rs = { s = 0x402de008 "\001", len = -1093029644}, ri = 1085865828, flags = 86}}, i2s = "\000\000\000\000\254\177=\000\000\000\200\000\200\233n\000\350\304\016\000\200d"} s = {s = 0x1b5 <Address 0x1b5 out of bounds>, len = 2623776} srevp = {0x419f3210, 0x386f70} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = { number = 2, string = 0x2 <Address 0x2 out of bounds>, str = {s = 0x2 <Address 0x2 out of bounds>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, { type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 3}, data = 0x0, attr = 0x0, select = 0x0}}, {type = MODFIXUP_ST, u = { number = 1077333124, string = 0x4036cc84 "X\314\066@ ", str = {s = 0x4036cc84 "X\314\066@ ", len = 3}, data = 0x4036cc84, attr = 0x4036cc84, select = 0x4036cc84}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #23 0x00036ce8 in run_actions (h=0xbed9b4a0, a=0x4030fbc0, msg=0x40408d18) at actionc:1583 t = 0x403160fc ret = 1 mod = 0x0 ms = 0 __FUNCTION__ = "run_actions" #24 0x00037538 in run_top_route (a=0x4030fbc0, msg=0x40408d18, c=0x0) at actionc:1669 ctx = {rec_lev = 4, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {303571655, 0, 135544, 0, 0, 0, 1073991680, -1093028724, -1093028840, 223592, 0 <repeats 16 times>, 1610612752, 966492, 1138420, 1215756, 2623776, -1093028532, -1093028532, 1076748296, -1093028492, 1085865828, 1077859196, 0, 14, 8388608, 7250220, 966460, 1138388, 1215756, 4, 162, 0, 303571655, -1093028388, 626540, 4124784, 4122588, 3607920, 0, 1077859224, 303571655, 1077872804, 1076748296, 0, 5, 1164317944, 1164522740, -1093028324, 1077972248}, __mask_was_saved = 0, __saved_mask = {__val = {4848676, 1, 13, 3749280, 3201938908, 1, 13, 0, 1085944084, 0, 135544, 303571655, 0, 1073741825, 1077972248, 0, 303571655, 3201939020, 1691804, 0, 135544, 3751672, 4848758, 1077972268, 82, 1, 1077972248, 225, 1073741825, 1077972220, 0, 0}}}}} p = 0xbed9b4a0 ret = 135544 sfbk = 0 #25 0x0013ce80 in receive_msg ( buf=0x49fc24 "INFO sip:9144@101009144:5060;transport=TCP;alias=101009144:36841~2 SIP/20\r\nVia: SIP/20/UDP 101009336:5062;rport;branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: ", len=657, rcv_info=0xbed9b8b4) at receivec:216 msg = 0x40408d18 ctx = {rec_lev = 0, run_flags = 1, last_retcode = -1093028124, jmp_env = {{__jmpbuf = {-1093028084, 1094111756, 0, 1077352742, 2127, 5, 0, 1074750288, 1075464840, 1073835920, 1094553688, 1100326448, 76, 8403020, 1075499064, 1073991680, -1093027904, 1073913596, 1073993916, 1075465032, 0, 0, 1, 2127, 1074750344, 1074749440, 105499, 1075499832, 88616, 1, 19, 268896657, 1073994528, 0, -1093027928, 105499, 1074750344, -1093027836, -1093027860, 1073836664, -1093027928, 1073994436, 0, 1074750344, 1, 0, 1, 1073994088, 2623776, -12717463, 1073994528, 0, -1093027856, 104072, 1074750328, 0, 1073994528, 1073836664, 1073994088, 1073994436, 0, 1074750328, 1, 0}, __mask_was_saved = 1, __saved_mask = {__val = {1073994088, 3201939468, 626540, 1075499064, 1074749440, 0, 108658490, 1073994528, 0, 7714291, 135544, 1073994088, 0, 1074349248, 0, 1, 4427664, 0, 1073991680, 3201939580, 1073854768, 1074750344, 1, 1, 0, 0, 1075499064, 303571655, 0, 135544, 0, 0}}}}} ret = 2487808 inb = { s = 0x49fc24 "INFO sip:9144@101009144:5060;transport=TCP;alias=101009144:36841~2 SIP/20\r\nVia: SIP/20/UDP 101009336:5062;rport;branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: ", len = 657} __FUNCTION__ = "receive_msg" #26 0x00266708 in udp_rcv_loop () at udp_serverc:521 len = 657 buf = "INFO sip:9144@101009144:5060;transport=TCP;alias=101009144:36841~2 SIP/20\r\nVia: SIP/20/UDP 101009336:5062;rport;branch=z9hG4bK897806495\r\nRoute: sip:101009336:5060;nat=yes;lr=on\r\nFrom: " tmp = 0x0 from = 0x402fe830 fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {610100234, 3201939668, 1935216, 1759989418}, addr32 = {610100234, 3201939668, 1935216, 1759989418}, addr16 = {25610, 9309, 47316, 48857, 34672, 29, 20138, 26855}, addr = "\nd]$\324\270\331\276p\207\035\000\252N\347h"}}, dst_ip = {af = 2, len = 4, u = {addrl = {610100234, 0, 0, 0}, addr32 = {610100234, 0, 0, 0}, addr16 = {25610, 9309, 0, 0, 0, 0, 0, 0}, addr = "\nd]$", '\000' <repeats 11 times>}}, src_port = 5062, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\306\nd]$\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50707, sin_addr = {s_addr = 610100234}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50707, sin6_flowinfo = 610100234, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x402fde78, proto = 1 '\001'} __FUNCTION__ = "udp_rcv_loop" #27 0x000c90a0 in main_loop () at mainc:1629 i = 0 pid = 0 si = 0x402fde78 si_desc = "udp receiver child=0 sock=101009336:5060 (101009336:5060)\000\300u1@h\t;\000 \236\060@\000\000\000\000x\021\002\000\024\260=@\000\000\000\000d\272\331\276\260\275'\000\230\331\070\000X\311\070\000\030\000\000\000\000\240[A\000\000\000\000\000\320\003@\244\303\230A" r = 1073991680 nrprocs = 1 __FUNCTION__ = "main_loop" #28 0x000d0830 in main (argc=5, argv=0xbed9bd04) at mainc:2581 cfg_stream = 0x4b5008 c = -1 r = 0 tmp = 0x4003dac4 "8c\017@\a" tmp_len = -1093026968 port = 1073836664 proto = -1093026900 options = 0x383880 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 629055291 rfd = 216 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x0 p = 0x5 <Address 0x5 out of bounds> __FUNCTION__ = "main"
---------------------------------------------------------------------------------------------------------
It seems the socket got error (uncertain) Does anyone have the similar experience?
Thanks
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466
The error log message is not related to the crash. It is about internal sockets being closed due to the crash of a process.
Looking at the code, it happens inside membar() which does some atomic operation to sync the memory. Behind it is some assembler code specific for CPU architecture. What is yours?
Paste here the output of "kamailio -v".
Does it happened many times or just once?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-170049987
version: kamailio 4.2.6 (armv41/linux) eaa70a-dirty flags: STATS: Off, EXTRA_DEBUG, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, USE_PTHREAD_MUTEX, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: eaa70a -dirty compiled on 11:24:16 Sep 22 2015 with arm-marvell-linux-gnueabi-gcc
This happened several times so far, but not very frequent...
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-170420909
Do you have any changes done by yourself? The version suggests that c code was changed...
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-172445799
Sorry for my late reply first since I was busy last 2 weeks...
Yes, I changed the C code for adding some customized functions , or modifying some loaded path for specific purpose.
Ex: I modified save() function to let it can pass one more parameter as a reason text.... I think they should be insignificant...
Btw, sometimes when my application stopped\restarted Kamailio, It can not stop successfully, and I forced to stop it by killing the process, remove the pid of Kamailio. Maybe this will cause some side effects on next start?
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-176629147
When it happens that it doesn't stop, check what the processes are doing by attaching with gdb and get the backtrace.
The code for membar() is standard, operating at assembler level. But the crash here could be a side effect of a buffer overflow. Can you doublecheck that your changes are not introducing such issue? Maybe you can give a try with stock kamailio to see if the issue is there.
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-183066625
I tested the stop issue using the script "sh kamctl restart" with several times and it can be reproduced. In my version which has changes in codes, it printed like this:
``` mv7:/tmp/kamailio/sbin# sh kamctl restart
\E[37;33mINFO: Stopping Kamailio : \E[37;33mINFO: stopped
\E[37;33mINFO: Starting Kamailio : 23370 XXX 0:00 ./kamailio -P /var/run/kamailio.pid -f /tmp/kamailio/etc/kamailio//kamailio.cfg 23371 XXX 0:00 ./kamailio -P /var/run/kamailio.pid -f /tmp/kamailio/etc/kamailio//kamailio.cfg 23478 XXX 0:00 /bin/egrep kamailio -rw-r--r-- 1 XXX XXX 6 Feb 15 16:14 /var/run/kamailio.pid \E[37;33mINFO: PID file exists (/var/run/kamailio.pid)! Kamailio already running? ``` The children processes above then terminated after a few seconds later, but the /var/run/kamailio.pid (may be or not)was still there, I had to remove it by myself to make the restart work.
And I also tried to build kamailio with the stock version - with just a few changes to make compiling and execution work, following is what I've done with it:
1. **build the config.mak for ARM**
make cfg include_modules="db_sqlite tls" mode=debug ARCH=armv41 CPU=cortex-a8 PREFIX=/tmp/kamailio 2. **modify the config.mak**
CC= /usr/gcc/bin/arm-marvell-linux-gnueabi-gcc LD= /usr/gcc/bin/arm-marvell-linux-gnueabi-gcc MKDEP= /usr/gcc/bin/arm-marvell-linux-gnueabi-gcc -MM -MG and remove -Wcast-align 3. **make all**
dbase.h:29:21: fatal error: sqlite3.h: No such file or directory => I add sqlite3.h in the modules/db_sqlite/ and change the db_sqlite.h //#include <sqlite3.h> #include "sqlite3.h"
/usr/bin/ld: cannot find -lsqlite3 => so I put the libsqlite3.so in the modules/db_sqlite and change the Makefile in the same directory #LIBS= -L$(LOCALBASE)/lib -lsqlite3 LIBS= -L./ -lsqlite3
After testing with this version, the error still occurs.
I check the unfinished children process by attaching with gdb, both two versions show the same result:
``` #0 0x400650b8 in wait () from /lib/libpthread.so.0 #1 0x000be6f0 in shutdown_children (sig=15, show_status=1) at main.c:708 #2 0x000bf288 in handle_sigs () at main.c:741 #3 0x000ca9bc in main_loop () at main.c:1757 #4 0x000d0830 in main (argc=5, argv=0xbeb82c94) at main.c:2581 ```
``` #0 0x40111a04 in __lll_lock_wait () from /lib/libpthread.so.0 #1 0x4010c0b8 in pthread_mutex_lock () from /lib/libpthread.so.0 #2 0x000bc7e0 in cfg_update_local (no_cbs=1) at cfg/cfg_struct.h:339 #3 0x000c21a4 in sig_usr (signo=15) at main.c:857 #4 <signal handler called> #5 0x40112450 in recvfrom () from /lib/libpthread.so.0 #6 0x00265848 in udp_rcv_loop () at udp_server.c:446 #7 0x000c90a0 in main_loop () at main.c:1629 #8 0x000d0830 in main (argc=5, argv=0xbee68c94) at main.c:2581 ```
or
``` #0 0x4012aa04 in __lll_lock_wait () from /lib/libpthread.so.0 #1 0x401250b8 in pthread_mutex_lock () from /lib/libpthread.so.0 #2 0x000bc7e0 in cfg_update_local (no_cbs=1) at cfg/cfg_struct.h:339 #3 0x000c21a4 in sig_usr (signo=15) at main.c:857 #4 <signal handler called> #5 0x4026fc5c in epoll_wait () from /lib/libc.so.6 #6 0x4109fc14 in io_wait_loop_epoll (h=0x410c9898, t=10, repeat=0) at ../../io_wait.h:1042 #7 0x410a4fc0 in io_listen_loop (fd_no=1, cs_lst=0x4ff960) at io_listener.c:292 #8 0x4107881c in mod_child (rank=0) at ctl.c:333 #9 0x001d83b0 in init_mod_child (m=0x403067c8, rank=0) at sr_module.c:921 #10 0x001d8090 in init_mod_child (m=0x40306ae0, rank=0) at sr_module.c:918 #11 0x001d8090 in init_mod_child (m=0x403078b8, rank=0) at sr_module.c:918 #12 0x001d8090 in init_mod_child (m=0x40307bd8, rank=0) at sr_module.c:918 #13 0x001d8770 in init_child (rank=0) at sr_module.c:947 #14 0x000c9b2c in main_loop () at main.c:1706 #15 0x000d0830 in main (argc=5, argv=0xbe8d3c94) at main.c:2581 ```
Any ideas??
--- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-184139851
Would you be able to test with version 4.4, there were some changes in approaching transaction handling and I wonder if that fixed the initial trace?
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-214176469
Any chance testing it with 4.4?
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-224087188
Okay, I will try it. :)
--- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/466#issuecomment-224140286
Reopen if still and issue with latest stable.
Closed #466.
-
BroWater -
Daniel-Constantin Mierla