<!-- Kamailio Project uses GitHub Issues only for bugs in the code or feature requests. If you have questions about using Kamailio or related to its configuration file, ask on sr-users mailing list: * http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users If you have questions about developing extensions to Kamailio or its existing C code, ask on sr-dev mailing list * http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev Please try to fill this template as much as possible for any issue. It helps the developers to troubleshoot the issue. If you submit a feature request (or enhancement), you can delete the text of the template and only add the description of what you would like to be added. If there is no content to be filled in a section, the entire section can be removed. You can delete the comments from the template sections when filling. You can delete next line and everything above before submitting (it is a comment). --> ### Description We have a Kamailio v4.4.5 Git-Rev. e9d576 running on Debian Wheezy 32bit KVM machine. We are experiencing random crashes since morning and so far Kamailio has crashed at least twice. <!-- Explain what you did, what you expected to happen, and what actually happened. --> ### Troubleshooting #### Reproduction <!-- If the issue can be reproduced, describe how it can be done. --> This issue occurs randomly. We are not able to reproduce it manually in our lab. #### Debugging Data <!-- If you got a core dump, use gdb to extract troubleshooting data - full backtrace, local variables and the list of the code at the issue location. gdb /path/to/kamailio /path/to/corefile bt full info locals list If you are familiar with gdb, feel free to attach more of what you consider to be relevant. --> ``` GNU gdb (GDB) 7.4.1-debian Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/adx-webrtc/sbin/kamailio...done. [New LWP 29499] warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1". Core was generated by `/usr/local/adx-webrtc/sbin/kamailio -f /usr/local/adx-webrtc/etc/kamailio/kamai'. Program terminated with signal 11, Segmentation fault. #0 0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223 1223 textops.c: No such file or directory. (gdb) bt full #0 0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223 time_str = {s = 0x0, len = 37} now = 1490005155 bd_time = 0xb775b6a0 hf = 0xb7117590 anchor = 0xb75f0f9c __FUNCTION__ = "append_time_request_f" #1 0x0806509e in do_action (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1054 ret = -5 v = 0 dst = {send_sock = 0x38, to = {s = {sa_family = 53644, sa_data = "2\bX\323\062\b9\000\000\000\314\317\062\b"}, sin = {sin_family = 53644, sin_port = 2098, sin_addr = { s_addr = 137548632}, sin_zero = "9\000\000\000\314\317\062\b"}, sin6 = {sin6_family = 53644, sin6_port = 2098, sin6_flowinfo = 137548632, sin6_addr = {__in6_u = { __u6_addr8 = "9\000\000\000\314\317\062\b\000\000\000\000\250\002U\377", __u6_addr16 = {57, 0, 53196, 2098, 0, 0, 680, 65365}, __u6_addr32 = {57, 137547724, 0, 4283761320}}}, sin6_scope_id = 1529706238}}, id = -1218478588, proto = -40 '\330', send_flags = {f = 205 '\315', blst_imask = 121 'y'}} tmp = 0x80e1081 "\213E\344\353\032\220\353\001\220\203", <incomplete sequence \340> new_uri = 0x832cfcc "core" end = 0x832d358 "add_lump_rpl2" crt = 0xbfdf8598 "x\217\337\277\"P\006\b|\233\337\277@\311h\266\200\372\246\266\360\304h\266\b\020_\266\360ꑶt" cmd = 0xb6619e00 len = -1230570520 user = -1075863780 uri = {user = {s = 0x20014 <Address 0x20014 out of bounds>, len = 1490002293}, passwd = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, host = { s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871080}, port = {s = 0x823f044 "\211E\364\203", <incomplete sequence \364>, len = 275}, params = { s = 0xb6762891 "uq_saas_domain)", len = 14}, sip_params = { s = 0x844b09c "exen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac(a)lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexen.vpn:48713>\r\nAccept: application/sdp\r\nContent-Length: 0\r\n\r\n", len = 13}, headers = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 642}, port_no = 4104, proto = 46687, type = 2133685845, flags = 196617, transport = {s = 0xbfdf82c8 "\350\257E\b\020", len = 1}, ttl = {s = 0xb65f1008 "\001", len = -1075871068}, user_param = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, maddr = {s = 0xbfdf9f1c "\020", len = -1230570520}, method = {s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135790452}, lr = { s = 0xbfdf9b7c "\003", len = -1234489668}, r2 = {s = 0xb6a6fa80 "\204\256", len = -1245799139}, gr = {s = 0xb6a6fa80 "\204\256", len = -1233770052}, transport_val = { s = 0x845afe8 "", len = 16}, ttl_val = {s = 0xbfdf9f1c "\020", len = -1230570520}, user_param_val = {s = 0xbfdf83b8 "x\204\337\277\353\004\016\b", len = -1245742364}, maddr_val = {s = 0x113 <Address 0x113 out of bounds>, len = -1233770351}, method_val = {s = 0xe <Address 0xe out of bounds>, len = 138719388}, lr_val = { s = 0xd <Address 0xd out of bounds>, len = 332}, r2_val = {s = 0xbfdf83c8 "\376", len = -1234654324}, gr_val = {s = 0xb668a73c "\a", len = 0}} next_hop = {user = {s = 0xbfdf833c "", len = 0}, passwd = {s = 0x0, len = -1230570616}, host = {s = 0xbfdf9f1c "\020", len = -1230570520}, port = { s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135806558}, params = {s = 0xbfdf9b7c "\003", len = -1230570880}, sip_params = {s = 0xbfdf8568 "", len = -1234648760}, headers = {s = 0x0, len = 0}, port_no = 43916, proto = 46696, type = ERROR_URI_T, flags = 262164, transport = {s = 0xb771161f "\201\303\001\255\002", len = -1245289828}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871064}, maddr = { s = 0x8070fc5 "\211E\360\213\225P\377\377\377\213E\b\211P\b\213\205L\377\377\377\200\344\376\211\302\vU\364\213E\b\211P\004\213E\360\311\303U\211\345\201\354", <incomplete sequence \330>, len = -1075871248}, method = {s = 0xb66b2ebc "g\006", len = -1230570880}, lr = {s = 0x60014 <Address 0x60014 out of bounds>, len = 0}, r2 = {s = 0x0, len = 1}, gr = { s = 0xb6a6fb88 "\002", len = -1075863780}, transport_val = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871272}, ttl_val = { s = 0xd1e9220b <Address 0xd1e9220b out of bounds>, len = 1625566052}, user_param_val = {s = 0x0, len = -1075871064}, maddr_val = { s = 0x8169901 "\211E\344", <incomplete sequence \351>, len = -1075864708}, method_val = {s = 0xb676d94c "\002", len = -1230570880}, lr_val = { s = 0x60014 <Address 0x60014 out of bounds>, len = -1494096780}, r2_val = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 628}, gr_val = { s = 0x39 <Address 0x39 out of bounds>, len = 64}} u = 0xb6a6fb88 port = 49119 dst_host = 0x7d0 ---Type <return> to continue, or q <return> to quit--- i = -1230570520 flags = 0 avp = 0xbfdf9f1c st = {flags = 2, id = 42812, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x0} sct = 0x1 sjt = 0x0 rve = 0xb66b6300 mct = 0xb75f3020 rv = 0xb6a6fa80 rv1 = 0x1 c1 = {cache_type = 30, val_type = 3062845320, c = {avp_val = {n = 0, s = {s = 0x0, len = -65536}, re = 0x0}, pval = {rs = {s = 0x0, len = -65536}, ri = -1217329915, flags = -1230570616}}, i2s = "\210\373\246\266\034\237\337\277\350\373\246\266\230\201\337\277%;\030\b\000"} s = {s = 0x8290229 "\205\300\017\211^\002", len = 138719371} srevp = {0xb6a6fbe8, 0xbfdf8198} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #2 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1549 t = 0xb668c940 ret = -1 mod = 0x25 ms = 0 __FUNCTION__ = "run_actions" #3 0x08065022 in do_action (h=0xbfdf9b7c, a=0xb668c9dc, msg=0xb6a6fa80) at action.c:1045 ret = 1 v = 1 dst = {send_sock = 0x20, to = {s = {sa_family = 42387, sa_data = ":\bC\304:\b\316\t\000\000\370\207:\b"}, sin = {sin_family = 42387, sin_port = 2106, sin_addr = { s_addr = 138069059}, sin_zero = "\316\t\000\000\370\207:\b"}, sin6 = {sin6_family = 42387, sin6_port = 2106, sin6_flowinfo = 138069059, sin6_addr = {__in6_u = { __u6_addr8 = "\316\t\000\000\370\207:\b\034\216\337\277\b\020_\266", __u6_addr16 = {2510, 0, 34808, 2106, 36380, 49119, 4104, 46687}, __u6_addr32 = {2510, 138053624, 3219099164, 3059683336}}}, sin6_scope_id = 8}}, id = -1217179589, proto = 1 '\001', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}} tmp = 0x1e <Address 0x1e out of bounds> new_uri = 0xb68f8c10 "\001" end = 0xbfdf9486 "D\b\244J\217\266\234bƵ\034\237\337\277\350\373\246\266\b\227\337\277\032\267&\bD]9\b\004" crt = 0x83475ac "(): memory allocation failure\n" cmd = 0x5 len = 4 user = 0 uri = {user = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075868328}, passwd = {s = 0x80c5317 "\211E\364\213E\364\351\373\001", len = 0}, host = { s = 0xb7710505 "\201Û\265\002", len = -1241058888}, port = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868284}, params = {s = 0x0, len = 28}, sip_params = {s = 0x0, len = 70696791}, headers = {s = 0x8 <Address 0x8 out of bounds>, len = -1075868120}, port_no = 4104, proto = 46687, type = URN_URI_T, flags = 0, transport = {s = 0xbfdf8da8 "؍\337\277\027S\f\b", len = -1241075329}, ttl = {s = 0xb65f1008 "\001", len = -1075868284}, user_param = {s = 0x8398904 "core", len = 0}, maddr = {s = 0xb773417d "(fwd->size & 0x4) == 0", len = 16777216}, method = {s = 0x364 <Address 0x364 out of bounds>, len = 8110980}, lr = { ---Type <return> to continue, or q <return> to quit--- s = 0xfffc9c <Address 0xfffc9c out of bounds>, len = 16777216}, r2 = {s = 0x4 <Address 0x4 out of bounds>, len = 13}, gr = {s = 0x0, len = -1075863780}, transport_val = { s = 0xbfdf8dd8 "\250\216\337\277\203\361$\b\005", len = 135025431}, ttl_val = {s = 0x0, len = -1230570520}, user_param_val = { s = 0xb76624e3 "\213U\344\003u\360\211B\024\353\257f\220\213E\020)\370\203\304\034[^_]Ív", len = 157513914}, maddr_val = {s = 0x838c064 "); Free fragment not found!\n", len = 256}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = 0}, lr_val = {s = 0xbfdf9f1c "\020", len = 0}, r2_val = { s = 0xbfdf8ea8 "\230\225\337\277\356\331+\b.\260D\b\005", len = 136638851}, gr_val = {s = 0x5 <Address 0x5 out of bounds>, len = 0}} next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0xbfdf92a8 "", len = 0}, host = {s = 0x0, len = 248}, port = {s = 0x2 <Address 0x2 out of bounds>, len = 136638062}, params = {s = 0x0, len = -1}, sip_params = {s = 0x14 <Address 0x14 out of bounds>, len = -1253735563}, headers = {s = 0x0, len = -1253660952}, port_no = 65535, proto = 65535, type = ERROR_URI_T, flags = 2772135936, transport = {s = 0x35 <Address 0x35 out of bounds>, len = -1514431728}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = { s = 0xb7710505 "\201Û\265\002", len = -1241058888}, maddr = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868412}, method = {s = 0x0, len = 28}, lr = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, r2 = {s = 0x0, len = -1}, gr = {s = 0xb65f1008 "\001", len = -1075863780}, transport_val = { s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075868376}, ttl_val = {s = 0xb606b17f "\213\223,\b", len = -1235283960}, user_param_val = {s = 0xbfdf8d04 "", len = 0}, maddr_val = {s = 0xb65c5ea8 "U\211\345WVS\203\354|\350%\327\377\377\201\303ަ\002", len = -1235450529}, method_val = {s = 0x1000000 <Address 0x1000000 out of bounds>, len = 72}, lr_val = {s = 0x7bc6a0 <Address 0x7bc6a0 out of bounds>, len = 16777144}, r2_val = {s = 0x1000000 <Address 0x1000000 out of bounds>, len = 4}, gr_val = { s = 0x1 <Address 0x1 out of bounds>, len = -1252044316}} u = 0x1 port = 46963 dst_host = 0x83475ca i = -1 flags = 10 avp = 0xffffffff st = {flags = 3219098728, id = 35944, name = {n = -1249644216, s = {s = 0xb583f148 "\f \001", len = 0}, re = 0xb583f148}, avp = 0xbfdf90bf} sct = 0xb7757ff4 sjt = 0x1b rve = 0xb668c4f0 mct = 0x963771c rv = 0xb583d328 rv1 = 0xb62d0564 c1 = {cache_type = 3077930996, val_type = 3077936064, c = {avp_val = {n = 157485576, s = {s = 0x9630a08 "71.234", len = -1075868616}, re = 0x9630a08}, pval = {rs = { s = 0x9630a08 "71.234", len = -1075868616}, ri = -1, flags = 0}}, i2s = '\000' <repeats 20 times>, " \n"} s = {s = 0xb581a8d8 "\205\300u\f\213D$\034\203\304([Ív", len = -1217031232} srevp = {0xbfdfa3ac, 0x21} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #4 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb6687218, msg=0xb6a6fa80) at action.c:1549 t = 0xb668c9dc ret = 1 mod = 0xbfdf9618 ms = 0 __FUNCTION__ = "run_actions" #5 0x08061d7e in do_action (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:678 ret = -5 ---Type <return> to continue, or q <return> to quit--- v = 0 dst = {send_sock = 0x11a, to = {s = {sa_family = 0, sa_data = "\000\000z\001\000\000\000\000\000\000\n\001\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 378}, sin_zero = "\000\000\000\000\n\001\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 378, sin6_addr = {__in6_u = { __u6_addr8 = "\000\000\000\000\n\001\000\000\000\000\000\000\005\000\000", __u6_addr16 = {0, 0, 266, 0, 0, 0, 5, 0}, __u6_addr32 = {0, 266, 0, 5}}}, sin6_scope_id = 0}}, id = 346, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}} tmp = 0x844b006 "Via: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguar"... new_uri = 0xb7710505 "\201Û\265\002" end = 0xbfdf9aa4 "" crt = 0xb68f5120 " " cmd = 0xbfdf9f1c len = 0 user = 8 uri = {user = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075864952}, passwd = {s = 0x8270416 "\203\370\002\017\216.\002", len = 137977156}, host = { s = 0x4 <Address 0x4 out of bounds>, len = -1223592560}, port = {s = 0x83988e7 "parse_headers", len = 318}, params = {s = 0x8395d44 "core", len = -1075865496}, sip_params = { s = 0x10 <Address 0x10 out of bounds>, len = 70696791}, headers = {s = 0xbfdf99dd "\261\006\266\b\020_\266\364\231\337\277h", len = -1075865496}, port_no = 20964, proto = 46431, type = SIPS_URI_T, flags = 0, transport = {s = 0x4001fa <Address 0x4001fa out of bounds>, len = 0}, ttl = {s = 0x0, len = 138807552}, user_param = { s = 0x16 <Address 0x16 out of bounds>, len = -1252044316}, maddr = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075865175}, method = { s = 0xbfdf9a38 " Q\217\266\006\260D\b\034\237\337\277\001", len = -1252118708}, lr = {s = 0xbfdf9934 "\350\373\246\266\210\232\337\277/\352(\b\376\257D\b\365\337\071\b\006", len = 0}, r2 = {s = 0xb <Address 0xb out of bounds>, len = -1075865288}, gr = {s = 0xbfdf9940 "\376\257D\b\365\337\071\b\006", len = -1218461980}, transport_val = { s = 0xd696910 <Address 0xd696910 out of bounds>, len = -1075865404}, ttl_val = {s = 0xb778cd76 "\213M\340\205\300u3\211\360\213]\364\213u\370\213}\374\211\354]Ít&", len = -1218418128}, user_param_val = {s = 0x4011fa <Address 0x4011fa out of bounds>, len = 0}, maddr_val = {s = 0x8054fde "libc.so.6", len = -1218498528}, method_val = { s = 0xbfdf0002 "", len = -1}, lr_val = {s = 0x8054fde "libc.so.6", len = -1216865232}, r2_val = {s = 0x102 <Address 0x102 out of bounds>, len = 0}, gr_val = { s = 0x1fa <Address 0x1fa out of bounds>, len = 0}} next_hop = {user = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = -1}, passwd = {s = 0x20 <Address 0x20 out of bounds>, len = 137978236}, host = { s = 0x83988e7 "parse_headers", len = 318}, port = {s = 0x8395d44 "core", len = 106}, params = {s = 0xb622d9a4 "p\330\002", len = 59}, sip_params = { s = 0x8401940 "{1 1 OPTIONS 76rjac(a)lifeguard.andrexen.vpn} ", len = 137565758}, headers = {s = 0xb65f1008 "\001", len = 0}, port_no = 47, proto = 0, type = 59, flags = 138418496, transport = {s = 0x833163e "ERROR", len = -1232122012}, ttl = {s = 0x44b1a6 <Address 0x44b1a6 out of bounds>, len = 88}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 1}, gr = {s = 0xb55ddf58 "\201Ìr\001", len = 0}, transport_val = { s = 0x0, len = -1}, ttl_val = {s = 0x0, len = -1514551080}, user_param_val = {s = 0x21 <Address 0x21 out of bounds>, len = 3}, maddr_val = {s = 0xb68f8c10 "\001", len = -1235283960}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = -1223592596}, lr_val = { s = 0x844b04f "Max-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac(a)lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexe"..., len = -1075865292}, r2_val = {s = 0x1c8 <Address 0x1c8 out of bounds>, len = 0}, gr_val = {s = 0x0, len = -1075863780}} u = 0x0 port = 0 dst_host = 0xfffc88 i = 4 flags = 136638851 avp = 0x0 st = {flags = 8, id = 0, name = {n = -1075863780, s = {s = 0xbfdf9f1c "\020", len = 0}, re = 0xbfdf9f1c}, avp = 0xbfdf9808} sct = 0xffffffff sjt = 0x30 rve = 0x2 mct = 0xa rv = 0x8395d44 rv1 = 0x0 ---Type <return> to continue, or q <return> to quit--- c1 = {cache_type = 16777216, val_type = RV_ACTION_ST, c = {avp_val = {n = -1245289828, s = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, re = 0xb5c6629c}, pval = {rs = { s = 0xb5c6629c "\340\060\t", len = -1075863780}, ri = -1230570520, flags = -1075865208}}, i2s = "\324\365&\b\b\020_\266\270K\217\266|a9\b\347\210\071\bN\001"} s = {s = 0xb65f1008 "\001", len = 16776200} srevp = {0x8, 0xb68f4b94} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __FUNCTION__ = "do_action" #6 0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:1549 t = 0xb66340c4 ret = -1 mod = 0x0 ms = 0 __FUNCTION__ = "run_actions" #7 0x0807105d in run_top_route (a=0xb66340c4, msg=0xb6a6fa80, c=0x0) at action.c:1635 ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {-1230570616, -1075863780, -1230570520, -1075864744, -774102517, 1625566052}, __mask_was_saved = 0, __saved_mask = {__val = {4, 3219102888, 3064396680, 3071374700, 3064396776, 3053849578, 3053849578, 3053908408, 3219103516, 3219102696, 3053849925, 3219062798, 48, 0, 3053908408, 3219103516, 3064396776, 3219102728, 3053850835, 138719229, 3053894780, 7, 1, 3219103516, 1, 3219102776, 135912396, 3064396416, 1073741825, 0, 134977546, 3064396416}}}}} p = 0xbfdf9b7c ret = 48 sfbk = 0 #8 0x081479fc in receive_msg ( buf=0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len=371, rcv_info=0xbfdf9ebc) at receive.c:240 msg = 0xb6a6fa80 ctx = {rec_lev = 5, run_flags = -1218441964, last_retcode = -1216865128, jmp_env = {{__jmpbuf = {-1216734968, -1, -1216737292, 134567086, 5, -1075864192}, __mask_was_saved = -1216799770, __saved_mask = {__val = {3078232768, 3076468768, 1, 5, 0, 5, 134565684, 138392384, 0, 3076525332, 961078559, 0, 3219104684, 3219103256, 3078191504, 3219103428, 3077456128, 33200, 3077641759, 961078559, 136537889, 18878, 3062860384, 4, 136537807, 2772170560, 4294967295, 3078230004, 134566635, 5, 3219103232, 3078167526}}}}} ret = 0 inb = { s = 0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len = 371} netinfo = {data = {s = 0xbfdf9db4 "\377\377\377\377\364\017z\267\353R\005\b\005", len = 134537572}, dst = 0xbfdf9da8, rcv = 0xb77a1a64} __FUNCTION__ = "receive_msg" #9 0x0823da26 in udp_rcv_loop () at udp_server.c:495 len = 371 buf = "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"... tmp = 0x8329042 "counters_prefork_init" from = 0xb68f8a5c fromlen = 16 ---Type <return> to continue, or q <return> to quit--- ri = {src_ip = {af = 2, len = 4, u = {addrl = {1378711125, 4, 961078559, 1}, addr32 = {1378711125, 4, 961078559, 1}, addr16 = {30293, 21037, 4, 0, 58655, 14664, 1, 0}, addr = "Uv-R\004\000\000\000\037\345H9\001\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1529706069, 0, 0, 0}, addr32 = {1529706069, 0, 0, 0}, addr16 = {30293, 23341, 0, 0, 0, 0, 0, 0}, addr = "Uv-[", '\000' <repeats 11 times>}}, src_port = 48713, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = { sa_family = 2, sa_data = "\276IUv-R\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 18878, sin_addr = {s_addr = 1378711125}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 18878, sin6_flowinfo = 1378711125, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0xb6611b94, proto = 1 '\001'} __FUNCTION__ = "udp_rcv_loop" #10 0x080eaf7b in main_loop () at main.c:1614 i = 2 pid = 0 si = 0xb6611b94 si_desc = "udp receiver child=2 sock=85.118.45.91:5080\000̛\206\245̛\206\245\000\000\000\000 \v\353\265\364\317\344\265\037\345H9\001\000\000\000\037\345H9\370\240\337\277\303I%\bHF\222\266\004\000\000\000\071x4\b\204\211\064\bn\000\000\000\064x4\b\024\000\000\000\037\345H9", '\000' <repeats 11 times> nrprocs = 4 woneinit = 1 __FUNCTION__ = "main_loop" #11 0x080f23d4 in main (argc=13, argv=0xbfdfa524) at main.c:2631 cfg_stream = 0x90c2008 c = -1 r = 0 tmp = 0xbfdfbd8f "" tmp_len = 1 port = 0 proto = 0 options = 0x833deb4 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 2912300657 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfdfa450 p = 0x805c9c8 "[\201\303H\347\071" st = {st_dev = 14, __pad1 = 0, st_ino = 6001, st_mode = 16832, st_nlink = 2, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1487716504, tv_nsec = 24863882}, st_mtim = {tv_sec = 1490002293, tv_nsec = 141472099}, st_ctim = {tv_sec = 1490002293, tv_nsec = 141472099}, __unused4 = 0, __unused5 = 0} __FUNCTION__ = "main" ``` #### Log Messages <!-- Check the syslog file and if there are relevant log messages printed by Kamailio, add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> Here are Kamailio logs in time DESC order. ``` Mar 20 12:02:15 webrtc-as kernel: [2291290.426195] kamailio[2963]: segfault at 23 ip b5a0be6b sp bfca2950 error 6 in textops.so[b59fb000+40000] Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 64) called from textops: textops.c: append_time_request_f(1216), module: textops; Free fragment not found! Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 64); Free fragment not found! Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found! Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found! Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found! Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es(a)lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found! ... Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1(a)172.16.171.16} ERROR: siptrace [siptrace.c:1603]: trace_send_duplicate(): bad host name in uri Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1(a)172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1(a)172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found! Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1(a)172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found! ... Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47(a)172.16.171.16} ERROR: siptrace [siptrace.c:1614]: trace_send_duplicate(): bad host name in uri Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47(a)172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47(a)172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found! Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47(a)172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found! ``` #### SIP Traffic <!-- If the issue is exposed by processing specific SIP messages, grab them with ngrep or save in a pcap file, then add them next, or attach to issue, or provide a link to download them (e.g., to a pastebin site). --> ``` No Data Available ``` ### Possible Solutions <!-- If you found a solution or workaround for the issue, describe it. Ideally, provide a pull request with a fix. --> Currently no idea. However one strange thing is that although siptrace module is load by kamailio but none of its functions are called or flags are set, i wonder why it shows up in kamailio log in the first place. ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 4.4.5 (i386/linux) e9d576 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, select. id: e9d576 -dirty compiled on 06:32:04 Jan 25 2017 with gcc 4.7.2 ``` * **Operating System**: <!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `uname -a`) --> ``` Linux webrtc-as 3.10-0.bpo.2-686-pae #1 SMP Debian 3.10.5-1~bpo70+1 (2013-08-11) i686 GNU/Linux ``` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1040