Description

We have a Kamailio v4.4.5 Git-Rev. e9d576 running on Debian Wheezy 32bit KVM machine. We are experiencing random crashes since morning and so far Kamailio has crashed at least twice.

Troubleshooting

Reproduction

This issue occurs randomly. We are not able to reproduce it manually in our lab.

Debugging Data

GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/adx-webrtc/sbin/kamailio...done.
[New LWP 29499]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Core was generated by `/usr/local/adx-webrtc/sbin/kamailio -f /usr/local/adx-webrtc/etc/kamailio/kamai'.
Program terminated with signal 11, Segmentation fault.
#0  0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223
1223	textops.c: No such file or directory.
(gdb) bt full
#0  0xb5a91e6b in append_time_request_f (msg=0xb6a6fa80, p1=0x0, p2=0x0) at textops.c:1223
        time_str = {s = 0x0, len = 37}
        now = 1490005155
        bd_time = 0xb775b6a0
        hf = 0xb7117590
        anchor = 0xb75f0f9c
        __FUNCTION__ = "append_time_request_f"
#1  0x0806509e in do_action (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1054
        ret = -5
        v = 0
        dst = {send_sock = 0x38, to = {s = {sa_family = 53644, sa_data = "2\bX\323\062\b9\000\000\000\314\317\062\b"}, sin = {sin_family = 53644, sin_port = 2098, sin_addr = {
                s_addr = 137548632}, sin_zero = "9\000\000\000\314\317\062\b"}, sin6 = {sin6_family = 53644, sin6_port = 2098, sin6_flowinfo = 137548632, sin6_addr = {__in6_u = {
                  __u6_addr8 = "9\000\000\000\314\317\062\b\000\000\000\000\250\002U\377", __u6_addr16 = {57, 0, 53196, 2098, 0, 0, 680, 65365}, __u6_addr32 = {57, 137547724, 0, 
                    4283761320}}}, sin6_scope_id = 1529706238}}, id = -1218478588, proto = -40 '\330', send_flags = {f = 205 '\315', blst_imask = 121 'y'}}
        tmp = 0x80e1081 "\213E\344\353\032\220\353\001\220\203", <incomplete sequence \340>
        new_uri = 0x832cfcc "core"
        end = 0x832d358 "add_lump_rpl2"
        crt = 0xbfdf8598 "x\217\337\277\"P\006\b|\233\337\277@\311h\266\200\372\246\266\360\304h\266\b\020_\266\360ꑶt"
        cmd = 0xb6619e00
        len = -1230570520
        user = -1075863780
        uri = {user = {s = 0x20014 <Address 0x20014 out of bounds>, len = 1490002293}, passwd = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, host = {
            s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871080}, port = {s = 0x823f044 "\211E\364\203", <incomplete sequence \364>, len = 275}, params = {
            s = 0xb6762891 "uq_saas_domain)", len = 14}, sip_params = {
            s = 0x844b09c "exen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexen.vpn:48713>\r\nAccept: application/sdp\r\nContent-Length: 0\r\n\r\n", len = 13}, headers = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 642}, port_no = 4104, proto = 46687, type = 2133685845, 
          flags = 196617, transport = {s = 0xbfdf82c8 "\350\257E\b\020", len = 1}, ttl = {s = 0xb65f1008 "\001", len = -1075871068}, user_param = {s = 0x1 <Address 0x1 out of bounds>, 
            len = 0}, maddr = {s = 0xbfdf9f1c "\020", len = -1230570520}, method = {s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135790452}, lr = {
            s = 0xbfdf9b7c "\003", len = -1234489668}, r2 = {s = 0xb6a6fa80 "\204\256", len = -1245799139}, gr = {s = 0xb6a6fa80 "\204\256", len = -1233770052}, transport_val = {
            s = 0x845afe8 "", len = 16}, ttl_val = {s = 0xbfdf9f1c "\020", len = -1230570520}, user_param_val = {s = 0xbfdf83b8 "x\204\337\277\353\004\016\b", len = -1245742364}, 
          maddr_val = {s = 0x113 <Address 0x113 out of bounds>, len = -1233770351}, method_val = {s = 0xe <Address 0xe out of bounds>, len = 138719388}, lr_val = {
            s = 0xd <Address 0xd out of bounds>, len = 332}, r2_val = {s = 0xbfdf83c8 "\376", len = -1234654324}, gr_val = {s = 0xb668a73c "\a", len = 0}}
        next_hop = {user = {s = 0xbfdf833c "", len = 0}, passwd = {s = 0x0, len = -1230570616}, host = {s = 0xbfdf9f1c "\020", len = -1230570520}, port = {
            s = 0xbfdf8398 "\230\205\337\277\273B\030\b\020\216\337\277\004", len = 135806558}, params = {s = 0xbfdf9b7c "\003", len = -1230570880}, sip_params = {s = 0xbfdf8568 "", 
            len = -1234648760}, headers = {s = 0x0, len = 0}, port_no = 43916, proto = 46696, type = ERROR_URI_T, flags = 262164, transport = {s = 0xb771161f "\201\303\001\255\002", 
            len = -1245289828}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871064}, maddr = {
            s = 0x8070fc5 "\211E\360\213\225P\377\377\377\213E\b\211P\b\213\205L\377\377\377\200\344\376\211\302\vU\364\213E\b\211P\004\213E\360\311\303U\211\345\201\354", <incomplete sequence \330>, len = -1075871248}, method = {s = 0xb66b2ebc "g\006", len = -1230570880}, lr = {s = 0x60014 <Address 0x60014 out of bounds>, len = 0}, r2 = {s = 0x0, len = 1}, gr = {
            s = 0xb6a6fb88 "\002", len = -1075863780}, transport_val = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075871272}, ttl_val = {
            s = 0xd1e9220b <Address 0xd1e9220b out of bounds>, len = 1625566052}, user_param_val = {s = 0x0, len = -1075871064}, maddr_val = {
            s = 0x8169901 "\211E\344", <incomplete sequence \351>, len = -1075864708}, method_val = {s = 0xb676d94c "\002", len = -1230570880}, lr_val = {
            s = 0x60014 <Address 0x60014 out of bounds>, len = -1494096780}, r2_val = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = 628}, gr_val = {
            s = 0x39 <Address 0x39 out of bounds>, len = 64}}
        u = 0xb6a6fb88
        port = 49119
        dst_host = 0x7d0
---Type <return> to continue, or q <return> to quit---
        i = -1230570520
        flags = 0
        avp = 0xbfdf9f1c
        st = {flags = 2, id = 42812, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x0}
        sct = 0x1
        sjt = 0x0
        rve = 0xb66b6300
        mct = 0xb75f3020
        rv = 0xb6a6fa80
        rv1 = 0x1
        c1 = {cache_type = 30, val_type = 3062845320, c = {avp_val = {n = 0, s = {s = 0x0, len = -65536}, re = 0x0}, pval = {rs = {s = 0x0, len = -65536}, ri = -1217329915, 
              flags = -1230570616}}, i2s = "\210\373\246\266\034\237\337\277\350\373\246\266\230\201\337\277%;\030\b\000"}
        s = {s = 0x8290229 "\205\300\017\211^\002", len = 138719371}
        srevp = {0xb6a6fbe8, 0xbfdf8198}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, 
              string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, 
              attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
              number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#2  0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb668c940, msg=0xb6a6fa80) at action.c:1549
        t = 0xb668c940
        ret = -1
        mod = 0x25
        ms = 0
        __FUNCTION__ = "run_actions"
#3  0x08065022 in do_action (h=0xbfdf9b7c, a=0xb668c9dc, msg=0xb6a6fa80) at action.c:1045
        ret = 1
        v = 1
        dst = {send_sock = 0x20, to = {s = {sa_family = 42387, sa_data = ":\bC\304:\b\316\t\000\000\370\207:\b"}, sin = {sin_family = 42387, sin_port = 2106, sin_addr = {
                s_addr = 138069059}, sin_zero = "\316\t\000\000\370\207:\b"}, sin6 = {sin6_family = 42387, sin6_port = 2106, sin6_flowinfo = 138069059, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\316\t\000\000\370\207:\b\034\216\337\277\b\020_\266", __u6_addr16 = {2510, 0, 34808, 2106, 36380, 49119, 4104, 46687}, __u6_addr32 = {2510, 138053624, 
                    3219099164, 3059683336}}}, sin6_scope_id = 8}}, id = -1217179589, proto = 1 '\001', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
        tmp = 0x1e <Address 0x1e out of bounds>
        new_uri = 0xb68f8c10 "\001"
        end = 0xbfdf9486 "D\b\244J\217\266\234bƵ\034\237\337\277\350\373\246\266\b\227\337\277\032\267&\bD]9\b\004"
        crt = 0x83475ac "(): memory allocation failure\n"
        cmd = 0x5
        len = 4
        user = 0
        uri = {user = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075868328}, passwd = {s = 0x80c5317 "\211E\364\213E\364\351\373\001", len = 0}, host = {
            s = 0xb7710505 "\201Û\265\002", len = -1241058888}, port = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868284}, params = {s = 0x0, 
            len = 28}, sip_params = {s = 0x0, len = 70696791}, headers = {s = 0x8 <Address 0x8 out of bounds>, len = -1075868120}, port_no = 4104, proto = 46687, type = URN_URI_T, 
          flags = 0, transport = {s = 0xbfdf8da8 "؍\337\277\027S\f\b", len = -1241075329}, ttl = {s = 0xb65f1008 "\001", len = -1075868284}, user_param = {s = 0x8398904 "core", 
            len = 0}, maddr = {s = 0xb773417d "(fwd->size & 0x4) == 0", len = 16777216}, method = {s = 0x364 <Address 0x364 out of bounds>, len = 8110980}, lr = {
---Type <return> to continue, or q <return> to quit---
            s = 0xfffc9c <Address 0xfffc9c out of bounds>, len = 16777216}, r2 = {s = 0x4 <Address 0x4 out of bounds>, len = 13}, gr = {s = 0x0, len = -1075863780}, transport_val = {
            s = 0xbfdf8dd8 "\250\216\337\277\203\361$\b\005", len = 135025431}, ttl_val = {s = 0x0, len = -1230570520}, user_param_val = {
            s = 0xb76624e3 "\213U\344\003u\360\211B\024\353\257f\220\213E\020)\370\203\304\034[^_]Ív", len = 157513914}, maddr_val = {s = 0x838c064 "); Free fragment not found!\n", 
            len = 256}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = 0}, lr_val = {s = 0xbfdf9f1c "\020", len = 0}, r2_val = {
            s = 0xbfdf8ea8 "\230\225\337\277\356\331+\b.\260D\b\005", len = 136638851}, gr_val = {s = 0x5 <Address 0x5 out of bounds>, len = 0}}
        next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0xbfdf92a8 "", len = 0}, host = {s = 0x0, len = 248}, port = {s = 0x2 <Address 0x2 out of bounds>, len = 136638062}, 
          params = {s = 0x0, len = -1}, sip_params = {s = 0x14 <Address 0x14 out of bounds>, len = -1253735563}, headers = {s = 0x0, len = -1253660952}, port_no = 65535, proto = 65535, 
          type = ERROR_URI_T, flags = 2772135936, transport = {s = 0x35 <Address 0x35 out of bounds>, len = -1514431728}, ttl = {s = 0xb6a6fb88 "\002", len = -1075863780}, user_param = {
            s = 0xb7710505 "\201Û\265\002", len = -1241058888}, maddr = {s = 0x8253671 "\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", len = -1075868412}, method = {s = 0x0, 
            len = 28}, lr = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, r2 = {s = 0x0, len = -1}, gr = {s = 0xb65f1008 "\001", len = -1075863780}, transport_val = {
            s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075868376}, ttl_val = {s = 0xb606b17f "\213\223,\b", len = -1235283960}, user_param_val = {s = 0xbfdf8d04 "", len = 0}, 
          maddr_val = {s = 0xb65c5ea8 "U\211\345WVS\203\354|\350%\327\377\377\201\303ަ\002", len = -1235450529}, method_val = {s = 0x1000000 <Address 0x1000000 out of bounds>, 
            len = 72}, lr_val = {s = 0x7bc6a0 <Address 0x7bc6a0 out of bounds>, len = 16777144}, r2_val = {s = 0x1000000 <Address 0x1000000 out of bounds>, len = 4}, gr_val = {
            s = 0x1 <Address 0x1 out of bounds>, len = -1252044316}}
        u = 0x1
        port = 46963
        dst_host = 0x83475ca
        i = -1
        flags = 10
        avp = 0xffffffff
        st = {flags = 3219098728, id = 35944, name = {n = -1249644216, s = {s = 0xb583f148 "\f \001", len = 0}, re = 0xb583f148}, avp = 0xbfdf90bf}
        sct = 0xb7757ff4
        sjt = 0x1b
        rve = 0xb668c4f0
        mct = 0x963771c
        rv = 0xb583d328
        rv1 = 0xb62d0564
        c1 = {cache_type = 3077930996, val_type = 3077936064, c = {avp_val = {n = 157485576, s = {s = 0x9630a08 "71.234", len = -1075868616}, re = 0x9630a08}, pval = {rs = {
                s = 0x9630a08 "71.234", len = -1075868616}, ri = -1, flags = 0}}, i2s = '\000' <repeats 20 times>, " \n"}
        s = {s = 0xb581a8d8 "\205\300u\f\213D$\034\203\304([Ív", len = -1217031232}
        srevp = {0xbfdfa3ac, 0x21}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, 
              string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, 
              attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
              number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#4  0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb6687218, msg=0xb6a6fa80) at action.c:1549
        t = 0xb668c9dc
        ret = 1
        mod = 0xbfdf9618
        ms = 0
        __FUNCTION__ = "run_actions"
#5  0x08061d7e in do_action (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:678
        ret = -5
---Type <return> to continue, or q <return> to quit---
        v = 0
        dst = {send_sock = 0x11a, to = {s = {sa_family = 0, sa_data = "\000\000z\001\000\000\000\000\000\000\n\001\000"}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 378}, 
              sin_zero = "\000\000\000\000\n\001\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 378, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\000\000\000\000\n\001\000\000\000\000\000\000\005\000\000", __u6_addr16 = {0, 0, 266, 0, 0, 0, 5, 0}, __u6_addr32 = {0, 266, 0, 5}}}, 
              sin6_scope_id = 0}}, id = 346, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
        tmp = 0x844b006 "Via: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguar"...
        new_uri = 0xb7710505 "\201Û\265\002"
        end = 0xbfdf9aa4 ""
        crt = 0xb68f5120 " "
        cmd = 0xbfdf9f1c
        len = 0
        user = 8
        uri = {user = {s = 0xb6a6fbe8 "\340\257D\bs\001", len = -1075864952}, passwd = {s = 0x8270416 "\203\370\002\017\216.\002", len = 137977156}, host = {
            s = 0x4 <Address 0x4 out of bounds>, len = -1223592560}, port = {s = 0x83988e7 "parse_headers", len = 318}, params = {s = 0x8395d44 "core", len = -1075865496}, sip_params = {
            s = 0x10 <Address 0x10 out of bounds>, len = 70696791}, headers = {s = 0xbfdf99dd "\261\006\266\b\020_\266\364\231\337\277h", len = -1075865496}, port_no = 20964, 
          proto = 46431, type = SIPS_URI_T, flags = 0, transport = {s = 0x4001fa <Address 0x4001fa out of bounds>, len = 0}, ttl = {s = 0x0, len = 138807552}, user_param = {
            s = 0x16 <Address 0x16 out of bounds>, len = -1252044316}, maddr = {s = 0x436bf57 <Address 0x436bf57 out of bounds>, len = -1075865175}, method = {
            s = 0xbfdf9a38 " Q\217\266\006\260D\b\034\237\337\277\001", len = -1252118708}, lr = {s = 0xbfdf9934 "\350\373\246\266\210\232\337\277/\352(\b\376\257D\b\365\337\071\b\006", 
            len = 0}, r2 = {s = 0xb <Address 0xb out of bounds>, len = -1075865288}, gr = {s = 0xbfdf9940 "\376\257D\b\365\337\071\b\006", len = -1218461980}, transport_val = {
            s = 0xd696910 <Address 0xd696910 out of bounds>, len = -1075865404}, ttl_val = {s = 0xb778cd76 "\213M\340\205\300u3\211\360\213]\364\213u\370\213}\374\211\354]Ít&", 
            len = -1218418128}, user_param_val = {s = 0x4011fa <Address 0x4011fa out of bounds>, len = 0}, maddr_val = {s = 0x8054fde "libc.so.6", len = -1218498528}, method_val = {
            s = 0xbfdf0002 "", len = -1}, lr_val = {s = 0x8054fde "libc.so.6", len = -1216865232}, r2_val = {s = 0x102 <Address 0x102 out of bounds>, len = 0}, gr_val = {
            s = 0x1fa <Address 0x1fa out of bounds>, len = 0}}
        next_hop = {user = {s = 0xffffffff <Address 0xffffffff out of bounds>, len = -1}, passwd = {s = 0x20 <Address 0x20 out of bounds>, len = 137978236}, host = {
            s = 0x83988e7 "parse_headers", len = 318}, port = {s = 0x8395d44 "core", len = 106}, params = {s = 0xb622d9a4 "p\330\002", len = 59}, sip_params = {
            s = 0x8401940 "{1 1 OPTIONS 76rjac@lifeguard.andrexen.vpn} ", len = 137565758}, headers = {s = 0xb65f1008 "\001", len = 0}, port_no = 47, proto = 0, type = 59, 
          flags = 138418496, transport = {s = 0x833163e "ERROR", len = -1232122012}, ttl = {s = 0x44b1a6 <Address 0x44b1a6 out of bounds>, len = 88}, user_param = {s = 0x0, len = 0}, 
          maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 1}, gr = {s = 0xb55ddf58 "\201Ìr\001", len = 0}, transport_val = {
            s = 0x0, len = -1}, ttl_val = {s = 0x0, len = -1514551080}, user_param_val = {s = 0x21 <Address 0x21 out of bounds>, len = 3}, maddr_val = {s = 0xb68f8c10 "\001", 
            len = -1235283960}, method_val = {s = 0x8 <Address 0x8 out of bounds>, len = -1223592596}, lr_val = {
            s = 0x844b04f "Max-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:5080;tag=5nwi2d\r\nCall-ID: 76rjac@lifeguard.andrexen.vpn\r\nCSeq: 1 OPTIONS\r\nContact: <sip:checksip@lifeguard.andrexe"..., len = -1075865292}, r2_val = {s = 0x1c8 <Address 0x1c8 out of bounds>, len = 0}, gr_val = {s = 0x0, len = -1075863780}}
        u = 0x0
        port = 0
        dst_host = 0xfffc88
        i = 4
        flags = 136638851
        avp = 0x0
        st = {flags = 8, id = 0, name = {n = -1075863780, s = {s = 0xbfdf9f1c "\020", len = 0}, re = 0xbfdf9f1c}, avp = 0xbfdf9808}
        sct = 0xffffffff
        sjt = 0x30
        rve = 0x2
        mct = 0xa
        rv = 0x8395d44
        rv1 = 0x0
---Type <return> to continue, or q <return> to quit---
        c1 = {cache_type = 16777216, val_type = RV_ACTION_ST, c = {avp_val = {n = -1245289828, s = {s = 0xb5c6629c "\340\060\t", len = -1075863780}, re = 0xb5c6629c}, pval = {rs = {
                s = 0xb5c6629c "\340\060\t", len = -1075863780}, ri = -1230570520, flags = -1075865208}}, i2s = "\324\365&\b\b\020_\266\270K\217\266|a9\b\347\210\071\bN\001"}
        s = {s = 0xb65f1008 "\001", len = 16776200}
        srevp = {0x8, 0xb68f4b94}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, 
              string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, 
              attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {
              number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#6  0x08070970 in run_actions (h=0xbfdf9b7c, a=0xb66340c4, msg=0xb6a6fa80) at action.c:1549
        t = 0xb66340c4
        ret = -1
        mod = 0x0
        ms = 0
        __FUNCTION__ = "run_actions"
#7  0x0807105d in run_top_route (a=0xb66340c4, msg=0xb6a6fa80, c=0x0) at action.c:1635
        ctx = {rec_lev = 3, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {-1230570616, -1075863780, -1230570520, -1075864744, -774102517, 1625566052}, __mask_was_saved = 0, 
              __saved_mask = {__val = {4, 3219102888, 3064396680, 3071374700, 3064396776, 3053849578, 3053849578, 3053908408, 3219103516, 3219102696, 3053849925, 3219062798, 48, 0, 
                  3053908408, 3219103516, 3064396776, 3219102728, 3053850835, 138719229, 3053894780, 7, 1, 3219103516, 1, 3219102776, 135912396, 3064396416, 1073741825, 0, 134977546, 
                  3064396416}}}}}
        p = 0xbfdf9b7c
        ret = 48
        sfbk = 0
#8  0x081479fc in receive_msg (
    buf=0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len=371, rcv_info=0xbfdf9ebc) at receive.c:240
        msg = 0xb6a6fa80
        ctx = {rec_lev = 5, run_flags = -1218441964, last_retcode = -1216865128, jmp_env = {{__jmpbuf = {-1216734968, -1, -1216737292, 134567086, 5, -1075864192}, 
              __mask_was_saved = -1216799770, __saved_mask = {__val = {3078232768, 3076468768, 1, 5, 0, 5, 134565684, 138392384, 0, 3076525332, 961078559, 0, 3219104684, 3219103256, 
                  3078191504, 3219103428, 3077456128, 33200, 3077641759, 961078559, 136537889, 18878, 3062860384, 4, 136537807, 2772170560, 4294967295, 3078230004, 134566635, 5, 
                  3219103232, 3078167526}}}}}
        ret = 0
        inb = {
          s = 0x844afe0 "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"..., len = 371}
        netinfo = {data = {s = 0xbfdf9db4 "\377\377\377\377\364\017z\267\353R\005\b\005", len = 134537572}, dst = 0xbfdf9da8, rcv = 0xb77a1a64}
        __FUNCTION__ = "receive_msg"
#9  0x0823da26 in udp_rcv_loop () at udp_server.c:495
        len = 371
        buf = "OPTIONS sip:777@85.118.45.91 SIP/2.0\r\nVia: SIP/2.0/UDP lifeguard.andrexen.vpn:48713;branch=z9hG4bKhjhs8ass877\r\nMax-Forwards: 69\r\nTo: sip:777@85.118.45.91\r\nFrom: sip:checksip@lifeguard.andrexen.vpn:508"...
        tmp = 0x8329042 "counters_prefork_init"
        from = 0xb68f8a5c
        fromlen = 16
---Type <return> to continue, or q <return> to quit---
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {1378711125, 4, 961078559, 1}, addr32 = {1378711125, 4, 961078559, 1}, addr16 = {30293, 21037, 4, 0, 58655, 14664, 1, 0}, 
              addr = "Uv-R\004\000\000\000\037\345H9\001\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1529706069, 0, 0, 0}, addr32 = {1529706069, 0, 0, 0}, addr16 = {30293, 
                23341, 0, 0, 0, 0, 0, 0}, addr = "Uv-[", '\000' <repeats 11 times>}}, src_port = 48713, dst_port = 5080, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {
              sa_family = 2, sa_data = "\276IUv-R\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 18878, sin_addr = {s_addr = 1378711125}, 
              sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 18878, sin6_flowinfo = 1378711125, sin6_addr = {__in6_u = {
                  __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0xb6611b94, 
          proto = 1 '\001'}
        __FUNCTION__ = "udp_rcv_loop"
#10 0x080eaf7b in main_loop () at main.c:1614
        i = 2
        pid = 0
        si = 0xb6611b94
        si_desc = "udp receiver child=2 sock=85.118.45.91:5080\000̛\206\245̛\206\245\000\000\000\000 \v\353\265\364\317\344\265\037\345H9\001\000\000\000\037\345H9\370\240\337\277\303I%\bHF\222\266\004\000\000\000\071x4\b\204\211\064\bn\000\000\000\064x4\b\024\000\000\000\037\345H9", '\000' <repeats 11 times>
        nrprocs = 4
        woneinit = 1
        __FUNCTION__ = "main_loop"
#11 0x080f23d4 in main (argc=13, argv=0xbfdfa524) at main.c:2631
        cfg_stream = 0x90c2008
        c = -1
        r = 0
        tmp = 0xbfdfbd8f ""
        tmp_len = 1
        port = 0
        proto = 0
        options = 0x833deb4 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
        ret = -1
        seed = 2912300657
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0xbfdfa450
        p = 0x805c9c8 "[\201\303H\347\071"
        st = {st_dev = 14, __pad1 = 0, st_ino = 6001, st_mode = 16832, st_nlink = 2, st_uid = 0, st_gid = 0, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, 
          st_atim = {tv_sec = 1487716504, tv_nsec = 24863882}, st_mtim = {tv_sec = 1490002293, tv_nsec = 141472099}, st_ctim = {tv_sec = 1490002293, tv_nsec = 141472099}, __unused4 = 0, 
          __unused5 = 0}
        __FUNCTION__ = "main"

Log Messages

Here are Kamailio logs in time DESC order.

Mar 20 12:02:15 webrtc-as kernel: [2291290.426195] kamailio[2963]: segfault at 23 ip b5a0be6b sp bfca2950 error 6 in textops.so[b59fb000+40000]
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 64) called from textops: textops.c: append_time_request_f(1216), module: textops; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 64); Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 520) called from core: rvalue.c: rval_new_empty(224), module: core; Free fragment not found!
Mar 20 12:02:15 webrtc-as adx-webrtc[2963]: {1 1 OPTIONS osj8es@lifeguard.andrexen.vpn} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 520); Free fragment not found!
...
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: siptrace [siptrace.c:1603]: trace_send_duplicate(): bad host name in uri
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found!
Mar 20 11:43:53 webrtc-as adx-webrtc[2963]: {1 102 ACK 62e26bfc56736b99020d5c1a3eaa34c1@172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found!
...
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: siptrace [siptrace.c:1614]: trace_send_duplicate(): bad host name in uri
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} CRITICAL: <core> [proxy.c:179]: hostent_cpy(): memory allocation failure
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: <core> [mem/f_malloc.c:498]: fm_malloc(): fm_malloc(0xb656b008, 16) called from core: proxy.c: hostent_cpy(177), module: core; Free fragment not found!
Mar 20 11:43:37 webrtc-as adx-webrtc[2963]: {1 102 ACK 2dcb43b73bc08adf29c5c7ed086a1b47@172.16.171.16} ERROR: <core> [mem/f_malloc.c:415]: fm_search_defrag(): fm_search_defrag(0xb656b008, 16); Free fragment not found!

SIP Traffic

No Data Available

Possible Solutions

Currently no idea. However one strange thing is that although siptrace module is load by kamailio but none of its functions are called or flags are set, i wonder why it shows up in kamailio log in the first place.

Additional Information

version: kamailio 4.4.5 (i386/linux) e9d576
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, select.
id: e9d576 -dirty
compiled on 06:32:04 Jan 25 2017 with gcc 4.7.2
Linux webrtc-as 3.10-0.bpo.2-686-pae #1 SMP Debian 3.10.5-1~bpo70+1 (2013-08-11) i686 GNU/Linux


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.