Re: [SR-Users] Segfault in dialog.so with 4.4.2 stable

(gdb) frame 1 #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at dlg_hash.c:921 921 dlg_lock( d_table, d_entry); (gdb) p *dlg $1 = {ref = 793790803, next = 0xa0d4b4f20303032, prev = 0x504953203a616956, h_id = 808333871, h_entry = 1346655535, state = 774976288, lifetime = 775107122, init_ts = 775435825, start_ts = 976303410, end_ts = 808857653, dflags = 1667592763, iflags = 1702259045, sflags = 825441636, toroute = 858927662, toroute_name = { s = 0x6172623b3135322e <Address 0x6172623b3135322e out of bounds>, len = 1030251374}, from_rr_nb = 894132788, tl = {next = 0x726f70723b646262, prev = 0xa0d303630353d74, timeout = 1836020294}, callid = {s = 0x20226e776f6e6b6e <Address 0x20226e776f6e6b6e out of bounds>, len = 1885958972}, from_uri = { s = 0x7340444c4f74656e <Address 0x7340444c4f74656e out of bounds>, len = 1999532137}, to_uri = {s = 0x743b3e74656e2e70 <Address 0x743b3e74656e2e70 out of bounds>, len = 1631414113}, req_uri = {s = 0x540a0d3536343766 <Address 0x540a0d3536343766 out of bounds>, len = 1008745071}, tag = {{ s = 0x363233313431332b <Address 0x363233313431332b out of bounds>, len = 892614711}, {s = 0x2e3836312e333232 <Address 0x2e3836312e333232 out of bounds>, len = 1043608370}}, cseq = {{s = 0x663330643473613d <Address 0x663330643473613d out of bounds>, len = 224671543}, {s = 0x3534203a44492d6c <Address 0x3534203a44492d6c out of bounds>, len = 909665638}}, route_set = {{s = 0x3433333435356635 <Address 0x3433333435356635 out of bounds>, len = 825582898}, { s = 0x7340353762316435 <Address 0x7340353762316435 out of bounds>, len = 1999532137}}, contact = {{s = 0x430a0d74656e2e70 <Address 0x430a0d74656e2e70 out of bounds>, len = 980510035}, {s = 0x65530a0d45594220 <Address 0x65530a0d45594220 out of bounds>, len = 1919252082}}, bind_addr = {0x70696f766c772e70, 0x6c410a0d74656e2e}, cbs = { first = 0x564e49203a776f6c, types = 742741065}, profile_links = 0x4c45434e4143202c, vars = 0x4e4f4954504f202c} On 07/15/2016 01:00 PM, Daniel-Constantin Mierla wrote: > From the second crash, can you get: > > frame 1 > > p *dlg > > So far it looks like either to a double free or some buffer overflow... > > Cheers, > Daniel > > > On 15/07/16 10:51, Dirk Teurlings - Signet B.V. wrote: >> Just got another segfault. >> >> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". >> Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg >> -P /var/run/kamailio/kamailio.'. >> Program terminated with signal 11, Segmentation fault. >> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 >> 74 return atomic_get_int(&(v->val)); >> (gdb) bt >> #0 atomic_get (v=0x7f6264d11378) at ../../mem/../atomic/atomic_common.h:74 >> #1 dlg_unref (dlg=dlg@entry=0x7f585c494b40, cnt=cnt@entry=1) at >> dlg_hash.c:921 >> #2 0x00007f5855912802 in dlg_run_event_route >> (dlg=dlg@entry=0x7f585c494b40, msg=msg@entry=0x7f587d4be8e8, >> ostate=<optimized out>, nstate=<optimized out>) at dlg_handlers.c:1630 >> #3 0x00007f585591416a in dlg_onroute (req=0x7f587d4be8e8, >> route_params=<optimized out>, param=<optimized out>) at dlg_handlers.c:1307 >> #4 0x00007f585965b0e2 in run_rr_callbacks >> (req=req@entry=0x7f587d4be8e8, rr_param=rr_param@entry=0x7f58598677a0) >> at rr_cb.c:96 >> #5 0x00007f58596452c5 in after_loose (_m=0x7f587d4be8e8, preloaded=0) >> at loose.c:919 >> #6 0x000000000042b618 in do_action (h=h@entry=0x7ffd6e277fd0, >> a=a@entry=0x7f587d264338, msg=msg@entry=0x7f587d4be8e8) at action.c:1060 >> #7 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e277fd0, >> a=0x7f587d264338, msg=0x7f587d4be8e8) at action.c:1549 >> #8 0x0000000000437544 in run_actions_safe (h=h@entry=0x7ffd6e279500, >> a=<optimized out>, msg=<optimized out>) at action.c:1614 >> #9 0x000000000053b2e8 in rval_get_int (h=0x7ffd6e279500, msg=<optimized >> out>, i=0x7ffd6e278430, rv=rv@entry=0x7f587d264d58, >> cache=cache@entry=0x0) at rvalue.c:912 >> #10 0x000000000054261c in rval_expr_eval_int (h=h@entry=0x7ffd6e279500, >> msg=msg@entry=0x7f587d4be8e8, res=res@entry=0x7ffd6e278430, >> rve=rve@entry=0x7f587d264d50) at rvalue.c:1910 >> #11 0x000000000042bc91 in do_action (h=h@entry=0x7ffd6e279500, >> a=a@entry=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1030 >> #12 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, >> a=0x7f587d268f88, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 >> #13 0x000000000042bcf2 in do_action (h=h@entry=0x7ffd6e279500, >> a=a@entry=0x7f587d2691e8, msg=msg@entry=0x7f587d4be8e8) at action.c:1049 >> #14 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, >> a=0x7f587d263f48, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 >> #15 0x000000000042bde0 in do_action (h=h@entry=0x7ffd6e279500, >> a=a@entry=0x7f587d073d70, msg=msg@entry=0x7f587d4be8e8) at action.c:678 >> #16 0x000000000042a10a in run_actions (h=h@entry=0x7ffd6e279500, >> a=a@entry=0x7f587d071698, msg=msg@entry=0x7f587d4be8e8) at action.c:1549 >> #17 0x00000000004375d0 in run_top_route (a=0x7f587d071698, >> msg=msg@entry=0x7f587d4be8e8, c=c@entry=0x0) at action.c:1635 >> #18 0x0000000000504386 in receive_msg (buf=<optimized out>, >> len=<optimized out>, rcv_info=<optimized out>) at receive.c:240 >> #19 0x00000000005f5bd4 in udp_rcv_loop () at udp_server.c:495 >> #20 0x00000000004b2625 in main_loop () at main.c:1600 >> #21 0x0000000000427e2b in main (argc=<optimized out>, argv=<optimized >> out>) at main.c:2616 >> >> >> Relevant logmessages before crash: >> Jul 15 10:37:55 server /usr/sbin/kamailio[12426]: NOTICE: dialog >> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old >> (0x7f585c4a6820 ref 4) >> Jul 15 10:37:55 server /usr/sbin/kamailio[12397]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param '70f.b9d1' [3847:7579] >> Jul 15 10:37:55 server /usr/sbin/kamailio[12395]: WARNING: dialog >> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg >> 0x7f585c4a6820 [3847:7579] with clid >> '4c41f08d317ecb9342b93f22738003f3@server' and tags 'as5f3a16b4' 'as71cb6036' >> Jul 15 10:40:13 server /usr/sbin/kamailio[12378]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param 'eb6.1e21' [1726:4833] >> Jul 15 10:40:13 server /usr/sbin/kamailio[12376]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param 'eb6.1e21' [1726:4833] >> Jul 15 10:40:14 server /usr/sbin/kamailio[12377]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param 'eb6.1e21' [1726:4833] >> Jul 15 10:40:16 server /usr/sbin/kamailio[12377]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param 'eb6.1e21' [1726:4833] >> Jul 15 10:40:16 server /usr/sbin/kamailio[12396]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param 'eb6.1e21' [1726:4833] >> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: sl >> [sl_funcs.c:363]: sl_reply_error(): ERROR: sl_reply_error used: I'm >> terribly sorry, server error occurred (1/SL) >> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm >> [t_reply.c:533]: _reply_light(): ERROR: _reply_light: can't generate 487 >> reply when a final 487 was sent out >> Jul 15 10:41:34 server /usr/sbin/kamailio[12396]: ERROR: tm >> [t_lookup.c:1471]: t_unref(): ERROR: t_unref: generation of a delayed >> stateful reply failed >> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog >> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old >> (0x7f585c49d5b0 ref 4) >> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog >> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old >> (0x7f585c604f18 ref 4) >> Jul 15 10:42:25 server /usr/sbin/kamailio[12426]: NOTICE: dialog >> [dlg_hash.c:245]: dlg_clean_run(): dialog in delete state is too old >> (0x7f585c494b40 ref 4) >> Jul 15 10:42:25 server /usr/sbin/kamailio[12383]: WARNING: dialog >> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg >> 0x7f585c604f18 [2396:9046] with clid >> '1b3ff5f0246fb7e82ed949544bcccbba@192.168.10.233:5060' and tags >> 'as4d83d6f8' '5788A162-2557E04D-3E86ED15' >> Jul 15 10:42:25 server /usr/sbin/kamailio[12395]: WARNING: dialog >> [dlg_handlers.c:1219]: dlg_onroute(): unable to find dialog for BYE with >> route param '6b3.c6b' [950:2924] >> Jul 15 10:42:25 server kernel: [209851.262461] kamailio[12376]: segfault >> at 7f6264d11378 ip 00007f585592a908 sp 00007ffd6e277330 error 4 in >> dialog.so[7f58558e0000+88000] >> Jul 15 10:42:25 server /usr/sbin/kamailio[12394]: WARNING: dialog >> [dlg_handlers.c:1348]: dlg_onroute(): inconsitent dlg timer data on dlg >> 0x7f585c49d5b0 [950:2924] with clid >> '45fe86ce065f5543342e51ad355d1b75@server' and tags 'as152f7465' 'as4d03f77d' >> Jul 15 10:42:26 server /usr/sbin/kamailio[12431]: CRITICAL: <core> >> [pass_fd.c:275]: receive_fd(): EOF on 32 >> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> >> [main.c:739]: handle_sigs(): child process 12376 exited by a signal 11 >> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: ALERT: <core> >> [main.c:742]: handle_sigs(): core was generated >> Jul 15 10:42:26 server /usr/sbin/kamailio[12370]: INFO: <core> >> [main.c:754]: handle_sigs(): terminating due to SIGCHLD >> >> >> Cheers, >> Dirk