Hi! We recently upgraded our Kamailio 4.1 to 4.2.1. With the newer version Kamailio crashes after just running a few minutes. After some debugging it looks as though the problem is in exec_msg (which is used in our config). After disabling this 4.2.1 seem to run just fine. Core file exists, here's the output:
(gdb) backtrace #0 0x00000000005ebf0f in fm_extract_free (frag=0x7f053ea08d18, qm=0x7f053e88e010) at mem/f_malloc.c:206 #1 fm_malloc (qm=0x7f053e88e010, size=<optimized out>, file=file@entry=0x7f053cbedfd4 "exec: exec_hf.c", func=func@entry=0x7f053cbef378 "replace_env", line=line@entry=375) at mem/f_malloc.c:490 #2 0x00007f053cbe7953 in replace_env (list=0x7f053ea08868) at exec_hf.c:375 #3 0x00007f053cbe862e in set_env (msg=0x7f053e91d690) at exec_hf.c:547 #4 0x00007f053cbeb835 in w_exec_msg (msg=0x7f053e87c480, cmd=0x7f053ea5e168 "X֤>\005\177", foo=<optimized out>) at exec_mod.c:164 #5 0x00000000004274f7 in do_action (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1094 #6 0x00000000004261a9 in run_actions (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1583 #7 0x0000000000432980 in run_top_route (a=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480, c=c@entry=0x0) at action.c:1669 #8 0x00007f053e610d2a in run_failure_handlers (t=t@entry=0x7f0534ecc770, rpl=0x7f053ea71040, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051 #9 0x00007f053e612aa3 in t_should_relay_response (Trans=Trans@entry=0x7f0534ecc770, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fff02e9d2a0, should_relay=should_relay@entry=0x7fff02e9d290, cancel_data=cancel_data@entry=0x7fff02e9d490, reply=reply@entry=0x7f053ea71040) at t_reply.c:1406 #10 0x00007f053e616126 in relay_reply (t=t@entry=0x7f0534ecc770, p_msg=p_msg@entry=0x7f053ea71040, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fff02e9d490, do_put_on_wait=do_put_on_wait@entry=1) at t_reply.c:1809 #11 0x00007f053e61a9f3 in reply_received (p_msg=0x7f053ea71040) at t_reply.c:2493 #12 0x00000000004920a6 in do_forward_reply (msg=msg@entry=0x7f053ea71040, mode=mode@entry=0) at forward.c:783 #13 0x0000000000493637 in forward_reply (msg=msg@entry=0x7f053ea71040) at forward.c:885 #14 0x00000000004f5634 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275 #15 0x00000000005d929d in udp_rcv_loop () at udp_server.c:521 #16 0x00000000004a73f1 in main_loop () at main.c:1629 #17 0x0000000000425085 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561 Kind regards,/Tobias
Hello,
this should be fixed in branch 4.2 -- you have to install the nightly builds (if you are using debian) or from sources branch 4.2.
We will have a release very soon, as 4.2.2 which will include it -- this most probably will happen sometime next week.
Cheers, Daniel
On 29/12/14 12:08, Tobias wrote:
Hi!
We recently upgraded our Kamailio 4.1 to 4.2.1. With the newer version Kamailio crashes after just running a few minutes. After some debugging it looks as though the problem is in exec_msg (which is used in our config). After disabling this 4.2.1 seem to run just fine.
Core file exists, here's the output:
(gdb) backtrace
#0 0x00000000005ebf0f in fm_extract_free (frag=0x7f053ea08d18, qm=0x7f053e88e010) at mem/f_malloc.c:206
#1 fm_malloc (qm=0x7f053e88e010, size=<optimized out>, file=file@entry=0x7f053cbedfd4 "exec: exec_hf.c", func=func@entry=0x7f053cbef378 "replace_env", line=line@entry=375) at mem/f_malloc.c:490
#2 0x00007f053cbe7953 in replace_env (list=0x7f053ea08868) at exec_hf.c:375
#3 0x00007f053cbe862e in set_env (msg=0x7f053e91d690) at exec_hf.c:547
#4 0x00007f053cbeb835 in w_exec_msg (msg=0x7f053e87c480, cmd=0x7f053ea5e168 "X֤>\005\177", foo=<optimized out>) at exec_mod.c:164
#5 0x00000000004274f7 in do_action (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1094
#6 0x00000000004261a9 in run_actions (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1583
#7 0x0000000000432980 in run_top_route (a=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480, c=c@entry=0x0) at action.c:1669
#8 0x00007f053e610d2a in run_failure_handlers (t=t@entry=0x7f0534ecc770, rpl=0x7f053ea71040, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051
#9 0x00007f053e612aa3 in t_should_relay_response (Trans=Trans@entry=0x7f0534ecc770, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fff02e9d2a0, should_relay=should_relay@entry=0x7fff02e9d290,
cancel_data=cancel_data@entry=0x7fff02e9d490,reply=reply@entry=0x7f053ea71040) at t_reply.c:1406
#10 0x00007f053e616126 in relay_reply (t=t@entry=0x7f0534ecc770, p_msg=p_msg@entry=0x7f053ea71040, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fff02e9d490, do_put_on_wait=do_put_on_wait@entry=1)
at t_reply.c:1809#11 0x00007f053e61a9f3 in reply_received (p_msg=0x7f053ea71040) at t_reply.c:2493
#12 0x00000000004920a6 in do_forward_reply (msg=msg@entry=0x7f053ea71040, mode=mode@entry=0) at forward.c:783
#13 0x0000000000493637 in forward_reply (msg=msg@entry=0x7f053ea71040) at forward.c:885
#14 0x00000000004f5634 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275
#15 0x00000000005d929d in udp_rcv_loop () at udp_server.c:521
#16 0x00000000004a73f1 in main_loop () at main.c:1629
#17 0x0000000000425085 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561
Kind regards,
/Tobias
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi again Daniel, We've upgraded to 4.2.2 and the recent changes in exec seem to still affect our usage of exec. From new coredump on 4.2.2:
(gdb) bt #0 0x00007f1c34dc404b in memcpy (__len=18446744073709551614, __src=0x7f1c2d4ecc09, __dest=0x7f1c368f4be2) at /usr/include/x86_64-linux-gnu/bits/string3.h:52 #1 print_hf_var (w=<optimized out>, offset=<optimized out>) at exec_hf.c:263 #2 print_var (w=<optimized out>, offset=<optimized out>) at exec_hf.c:296 #3 create_vars (list=<optimized out>, offset=<optimized out>) at exec_hf.c:346 #4 set_env (msg=0x7f1c368f4a08) at exec_hf.c:544 #5 0x00007f1c34dc6835 in w_exec_msg (msg=0x7f1c36839480, cmd=0x7f1c3692b298 "", foo=<optimized out>) at exec_mod.c:164 #6 0x00000000004275d7 in do_action (h=h@entry=0x7fffdcd30740, a=a@entry=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480) at action.c:1094 #7 0x0000000000426289 in run_actions (h=h@entry=0x7fffdcd30740, a=a@entry=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480) at action.c:1583 #8 0x0000000000432a90 in run_top_route (a=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480, c=c@entry=0x0) at action.c:1669 #9 0x00007f1c365cdd9a in run_failure_handlers (t=t@entry=0x7f1c2d4f9d68, rpl=0x7f1c3693b0c0, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051 #10 0x00007f1c365cfb13 in t_should_relay_response (Trans=Trans@entry=0x7f1c2d4f9d68, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fffdcd30a50, should_relay=should_relay@entry=0x7fffdcd30a40, cancel_data=cancel_data@entry=0x7fffdcd30c40, reply=reply@entry=0x7f1c3693b0c0) at t_reply.c:1406 #11 0x00007f1c365d3196 in relay_reply (t=t@entry=0x7f1c2d4f9d68, p_msg=p_msg@entry=0x7f1c3693b0c0, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fffdcd30c40, do_put_on_wait=do_put_on_wait@entry=1) at t_reply.c:1809 #12 0x00007f1c365d7a63 in reply_received (p_msg=0x7f1c3693b0c0) at t_reply.c:2493 #13 0x00000000004922b6 in do_forward_reply (msg=msg@entry=0x7f1c3693b0c0, mode=mode@entry=0) at forward.c:783 #14 0x0000000000493847 in forward_reply (msg=msg@entry=0x7f1c3693b0c0) at forward.c:885 #15 0x00000000004f5974 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275 #16 0x00000000005d998d in udp_rcv_loop () at udp_server.c:521 #17 0x00000000004a7601 in main_loop () at main.c:1629 #18 0x0000000000425165 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561 Can be reproduced by sending a SIP INVITE containing a custom header that is empty/has no data, ex:
"X-model-id: ."
modparam("exec", "setvars", 0) is currently used as a workaround. Kind regards,/Tobias Date: Mon, 29 Dec 2014 12:13:19 +0100 From: miconda@gmail.com To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio 4.2.1 crashes on using exec_msg
Hello,
this should be fixed in branch 4.2 -- you have to install the nightly builds (if you are using debian) or from sources branch 4.2.
We will have a release very soon, as 4.2.2 which will include it -- this most probably will happen sometime next week.
Cheers,
Daniel
On 29/12/14 12:08, Tobias wrote:
Hi!
We recently upgraded our Kamailio 4.1 to 4.2.1. With the newer version Kamailio crashes after just running a few minutes. After some debugging it looks as though the problem is in exec_msg (which is used in our config). After disabling this 4.2.1 seem to run just fine.
Core file exists, here's the output:
(gdb) backtrace #0 0x00000000005ebf0f in fm_extract_free (frag=0x7f053ea08d18, qm=0x7f053e88e010) at mem/f_malloc.c:206 #1 fm_malloc (qm=0x7f053e88e010, size=<optimized out>, file=file@entry=0x7f053cbedfd4 "exec: exec_hf.c", func=func@entry=0x7f053cbef378 "replace_env", line=line@entry=375) at mem/f_malloc.c:490 #2 0x00007f053cbe7953 in replace_env (list=0x7f053ea08868) at exec_hf.c:375 #3 0x00007f053cbe862e in set_env (msg=0x7f053e91d690) at exec_hf.c:547 #4 0x00007f053cbeb835 in w_exec_msg (msg=0x7f053e87c480, cmd=0x7f053ea5e168 "X֤>\005\177", foo=<optimized out>) at exec_mod.c:164 #5 0x00000000004274f7 in do_action (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1094 #6 0x00000000004261a9 in run_actions (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1583 #7 0x0000000000432980 in run_top_route (a=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480, c=c@entry=0x0) at action.c:1669 #8 0x00007f053e610d2a in run_failure_handlers (t=t@entry=0x7f0534ecc770, rpl=0x7f053ea71040, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051 #9 0x00007f053e612aa3 in t_should_relay_response (Trans=Trans@entry=0x7f0534ecc770, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fff02e9d2a0, should_relay=should_relay@entry=0x7fff02e9d290,
cancel_data=cancel_data@entry=0x7fff02e9d490, reply=reply@entry=0x7f053ea71040) at t_reply.c:1406 #10 0x00007f053e616126 in relay_reply (t=t@entry=0x7f0534ecc770, p_msg=p_msg@entry=0x7f053ea71040, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fff02e9d490, do_put_on_wait=do_put_on_wait@entry=1) at t_reply.c:1809 #11 0x00007f053e61a9f3 in reply_received (p_msg=0x7f053ea71040) at t_reply.c:2493 #12 0x00000000004920a6 in do_forward_reply (msg=msg@entry=0x7f053ea71040, mode=mode@entry=0) at forward.c:783 #13 0x0000000000493637 in forward_reply (msg=msg@entry=0x7f053ea71040) at forward.c:885 #14 0x00000000004f5634 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275 #15 0x00000000005d929d in udp_rcv_loop () at udp_server.c:521 #16 0x00000000004a73f1 in main_loop () at main.c:1629 #17 0x0000000000425085 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561
Kind regards, /Tobias
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
thanks for the report and details, I just pushed a fix (master, 4.2 and 4.1 branches) for properly dealing with empty headers after the patch for exec_bash_safety.
Let me know if works ok.
Cheers, Daniel
On 13/01/15 11:56, Tobias wrote:
Hi again Daniel,
We've upgraded to 4.2.2 and the recent changes in exec seem to still affect our usage of exec.
From new coredump on 4.2.2:
(gdb) bt
#0 0x00007f1c34dc404b in memcpy (__len=18446744073709551614, __src=0x7f1c2d4ecc09, __dest=0x7f1c368f4be2) at /usr/include/x86_64-linux-gnu/bits/string3.h:52
#1 print_hf_var (w=<optimized out>, offset=<optimized out>) at exec_hf.c:263
#2 print_var (w=<optimized out>, offset=<optimized out>) at exec_hf.c:296
#3 create_vars (list=<optimized out>, offset=<optimized out>) at exec_hf.c:346
#4 set_env (msg=0x7f1c368f4a08) at exec_hf.c:544
#5 0x00007f1c34dc6835 in w_exec_msg (msg=0x7f1c36839480, cmd=0x7f1c3692b298 "", foo=<optimized out>) at exec_mod.c:164
#6 0x00000000004275d7 in do_action (h=h@entry=0x7fffdcd30740, a=a@entry=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480) at action.c:1094
#7 0x0000000000426289 in run_actions (h=h@entry=0x7fffdcd30740, a=a@entry=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480) at action.c:1583
#8 0x0000000000432a90 in run_top_route (a=0x7f1c3692a9c0, msg=msg@entry=0x7f1c36839480, c=c@entry=0x0) at action.c:1669
#9 0x00007f1c365cdd9a in run_failure_handlers (t=t@entry=0x7f1c2d4f9d68, rpl=0x7f1c3693b0c0, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051
#10 0x00007f1c365cfb13 in t_should_relay_response (Trans=Trans@entry=0x7f1c2d4f9d68, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fffdcd30a50,
should_relay=should_relay@entry=0x7fffdcd30a40,cancel_data=cancel_data@entry=0x7fffdcd30c40, reply=reply@entry=0x7f1c3693b0c0) at t_reply.c:1406
#11 0x00007f1c365d3196 in relay_reply (t=t@entry=0x7f1c2d4f9d68, p_msg=p_msg@entry=0x7f1c3693b0c0, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fffdcd30c40,
do_put_on_wait=do_put_on_wait@entry=1) at t_reply.c:1809#12 0x00007f1c365d7a63 in reply_received (p_msg=0x7f1c3693b0c0) at t_reply.c:2493
#13 0x00000000004922b6 in do_forward_reply (msg=msg@entry=0x7f1c3693b0c0, mode=mode@entry=0) at forward.c:783
#14 0x0000000000493847 in forward_reply (msg=msg@entry=0x7f1c3693b0c0) at forward.c:885
#15 0x00000000004f5974 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275
#16 0x00000000005d998d in udp_rcv_loop () at udp_server.c:521
#17 0x00000000004a7601 in main_loop () at main.c:1629
#18 0x0000000000425165 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561
Can be reproduced by sending a SIP INVITE containing a custom header that is empty/has no data, ex:
"X-model-id: ."
modparam("exec", "setvars", 0) is currently used as a workaround.
Kind regards, /Tobias
Date: Mon, 29 Dec 2014 12:13:19 +0100 From: miconda@gmail.com To: sr-users@lists.sip-router.org Subject: Re: [SR-Users] Kamailio 4.2.1 crashes on using exec_msg
Hello,
this should be fixed in branch 4.2 -- you have to install the nightly builds (if you are using debian) or from sources branch 4.2.
We will have a release very soon, as 4.2.2 which will include it -- this most probably will happen sometime next week.
Cheers, Daniel
On 29/12/14 12:08, Tobias wrote:
Hi! We recently upgraded our Kamailio 4.1 to 4.2.1. With the newer version Kamailio crashes after just running a few minutes. After some debugging it looks as though the problem is in exec_msg (which is used in our config). After disabling this 4.2.1 seem to run just fine. Core file exists, here's the output: (gdb) backtrace #0 0x00000000005ebf0f in fm_extract_free (frag=0x7f053ea08d18, qm=0x7f053e88e010) at mem/f_malloc.c:206 #1 fm_malloc (qm=0x7f053e88e010, size=<optimized out>, file=file@entry=0x7f053cbedfd4 "exec: exec_hf.c", func=func@entry=0x7f053cbef378 "replace_env", line=line@entry=375) at mem/f_malloc.c:490 #2 0x00007f053cbe7953 in replace_env (list=0x7f053ea08868) at exec_hf.c:375 #3 0x00007f053cbe862e in set_env (msg=0x7f053e91d690) at exec_hf.c:547 #4 0x00007f053cbeb835 in w_exec_msg (msg=0x7f053e87c480, cmd=0x7f053ea5e168 "X֤>\005\177", foo=<optimized out>) at exec_mod.c:164 #5 0x00000000004274f7 in do_action (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1094 #6 0x00000000004261a9 in run_actions (h=h@entry=0x7fff02e9cf90, a=a@entry=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480) at action.c:1583 #7 0x0000000000432980 in run_top_route (a=0x7f053ea5cfb8, msg=msg@entry=0x7f053e87c480, c=c@entry=0x0) at action.c:1669 #8 0x00007f053e610d2a in run_failure_handlers (t=t@entry=0x7f0534ecc770, rpl=0x7f053ea71040, code=486, extra_flags=extra_flags@entry=64) at t_reply.c:1051 #9 0x00007f053e612aa3 in t_should_relay_response (Trans=Trans@entry=0x7f0534ecc770, new_code=new_code@entry=486, branch=branch@entry=0, should_store=should_store@entry=0x7fff02e9d2a0, should_relay=should_relay@entry=0x7fff02e9d290, cancel_data=cancel_data@entry=0x7fff02e9d490, reply=reply@entry=0x7f053ea71040) at t_reply.c:1406 #10 0x00007f053e616126 in relay_reply (t=t@entry=0x7f0534ecc770, p_msg=p_msg@entry=0x7f053ea71040, branch=0, msg_status=msg_status@entry=486, cancel_data=cancel_data@entry=0x7fff02e9d490, do_put_on_wait=do_put_on_wait@entry=1) at t_reply.c:1809 #11 0x00007f053e61a9f3 in reply_received (p_msg=0x7f053ea71040) at t_reply.c:2493 #12 0x00000000004920a6 in do_forward_reply (msg=msg@entry=0x7f053ea71040, mode=mode@entry=0) at forward.c:783 #13 0x0000000000493637 in forward_reply (msg=msg@entry=0x7f053ea71040) at forward.c:885 #14 0x00000000004f5634 in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at receive.c:275 #15 0x00000000005d929d in udp_rcv_loop () at udp_server.c:521 #16 0x00000000004a73f1 in main_loop () at main.c:1629 #17 0x0000000000425085 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2561 Kind regards, /Tobias _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users-- Daniel-Constantin Mierla http://twitter.com/#!/miconda http://twitter.com/#%21/miconda - http://www.linkedin.com/in/miconda
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Tobias