Re: [SR-Users] kamailio crash periodically in timer handler

Hello,

got dragged by some other tasks for a while...

Can you list here all your loaded modules? Also, are you having event routes in your config?

At this moment it looks like either a buffer overflow somewhere or a free of an shared memory element without detaching from timer. I will need more details to narrow it down.

Cheers, Daniel

On 23/04/15 17:54, Péter Barabás wrote:

Hi,

here it is:

$1 = {next = 0x0, prev = 0x7f64eb5b6bd0, expire = 1432560038, initial_timeout = 80, data = 0x7f64ebc3a888, f = 0x7f6510a2c165 <wait_handler>, flags = 513, slow_idx = 0}

*Dr. Péter Barabás*

Director of Software Development

*Arenim Technologies AB.*

Mob: +36 70 314 56 57

Tel: +36 1 8 555 602

E-mail: peter.barabas@arenim.com mailto:peter.barabas@arenim.com

Postal: Sweden – 11435 Stockholm, Stureplan 4c, 4th. Floor

Arenim_logo http://www.arenim.com/

CryptTalk_Logo http://www.crypttalk.com/

*From:*sr-users [mailto:sr-users-bounces@lists.sip-router.org] *On Behalf Of *Daniel-Constantin Mierla *Sent:* Thursday, April 23, 2015 5:07 PM *To:* Kamailio (SER) - Users Mailing List *Subject:* Re: [SR-Users] kamailio crash periodically in timer handler

Hello,

can you get from gdb the output of:

p *tl

Cheers, Daniel

On 23/04/15 17:00, Péter Barabás wrote:

Hi,



we have a serious problem with Kamailio v4.2.3, it crashes after
some days. We have checked the core dump and the gdb output is the
following for bt and full backtrace:



#0  0x0000000000602291 in timer_list_expire (t=1432559962,
h=0x7f64eb5b6bd0, slow_l=0x7f64eb5ba328, slow_mark=43873) at
timer.c:877

#1  0x0000000000602746 in timer_handler () at timer.c:953

#2  0x0000000000602bb9 in timer_main () at timer.c:992

#3  0x00000000004a833c in main_loop () at main.c:1700

#4  0x00000000004ad857 in main (argc=13, argv=0x7fff96697d38) at
main.c:2578



#0  0x0000000000602291 in timer_list_expire (t=1432559962,
h=0x7f64eb5b6bd0, slow_l=0x7f64eb5ba328, slow_mark=43873) at
timer.c:877

        tl = 0x7f64ebc3a908

        ret = 0

#1  0x0000000000602746 in timer_handler () at timer.c:953

        saved_ticks = 1432559962

        run_slow_timer = 0

        i = 865

        __FUNCTION__ = "timer_handler"

#2  0x0000000000602bb9 in timer_main () at timer.c:992

No locals.

#3  0x00000000004a833c in main_loop () at main.c:1700

        i = 4

        pid = 0

        si = 0x0

        si_desc = "udp receiver child=3
sock=127.0.0.1:5060\000\375\362\250/\313\345\v\004\000\000\000\000\000\000\000\004\257\263=\000\000\000\000\200TA\000\000\000\000\000\060}i\226\377\177",
'\000' <repeats 19 times>,
"{i\226\377\177\000\000\376\366R\000\000\000\000\000\377\377\377\377\377\177\000\000@\350\250\000\000\000\000"

        nrprocs = 4

        __FUNCTION__ = "main_loop"

#4  0x00000000004ad857 in main (argc=13, argv=0x7fff96697d38) at
main.c:2578

        cfg_stream = 0x1b94010

        c = -1

        r = 0

        tmp = 0x7fff96699b17 ""

        tmp_len = 0

        port = 1

        proto = 32767

        options = 0x703e08
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:"

        ret = -1

        seed = 2591880768

        rfd = 4

        debug_save = 0

        debug_flag = 0

        dont_fork_cnt = 0

        n_lst = 0x0

        p = 0x7fff96697c50 ""

        __FUNCTION__ = "main"



In the logs it can be seen:



Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4635]: CRITICAL:
<core> [pass_fd.c:293]: receive_fd(): EOF on 21

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4612]: ALERT:
<core> [main.c:784]: handle_sigs(): child process 4623 exited by a
signal 11

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4612]: ALERT:
<core> [main.c:787]: handle_sigs(): core was generated

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4612]: INFO:
<core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4631]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4628]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4634]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4615]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4617]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4626]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4625]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4624]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4632]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4622]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4621]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4620]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4619]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4618]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4616]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4635]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4614]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4629]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4627]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4633]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4630]: INFO:
<core> [main.c:850]: sig_usr(): signal 15 received

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4612]: INFO:
<core> [mem/f_malloc.c:599]: fm_free(): freeing a free fragment
(0x7f64eca19950/0x7f64eca19988) - ignore

Apr 23 08:07:31 ctsip1 /usr/local/sbin/kamailio[4612]: INFO:
<core> [mem/f_malloc.c:599]: fm_free(): freeing a free fragment
(0x7f64ed587f20/0x7f64ed587f58) - ignore



Kamailio runs on Ubuntu trusty. The interesting thing that we run
Kamailio 4.2.3 in an other environment (on Debian Squeeze) and it
works perfect, no crashes. The main difference is that in Debian
environment we have less number of users and the firewall is not
so strength. But the configuration and the source is the same for
both.



Please help us to solve this problem. Thanks.



*Dr. Péter Barabás*

Director of Software Development

*Arenim Technologies AB.*

* *

Mob:           +36 70 314 56 57

Tel:             +36 1 8 555 602

E-mail:        peter.barabas@arenim.com
<mailto:peter.barabas@arenim.com>

Postal:        Sweden – 11435 Stockholm, Stureplan 4c, 4th. Floor



Arenim_logo <http://www.arenim.com/>



CryptTalk_Logo <http://www.crypttalk.com/>






_______________________________________________

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list

sr-users@lists.sip-router.org <mailto:sr-users@lists.sip-router.org>

http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda http://twitter.com/#%21/miconda - http://www.linkedin.com/in/miconda Kamailio World Conference, May 27-29, 2015 Berlin, Germany - http://www.kamailioworld.com