So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? () No symbol table info available. #1 0x000000000053542a in ?? () No symbol table info available. #2 0x00000000005356c7 in timer_main () No symbol table info available. #3 0x000000000046d572 in main_loop () No symbol table info available. #4 0x000000000047030b in main () No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2ff1683000 cbs = 0x0 cbs_tmp = 0x7f5bf198e508 __FUNCTION__ = "free_cell" #6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote: Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably have...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote: Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

I'm afraid this is also not the correct core. Can you check the timestamp on the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably have...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus daniel.ciprus@acision.comwrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- *Daniel Ciprus* Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- *Daniel Ciprus* Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens:

2014-03-13 17:22 GMT+01:00 Daniel Ciprus daniel.ciprus@acision.com:

There are no more core files on the filesystem :-(

On 03/13/2014 12:18 PM, Jason Penton wrote:

I'm afraid this is also not the correct core. Can you check the timestamp on the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus daniel.ciprus@acision.com wrote:

...

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably have...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus daniel.ciprus@acision.com wrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

I don't think these cores indicate the real crash... I'd like to get some more detail on what actually happened? Daniel, can you re-create? Keep in mind that if your core dump config on your box is not configured to name your cores according to process id or timestamp one core will overwrite the other..... as a result you will never see the core that is the root cause.

Which OS are you running?

if Linux, I use the following in /etc/sysctl.conf:

kernel.core_pattern=/tmp/core.%e.%p.%h.%t

On Thu, Mar 13, 2014 at 8:45 PM, Carsten Bock carsten@ng-voice.com wrote:

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens: mem_safety=1 See: http://www.kamailio.org/wiki/cookbooks/devel/core#mem_safety

Kind regards, Carsten

2014-03-13 19:44 GMT+01:00 Carsten Bock carsten@ng-voice.com:

...

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens:

2014-03-13 17:22 GMT+01:00 Daniel Ciprus daniel.ciprus@acision.com:

...

There are no more core files on the filesystem :-(

On 03/13/2014 12:18 PM, Jason Penton wrote:

I'm afraid this is also not the correct core. Can you check the

timestamp on

...

...

the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus <

daniel.ciprus@acision.com>

...

...

wrote:

...

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528)

at

...

...

...

uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably

have...

...

...

...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus <

daniel.ciprus@acision.com>

...

...

...

wrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728)

at

...

...

...

...

uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728)

at

...

...

...

...

uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.com

Office +49 40 34927219 Fax +49 40 34927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.com

Office +49 40 34927219 Fax +49 40 34927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

Jason,

I've tried multiple combinations for pattern but I'm getting only 2 core files ...

Details:

~]# cat /proc/sys/kernel/core_pattern /tmp/core.%e.sig%s.%p

~]# lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 6.5 (Santiago) Release: 6.5 Codename: Santiago

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? () No symbol table info available. #1 0x000000000053542a in ?? () No symbol table info available. #2 0x00000000005356c7 in timer_main () No symbol table info available. #3 0x000000000046d572 in main_loop () No symbol table info available. #4 0x000000000047030b in main () No symbol table info available. (gdb)

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd3221000 cbs = 0x0 cbs_tmp = 0x7f23d35386b8 __FUNCTION__ = "free_cell" #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 p_cell = 0x7f23d3319a70 tmp_cell = 0x7f23d353dca0 i = 580 __FUNCTION__ = "free_hash_table" #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 02:58 PM, Jason Penton wrote: I don't think these cores indicate the real crash... I'd like to get some more detail on what actually happened? Daniel, can you re-create? Keep in mind that if your core dump config on your box is not configured to name your cores according to process id or timestamp one core will overwrite the other..... as a result you will never see the core that is the root cause.

Which OS are you running?

if Linux, I use the following in /etc/sysctl.conf:

kernel.core_pattern=/tmp/core.%e.%p.%h.%t

On Thu, Mar 13, 2014 at 8:45 PM, Carsten Bock <carsten@ng-voice.commailto:carsten@ng-voice.com> wrote: It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens: mem_safety=1 See: http://www.kamailio.org/wiki/cookbooks/devel/core#mem_safety

Kind regards, Carsten

2014-03-13 19:44 GMT+01:00 Carsten Bock <carsten@ng-voice.commailto:carsten@ng-voice.com>:

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens:

2014-03-13 17:22 GMT+01:00 Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com>:

...

There are no more core files on the filesystem :-(

On 03/13/2014 12:18 PM, Jason Penton wrote:

I'm afraid this is also not the correct core. Can you check the timestamp on the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote:

...

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably have...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.commailto:carsten@ng-voice.com

Office +49 40 34927219tel:%2B49%2040%2034927219 Fax +49 40 34927220tel:%2B49%2040%2034927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.commailto:carsten@ng-voice.com

Office +49 40 34927219tel:%2B49%2040%2034927219 Fax +49 40 34927220tel:%2B49%2040%2034927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

Looks like kamailio-debuginfo rpm was from older version of kamailio. I'm not able to reproduce core file anymore. Could somebody please be so kind and explain why ?

Secondly:

Mar 13 18:12:57 ricvmf-fusion01 kam-scscf[13524]: WARNING: tm [t_lookup.c:1536]: t_unref(): WARNING: script writer didn't release transaction Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13657]: ERROR: <core> [tcp_main.c:4237]: handle_tcpconn_ev(): connect 10.67.64.29:1305 failed Mar 13 18:12:57 ricvmf-fusion01 kam-scscf[13524]: INFO: ims_registrar_scscf [cxdx_sar.c:79]: create_return_code(): created AVP successfully : [saa_return_code] - [1] Mar 13 18:12:57 ricvmf-fusion01 kam-scscf[13524]: WARNING: tm [t_lookup.c:1536]: t_unref(): WARNING: script writer didn't release transaction Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:12:57 ricvmf-fusion01 kam-pcscf[13639]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact

As far as I remember there was configurable called "mem=XXX" but I don't see it in the devel cookbook anymore. Any idea what replaced this variable ?

@Hugh: we'll have beer on me once you get here ;)

On 03/13/2014 05:47 PM, Hugh Waite wrote: Dan, There are two cores because of a crash in one process followed by a crash when the other processes are trying to shutdown.

What's interesting is that the bt doesn't show useful pointers. If you have installed from RPMs make sure the kamailio-debuginfo is from the same build as the other RPMs.

Also, do the logs say anything? There should be a log entry from the kernel for the segfault/signal that says which module crashed (e.g. registrar.so) and possibly (hopefully) an error message just before that.

Hugh

On 13/03/2014 19:53, Daniel Ciprus wrote: Jason,

I've tried multiple combinations for pattern but I'm getting only 2 core files ...

Details:

~]# cat /proc/sys/kernel/core_pattern /tmp/core.%e.sig%s.%p

~]# lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 6.5 (Santiago) Release: 6.5 Codename: Santiago

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? () No symbol table info available. #1 0x000000000053542a in ?? () No symbol table info available. #2 0x00000000005356c7 in timer_main () No symbol table info available. #3 0x000000000046d572 in main_loop () No symbol table info available. #4 0x000000000047030b in main () No symbol table info available. (gdb)

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd3221000 cbs = 0x0 cbs_tmp = 0x7f23d35386b8 __FUNCTION__ = "free_cell" #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 p_cell = 0x7f23d3319a70 tmp_cell = 0x7f23d353dca0 i = 580 __FUNCTION__ = "free_hash_table" #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 02:58 PM, Jason Penton wrote: I don't think these cores indicate the real crash... I'd like to get some more detail on what actually happened? Daniel, can you re-create? Keep in mind that if your core dump config on your box is not configured to name your cores according to process id or timestamp one core will overwrite the other..... as a result you will never see the core that is the root cause.

Which OS are you running?

if Linux, I use the following in /etc/sysctl.conf:

kernel.core_pattern=/tmp/core.%e.%p.%h.%t

On Thu, Mar 13, 2014 at 8:45 PM, Carsten Bock <carsten@ng-voice.commailto:carsten@ng-voice.com> wrote: It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens: mem_safety=1 See: http://www.kamailio.org/wiki/cookbooks/devel/core#mem_safety

Kind regards, Carsten

2014-03-13 19:44 GMT+01:00 Carsten Bock <carsten@ng-voice.commailto:carsten@ng-voice.com>:

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens:

2014-03-13 17:22 GMT+01:00 Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com>:

...

There are no more core files on the filesystem :-(

On 03/13/2014 12:18 PM, Jason Penton wrote:

I'm afraid this is also not the correct core. Can you check the timestamp on the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote:

...

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably have...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus <daniel.ciprus@acision.commailto:daniel.ciprus@acision.com> wrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe (lack=0x7fb4d8b31728) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601tel:%2B1%20804%20762%205601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.commailto:carsten@ng-voice.com

Office +49 40 34927219tel:%2B49%2040%2034927219 Fax +49 40 34927220tel:%2B49%2040%2034927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.commailto:carsten@ng-voice.com

Office +49 40 34927219tel:%2B49%2040%2034927219 Fax +49 40 34927220tel:%2B49%2040%2034927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

_______________________________________________ sr-dev mailing list sr-dev@lists.sip-router.orgmailto:sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Hugh Waite Principal Design Engineer Crocodile RCS Ltd.

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.commailto:daniel.ciprus@acision.com

________________________________ This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

Hey Daniel,

Can you build from source instead of using debian packages. This looks like an inconsistent build that you are running. I suspect some modules have not been built using the correct sources...

On Fri, Mar 14, 2014 at 12:22 AM, Daniel Ciprus daniel.ciprus@acision.comwrote:

For now iscscf crashed.

mem_safety=1

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in qm_debug_frag (qm=0x7f1b2794e000, f=0x7f1b27a69a00) at mem/q_malloc.c:142 __FUNCTION__ = "qm_debug_frag" #3 0x000000000054853a in qm_free (qm=0x7f1b2794e000, p=0x7f1b27a69a30, file=0x7f1b2ce7b0ea "tm: uac.c", func=0x7f1b2ce7baa0 "free_local_ack_unsafe", line=600) at mem/q_malloc.c:464 f = 0x7f1b27a69a00 size = 695784701953 next = 0x7f1b2ce60e08 prev = 0x7f1b2ce60c6d __FUNCTION__ = "qm_free" #4 0x00007f1b2ce5c7de in free_local_ack_unsafe (lack=0x7f1b27a69a30) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f1b2cdf2a57 in free_cell (dead_cell=0x7f1b27a69730) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x1
    cbs = 0x0
    cbs_tmp = 0x7c00000001
    __FUNCTION__ = "free_cell"

#6 0x00007f1b2cdf32ee in free_hash_table () at h_table.c:441 p_cell = 0x7f1b27a69730 tmp_cell = 0x7f1b27bff350

    i = 580
    __FUNCTION__ = "free_hash_table"

#7 0x00007f1b2ce06fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () at sr_module.c:817 t = 0x7f1b2d0d7df0 foo = 0x0 __FUNCTION__ = "destroy_modules" #9 0x0000000000466e63 in cleanup (show_status=1) at main.c:560 memlog = 32767 __FUNCTION__ = "cleanup" #10 0x0000000000467f65 in shutdown_children (sig=15, show_status=1) at main.c:702 __FUNCTION__ = "shutdown_children" #11 0x0000000000469679 in handle_sigs () at main.c:793 chld = 0 chld_status = 139 memlog = 0 __FUNCTION__ = "handle_sigs" #12 0x000000000046db19 in main_loop () at main.c:1746 i = 8 pid = 13103 si = 0x0 si_desc = "udp receiver child=7 sock=10.67.97.63:5060\000-\033\177\000\000\001\000\000\000z\000\000\000\376}^\000\000\000\000\000\340o^\000\000\000\000\000\026\070\266M\000\000\000\000\300LA\000\000\000\000\000\260\236\033\346\377\177", '\000' <repeats 18 times>"\360, \234\033\346\377\177\000\000.\241K\000\000\000\000" nrprocs = 8 __FUNCTION__ = "main_loop" #13 0x000000000047030b in main (argc=13, argv=0x7fffe61b9eb8) at main.c:2533 cfg_stream = 0x2753010 c = -1 r = 0 tmp = 0x7fffe61bae4b "" tmp_len = 0 port = 0 proto = 0 options = 0x5e0340 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" ret = -1 seed = 3395117202 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x31ba40fba0 p = 0x5ca460 "H\211l$\330L\211d$\340H\215-\217\233*" __FUNCTION__ = "main" (gdb)

Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_usrloc_pcscf [udomain.c:400]: update_pcontact(): no more shm_mem Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13641]: ERROR: ims_registrar_pcscf [save.c:208]: update_contacts(): failed to update pcscf contact Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13657]: ERROR: <core> [tcp_main.c:4237]: handle_tcpconn_ev(): connect 10.67.64.29:1305 failed Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13657]: ERROR: <core> [tcp_main.c:4237]: handle_tcpconn_ev(): connect 10.67.64.29:1305 failed Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13657]: ERROR: <core> [tcp_main.c:4237]: handle_tcpconn_ev(): connect 10.67.64.29:1325 failed Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13657]: ERROR: <core> [tcp_main.c:4237]: handle_tcpconn_ev(): connect 10.67.64.29:1325 failed Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13638]: WARNING: ims_registrar_pcscf [service_routes.c:440]: assert_identity(): Contact not found based on Contact, trying IP/Port/Proto Mar 13 18:15:51 ricvmf-fusion01 kam-scscf[13512]: ERROR: <script>: SUBSCRIBE (sip:18047621111@lab.ricuc.com (10.67.97.63:5090) to sip:18047621111@lab.ricuc.com, d1dc7671-e6fb-0071-1042-edba53716ada) Mar 13 18:15:51 ricvmf-fusion01 kam-scscf[13512]: ERROR: tm [tm.c:1302]: w_t_reply(): ERROR: t_reply: cannot send a t_reply to a message for which no T-state has been established Mar 13 18:15:51 ricvmf-fusion01 kam-pcscf[13638]: WARNING: ims_registrar_pcscf [service_routes.c:440]: assert_identity(): Contact not found based on Contact, trying IP/Port/Proto Mar 13 18:15:51 ricvmf-fusion01 kam-scscf[13512]: ERROR: <script>: SUBSCRIBE (sip:18047621111@lab.ricuc.com (10.67.97.63:5090) to sip:18047621111@lab.ricuc.com, d1dc7671-e6fb-0071-1042-edba53716ada) Mar 13 18:15:51 ricvmf-fusion01 kam-scscf[13512]: ERROR: tm [tm.c:1302]: w_t_reply(): ERROR: t_reply: cannot send a t_reply to a message for which no T-state has been established Mar 13 18:15:52 ricvmf-fusion01 kernel: kamailio[13083]: segfault at 7f1b27c05af0 ip 00007f1b27c05af0 sp 00007fffe61b9bb8 error 15 in zero (deleted)[7f1b2794e000+2000000] Mar 13 18:15:52 ricvmf-fusion01 kernel: kamailio[13083]: segfault at 7f1b27c05af0 ip 00007f1b27c05af0 sp 00007fffe61b9bb8 error 15 in zero (deleted)[7f1b2794e000+2000000] Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: ALERT: <core> [main.c:775]: handle_sigs(): child process 13083 exited by a signal 11 Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13103]: : <core> [pass_fd.c:293]: receive_fd(): ERROR: receive_fd: EOF on 18 Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: ALERT: <core> [main.c:778]: handle_sigs(): core was generated Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: INFO: <core> [main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLD Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13103]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13102]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13101]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13100]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13099]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13098]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13097]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13095]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13096]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13091]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13094]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13088]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13093]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: ALERT: <core> [main.c:775]: handle_sigs(): child process 13083 exited by a signal 11 Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13103]: : <core> [pass_fd.c:293]: receive_fd(): ERROR: receive_fd: EOF on 18 Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: ALERT: <core> [main.c:778]: handle_sigs(): core was generated Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: INFO: <core> [main.c:790]: handle_sigs(): INFO: terminating due to SIGCHLD Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13103]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13102]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13101]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13100]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13099]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13098]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13097]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13095]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13096]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13091]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13094]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13088]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13093]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13086]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13086]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13076]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13084]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13077]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13082]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13085]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13078]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13080]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13087]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13076]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13084]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13077]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13082]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13085]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13078]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13080]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13087]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13075]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13089]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13079]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13081]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13090]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13092]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13075]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13089]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13079]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13081]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13090]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13092]: INFO: <core> [main.c:841]: sig_usr(): INFO: signal 15 received Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: INFO: cdp [mod.c:251]: cdp_exit(): CDiameterPeer child stopping ... Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: INFO: cdp [diameter_peer.c:428]: diameter_peer_destroy(): destroy_diameter_peer(): Terminating all children... Mar 13 18:15:52 ricvmf-fusion01 kam-icscf[13074]: INFO: cdp [diameter_peer.c:435]: diameter_peer_destroy(): destroy_diameter_peer(): Waiting for child [13094] to terminate..

On 03/13/2014 05:47 PM, Hugh Waite wrote:

Dan, There are two cores because of a crash in one process followed by a crash when the other processes are trying to shutdown.

What's interesting is that the bt doesn't show useful pointers. If you have installed from RPMs make sure the kamailio-debuginfo is from the same build as the other RPMs.

Also, do the logs say anything? There should be a log entry from the kernel for the segfault/signal that says which module crashed (e.g. registrar.so) and possibly (hopefully) an error message just before that.

Hugh

On 13/03/2014 19:53, Daniel Ciprus wrote:

Jason,

I've tried multiple combinations for pattern but I'm getting only 2 core files ...

Details:

~]# cat /proc/sys/kernel/core_pattern /tmp/core.%e.sig%s.%p

~]# lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 6.5 (Santiago) Release: 6.5 Codename: Santiago

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? () No symbol table info available. #1 0x000000000053542a in ?? () No symbol table info available. #2 0x00000000005356c7 in timer_main () No symbol table info available. #3 0x000000000046d572 in main_loop () No symbol table info available. #4 0x000000000047030b in main () No symbol table info available. (gdb)

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f23d98f87de in free_local_ack_unsafe (lack=0x7f23d3319d70) at uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f23d988ea57 in free_cell (dead_cell=0x7f23d3319a70) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd3221000 cbs = 0x0 cbs_tmp = 0x7f23d35386b8 __FUNCTION__ = "free_cell" #6 0x00007f23d988f2ee in free_hash_table () at h_table.c:441 p_cell = 0x7f23d3319a70 tmp_cell = 0x7f23d353dca0 i = 580 __FUNCTION__ = "free_hash_table" #7 0x00007f23d98a2fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 02:58 PM, Jason Penton wrote:

I don't think these cores indicate the real crash... I'd like to get some more detail on what actually happened? Daniel, can you re-create? Keep in mind that if your core dump config on your box is not configured to name your cores according to process id or timestamp one core will overwrite the other..... as a result you will never see the core that is the root cause.

Which OS are you running?

if Linux, I use the following in /etc/sysctl.conf:

kernel.core_pattern=/tmp/core.%e.%p.%h.%t

On Thu, Mar 13, 2014 at 8:45 PM, Carsten Bock carsten@ng-voice.comwrote:

...

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens: mem_safety=1 See: http://www.kamailio.org/wiki/cookbooks/devel/core#mem_safety

Kind regards, Carsten

2014-03-13 19:44 GMT+01:00 Carsten Bock carsten@ng-voice.com:

...

It looks a little bit like a "double free".

You could try to disable the call to "abort()" in case this happens:

2014-03-13 17:22 GMT+01:00 Daniel Ciprus daniel.ciprus@acision.com:

...

There are no more core files on the filesystem :-(

On 03/13/2014 12:18 PM, Jason Penton wrote:

I'm afraid this is also not the correct core. Can you check the

timestamp on

...

...

the cores? Can you re-create the crash and send me the correct core?

On Thu, Mar 13, 2014 at 5:36 PM, Daniel Ciprus <

daniel.ciprus@acision.com>

...

...

wrote:

...

So I cleaned up my junkyard and I got 2 core files:

(gdb) bt #0 0x00000000005350b0 in ?? () #1 0x000000000053542a in ?? () #2 0x00000000005356c7 in timer_main () #3 0x000000000046d572 in main_loop () #4 0x000000000047030b in main () (gdb) bt full #0 0x00000000005350b0 in ?? ()

No symbol table info available. #1 0x000000000053542a in ?? ()

No symbol table info available. #2 0x00000000005356c7 in timer_main ()

No symbol table info available. #3 0x000000000046d572 in main_loop ()

No symbol table info available. #4 0x000000000047030b in main ()

No symbol table info available. (gdb)

(gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007f5bf7d5a7de in free_local_ack_unsafe (lack=0x7f5bf1894528)

at

...

...

...

uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007f5bf7cf0a57 in free_cell (dead_cell=0x7f5bf1894228) at h_table.c:217

    b = 0x0
    i = 0
    rpl = 0x0
    tt = 0x0
    foo = 0x2ff1683000
    cbs = 0x0
    cbs_tmp = 0x7f5bf198e508
    __FUNCTION__ = "free_cell"

#6 0x00007f5bf7cf12ee in free_hash_table () at h_table.c:441 p_cell = 0x7f5bf1894228 tmp_cell = 0x7f5bf1894228 i = 3533 __FUNCTION__ = "free_hash_table" #7 0x00007f5bf7d04fca in tm_shutdown () at t_funcs.c:122

    __FUNCTION__ = "tm_shutdown"

#8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

On 03/13/2014 11:18 AM, Jason Penton wrote:

Hi Daniel,

this is the wrong core file. This is the one created on shutdown of kamailio. Can you do a bt on the other core file that you probably

have...

...

...

...

Cheers Jason

On Thu, Mar 13, 2014 at 5:05 PM, Daniel Ciprus <

daniel.ciprus@acision.com>

...

...

...

wrote:

...

Folks,

This is happening during the registration on SCSCF.

Server:: kamailio (4.2.0-dev2 (x86_64/linux)) Build:: mi_core.c compiled on 10:01:09 Mar 13 2014 with gcc 4.4.6 Flags:: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES GIT:: unknown Now:: Thu Mar 13 11:04:47 2014 Up since:: Thu Mar 13 10:58:12 2014 Up time:: 395 [sec]

(gdb) bt #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 #2 0x0000000000546750 in ?? () #3 0x000000000054853a in qm_free () #4 0x00007fb4def5b7de in free_local_ack_unsafe

(lack=0x7fb4d8b31728) at

...

...

...

...

uac.c:600 #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 #8 0x00000000004f7c7a in destroy_modules () #9 0x0000000000466e63 in cleanup () #10 0x0000000000467f65 in ?? () #11 0x0000000000469679 in handle_sigs () #12 0x000000000046db19 in main_loop () #13 0x000000000047030b in main () (gdb) bt full #0 0x00000031ba432925 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00000031ba434105 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x0000000000546750 in ?? () No symbol table info available. #3 0x000000000054853a in qm_free () No symbol table info available. #4 0x00007fb4def5b7de in free_local_ack_unsafe

(lack=0x7fb4d8b31728) at

...

...

...

...

uac.c:600 __FUNCTION__ = "free_local_ack_unsafe" #5 0x00007fb4deef1a57 in free_cell (dead_cell=0x7fb4d8b31428) at h_table.c:217 b = 0x0 i = 0 rpl = 0x0 tt = 0x0 foo = 0x2fd8a8b000 cbs = 0x0 cbs_tmp = 0x7fb4d8d9c9e0 __FUNCTION__ = "free_cell" #6 0x00007fb4deef22ee in free_hash_table () at h_table.c:441 p_cell = 0x7fb4d8b31428 tmp_cell = 0x7fb4d8b31428 i = 11517 __FUNCTION__ = "free_hash_table" #7 0x00007fb4def05fca in tm_shutdown () at t_funcs.c:122 __FUNCTION__ = "tm_shutdown" #8 0x00000000004f7c7a in destroy_modules () No symbol table info available. #9 0x0000000000466e63 in cleanup () No symbol table info available. #10 0x0000000000467f65 in ?? () No symbol table info available. #11 0x0000000000469679 in handle_sigs () No symbol table info available. #12 0x000000000046db19 in main_loop () No symbol table info available. #13 0x000000000047030b in main () No symbol table info available. (gdb)

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel Ciprus Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be

copied,

...

...

disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.com

Office +49 40 34927219 Fax +49 40 34927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- Carsten Bock CEO (Geschäftsführer)

ng-voice GmbH Schomburgstr. 80 D-22767 Hamburg / Germany

http://www.ng-voice.com mailto:carsten@ng-voice.com

Office +49 40 34927219 Fax +49 40 34927220

Sitz der Gesellschaft: Hamburg Registergericht: Amtsgericht Hamburg, HRB 120189 Geschäftsführer: Carsten Bock Ust-ID: DE279344284

Hier finden Sie unsere handelsrechtlichen Pflichtangaben: http://www.ng-voice.com/imprint/

-- *Daniel Ciprus* Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Hugh Waite Principal Design Engineer Crocodile RCS Ltd.

-- *Daniel Ciprus* Integration engineer http://www.acision.com

9954 Mayland Dr Suite 3100 Richmond, VA 23233 USA T: +1 804 762 5601 E: daniel.ciprus@acision.com

---

This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

---

sr-dev mailing list sr-dev@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev