Sure, here is the list of module in given order,

-- loadmodule "db_mysql.so" loadmodule "mi_fifo.so" loadmodule "mi_datagram.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "outbound.so" loadmodule "rr.so" loadmodule "path.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "sdpops.so" loadmodule "textops.so" loadmodule "textopsx.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" loadmodule "mi_rpc.so" loadmodule "dialog.so" loadmodule "acc.so" loadmodule "uac.so" loadmodule "rtimer.so" loadmodule "sqlops.so" loadmodule "ndb_redis.so" loadmodule "app_perl.so" loadmodule "permissions.so" loadmodule "domain.so" loadmodule "async.so" loadmodule "stun.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "alias_db.so" loadmodule "speeddial.so" loadmodule "presence.so" loadmodule "presence_mwi.so" loadmodule "presence_xml.so" loadmodule "presence_profile.so" loadmodule "nathelper.so" loadmodule "rtpengine.so" loadmodule "tls.so" loadmodule "htable.so" loadmodule "pike.so" loadmodule "xmlrpc.so" loadmodule "debugger.so" loadmodule "xhttp.so" loadmodule "xhttp_rpc.so" loadmodule "xhttp_pi.so" loadmodule "xcap_server.so" loadmodule "pua.so" loadmodule "pua_mi.so" loadmodule "rls.so" loadmodule "cfgutils.so" loadmodule "htable.so" loadmodule "msrp.so" loadmodule "websocket.so" loadmodule "msilo.so" loadmodule "siptrace.so" --

In "top" i see at least 6 kamailio processes using very high cpu (perhaps these are the 6 child processes involved in that single call processing).

Thank you.

On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

Hello,

I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here the list of modules you are using? The loadmodule lines in kamailio.cfg.

Cheers, Daniel

On 14/09/15 10:51, M S wrote:

Hi,

Over last weekend i upgraded one of my test servers from Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes crazy even with single call (I am using same db and configuration of course).

As soon as call establishes system load average (as seen in top command) starts increasing and soon it increases beyond 6.0 and system becomes completely unresponsive, sip messages are no longer being processed by kamailio service. Even after call hangup, system remains under high load. The "htop" indicates that IO Wait time has increased substantially.

Any idea what is causing this? For now i have reverted by to v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or testing.

Thank you.

---

sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

I found a regression when detecting spiraled dialogs with state deleted. That would be the case when you create a dialog for an INVITE, but you reply with a negative response code from kamailio and then quickly a new invite with same callid and from tag arrives.

Can you upgrade to latest version in branch 4.3, try and see if works fine now?

Cheers, Daniel

On 14/09/15 11:55, M S wrote:

OK runtime debugging shows something is wrong with dialog module, all 6 processes are locked in trying to access dialog module when 200 OK is received for the call. For example, here is the BT of one the processes,

-- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643 #5 0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at dlg_handlers.c:429 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358, req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295 #8 0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4) at t_lookup.c:966 #9 0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1069 #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1111 #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134 #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at forward.c:747 #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849 #14 0x08138aab in receive_msg ( buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia: SIP/2.0/UDP 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"..., len=1103, rcv_info=0xbff3a93c) at receive.c:255 #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495 #16 0x080dec65 in main_loop () at main.c:1573

#17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533

Here is BT of second process,

-- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643 #5 0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072, param=0xbff3a7ac) at dlg_handlers.c:398 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464, type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424, req=0x0, rpl=0x0, code=0) at t_hooks.c:295 #8 0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128 #9 0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c, data=0xa5985424) at timer.c:648 #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718, slow_l=0xa530e7ec, slow_mark=6) at timer.c:873 #11 0x0820dcc0 in timer_handler () at timer.c:938 #12 0x0820e0d8 in timer_main () at timer.c:977 #13 0x080df4e0 in main_loop () at main.c:1644

#14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533

Does this makes any sense to you? Let me know if you need anything else.

Thank you.

On Mon, Sep 14, 2015 at 11:36 AM, M S <shaheryarkh@gmail.com mailto:shaheryarkh@gmail.com> wrote:

Sure, here is the list of module in given order,

--
loadmodule "db_mysql.so"
loadmodule "mi_fifo.so"
loadmodule "mi_datagram.so"
loadmodule "kex.so"
loadmodule "corex.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "outbound.so"
loadmodule "rr.so"
loadmodule "path.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "usrloc.so"
loadmodule "registrar.so"
loadmodule "sdpops.so"
loadmodule "textops.so"
loadmodule "textopsx.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "cfg_rpc.so"
loadmodule "mi_rpc.so"
loadmodule "dialog.so"
loadmodule "acc.so"
loadmodule "uac.so"
loadmodule "rtimer.so"
loadmodule "sqlops.so"
loadmodule "ndb_redis.so"
loadmodule "app_perl.so"
loadmodule "permissions.so"
loadmodule "domain.so"
loadmodule "async.so"
loadmodule "stun.so"
loadmodule "auth.so"
loadmodule "auth_db.so"
loadmodule "alias_db.so"
loadmodule "speeddial.so"
loadmodule "presence.so"
loadmodule "presence_mwi.so"
loadmodule "presence_xml.so"
loadmodule "presence_profile.so"
loadmodule "nathelper.so"
loadmodule "rtpengine.so"
loadmodule "tls.so"
loadmodule "htable.so"
loadmodule "pike.so"
loadmodule "xmlrpc.so"
loadmodule "debugger.so"
loadmodule "xhttp.so"
loadmodule "xhttp_rpc.so"
loadmodule "xhttp_pi.so"
loadmodule "xcap_server.so"
loadmodule "pua.so"
loadmodule "pua_mi.so"
loadmodule "rls.so"
loadmodule "cfgutils.so"
loadmodule "htable.so"
loadmodule "msrp.so"
loadmodule "websocket.so"
loadmodule "msilo.so"
loadmodule "siptrace.so"
--

In "top" i see at least 6 kamailio processes using very high cpu
(perhaps these are the 6 child processes involved in that single
call processing).

Thank you.



On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla
<miconda@gmail.com <mailto:miconda@gmail.com>> wrote:

    Hello,

    I will review the changes pushed to 4.3.2 vs 4.3.1. Can you
    send here the list of modules you are using? The loadmodule
    lines in kamailio.cfg.

    Cheers,
    Daniel


    On 14/09/15 10:51, M S wrote:

...

    Hi,

    Over last weekend i upgraded one of my test servers from
    Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f
    and now kamailio goes crazy even with single call (I am using
    same db and configuration of course).

    As soon as call establishes system load average (as seen in
    top command) starts increasing and soon it increases beyond
    6.0 and system becomes completely unresponsive, sip messages
    are no longer being processed by kamailio service. Even after
    call hangup, system remains under high load. The "htop"
    indicates that IO Wait time has increased substantially.

    Any idea what is causing this? For now i have reverted by to
    v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need
    further info or testing.

    Thank you.


    _______________________________________________
    sr-dev mailing list
    sr-dev@lists.sip-router.org <mailto:sr-dev@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

    -- 
    Daniel-Constantin Mierla
    http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda
    Book: SIP Routing With Kamailio - http://www.asipto.com
    Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat


    _______________________________________________
    SIP Express Router (SER) and Kamailio (OpenSER) - sr-users
    mailing list
    sr-users@lists.sip-router.org
    <mailto:sr-users@lists.sip-router.org>
    http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

Hi,

After doing all the test cases, it seems the problem has been resolved in v4.3.2-3763f6.

Thank you so much for your help in this timely fix.

On Mon, Sep 14, 2015 at 12:52 PM, M S shaheryarkh@gmail.com wrote:

OK, let me test it and get back to you.

Thank you.

On Mon, Sep 14, 2015 at 12:18 PM, Daniel-Constantin Mierla < miconda@gmail.com> wrote:

...

I found a regression when detecting spiraled dialogs with state deleted. That would be the case when you create a dialog for an INVITE, but you reply with a negative response code from kamailio and then quickly a new invite with same callid and from tag arrives.

Can you upgrade to latest version in branch 4.3, try and see if works fine now?

Cheers, Daniel

On 14/09/15 11:55, M S wrote:

OK runtime debugging shows something is wrong with dialog module, all 6 processes are locked in trying to access dialog module when 200 OK is received for the call. For example, here is the BT of one the processes,

-- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11302) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa598a2b0) at dlg_hash.c:643 #5 0xb5a14143 in dlg_onreply (t=0xa5989358, type=2, param=0xbff3a0ac) at dlg_handlers.c:429 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5989398, type=2, trans=0xa5989358, params=0xbff3a0ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=2, trans=0xa5989358, req=0xa598a308, rpl=0xb68cc6e8, code=200) at t_hooks.c:295 #8 0xb5f84b14 in t_reply_matching (p_msg=0xb68cc6e8, p_branch=0xbff3a5a4) at t_lookup.c:966 #9 0xb5f868c4 in t_check_msg (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1069 #10 0xb5f871fd in t_check (p_msg=0xb68cc6e8, param_branch=0xbff3a5a4) at t_lookup.c:1111 #11 0xb5fc9b4e in reply_received (p_msg=0xb68cc6e8) at t_reply.c:2134 #12 0x080c9142 in do_forward_reply (msg=0xb68cc6e8, mode=0) at forward.c:747 #13 0x080ca61c in forward_reply (msg=0xb68cc6e8) at forward.c:849 #14 0x08138aab in receive_msg ( buf=0x840f6c0 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP X.X.X.X;rport=5060;branch=z9hG4bK719e.844788c1e820e6096cb00d2b2f6c613d.0\r\nVia: SIP/2.0/UDP 192.168.3.10;received=Y.Y.Y.Y;rport=5060;branch=z9hG4bKlzzjpctt\r\nRe"..., len=1103, rcv_info=0xbff3a93c) at receive.c:255 #15 0x082188d0 in udp_rcv_loop () at udp_server.c:495 #16 0x080dec65 in main_loop () at main.c:1573

#17 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533

Here is BT of second process,

-- #0 0xb77a0424 in __kernel_vsyscall () #1 0xb76c140c in sched_yield () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 #2 0xb5a20206 in get_lock (lock=0xa55058e4) at ../../mem/../fastlock.h:277 #3 0xb5a25a9f in dlg_lookup (h_entry=2405, h_id=11301) at dlg_hash.c:610 #4 0xb5a26446 in dlg_get_by_iuid (diuid=0xa5985060) at dlg_hash.c:643 #5 0xb5a140c4 in dlg_ontdestroy (t=0xa5985424, type=131072, param=0xbff3a7ac) at dlg_handlers.c:398 #6 0xb5f78092 in run_trans_callbacks_internal (cb_lst=0xa5985464, type=131072, trans=0xa5985424, params=0xbff3a7ac) at t_hooks.c:268 #7 0xb5f781a3 in run_trans_callbacks (type=131072, trans=0xa5985424, req=0x0, rpl=0x0, code=0) at t_hooks.c:295 #8 0xb5f3cbbb in free_cell (dead_cell=0xa5985424) at h_table.c:128 #9 0xb5f7cb20 in wait_handler (ti=1848316837, wait_tl=0xa598546c, data=0xa5985424) at timer.c:648 #10 0x0820d8f8 in timer_list_expire (t=1848316837, h=0xa530e718, slow_l=0xa530e7ec, slow_mark=6) at timer.c:873 #11 0x0820dcc0 in timer_handler () at timer.c:938 #12 0x0820e0d8 in timer_main () at timer.c:977 #13 0x080df4e0 in main_loop () at main.c:1644

#14 0x080e4c99 in main (argc=13, argv=0xbff3ad44) at main.c:2533

Does this makes any sense to you? Let me know if you need anything else.

Thank you.

On Mon, Sep 14, 2015 at 11:36 AM, M S shaheryarkh@gmail.com wrote:

...

Sure, here is the list of module in given order,

-- loadmodule "db_mysql.so" loadmodule "mi_fifo.so" loadmodule "mi_datagram.so" loadmodule "kex.so" loadmodule "corex.so" loadmodule "tm.so" loadmodule "tmx.so" loadmodule "sl.so" loadmodule "outbound.so" loadmodule "rr.so" loadmodule "path.so" loadmodule "pv.so" loadmodule "maxfwd.so" loadmodule "usrloc.so" loadmodule "registrar.so" loadmodule "sdpops.so" loadmodule "textops.so" loadmodule "textopsx.so" loadmodule "siputils.so" loadmodule "xlog.so" loadmodule "sanity.so" loadmodule "ctl.so" loadmodule "cfg_rpc.so" loadmodule "mi_rpc.so" loadmodule "dialog.so" loadmodule "acc.so" loadmodule "uac.so" loadmodule "rtimer.so" loadmodule "sqlops.so" loadmodule "ndb_redis.so" loadmodule "app_perl.so" loadmodule "permissions.so" loadmodule "domain.so" loadmodule "async.so" loadmodule "stun.so" loadmodule "auth.so" loadmodule "auth_db.so" loadmodule "alias_db.so" loadmodule "speeddial.so" loadmodule "presence.so" loadmodule "presence_mwi.so" loadmodule "presence_xml.so" loadmodule "presence_profile.so" loadmodule "nathelper.so" loadmodule "rtpengine.so" loadmodule "tls.so" loadmodule "htable.so" loadmodule "pike.so" loadmodule "xmlrpc.so" loadmodule "debugger.so" loadmodule "xhttp.so" loadmodule "xhttp_rpc.so" loadmodule "xhttp_pi.so" loadmodule "xcap_server.so" loadmodule "pua.so" loadmodule "pua_mi.so" loadmodule "rls.so" loadmodule "cfgutils.so" loadmodule "htable.so" loadmodule "msrp.so" loadmodule "websocket.so" loadmodule "msilo.so" loadmodule "siptrace.so" --

In "top" i see at least 6 kamailio processes using very high cpu (perhaps these are the 6 child processes involved in that single call processing).

Thank you.

On Mon, Sep 14, 2015 at 11:28 AM, Daniel-Constantin Mierla < miconda@gmail.commiconda@gmail.com> wrote:

...

Hello,

I will review the changes pushed to 4.3.2 vs 4.3.1. Can you send here the list of modules you are using? The loadmodule lines in kamailio.cfg.

Cheers, Daniel

On 14/09/15 10:51, M S wrote:

Hi,

Over last weekend i upgraded one of my test servers from Kamailio v4.3.1-4d1b65 to latest stable release v4.3.2-07690f and now kamailio goes crazy even with single call (I am using same db and configuration of course).

As soon as call establishes system load average (as seen in top command) starts increasing and soon it increases beyond 6.0 and system becomes completely unresponsive, sip messages are no longer being processed by kamailio service. Even after call hangup, system remains under high load. The "htop" indicates that IO Wait time has increased substantially.

Any idea what is causing this? For now i have reverted by to v4.3.1-4d1b65 but can go to v4.3.2-07690f again if you need further info or testing.

Thank you.

---

sr-dev mailing listsr-dev@lists.sip-router.orghttp://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat

---

SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Sep 28-30, 2015, in Berlin - http://asipto.com/u/kat