[SR-Users] Kamailio IMS heavy load

Hi! Previously, I created a ticket on github https://github.com/kamailio/kamailio/issues/3071 and tried to deal with the problem there, but they sent me here for advice. I set up Kamailio IMS , in the end everything turned out great, calls are made, and I decided to check how Kamailio copes with the load. I am creating a load using the sipp application. There is no problem with a low call generation rate, but if you raise the call generation to more than 20 calls per second, some of the calls fail. I compared the calls and saw that the ICSCF does not send a 180 Ringing message towards the S-CSCF [image: image.png] [image: image.png] As I found out further and suggested, the problem is that some calls are cut on the S-CSCF and this is probably due to some kind of timer, but I could not figure out which one. Full logs and dump are attached to the issue on github, below is a log sample by bad dialog 0x7fc133ce61d8 ``` Apr 5 08:24:33 ims-core /usr/local/sbin/kamailio[26061]: DEBUG: ims_dialog [dlg_handlers.c:284]: dlg_iuid_sfree(): freeing dlg iuid [465:10905] (0x7fc133ce61d8) Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1025]: link_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:251]: run_create_callbacks(): dialog=0x7fc133ce61d8 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:789]: dlg_onreq(): dialog [0x7fc133ce61d8] added to tm callbacks Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1293]: next_state_dlg(): dialog 0x7fc133ce61d8 changed from state 1 to state 6, due event 4 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:1489]: dlg_onreply(): dialog 0x7fc133ce61d8 failed (negative reply) Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=4 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 0 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): ref <=0 for dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:196]: destroy_dlg(): destroying dialog 0x7fc133ce61d8 Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:207]: destroy_dlg(): removed timer for dlg 0x7fc133ce61d8 [3791:1419] with clid &#39;43-3057631(a)10.2.16.63&#39; and tags '78885556600' Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=8192 ``` What could be the reasons for this behavior and how to diagnose them? I did not observe problems with MEM or CPU during the tests. I will be grateful for any advice.