Hello,

as mentioned, I would start by monitoring my dependencies, in particular related to I/O, like database or monitoring.

Just to be sure, you are not running the tests with enabled debug log level? This would certainly cause problems.

Cheers,

Henning

Henning Westerholt – https://skalatan.de/blog/

From: sr-users <sr-users-bounces@lists.kamailio.org> On Behalf Of ??????? ?????????
Sent: Monday, April 11, 2022 5:23 PM
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Kamailio IMS heavy load

Hi!

Previously, I created a ticket on github https://github.com/kamailio/kamailio/issues/3071 and tried to deal with the problem there, but they sent me here for advice.

I set up Kamailio IMS , in the end everything turned out great, calls are made, and I decided to check how Kamailio copes with the load. I am creating a load using the sipp application. There is no problem with a low call generation rate, but if you raise the call generation to more than 20 calls per second, some of the calls fail.

I compared the calls and saw that the ICSCF does not send a 180 Ringing message towards the S-CSCF

As I found out further and suggested, the problem is that some calls are cut on the S-CSCF and this is probably due to some kind of timer, but I could not figure out which one.

Full logs and dump are attached to the issue on github, below is a log sample by bad dialog 0x7fc133ce61d8

```
Apr 5 08:24:33 ims-core /usr/local/sbin/kamailio[26061]: DEBUG: ims_dialog [dlg_handlers.c:284]: dlg_iuid_sfree(): freeing dlg iuid [465:10905] (0x7fc133ce61d8)
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1025]: link_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:251]: run_create_callbacks(): dialog=0x7fc133ce61d8
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:789]: dlg_onreq(): dialog [0x7fc133ce61d8] added to tm callbacks
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1293]: next_state_dlg(): dialog 0x7fc133ce61d8 changed from state 1 to state 6, due event 4
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:1489]: dlg_onreply(): dialog 0x7fc133ce61d8 failed (negative reply)
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=4
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 0
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): ref <=0 for dialog 0x7fc133ce61d8
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:196]: destroy_dlg(): destroying dialog 0x7fc133ce61d8
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:207]: destroy_dlg(): removed timer for dlg 0x7fc133ce61d8 [3791:1419] with clid '43-3057631@10.2.16.63' and tags '78885556600'
Apr 5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=8192
```

What could be the reasons for this behavior and how to diagnose them? I did not observe problems with MEM or CPU during the tests.

I will be grateful for any advice.