Hello,

checking CPU/I/O performance of the database during the load test would be a start. You can also monitor database access e.g. with a network debugging tools.

Cheers,

Henning

--

Henning Westerholt – https://skalatan.de/blog/

Kamailio services – https://gilawa.com

From: Евгений Мастратов <dgeka25594@gmail.com>
Sent: Tuesday, April 12, 2022 10:03 AM
To: Henning Westerholt <hw@gilawa.com>
Subject: Re: [SR-Users] Kamailio IMS heavy load

Hey Henning!

Thanks for the advice, I understand about logging, I will make edits and re-check. And how can I check the fact that what are the gaps when reading / writing the database? I am using MySQL.

вт, 12 апр. 2022 г. в 10:52, Henning Westerholt <hw@gilawa.com>:

Hello,

as mentioned, I would start by monitoring my dependencies, in particular related to I/O, like database or monitoring.

Just to be sure, you are not running the tests with enabled debug log level? This would certainly cause problems.

Cheers,

Henning

--

Henning Westerholt – https://skalatan.de/blog/

Kamailio services – https://gilawa.com

From: sr-users <sr-users-bounces@lists.kamailio.org> On Behalf Of ??????? ?????????
Sent: Monday, April 11, 2022 5:23 PM
To: sr-users@lists.kamailio.org
Subject: [SR-Users] Kamailio IMS heavy load

Hi!

Previously, I created a ticket on github https://github.com/kamailio/kamailio/issues/3071 and tried to deal with the problem there, but they sent me here for advice.

I set up Kamailio IMS , in the end everything turned out great, calls are made, and I decided to check how Kamailio copes with the load. I am creating a load using the sipp application. There is no problem with a low call generation rate, but if you raise the call generation to more than 20 calls per second, some of the calls fail.

I compared the calls and saw that the ICSCF does not send a 180 Ringing message towards the S-CSCF

[..]    

As I found out further and suggested, the problem is that some calls are cut on the S-CSCF and this is probably due to some kind of timer, but I could not figure out which one.

Full logs and dump are attached to the issue on github, below is a log sample by bad dialog 0x7fc133ce61d8

```
Apr  5 08:24:33 ims-core /usr/local/sbin/kamailio[26061]: DEBUG: ims_dialog [dlg_handlers.c:284]: dlg_iuid_sfree(): freeing dlg iuid [465:10905] (0x7fc133ce61d8)
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1025]: link_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:251]: run_create_callbacks(): dialog=0x7fc133ce61d8
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:789]: dlg_onreq(): dialog [0x7fc133ce61d8] added to tm callbacks
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:35 ims-core /usr/local/sbin/kamailio[26059]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:887]: lookup_dlg(): ref dlg 0x7fc133ce61d8 with 1 -> 2
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1293]: next_state_dlg(): dialog 0x7fc133ce61d8 changed from state 1 to state 6, due event 4
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_handlers.c:1489]: dlg_onreply(): dialog 0x7fc133ce61d8 failed (negative reply)
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=4
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 1
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): unref dlg 0x7fc133ce61d8 with 1 -> 0
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:1066]: unref_dlg(): ref <=0 for dialog 0x7fc133ce61d8
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:196]: destroy_dlg(): destroying dialog 0x7fc133ce61d8
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_hash.c:207]: destroy_dlg(): removed timer for dlg 0x7fc133ce61d8 [3791:1419] with clid '43-3057631@10.2.16.63' and tags '78885556600'
Apr  5 08:24:38 ims-core /usr/local/sbin/kamailio[26056]: DEBUG: ims_dialog [dlg_cb.c:277]: run_dlg_callbacks(): dialog=0x7fc133ce61d8, type=8192
```

What could be the reasons for this behavior and how to diagnose them? I did not observe problems with MEM or CPU during the tests.

I will be grateful for any advice.