Hey Alex,

Thanks for the insight. I remember reading your article before and it was super helpful. I think we disabled all of the heavy database operations, but I will double check.

We are building the RURI using the cfg_rpc module, so we aren’t hitting a database. We have a failure route that causes another RURI to be created, which sends the call to a Freeswitch server to play a message.

Let me know if you see any issue with this, I will double check the database calls to be safe.

Sent from my iPhone

On May 2, 2018, at 11:36 PM, Alex Balashov abalashov@evaristesys.com wrote:

Hi Mack,

What sort of end-to-end delays are you seeing, from receipt of request until immediately before t_relay()?

You can profile this with the benchmark module:

https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html

Or my preferred method, the hard way:

   # At the practical start of request processing.

   $avp(proc_start) = $TV(Sn);

   # At the end, right before calling t_relay() or forward().

   $var(cur_time) = $TV(Sn);

   $var(proc_diff) = (
           (((
             $(var(cur_time){s.select,0,.}{s.int}) -
             $(avp(proc_start){s.select,0,.}{s.int})
           ) * 1000000)
           +
           (
            $(var(cur_time){s.select,1,.}{s.int}) -
            $(avp(proc_start){s.select,1,.}{s.int})
           ) / 1000) mod 1000
   );

   xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n"););

Also, for general background on this topic, you might consider my article on the subject of Kamailio performance characteristics:

http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performan...

If I had to guess, shooting from the hip, you've got some really slow database queries somewhere.

-- Alex

...

On Wed, May 02, 2018 at 06:19:00PM -0400, Mack Hendricks wrote:

Hey All,

We are load testing Kamailio 4.2.3 and we are seeing the waiting within the transaction manager increasing. I’ve increased the children from 8 to 16 to 24, but we are still seeing a large amount of waiting. I don’t see an errors in the logs. Can anyone give me a clue on where to look.

05-02-18_03:07:38 PM tm.stats { current: 390 waiting: 226 total: 2361 total_local: 0 replied_locally: 1406 6xx: 0 5xx: 0 4xx: 88 3xx: 0 2xx: 3180 created: 2361 freed: 1971 delayed_free: 0 }

Mack Hendricks / Head of Support / dOpenSource web: http://dopensource.com http://dopensource.com/ support: +888-907-2085 dSIPRouter http://dsiprouter.org/ - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services

...

---

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

Hello,

do not forget about the core parameters related to latency, they can be useful to detect slow actions or db queries:

- https://www.kamailio.org/wiki/cookbooks/devel/core#latency_cfg_log

Cheers, Daniel

On 03.05.18 06:49, Alex Balashov wrote:

Hi Mack,

Happy to help!

There must be a hidden source of I/O wait somewhere. It may not be database calls, but there's something. This shouldn't be happening in a mostly static config.

Can you share the numbers from your load testing? At what point did you start seeing retransmissions, if any? What kind of delays are you seeing?

-- Alex

On Thu, May 03, 2018 at 12:48:38AM -0400, Mack Hendricks wrote:

...

Hey Alex,

Thanks for the insight. I remember reading your article before and it was super helpful. I think we disabled all of the heavy database operations, but I will double check.

We are building the RURI using the cfg_rpc module, so we aren’t hitting a database. We have a failure route that causes another RURI to be created, which sends the call to a Freeswitch server to play a message.

Let me know if you see any issue with this, I will double check the database calls to be safe.

Sent from my iPhone

...

On May 2, 2018, at 11:36 PM, Alex Balashov abalashov@evaristesys.com wrote:

Hi Mack,

What sort of end-to-end delays are you seeing, from receipt of request until immediately before t_relay()?

You can profile this with the benchmark module:

https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html

Or my preferred method, the hard way:

   # At the practical start of request processing.

   $avp(proc_start) = $TV(Sn);

   # At the end, right before calling t_relay() or forward().

   $var(cur_time) = $TV(Sn);

   $var(proc_diff) = (
           (((
             $(var(cur_time){s.select,0,.}{s.int}) -
             $(avp(proc_start){s.select,0,.}{s.int})
           ) * 1000000)
           +
           (
            $(var(cur_time){s.select,1,.}{s.int}) -
            $(avp(proc_start){s.select,1,.}{s.int})
           ) / 1000) mod 1000
   );

   xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n"););

Also, for general background on this topic, you might consider my article on the subject of Kamailio performance characteristics:

http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performan...

If I had to guess, shooting from the hip, you've got some really slow database queries somewhere.

-- Alex

...

On Wed, May 02, 2018 at 06:19:00PM -0400, Mack Hendricks wrote:

Hey All,

We are load testing Kamailio 4.2.3 and we are seeing the waiting within the transaction manager increasing. I’ve increased the children from 8 to 16 to 24, but we are still seeing a large amount of waiting. I don’t see an errors in the logs. Can anyone give me a clue on where to look.

05-02-18_03:07:38 PM tm.stats { current: 390 waiting: 226 total: 2361 total_local: 0 replied_locally: 1406 6xx: 0 5xx: 0 4xx: 88 3xx: 0 2xx: 3180 created: 2361 freed: 1971 delayed_free: 0 }

Mack Hendricks / Head of Support / dOpenSource web: http://dopensource.com http://dopensource.com/ support: +888-907-2085 dSIPRouter http://dsiprouter.org/ - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services

---

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

Hey Daniel,

Yes, in this case we are using the cfg_rpc module:

This is what we were using:

$ru = "sip:" + $rU + "@" + $sel(cfg_get.teleblock.gw_ip) + ":" + $sel(cfg_get.teleblock.gw_port);

Now, I will hardcode it to see if it make a difference

$ru = "sip:" + $rU + "@" + “66.203.XX.XXX:5033”;

I turned on the general log and slow log in mysql and we hard hitting the database except during startup and subscriber lookup, which isn’t in play here.

Mack Hendricks / Head of Support / dOpenSource web: http://dopensource.com http://dopensource.com/ support: +888-907-2085 dSIPRouter http://dsiprouter.org/ - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services

On May 3, 2018, at 2:11 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:

Hello,

On 03.05.18 06:48, Mack Hendricks wrote:

...

Hey Alex,

Thanks for the insight. I remember reading your article before and it was super helpful. I think we disabled all of the heavy database operations, but I will double check.

We are building the RURI using the cfg_rpc module,

are you really using cfg_rpc module to build the r-uri? Or did you mean someting like jsonrpcc?

Cheers, Daniel

...

so we aren’t hitting a database. We have a failure route that causes another RURI to be created, which sends the call to a Freeswitch server to play a message.

Let me know if you see any issue with this, I will double check the database calls to be safe.

Sent from my iPhone

...

On May 2, 2018, at 11:36 PM, Alex Balashov abalashov@evaristesys.com wrote:

Hi Mack,

What sort of end-to-end delays are you seeing, from receipt of request until immediately before t_relay()?

You can profile this with the benchmark module:

https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html

Or my preferred method, the hard way:

  # At the practical start of request processing.

  $avp(proc_start) = $TV(Sn);

  # At the end, right before calling t_relay() or forward().

  $var(cur_time) = $TV(Sn);

  $var(proc_diff) = (
          (((
            $(var(cur_time){s.select,0,.}{s.int}) -
            $(avp(proc_start){s.select,0,.}{s.int})
          ) * 1000000)
          +
          (
           $(var(cur_time){s.select,1,.}{s.int}) -
           $(avp(proc_start){s.select,1,.}{s.int})
          ) / 1000) mod 1000
  );

  xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n"););

Also, for general background on this topic, you might consider my article on the subject of Kamailio performance characteristics:

http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performan...

If I had to guess, shooting from the hip, you've got some really slow database queries somewhere.

-- Alex

...

On Wed, May 02, 2018 at 06:19:00PM -0400, Mack Hendricks wrote:

Hey All,

We are load testing Kamailio 4.2.3 and we are seeing the waiting within the transaction manager increasing. I’ve increased the children from 8 to 16 to 24, but we are still seeing a large amount of waiting. I don’t see an errors in the logs. Can anyone give me a clue on where to look.

05-02-18_03:07:38 PM tm.stats { current: 390 waiting: 226 total: 2361 total_local: 0 replied_locally: 1406 6xx: 0 5xx: 0 4xx: 88 3xx: 0 2xx: 3180 created: 2361 freed: 1971 delayed_free: 0 }

Mack Hendricks / Head of Support / dOpenSource web: http://dopensource.com http://dopensource.com/ support: +888-907-2085 dSIPRouter http://dsiprouter.org/ - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services

---

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Alex Balashov | Principal | Evariste Systems LLC

Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

---

Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com http://www.kamailioworld.com/