On May 2, 2018, at 11:36 PM, Alex Balashov <abalashov(a)evaristesys.com> wrote: Hi Mack, What sort of end-to-end delays are you seeing, from receipt of request until immediately before t_relay()? You can profile this with the benchmark module: https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html Or my preferred method, the hard way: # At the practical start of request processing. $avp(proc_start) = $TV(Sn); # At the end, right before calling t_relay() or forward(). $var(cur_time) = $TV(Sn); $var(proc_diff) = ( ((( $(var(cur_time){s.select,0,.}{s.int}) - $(avp(proc_start){s.select,0,.}{s.int}) ) * 1000000) + ( $(var(cur_time){s.select,1,.}{s.int}) - $(avp(proc_start){s.select,1,.}{s.int}) ) / 1000) mod 1000 ); xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n");); Also, for general background on this topic, you might consider my article on the subject of Kamailio performance characteristics: http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performa… If I had to guess, shooting from the hip, you've got some really slow database queries somewhere. -- Alex -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

Hi Mack, Happy to help! There must be a hidden source of I/O wait somewhere. It may not be database calls, but there's something. This shouldn't be happening in a mostly static config. Can you share the numbers from your load testing? At what point did you start seeing retransmissions, if any? What kind of delays are you seeing? -- Alex On Thu, May 03, 2018 at 12:48:38AM -0400, Mack Hendricks wrote: > Hey Alex, > > Thanks for the insight. I remember reading your article before and it was super helpful. I think we disabled all of the heavy database operations, but I will double check. > > We are building the RURI using the cfg_rpc module, so we aren’t hitting a database. We have a failure route that causes another RURI to be created, which sends the call to a Freeswitch server to play a message. > > Let me know if you see any issue with this, I will double check the database calls to be safe. > > Sent from my iPhone > >> On May 2, 2018, at 11:36 PM, Alex Balashov &lt;abalashov(a)evaristesys.com&gt; wrote: >> >> Hi Mack, >> >> What sort of end-to-end delays are you seeing, from receipt of request >> until immediately before t_relay()? >> >> You can profile this with the benchmark module: >> >> https://kamailio.org/docs/modules/5.1.x/modules/benchmark.html >> >> Or my preferred method, the hard way: >> >> # At the practical start of request processing. >> >> $avp(proc_start) = $TV(Sn); >> >> # At the end, right before calling t_relay() or forward(). >> >> $var(cur_time) = $TV(Sn); >> >> $var(proc_diff) = ( >> ((( >> $(var(cur_time){s.select,0,.}{s.int}) - >> $(avp(proc_start){s.select,0,.}{s.int}) >> ) * 1000000) >> + >> ( >> $(var(cur_time){s.select,1,.}{s.int}) - >> $(avp(proc_start){s.select,1,.}{s.int}) >> ) / 1000) mod 1000 >> ); >> >> xlog("L_INFO", "[R-MAIN:$ci] -> Request processing delay: $var(proc_diff) ms\n");); >> >> Also, for general background on this topic, you might consider my >> article on the subject of Kamailio performance characteristics: >> >> http://blog.csrpswitch.com/tuning-kamailio-for-high-throughput-and-performa… >> >> If I had to guess, shooting from the hip, you've got some really slow >> database queries somewhere. >> >> -- Alex >> >>> On Wed, May 02, 2018 at 06:19:00PM -0400, Mack Hendricks wrote: >>> >>> Hey All, >>> >>> We are load testing Kamailio 4.2.3 and we are seeing the waiting within the transaction manager increasing. I’ve increased the children from 8 to 16 to 24, but we are still seeing a large amount of waiting. I don’t see an errors in the logs. Can anyone give me a clue on where to look. >>> >>> 05-02-18_03:07:38 PM >>> tm.stats >>> { >>> current: 390 >>> waiting: 226 >>> total: 2361 >>> total_local: 0 >>> replied_locally: 1406 >>> 6xx: 0 >>> 5xx: 0 >>> 4xx: 88 >>> 3xx: 0 >>> 2xx: 3180 >>> created: 2361 >>> freed: 1971 >>> delayed_free: 0 >>> } >>> >>> Mack Hendricks / Head of Support / dOpenSource >>> web: http://dopensource.com <http://dopensource.com/> >>> support: +888-907-2085 >>> dSIPRouter <http://dsiprouter.org/> - GUI focused on implementing Kamailio to provide SIP Trunking and PBX Hosting Services >>> >>> _______________________________________________ >>> Kamailio (SER) - Users Mailing List >>> sr-users(a)lists.kamailio.org >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Alex Balashov | Principal | Evariste Systems LLC >> >> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) >> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/

On May 3, 2018, at 2:11 AM, Daniel-Constantin Mierla &lt;miconda(a)gmail.com&gt; wrote: Hello, On 03.05.18 06:48, Mack Hendricks wrote: are you really using cfg_rpc module to build the r-uri? Or did you mean someting like jsonrpcc? Cheers, Daniel -- Daniel-Constantin Mierla www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio World Conference - May 14-16, 2018 - www.kamailioworld.com <http://www.kamailioworld.com/>