Hello,
if you keep coming with some reports, but don't follow the requests for
further details, nobody will be able to help.
Not being able to gather basic troubleshooting data, is not something
that should make you look good as a vendor. Moreover, you should know
that we can help if help troubleshooting. Even few days ago I replied to
another similar email from you and instead of following there, you come
with another thread.
Also, Olle concluded that needs to investigate more as there is no real
high CPU load, but system load -- however, it was no follow up.
I run many instances of kamailio in VMs with Debian, with async enabled,
no symptoms as you reported.
If you run on centos, I noticed a high cpu from rtpproxy on a test box,
with no or really low traffic, that rtpproxy (official package installed
from distro rpm, no custom changes) eats 100% with backtrace reporting
it is executing recvfrom(). So, the problem might be combination of
vm+centos. I did investigated on rtpproxy, couldn't find anything
different than what I run on Debian (same version 1.2.1, from 2013).
Some similar reports on the web for other apps suggested upgrades to kernel.
Regarding the blocking, I asked whether is high CPU or not, because if
not, the blocking can be an I/O operation (database, dns), not tm
module. I pointed you in my previous response at 'kamctl trap', which
you can easily look inside and extract the gdb commands to get the
backtrace from command line.
The latest version of async framework in branch 4.2 is quite the same as
in 4.1, as the extra dedicated lock was reverted.
Your way to deal with these reports and not follow up when asked for
more details doesn't encourage developers (at least me) to assist -- it
is anything but constructive, waste of time here keep requesting same
details and not getting the responses.
Daniel
On 10/06/15 16:59, Alex Balashov wrote:
Hello,
I am looking for some recommendations on how to move forward with
asynchronous call processing.
Our adoption and standardisation of async processing in >= 4.2 into
our core product has been a disaster. I don't mean that to sound
accusatory; it's open source, there's no reason to blame anyone. It's
just a matter of fact.
1) We can't use async_task_route()/the standard async task worker
approach in the 'async' module because of this problem, which both
Olle and I have reported:
http://sr-dev.sip-router.narkive.com/5Sfc5cUU/async-module-cpu-load
Most users run Kamailio inside a VM and the problem shows up for ~50%
of them.
2) Using t_suspend -> mqueue -> rtimer -> t_continue(), we continue to
see deadlocks and occasional crashes. They are rare, and are most
likely to happen in high-throughput, short-duration environments, but
when they do happen, they're politically disastrous. We've had to roll
back use of this method of async processing for pretty much all
customers for whom it was enabled.
After the last time we visited this issue earlier this spring, the
problem has shifted away from crashes and mostly toward deadlocks.
Regardless, the customer's enthusiasm for pausing call processing long
enough to attach a debugger and grab a backtrace or something like
that is exactly 0.0%. I think most of them have more of an enthusiasm
for firing us as a vendor than for doing any diagnostic work.
I know there's a way to invoke a process in such a way that when it
crashes, gdb auto-attaches and pulls a backtrace, then restarts the
process. I've written such a wrapper script before in the distant
past. I just don't remember how to do it, especially with modern
versions of GDB; any suggestions would be appreciated.
Otherwise, I don't really know what to do. We need async processing
for higher-CPS systems, and would like to standardise upon it in
principle, but so far it has, from a strictly functional point of
view, been an enormous economic blunder.
I still prefer to be an "early adopter" of such novelties - when
useful - in high-volume production systems in order to contribute the
testing and feedback back to the project. But I have to strike some
realistic balance here and not lose the customers. :-)
Thanks!
-- Alex
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda -
http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio -
http://www.asipto.com