Another thing I have found is that having a certain amount of async workers running, even if they are not doing anything, appears to cause unexplained CPU load, even if the Kamailio instance is completely idle and not processing any calls.
Here is the baseline load with no async workers:
[root@centosity6 ~]# uptime 14:06:28 up 2 min, 2 users, load average: 0.01, 0.00, 0.00
If I start Kamailio with 1 async worker...
modparam("async", "workers", 1) async_workers=1
14:07:59 up 3 min, 2 users, load average: 0.00, 0.00, 0.00
Still nothing.
Now I try 2:
14:08:54 up 4 min, 2 users, load average: 0.45, 0.11, 0.04 14:08:59 up 4 min, 2 users, load average: 0.49, 0.12, 0.04 14:09:04 up 4 min, 2 users, load average: 0.53, 0.14, 0.04 14:09:09 up 4 min, 2 users, load average: 0.57, 0.15, 0.05 14:09:14 up 5 min, 2 users, load average: 0.60, 0.17, 0.05 14:09:19 up 5 min, 2 users, load average: 0.63, 0.18, 0.06 14:09:31 up 5 min, 2 users, load average: 0.69, 0.21, 0.07 14:09:37 up 5 min, 2 users, load average: 0.71, 0.22, 0.07 14:10:35 up 6 min, 2 users, load average: 0.89, 0.36, 0.13 14:11:07 up 6 min, 2 users, load average: 0.93, 0.42, 0.16 ...
I don't have time right now to see if it will rise to infinity, but it just keeps going up.
There are no SIP messages being received at all, and there is absolutely nothing going on.
The ascent of the load average is much higher with >= 4 threads. I have 8 "CPUs" available from /proc/cpuinfo (quad-core processor with HyperThreading).
When I attach strace to the PID of the Async Worker, there's nothing going on except the normal recvfrom() blocking wait:
[root@centosity6 kamailio]# kamctl ps | grep Async which: no greadlink in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/) Process:: ID=11 PID=1559 Type=Async Task Worker Process:: ID=12 PID=1560 Type=Async Task Worker
[root@centosity6 kamailio]# strace -p 1559 Process 1559 attached - interrupt to quit recvfrom(9,
[root@centosity6 ~]# strace -p 1560 Process 1560 attached - interrupt to quit recvfrom(9,
Nor do the other Kamailio threads seem to be doing anything unusual. 'ps aux' and 'top' show 0.0% CPU load on the Kamailio processes themselves, and 'iowait' shows the system as 98-99% idle.
I suppose it's possible that this is a peculiarity of the testing environment: I am running this on CentOS 6.5 inside VirtualBox, which I think is KVM-based. I have not had a chance to duplicate this experiment in a different kind of execution environment yet. It's possible that the load average is simply being improperly indicated because of the VM environment.