Kamailio is using spin locks (for mutexes) and that can cause high cpu on some systems. I am not very familair with dmq internals, but it looks like it uses mutexes to trigger the presence of a new task for its worker processes, meaning that a worker stays in mutex-get operation (spinning), until a new task arrives. Maybe @charlesrchance can shed more light here.

Of course, this high cpu can show up in case of a deadlock, but if you say all works fine whether there is traffic to handle, then is not a deadlock.

The option is to compile to use POSIX semaphores -- the Makefile.defs needs to be changed and all recompiled.

Apologies, I have been away for a couple of weeks. DMQ does indeed use mutexes to trigger processing of a new task by its workers.

Your log shows everything is as expected with DMQ - i.e. with no other traffic, a lock is acquired/released once every minute with each peer notification (node ping). There is nothing else unusual about your log (or GDB output) that I can see.

So it does seem likely that something in the way these locks are being implemented is causing the high load on your particular system. Maybe someone else can suggest another alternative, if enabling POSIX semaphores has not resolved the issue for you.

@charlesrchance - a solution is to use a shared memory variable to say if something needs to be consumed, like:

``` // mod init int *dmq_usrloc_tasks = shm_malloc(sizeof(int)); *dmq_usrloc_tasks = 0; gen_lock_t *dmq_usrloc_tasks_lock = ... // usual lock initialization

// runtime - producer lock_get(dmq_usrloc_tasks_lock); *dmq_usrloc_tasks++ lock_release(dmq_usrloc_tasks_lock);

// runtime - consumer while (1) { lock_get(dmq_usrloc_tasks_lock); while(*dmq_usrloc_tasks>0) *dmq_usrloc_tasks++; lock_release(dmq_usrloc_tasks_lock); sleep_us(dmq_usrloc_tasks_sleep); } ```

The incrementing/decrementing needs to be accompanied by producing/consuming the tasks.

The key here is the sleep_us(), where dmq_usrloc_tasks_sleep can be a new mod param to specify the miliseconds (or microseconds) to sleep. It proved that triggering a sleep takes the cpu from process and cpu load is kept low. One can tune its value to suit better its environment.

IIRC, this is used by maybe in async module for async_route().

An alternative is using in-memory sockets to pass tasks from producers to consumers. The consumer should be blocked in a read() and that should not consume cpu, although on some systems I got read() returning fast with error, ending in a loop of fast reads and high cpu again. This should be used by evapi to pass events between sip workers and evapi worker.

Any progress?

@charlesrchance do you need a lock at all?

https://github.com/tombeard/kamailio/commit/310c0f91858a6baf4b17a00798689f7f...

No idea how this performs on other platforms or in other situations, but for my specific requirement it solves the high cpu condition on FreeBSD waiting for lock yeild and I've had it running in production with dmq_usrloc for several months now without issue. Of course, your mileage may vary.

@tombeard thanks - I guess not! Pull request?

@charlesrchance ignoring my second comment, I think that probably is safe. Either way, there's a pull request for testing / fixing.

I am not sure if removing the lock/mutex is the proper solution here, as I didn't analyze the module properly, so it might be better to make it optional via modparam to run in old mode with locks and new mode with sleep. I guess it should not be that complex, given that the new patch is rather small.

Closed #822.