On 03/05/16 09:01, Juha Heinanen wrote:
Daniel-Constantin Mierla writes:


These ones are no longer related to xmlrpc, but triggered by an sqlops
query - first bracktrace, the other two are during the signal
handling.

OK, but there must be some kind of relation to xmlrpc, because I have
had this same test system running without any problems for months and
now suddenly I started to experience crashes, which happen always after
xmlrpc queries.


The one for sqlops leads quite a bit inside the libmysqlclient, which
can happen because there is a buffer overflow in kamailio for the
operations in the system memory or an issue in the libmysqlclient.

Trying to investigate on the first option -- does it happen always after
an xmlrpc command? If yes, it is any kind of command or a specific
xmlrpc command (I see you used htable.reload)?

Yes.  I made another test.  I sent three xmlrpc queries (htable.reload)
and got crash after the third.  Why after third could be explained by
this sql_query that gets executed once per minute:

route [cleaner] {  # Clean old statistics from database
    sql_query_async("sip_proxy_usage", "DELETE FROM statistics WHERE (unix_timestamp() - time_stamp) > 86400");
}

So xmlrpc query or queries did something and that something triggered
subsequent sql_query to crash.
What is the version of kamailio running there?

xmlrpc is compiled by default to use system memory manager (expecting to need big chunks for some responses), can you recompile it to use the pkg? You have to edit modules/xmlrpc/Makefile and remove the line:

DEFS+=-DXMLRPC_SYSTEM_MALLOC

If there is a buffer overflow, it should be easier to catch using the internal memory manager checks -- some error messages should be printed inside the syslog as well.

Cheers,
Daniel
-- 
Daniel-Constantin Mierla
http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Kamailio World Conference, Berlin, May 18-20, 2016 - http://www.kamailioworld.com