Hello,

the backtrace shows a crash in tm module, not sqlops.

Can you tell which of the log files correspond to the instance that produced the core file from where you took the back trace you attached?

Can you give the backtraces from all cores? You say there were 3 crashes. It is important to see if the backtrace is the same everywhere.

Cheers,
Daniel

On 09/01/14 13:28, Uri Shacked wrote:

Hi,  

Our system had crashed again.

Here are the exact details and there are logs and corefile attached.

 

We use 6 kamailio servers over 3 sites.

All the servers are connecting to 1 DB server running MYSQL 5.6.12.

 

The sql server supplies 2 things for kamailio servers.

1.   Some SQLOPS that the servers need.

2.   The data for the MTREE, HTABLE, CARRIERROUTE etc..

 

We use "kamctl" command in shell script to reload the data for the modules every 1 minute. In normal times, the reload takes about 2-3 sec each minute.

 

When the sql server is down, the queries and reloads fail and it is OK.

 

Yesterday, we had a different failure on the sql server – The server is set ona centos 6.2 that is under a VM cluster. The VM cluster had some disk writing errors, and the sql server could not write or read fast enough.

 

On 3 of the 6 servers (one at each site) kamailio crashed.

 

I have no idea why. When the sql server is down, it does not happen.

 

I attached all 6 servers log, 1 server core file, 1 server var/log/messages and the mysql slow-query log.

 

Any ideas how to handle this?


kamailio version 3.3.2


 

BR,

Uri


-- 
Daniel-Constantin Mierla
http://www.asipto.com