Hi,  

Our system had crashed again.

Here are the exact details and there are logs and corefile attached.

 

We use 6 kamailio servers over 3 sites.

All the servers are connecting to 1 DB server running MYSQL 5.6.12.

 

The sql server supplies 2 things for kamailio servers.

1.   Some SQLOPS that the servers need.

2.   The data for the MTREE, HTABLE, CARRIERROUTE etc..

 

We use "kamctl" command in shell script to reload the data for the modules every 1 minute. In normal times, the reload takes about 2-3 sec each minute.

 

When the sql server is down, the queries and reloads fail and it is OK.

 

Yesterday, we had a different failure on the sql server – The server is set ona centos 6.2 that is under a VM cluster. The VM cluster had some disk writing errors, and the sql server could not write or read fast enough.

 

On 3 of the 6 servers (one at each site) kamailio crashed.

 

I have no idea why. When the sql server is down, it does not happen.

 

I attached all 6 servers log, 1 server core file, 1 server var/log/messages and the mysql slow-query log.

 

Any ideas how to handle this?

kamailio version 3.3.2

 

BR,

Uri