Hi all!
Having an issue with MySQL db connections being dropped in a system running 4.4.7.
We're using db_mysql and db_cluster modules setup a cluster connecting two different DB servers. We have two cluster connections, one for acc and one for "other queries". One DB (A) is on the same network, another DB (B) is on another network. The default DB connection is for the remote server B. Auto reconnect is enabled.
The specific issue seen is that the "timer" child process looses/drops both connections to DB A and B. Looking at the output from lsof when this happens, the connections usually does not both drop connections for A and B at the same time. Sometimes the connections keep up for ~24h, sometimes for 10 minutes, but normally the problem re-occurs every 6 hours or so. We're seeing this problem on two Kamailio servers, both handling fairly high amount of calls.
None of the other Kamailio child processes seems to get their connections dropped, only the "timer" process. To solve this we need to restart Kamailio.
Lately I've added the timer.so module to make a simple query on each cluster connection each 10 seconds.
This is an example output from when the problem appears and connections are dropped:
Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)
Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013)
Jun 15 09:39:12 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Can't connect to MySQL server on 'xxx' (4) (2003)
Jun 15 09:39:13 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server during query (2013)
Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:14 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:15 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:16 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:17 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:18 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:27 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:28 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:32 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:34 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:35 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:37 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:38 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:47 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:48 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:235]: db_do_insert_cmd(): error while submitting query
Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:50 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: db_mysql [km_dbase.c:128]: db_mysql_submit_query(): driver error on query: Lost connection to MySQL server at 'waiting for initial communication packet', system error: 4 (2013)
Jun 15 09:39:57 /usr/sbin/kamailio[10439]: ERROR: <core> [db_query.c:181]: db_do_raw_query(): error while submitting query
Again, it's always and only the timer child process where these error messages are seen. Have been trying to debug this for some time now but I'm running out of ideas. Do you have any ideas on what I could look for? Any idea on why it's just the timer-process?
Kind regards,
/Tobias
Hi,
I am trying to configure kamailio to play early media and i am using
rtpproxy_stream2uac(), but the audio file plays only after the call is
connected.
How to use rtpproxy_stream2uac() to play only the early media.
Thank you
vinod.M.N
--
To download the app, click: iOS <http://goo.gl/ZJx3y1> | Android
<http://goo.gl/G1ehjZ>
--
***This transmission contains information that is confidential and
proprietary of Kirusa Inc.,its affiliates and/or its partners and
customers. This information is intended to be shared only with the
recipient of this e-mail who is bound by confidentiality. If you received
this e-mail in error, please delete the e-mail and notify the sender by
return e-mail.***
Greetings,
What kind of regex does textops module uses?
I'm trying to find the string "application\sdp" but i can't the escape
character for the "\".
Thanks in advance
Hi all,
Few days back only, I started working on Kamailio IMS. I set up PSCSF,
ICSCF, SCSCF and HSS(from open hss).
When I try to send Register request from Zoiper, ICSCF is giving request
timeout. At SCSCF, MAR failed error is logged.
Kindly, let me know where I am doing wrong?
>From my observation, At HSS, I can't see MAR request processing, It is only
processing UAR from ICSCF.
Find the attached log file.
--
With Best Regards,
Ashok Kumar.
Hi all,
Few days back only, I started working on Kamailio IMS. I set up PSCSF,
ICSCF, SCSCF and HSS(from open hss).
When I try to send Register request from Zoiper, ICSCF is giving request
timeout. At SCSCF, MAR failed error is logged.
I am attaching the screenshot of log. Kindly, let me know where I am doing
wrong?
>From my observation, At HSS, I can't see MAR request processing, It is only
processing UAR from ICSCF.
--
With Best Regards,
Ashok Kumar
Hi all,
I was playing around with the "config" modparam to specify and use a TLS
configuration file instead of inline modparams inside kamailio.cfg.
I did some testing and when I went to try with IPv6 I run into a little
doubt??
tls.cfg config example:
[server:1.2.3.4:5061] <----- How do you put an IPv6 in there?
method = TLSv1
verify_certificate = yes
require_certificate = no
private_key = /usr/local/etc/kamailio/tls/local_key.pem
certificate = /usr/local/etc/kamailio/tls/local_cert.pem
verify_depth = 3
ca_list = local_ca.pem
crl = local_crl.pem
Applying the logic of the "listen" param, it should be:
[server:[AAAA:BBBB:CCCC:DDDD:EEEE:FFFF:GGGG:HHHH]:5061]
But it just seems totally wrong and I haven't found anything in the docs.
Any suggestions?
Thanks,
Joel.
Greetings,
Please help me understand the reason and source of this behaviour on
Kamailio.
I receive a 200OK to a RE-INVITE that has :
Session Name (s) : MySession
Connection Information (c) : IN IP4 xxx.xxx.xxx.xxx
Then, Kamailio routes the 200OK to it's destination, and it leaves Kamailio
like this :
Session Name (s) : MySession sdpc=IN xxx.xxx.xxx.xxx
The line Connection Information got deleted and it's info is now on
Session Name. How can i disable this ?
Thanks
Hello everybody,
I have a question about the function "aliases". With my server no alias entry is registered and there is no DNS server (reverse DNS is not possible) in the network. Nevertheless, the SIP server listens to a defined alias. Is there next to the config file and the kamctlrc file another place where an alias can be entered?
Cheers,
Kai
