[SR-Users] Possible memory leak in mysql driver

Hello list, We’re using dev version of Kamailio: version: kamailio 5.0.0-dev4 (x86_64/linux) ff63e5 flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: ff63e5 compiled on 15:46:49 May 31 2016 with gcc 4.9.2 Sometimes we encounter such issue: Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:188]: db_mysql_convert_rows(): error while converting row #16 Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_res.c:217]: db_mysql_convert_result(): error while converting rows Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:261]: db_mysql_store_result(): error while converting result Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_query.c:139]: db_do_query_internal(): error while storing result Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: permissions [trusted.c:91]: reload_trusted_table(): failed to query database Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> [db_row.c:114]: db_allocate_row(): no private memory left Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_row.c:57]: db_mysql_convert_row(): could not allocate row Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql [km_dbase.c:444]: db_mysql_fetch_result(): error while converting row #15 Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: htable [ht_db.c:234]: ht_db_load_table(): Error while fetching result Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: : <core> [mem/q_malloc.c:468]: qm_free(): BUG: qm_free: freeing already freed pointer (0x7f5ebda8ae18), called from db_mysql: km_dbase.c: db_mysql_free_result(305), first free db_mysql: km_dbase.c: db_mysql_free_result(305) - aborting Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12281]: CRITICAL: <core> [pass_fd.c:275]: receive_fd(): EOF on 16 Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:739]: handle_sigs(): child process 12276 exited by a signal 6 Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> [main.c:742]: handle_sigs(): core was not generated Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: INFO: <core> [main.c:754]: handle_sigs(): terminating due to SIGCHLD The thing is we heavily use mysql module, but only to update the in-memory tables by kamcmd. Each N minutes a special script updates the trusted,address and htable executing kamcmd. Kamailio (and kamcmd as well) talks only with localhost mysql server. What I saw when encountered that issue on a live machine is that issue happens only with one of child processes, any other are ok. Interesting thing is that happens at the same time with machines on the same «set», I mean that issue happened simultaneously with two our test machines which actually didn’t have any load on them. The common thing between those machines is that they are in same subnet and local mysql databases are filled by scripts which query same external db. I can’t confirm if there were or there weren’t any networking issues at that time with those machines, but as soon as kamcmd queries localhost that shouldn’t be the source of the issue. So my questions are: 1. Has anyone encountered such thing? 2. Maybe the issue is already localized so it has sense to update? We actually use that on production (pls don’t throw too much rocks at me), so maintenance should be properly planned and I must be sure update won’t break anything. 3. If update is proposed - how to do it? I mean - follow the guide https://www.kamailio.org/wiki/install/devel/git or there are some other tips? I suppose in ideal world I don’t even stop the binary, only restart after make all && make install are done, as everything is in-memory. Am I correct? 4. When can we expect stable 5.0 version? (at least tell if it’s months/years) Thanks in advance!