8 May
2020
8 May
'20
5:09 p.m.
Hello,
I'm trying to replicate a hash table to another 2 nodes by using DMQ
Module. That's the related configuration:
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "dialog.so"
loadmodule "dmq.so"
loadmodule "htable.so"
modparam("dmq", "server_address",
"sip:MY_INTERNAL_IP_ADDR:5062")
modparam("dmq", "notification_address",
"sip:dmq.domain.local:5062")
modparam("dmq", "multi_notify", 1)
modparam("dmq", "num_workers", 4)
modparam("dmq", "ping_interval", 90)
modparam("htable", "enable_dmq", 1)
modparam("htable", "dmq_init_sync", 1)
modparam("htable", "htable",
"test=>size=10;initval=0;dmqreplicate=1;")
if(is_method("KDMQ")) {
dmq_handle_message();
exit;
}
After starting Kamailio - everything seems to be working fine, Kamailio is
able to discover other nodes and replications is working fine, until after
a few minutes I'm getting such errors in the Kamailio Log file:
May 8 17:14:26 kamailio kamailio[28064]: CRITICAL: <core>
[core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer
(0x7f0ac62493e0), called from dialog: dlg_var.c: free_local_varlist(145),
first free dialog: dlg_var.c: set_dlg_variable_unsafe(205) - ignoring
Then, if I'm getting any SIP messages from any Device - Kamailio is
crashing with the following info in the log file:
May 8 16:03:19 kamailio kam[10963]: CRITICAL: {REGISTER (1) | Seq=45534 |
Source IP=192.168.1.5 (tcp) | Call ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7}
<core> [core/mem/q_malloc.c:150]: qm_debug_check_frag(): BUG: qm: prev.
fragm. tail overwritten(0, 0)[0x7f62cc746a10:0x7f62cc746a48]!
Memory allocator was called from dialog: dlg_var.c:147. Fragment marked by
dialog: dlg_var.c:108. Exec from core/mem/q_malloc.c:511.
May 8 16:03:21 kamailio kam[10966]: CRITICAL: <core> [core/pass_fd.c:277]:
receive_fd(): EOF on 59
May 8 16:03:21 kamailio kam[10966]: DEBUG: <core> [core/tcp_main.c:3825]:
handle_ser_child(): dead child 37, pid 10963 (shutting down?)
May 8 16:03:21 kamailio kam[10966]: DEBUG: <core> [core/io_wait.h:599]:
io_watch_del(): DBG: io_watch_del (0xa9e1c0, 59, -1, 0x0) fd_no=48 called
May 8 16:03:21 kamailio kam[10966]: DEBUG: <core> [core/tcp_main.c:3593]:
handle_tcp_child(): dead tcp child 1 (pid 10963, no 37) (shutting down?)
May 8 16:03:21 kamailio kam[10966]: DEBUG: <core> [core/io_wait.h:599]:
io_watch_del(): DBG: io_watch_del (0xa9e1c0, 62, -1, 0x0) fd_no=47 called
May 8 16:03:21 kamailio kam[10919]: ALERT: <core> [main.c:767]:
handle_sigs(): child process 10963 exited by a signal 6
May 8 16:03:21 kamailio kam[10919]: ALERT: <core> [main.c:770]:
handle_sigs(): core was generated
May 8 16:03:21 kamailio kam[10919]: INFO: <core> [main.c:792]:
handle_sigs(): terminating due to SIGCHLD
May 8 16:03:21 kamailio kam[10919]: DEBUG: <core> [main.c:794]:
handle_sigs(): terminating due to SIGCHLD
May 8 16:03:21 kamailio kam[10966]: INFO: <core> [main.c:847]: sig_usr():
signal 15 received
This is what i got from the dump core file:
uri = {user = {s = 0x3 <Address 0x3 out of bounds>, len = -264573566},
passwd = {s = 0x788 <Address 0x788 out of bounds>, len = 8208}, host = {s =
0x393100000000 <Address 0x393100000000 out of bounds>, len = 1588950199},
port = {s = 0x2 <Address 0x2 out of bounds>,
len = 24086112}, params = {s = 0x7ffcbb1600c0 "\200", len =
-261173312}, sip_params = {s = 0x7ffcbb1601a8 "\300\317n\360b\177", len =
24086112}, headers = {s = 0x0, len = -261173312}, port_no = 424, proto =
47894, type = 32764,
flags = (URI_SIP_USER_PHONE | unknown: 4031456108), transport = {
s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL: {REGISTER
(1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core> [core/mem/q_malloc.c:150]:
qm_debug_check_frag(): BUG: qm: pr"..., len = -261163168}, ttl = {
s = 0x8 <Address 0x8 out of bounds>, len = -263517049},
user_param = {
s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL: {REGISTER
(1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core> [core/mem/q_malloc.c:150]:
qm_debug_check_frag(): BUG: qm: pr"..., len = -264350686}, maddr = {
s = 0x3336393031000080 <Address 0x3336393031000080 out of
bounds>, len = 8187}, method = {s = 0x0, len = 24086112}, lr = {s =
0x820ce8 "%s: %s%s(): cseq <%.*s>: <%.*s> <%.*s>\n", len =
-1156185384}, r2
= {s = 0xb <Address 0xb out of bounds>, len = 21}, gr = {
s = 0xffffffff <Address 0xffffffff out of bounds>, len =
-264780793}, transport_val = {s = 0x1ff4 <Address 0x1ff4 out of bounds>,
len = 24086720}, ttl_val = {s = 0x7f62f06f0760 <main_arena> "", len =
128},
user_param_val = {s = 0x16f88b0 "`\002", len = 144},
maddr_val = {s = 0x15 <Address 0x15 out of bounds>, len =
-264573566}, method_val = {s = 0x788 <Address 0x788 out of bounds>, len =
8208}, lr_val = {s = 0x0, len = 1588950199}, r2_val = {s = 0x2 <Address 0x2
out of bounds>, len = 24086112}, gr_val = {
s = 0x7ffcbb1601f0 "\006", len = -261173312}}
next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0},
host = {s = 0x7f62f06eef20 <_nl_C_LC_NUMERIC+32> "", len = 0}, port = {s
=
0x0, len = 0}, params = {s = 0x7f620000000b <Address 0x7f620000000b out of
bounds>, len = 0}, sip_params = {
s = 0x7f62f03762cd <vfprintf+19661>
"\200\275\360\372\377\377",
len = 0}, headers = {s = 0x0, len = -1}, port_no = 25293, proto = 61495,
type = 32610, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE), transport
= {s = 0x7ffcbb160038 "\267t\265^", len = 24086112},
ttl = {s = 0xffffffff <Address 0xffffffff out of bounds>, len =
0}, user_param = {s = 0x7d62e8 "%s: %.*s%s%s(): preparing to run routing
scripts...\n", len = 11}, maddr = {s = 0xbd <Address 0xbd out of bounds>,
len = 3440}, method = {s = 0x7d631c "", len = -263511186},
lr = {s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL:
{REGISTER (1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core> [core/mem/q_malloc.c:150]:
qm_debug_check_frag(): BUG: qm: pr"..., len = -261163168}, r2 = {
s = 0x8 <Address 0x8 out of bounds>, len = -263517049}, gr = {
s = 0x16f88cc "16:03:19 kamailio[10963]: CRITICAL: {REGISTER
(1) | Seq=45534 | Source IP=192.168.1.5 (tcp) | Call
ID=Kyb-KJjf.U9ttdOwQNYu6mUWCKb2bvk7} <core> [core/mem/q_malloc.c:150]:
qm_debug_check_frag(): BUG: qm: pr"..., len = -264350686}, transport_val = {
s = 0x7d62ef "s%s%s(): preparing to run routing scripts...\n",
len = 8187}, ttl_val = {s = 0x0, len = -264345823}, user_param_val = {s =
0x3000000010 <Address 0x3000000010 out of bounds>, len = -1156184608},
maddr_val = {
s = 0x7ffcbb160520 "\341+:\213'H\027\203\341+\342ޙ>\356|",
len
= 21}, method_val = {s = 0x7ffcbb160250 "\023", len = -1156186080}, lr_val
= {s = 0x7f62f06eef80 <_nl_C_LC_TIME> "r\036K\360b\177", len =
-1156185648}, r2_val = {s = 0x17ff710 "IST", len = -1156186048},
gr_val = {s = 0x0, len = -1156186058}}
u = 0x400001ff4
By disabling DMQ Module i haven't noticed any crashes or issues like that,
so i suppose that Kamailio is crashing because of this module, but i can be
wrong. I've tried with 2 versions of Kamailio (5.2.6 and 5.3.4), both of
them having the same issue.
Thanks