[kamailio/kamailio] Kamailio 5 coredump 11 (SEGV) upcon receiving MESSAGE via jsonrpcs (#1030)

List overview All Threads
Download

newer

older

Forking and Call-ID Hiding in TOPOS

git:5.0:f81c128d: jsonrpcs: safety...

Anthony Messina

11 Mar 2017 11 Mar '17

2:12 a.m.

Using python-requests to generate a MESSAGE injected via jsonrpcs, Kamailio (built from the 5.0 branch @ 49bafac) dumps core. This did not occur in 4.4.5. See attached gdb output. [kamailio-coredump-jsonrpcs-message.txt](https://github.com/kamailio/kamailio/files/835428/kamailio-coredump-jsonrpcs...)

``` """ Generate a SIP MESSAGE via Kamailio's jsonrpcs module """ headers = ("From: <{0}>\r\n" "To: <{1}>\r\n" "Max-Forwards: 1\r\n" "X-Kamailio-Message: noreply\r\n" "Content-Type: text/plain; " "charset=utf-8\r\n").format(self.sender, self.recipient)

content = { 'id': 1, 'jsonrpc': '2.0', 'method': 'tm.t_uac_start', 'params': [ 'MESSAGE', self.recipient, SIP_NEXTHOP, '.', headers, self.body ] }

# http://docs.python-requests.org/en/latest/user/quickstart/ try: r = requests.post(KAMAILIO_RPC_URL, allow_redirects=False, json=content, timeout=5) ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030

Attachments:

attachment.html (text/html — 2.9 KB)

Show replies by date

Daniel-Constantin Mierla

11 Mar 11 Mar

8:32 a.m.

Can this be reproduced? If you get other corefile, are the details of frame 0 missing again? Without it, no details of the code it effective crashed are shown.

...

From this corefile, can you get the output of:

``` frame 0 p *ctx ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-285852714

Anthony Messina

1:43 p.m.

This is reproducable every time. Here is the output you requested: ``` (gdb) frame 1 #1 0x00007f1426c9f5d1 in jsonrpc_send (ctx=0x7f1426ec88e0 <_jsonrpc_ctx>) at jsonrpcs_mod.c:395 395 xhttp_api.reply(ctx->msg, ctx->http_code, &ctx->http_text, (gdb) p *ctx $1 = {msg = 0x7ffdd76deda0, msg_shm_block_size = 0, method = 0x23fe850 "tm.t_uac_start", flags = 0, jreq = 0x2402810, req_node = 0x0, jrpl = 0x2402f40, rpl_node = 0x0, reply_sent = 1, error_code = 0, http_code = 200, http_text = { s = 0x7f1426cbec14 "OK", len = 2}, transport = 1} ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-285867501

Daniel-Constantin Mierla

13 Mar 13 Mar

2:37 p.m.

Can you also get the content of msg field, respectively:

``` p *ctx->msg ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-286126055

Anthony Messina

11:11 p.m.

See attached. I have replaced my domain with "example.com" and my username with "user". [kamailio-coredump-jsonrpcs-message_2.txt](https://github.com/kamailio/kamailio/files/839849/kamailio-coredump-jsonrpcs...)

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-286272391

Daniel-Constantin Mierla

14 Mar 14 Mar

1:16 p.m.

Couldn't spot anything relevant wrong for sending the response.

Can you grab all the log messages with debug=3 in kamailio.cfg? Hopefully it will show more leads about what frame 0 can be.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-286417462

Anthony Messina

10:18 p.m.

I had to send another message to do this, but here it is: [kamailio-coredump-jsonrpcs-message-script-debug.txt](https://github.com/kamailio/kamailio/files/842912/kamailio-coredump-jsonrpcs...)

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-286578930

Anthony Messina

10:21 p.m.

The strange thing is, the MESSAGE is received and transmitted to the recipient, then Kamailio fails, apparently on the reply.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-286579620

Daniel-Constantin Mierla

16 Mar 16 Mar

10:05 a.m.

Can you try with latest branch 5.0 and grab again the log messages with the debug=3 in case the issue is still there.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287010991

Anthony Messina

9:13 p.m.

I see a97701575863993f8a19f9dd611156b37ffca247 in the master branch, but not the 5.0 branch. Do you want me to include that in the testing as well?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287192751

Anthony Messina

9:31 p.m.

@miconda attached is the debug log using the 5.0 branch at f5889fcc9e42cae01c58357a622d2270f994b8d7. [kamailio.script.debug.3.txt](https://github.com/kamailio/kamailio/files/849081/kamailio.script.debug.3.tx...)

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287197443

Daniel-Constantin Mierla

9:50 p.m.

The log messages are not from syslog, so they are missing date/time and which PID printed them. Can you redo the test and grab via syslog?

You can also test with a977015 .

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287201871

Anthony Messina

10 p.m.

The new log is from a rebuild of the 5.0 branch as before with 0350885c076fa691de6106eb821ec5890572cee6, fa50c85a9e61dd6b91ce88f3f74c433af5d414e2, and a97701575863993f8a19f9dd611156b37ffca247 cherry-picked onto it. Still dumps. [kamailio.script.stateless.debug.3.txt](https://github.com/kamailio/kamailio/files/849124/kamailio.script.stateless....)

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287204075

Anthony Messina

10:04 p.m.

Just in case, these are some of the script settings: ``` # ----- jsonrpcs params ----- modparam("jsonrpcs", "pretty_format", 1) /* set the path to RPC fifo control file */ # modparam("jsonrpcs", "fifo_name", "/run/kamailio/kamailio_rpc.fifo") /* set the path to RPC unix socket control file */ # modparam("jsonrpcs", "dgram_socket", "/run/kamailio/kamailio_rpc.sock")

# ----- ctl params ----- /* set the path to RPC unix socket control file */ # modparam("ctl", "binrpc", "unix:/run/kamailio/kamailio_ctl") ```

Probably unrelated, but a few xhttp-related compiler warnings: ``` make[2]: Entering directory '/builddir/build/BUILD/kamailio-5.0.0/src/modules/xhttp_rpc' Makefile.defs defs skipped Makefile.defs defs skipped gcc -fPIC -DPIC -g -funroll-loops -Wcast-align -m64 -minline-all-stringops -falign-loops -ftree-vectorize -fno-strict-overflow -Wall -DNAME='"kamailio"' -DVERSION='"5.0.0"' -DARCH='"x86_64"' -DOS='linux_' -DOS_QUOTED='"linux"' -DCOMPILER='"gcc 6.3.1"' -D__CPU_x86_64 -D__OS_linux -DSER_VER=5000000 -DCFG_DIR='"/etc/kamailio/"' -DRUN_DIR='"/var/run/kamailio/"' -DPKG_MALLOC -DSHM_MEM -DSHM_MMAP -DDNS_IP_HACK -DUSE_MCAST -DUSE_TCP -DDISABLE_NAGLE -DHAVE_RESOLV_RES -DUSE_DNS_CACHE -DUSE_DNS_FAILOVER -DUSE_DST_BLACKLIST -DUSE_NAPTR -DWITH_XAVP -DMEM_JOIN_FREE -DF_MALLOC -DQ_MALLOC -DTLSF_MALLOC -DDBG_SR_MEMORY -DUSE_TLS -DTLS_HOOKS -DUSE_CORE_STATS -DSTATISTICS -DMALLOC_STATS -DWITH_AS_SUPPORT -DUSE_SCTP -DFAST_LOCK -DADAPTIVE_WAIT -DADAPTIVE_WAIT_LOOPS=1024 -DCC_GCC_LIKE_ASM -DHAVE_GETHOSTBYNAME2 -DHAVE_UNION_SEMUN -DHAVE_SCHED_YIELD -DHAVE_MSG_NOSIGNAL -DHAVE_MSGHDR_MSG_CONTROL -DHAVE_ALLOCA_H -DHAVE_TIMEGM -DHAVE_SCHED_SETSCHEDULER -DHAVE_IP_MREQN -DHAVE_EPOLL -DHAVE_SIGIO_RT -DSIGINFO64_WORKARROUND -DUSE_FUTEX -DHAVE_SELECT -DKAMAILIO_MOD_INTERFACE -DMOD_NAME='"xhttp_rpc"' -c xhttp_rpc.c -o xhttp_rpc.o -MMD -MP gcc -fPIC -DPIC -g -funroll-loops -Wcast-align -m64 -minline-all-stringops -falign-loops -ftree-vectorize -fno-strict-overflow -Wall -DNAME='"kamailio"' -DVERSION='"5.0.0"' -DARCH='"x86_64"' -DOS='linux_' -DOS_QUOTED='"linux"' -DCOMPILER='"gcc 6.3.1"' -D__CPU_x86_64 -D__OS_linux -DSER_VER=5000000 -DCFG_DIR='"/etc/kamailio/"' -DRUN_DIR='"/var/run/kamailio/"' -DPKG_MALLOC -DSHM_MEM -DSHM_MMAP -DDNS_IP_HACK -DUSE_MCAST -DUSE_TCP -DDISABLE_NAGLE -DHAVE_RESOLV_RES -DUSE_DNS_CACHE -DUSE_DNS_FAILOVER -DUSE_DST_BLACKLIST -DUSE_NAPTR -DWITH_XAVP -DMEM_JOIN_FREE -DF_MALLOC -DQ_MALLOC -DTLSF_MALLOC -DDBG_SR_MEMORY -DUSE_TLS -DTLS_HOOKS -DUSE_CORE_STATS -DSTATISTICS -DMALLOC_STATS -DWITH_AS_SUPPORT -DUSE_SCTP -DFAST_LOCK -DADAPTIVE_WAIT -DADAPTIVE_WAIT_LOOPS=1024 -DCC_GCC_LIKE_ASM -DHAVE_GETHOSTBYNAME2 -DHAVE_UNION_SEMUN -DHAVE_SCHED_YIELD -DHAVE_MSG_NOSIGNAL -DHAVE_MSGHDR_MSG_CONTROL -DHAVE_ALLOCA_H -DHAVE_TIMEGM -DHAVE_SCHED_SETSCHEDULER -DHAVE_IP_MREQN -DHAVE_EPOLL -DHAVE_SIGIO_RT -DSIGINFO64_WORKARROUND -DUSE_FUTEX -DHAVE_SELECT -DKAMAILIO_MOD_INTERFACE -DMOD_NAME='"xhttp_rpc"' -c xhttp_rpc_fnc.c -o xhttp_rpc_fnc.o -MMD -MP xhttp_rpc_fnc.c:188:18: warning: 'XHTTP_RPC_Post_1a' defined but not used [-Wunused-const-variable=] static const str XHTTP_RPC_Post_1a = str_init("\n"\ ^~~~~~~~~~~~~~~~~ xhttp_rpc_fnc.c:176:18: warning: 'XHTTP_RPC_ATTR_VAL_SEPARATOR' defined but not used [-Wunused-const-variable=] static const str XHTTP_RPC_ATTR_VAL_SEPARATOR = str_init("="); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ xhttp_rpc_fnc.c:175:18: warning: 'XHTTP_RPC_ATTR_SEPARATOR' defined but not used [-Wunused-const-variable=] static const str XHTTP_RPC_ATTR_SEPARATOR = str_init(" "); ^~~~~~~~~~~~~~~~~~~~~~~~ xhttp_rpc_fnc.c:174:18: warning: 'XHTTP_RPC_NODE_SEPARATOR' defined but not used [-Wunused-const-variable=] static const str XHTTP_RPC_NODE_SEPARATOR = str_init(":: "); ^~~~~~~~~~~~~~~~~~~~~~~~ gcc -shared -m64 -Wl,-O2 -Wl,-E xhttp_rpc.o xhttp_rpc_fnc.o -o xhttp_rpc.so make[2]: Leaving directory '/builddir/build/BUILD/kamailio-5.0.0/src/modules/xhttp_rpc' ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287205186

Anthony Messina

18 Mar 18 Mar

5:12 a.m.

After applying f81c128, no coredump, but I get `jsonrpcs [jsonrpcs_mod.c:1174]: jsonrpc_dispatch(): jsonrpc over http not initialized - check transport param`

FYI, The request is sent in via 127.0.0.1:5060 TCP, with nexthop set as 127.0.0.1:5060 so the request will route through the script and eventually gets looked up by lookup_branches--finding the contact which is a SIP/TLS endpoint.

Attached log. [kamailio-coredump-jsonrpcs-message_3.txt](https://github.com/kamailio/kamailio/files/852183/kamailio-coredump-jsonrpcs...)

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287517348

Anthony Messina

5:23 a.m.

Well, I'll be damned. Your log message gave me the clue. I was simply missing `modparam("jsonrpcs", "transport", 1)`. I must have missed that in the upgrade.

Maybe that parameter should be added to the default kamailio.cfg since the `Default value is '6' (fifo and datagram transport).`

Sorry @miconda. But I guess it shouldn't segfault either ;) Thanks again.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287517767

Anthony Messina

5:23 a.m.

Closed #1030.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#event-1005402855

Daniel-Constantin Mierla

7:04 a.m.

It was a good catch after all, indeed it should not segfault and now should be fixed.

Yesterday I didn't have time to add here the comments about it, but as jsonrpcs replaced the mi_fifo, its default transports were changed in 5.0, given that xhttp is not loaded by default.

I also added a note in the upgrade guide from 4.4 to 5.0. Thanks for assisting with troubleshooting.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1030#issuecomment-287521722

2899

Age (days ago)

2906

Last active (days ago)

sr-dev@lists.kamailio.org

17 comments

2 participants

tags (0)

participants (2)

Anthony Messina
Daniel-Constantin Mierla