Hello,
Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:
Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf. That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced. In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.
Other info: This is Kamailio 4.4.4 on x86_64. I've not had success trying to reproduce this yet. Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.
Thoughts are welcome. Thanks!
Armen
Hello,
I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there?
Can you also give the locals?
frame 0
info locals
Cheers, Daniel
On 05.06.17 05:19, Armen Babikyan wrote:
Hello,
Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:
Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf. That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced. In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.
Other info: This is Kamailio 4.4.4 on x86_64. I've not had success trying to reproduce this yet. Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.
Thoughts are welcome. Thanks!
Armen
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
The server is running other protocols as well, yes, but those requests are handled on other ports (e.g. WSS on 443/tcp, TLS on 5061/tcp).
Regarding the locals, I have updated the pastebin.
Many thanks!
Armen
On Mon, Jun 5, 2017 at 1:23 AM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there?
Can you also give the locals?
frame 0
info locals
Cheers, Daniel
On 05.06.17 05:19, Armen Babikyan wrote:
Hello,
Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:
Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf. That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced. In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.
Other info: This is Kamailio 4.4.4 on x86_64. I've not had success trying to reproduce this yet. Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.
Thoughts are welcome. Thanks!
Armen
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierlawww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio Advanced Training - www.asipto.com Kamailio World Conference - www.kamailioworld.com
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
I analyzed the code and couldn't find a reason for that pointer to be out of range. It could be some memory corruption, independent of the software, given that's one bit shifted one position in 'parsed', all other bits are the same as for buf/start/pos, which should be the good value.
But first to dig into it a bit more ...
- from frame 0, let's see if there is something in the read buffer, get:
p r->buf[0] p r->buf[1] p r->buf[2] p r->buf[3]
- from frame 3, get:
info locals p *h p *fm
Cheers, Daniel
On 05.06.17 16:58, Armen Babikyan wrote:
Hi Daniel,
The server is running other protocols as well, yes, but those requests are handled on other ports (e.g. WSS on 443/tcp, TLS on 5061/tcp).
Regarding the locals, I have updated the pastebin.
Many thanks!
Armen
On Mon, Jun 5, 2017 at 1:23 AM, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there? Can you also give the locals? frame 0 info locals Cheers, Daniel On 05.06.17 05:19, Armen Babikyan wrote:Hello, Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here: https://pastebin.com/qJ3ypnVz Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf. That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced. In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf. Other info: This is Kamailio 4.4.4 on x86_64. I've not had success trying to reproduce this yet. Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash. Thoughts are welcome. Thanks! Armen _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>-- Daniel-Constantin Mierla www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Kamailio Advanced Training - www.asipto.com <http://www.asipto.com> Kamailio World Conference - www.kamailioworld.com <http://www.kamailioworld.com> _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
-
Armen Babikyan -
Daniel-Constantin Mierla