I am running into a situation where the nathelper module is causing openser/kamailio to crash. The crash occurs when opensbc sends and invite to openser with an extra CR in the message body. I already contacted the opensbc folks about this. I just wanted you all to be aware that this could cause serious service interruptions.
Error log:
kernel: [599650.771067] kamailio[25461]: segfault at 18 ip b7a0a6c7 sp bfbaaa70 error 4 in nathelper.so[b79f7000+18000]
Sample invite sent to kamailio.. Please look at the extra CR/LF after Supported. The nathelper module crashes after this.
INVITE sip:phonenumber@chgw.commx.net SIP/2.0 From: "South Test" sip:phonenumber@orig.domain.com;user=phone;tag=392385664-1238777567965- To: sip:phonenumber@chgw.commx.net Via: SIP/2.0/UDP opensbc:5060;branch=z9hG4bKdec1a187dd1ede1193b6b6f3163a7dda;uas-addr=openser;rport CSeq: 913493359 INVITE Call-ID: BW1252479650304091860611707@orig.domain.com-0x000a Contact: sip:+18135146513@opensbc Max-Forwards: 9 Accept: multipart/mixed, application/dtmf-relay, application/media_control+xml, application/sdp Allow: ACK, BYE, CANCEL, INFO, INVITE, OPTIONS, PRACK, REFER, NOTIFY, UPDATE Proxy-Require: privacy Supported: 100rel
Content-Type: application/sdp Content-Length: 214
v=0 o=BroadWorks 3675 1 IN IP4 gw1.domain.com s=- c=IN IP4 opensbc t=0 0 m=audio 10074 RTP/AVP 18 0 101 a=rtpmap:18 G729/8000 a=fmtp:18 annexb=no a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000
Carlos A. Alvarez writes:
I am running into a situation where the nathelper module is causing openser/kamailio to crash. The crash occurs when opensbc sends and invite to openser with an extra CR in the message body. I already contacted the opensbc folks about this. I just wanted you all to be aware that this could cause serious service interruptions.
i think daniel fixed this a few days ago. try with kamailio 1.5. perhaps the fix is also in 1.4, but i'm not sure.
-- juha
Thanks for your reply, I am using v 1.5, perhaps downloading the latest from CVS?
________________________________________ From: Juha Heinanen [jh@tutpro.com] Sent: Saturday, April 04, 2009 3:55 AM To: Carlos A. Alvarez Cc: users@lists.kamailio.org Subject: [Kamailio-Users] Nethelper Causes Kamailio to Crash
Carlos A. Alvarez writes:
I am running into a situation where the nathelper module is causing openser/kamailio to crash. The crash occurs when opensbc sends and invite to openser with an extra CR in the message body. I already contacted the opensbc folks about this. I just wanted you all to be aware that this could cause serious service interruptions.
i think daniel fixed this a few days ago. try with kamailio 1.5. perhaps the fix is also in 1.4, but i'm not sure.
-- juha
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.285 / Virus Database: 270.11.40/2039 - Release Date: 04/03/09 17:54:00
Juha,
I tried the latest from svn and I still have the same problem. Any other ideas?
-----Original Message----- From: Juha Heinanen [mailto:jh@tutpro.com] Sent: Saturday, April 04, 2009 8:50 AM To: Carlos A. Alvarez Cc: users@lists.kamailio.org Subject: RE: [Kamailio-Users] Nethelper Causes Kamailio to Crash
Carlos A. Alvarez writes:
I am using v 1.5, perhaps downloading the latest from CVS?
yes, check out 1.5 branch from svn.
-- juha
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.285 / Virus Database: 270.11.40/2039 - Release Date: 04/03/09 17:54:00
This is what I get with debug enabled.
DBG:carrierroute:rewrite_on_rule: searching for matching routing rules [847853.158339] kamailio[10357]: segfault at 18 ip b7a06d17 sp bf9a8a10 error 4 in nathelper.so[b79f0000+1b000] [10357]: DBG:carrierroute:hash_func: hash: 304942355 % 1000 = 355 [10357]: INFO:carrierroute:cr_do_route: uri 5555555555 was rewritten to sip:5555555555@ip_address, carrier 1, domain 9 [10357]: cr_route("1", "ip_address", "5555555555", "5555555555", "call_id",""); [10357]: DBG:core:parse_headers: flags=ffffffffffffffff [10370]: CRITICAL:core:receive_fd: EOF on 7 [10370]: DBG:core:handle_ser_child: dead child 2, pid 10357 (shutting down?) [10355]: INFO:core:handle_sigs: child process 10357 exited by a signal 11 [10355]: INFO:core:handle_sigs: core was not generated [10355]: INFO:core:handle_sigs: terminating due to SIGCHLD [10362]: INFO:core:sig_usr: signal 15 received [10360]: INFO:core:sig_usr: signal 15 received [10362]: Memory status (pkg): [10360]: Memory status (pkg): [10362]: fm_status (0x8143cc0): [10360]: fm_status (0x8143cc0): [10362]: heap size= 4194304 [10360]: heap size= 4194304 [10362]: used= 125968, used+overhead=154096, free=4040208 [10360]: used= 125968, used+overhead=154088, free=4040216 [10362]: max used (+overhead)= 154104 [10364]: INFO:core:sig_usr: signal 15 received [10364]: Memory status (pkg): [10364]: fm_status (0x8143cc0): [10364]: heap size= 4194304 [10364]: used= 183040, used+overhead=211072, free=3983232 [10364]: max used (+overhead)= 211072 [10364]: dumping free list: [10364]: hash = 1 fragments no.: 3, unused: 0#012#011#011 bucket size: 8 - 8 (first 8) [10364]: hash = 2056 fragments no.: 1, unused: 0#012#011#011 bucket size: 2097152 - 4194304 (first 3983448) [10364]: TOTAL: 4 free fragments = 3983472 free bytes [10364]: -----------------------------
-----Original Message----- From: Juha Heinanen [mailto:jh@tutpro.com] Sent: Monday, April 06, 2009 8:38 AM To: Carlos A. Alvarez Cc: users@lists.kamailio.org Subject: RE: [Kamailio-Users] Nethelper Causes Kamailio to Crash
Carlos A. Alvarez writes:
I tried the latest from svn and I still have the same problem. Any other ideas?
get core dump and check with gdb 'where' command where the crash takes place.
-- juha
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.285 / Virus Database: 270.11.40/2039 - Release Date: 04/06/09 06:22:00
On Monday 06 April 2009, Carlos A. Alvarez wrote:
signal 11 [10355]: INFO:core:handle_sigs: core was not generated
Hi Carlos,
the server don't generated a core file. Can you try to create one, please see here for more informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
Cheers,
Henning
I'm sorry the server is not crashing, it's the kamailio application that is crashing. You see a segfault in the log that I sent.
-----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Monday, April 06, 2009 11:03 AM To: users@lists.kamailio.org Cc: Carlos A. Alvarez; Juha Heinanen Subject: Re: [Kamailio-Users] Nethelper Causes Kamailio to Crash
On Monday 06 April 2009, Carlos A. Alvarez wrote:
signal 11 [10355]: INFO:core:handle_sigs: core was not generated
Hi Carlos,
the server don't generated a core file. Can you try to create one, please see here for more informations:
http://www.kamailio.org/dokuwiki/doku.php/troubleshooting:corefiles
Cheers,
Henning
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.285 / Virus Database: 270.11.40/2039 - Release Date: 04/06/09 06:22:00
On Monday 06 April 2009, Carlos A. Alvarez wrote:
I'm sorry the server is not crashing, it's the kamailio application that is crashing. You see a segfault in the log that I sent. [..]
Hi Carlos,
yes, i was refering to the kamailio server, sorry if i was a bit unclear. :-) It would be really helpful if you could try to generate a core file and then print a backtrace as described in the wiki article given previously.
Cheers,
Henning
-
Carlos A. Alvarez -
Henning Westerholt -
jh＠tutpro.com