[Users] Re: Memory issues - sr-users

12 Jun 2007

Hello,

We have exactly the same problem. Openser after a few hours starts loging "out of
memory" errors and soon after that it cannot process any messages. I am using 1.2
branch from SVN and the last export that I got was at the begining of June, the revision
number was 2333. Today I was able to catch the problem while it is happening and I did a
memory dump on the process that run out of memory (process 4197). ps shows that this
process is a receiver with id=1 and child=0. I have not restarted the server yet, but
pretty soon I will have to restart because more processes will run out of memory. If there
is some additional information (statistics, ngrep etc) that I could gather while the
server is still running, please let me know. I don't have quick access to a public ftp
site, so if someone can take a look at the log file and the memory dump, please let me
know and I will email it (1MB).

thank you for any help.

George

Date: Mon, 11 Jun 2007 17:52:08 +0200
From: Klaus Darilion &lt;klaus.mailinglists(a)pernau.at&gt;
Subject: Re: [Users] Memory issues
To: Christian &lt;mailing-lists(a)switzernet.com&gt;
Cc: users(a)openser.org
Message-ID: &lt;466D6FA8.9010201(a)pernau.at&gt;
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Christian wrote:
...
  Thank you Bogdan,

 For the moment I am not very sure to switch to SVN branch (this is a 
 production server, so every minor bug can be dramatic) SVN != unstable
SVN is the code repository. In the repository there are different branches. E.g. there is
1.1: openser 1.1 branch
1.2: openser 1.2 branch
trunk: openser development branch
Thus, you should update to the 1.2 branch from SVN. This is the currents table release
with all bugfixes:
...
  But I will at least upgrade to 1.2.1 asap, and see if
the issue gets 
 solved. 1.2.1 is a snapshot taken from 1.2 branch
regards
klaus
...
  The main issue here is that the memory leak
doesn't crash openser, but 
 puts it in a state where messages are accepted but not relayed.
 Shouldn't it rather send a SIP error reply (500 I guess), so that 
 handover on other servers gets properly handled?

 Regards,
 Christian

 Bogdan-Andrei Iancu a ιcrit :
  it looks like PKG memory problem.
 Christian, try to update to the latest 1.2 version from SVN and see 
 if the problem disappears - there were some pkg mem leak fixed 
 since the 1.2 release.

 regards,
 bogdan

 Klaus Darilion wrote:

http://www.openser.org/dokuwiki/doku.php/troubleshooting:memory
<http://www.openser.org/dokuwiki/doku.php/troubleshooting:memory> 

 Make sure to use newest 1.2 (make a SVN checkout of /branches/1.2/ )

 regards
 klaus

 Christian wrote:
  Hi All,

 I have a recurrent memory problem with openser 1.2.0. In this case 
 it only acts as a SIP router. No registration, accounting etc. and 
 only approx. 20 simultaneous calls.

 Here is the problem: everything works fine (calls from/to PSTN 
 through different SIP servers), but after a few days, "out of 
 memory" errors start to occur. After this, calls don't work 
 anymore, even if openser continues to run.

 Here is a part of the log, showing the first errors:

 Jun 5 17:25:43 localhost SER[27772]: ERROR: 
 build_res_buf_from_sip_res: out of mem Jun 5 17:25:43 localhost 
 SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer 
 Jun 5 17:25:43 localhost SER[27772]: ERROR: via_builder: out of 
 memory Jun 5 17:25:43 localhost SER[27772]: ERROR: build_local: no 
 via header got from builder Jun 5 17:25:43 localhost SER[27772]: 
 ERROR: attempt to build a CANCEL failed Jun 5 17:25:50 localhost 
 SER[27772]: ERROR:
 build_res_buf_from_sip_res: out of mem Jun 5 17:25:50 localhost 
 SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer 
 Jun 5 17:25:50 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_req: out of memory ; needs 410 Jun 5 
 17:26:11 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_res: out of mem Jun 5 17:26:11 localhost 
 SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer 
 Jun 5 17:26:11 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_req: out of memory ; needs 404 Jun 5 
 17:26:25 localhost SER[27772]: eval_elem: no more memory Jun 5 
 17:26:25 localhost SER[27772]: do_assign: no value in right 
 expression Jun 5 17:26:25 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_res: out of mem Jun 5 17:26:25 localhost 
 SER[27772]: ERROR:tm:relay_reply: no mem for outbound reply buffer 
 Jun 5 17:26:25 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_req: out of memory ; needs 420 Jun 5 
 17:26:25 localhost SER[27771]: ERROR:forward_reply: no 2nd via 
 found in reply Jun 5 17:26:31 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_res: out of mem Jun 5 17:26:31 localhost 
 SER[27772]: ERROR:forward_reply: building rpl from req failed Jun 
 5 17:26:33 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_res: out of mem Jun 5 17:26:33 localhost 
 SER[27772]: ERROR:forward_reply: building rpl from req failed Jun 
 5 17:26:38 localhost SER[27772]: ERROR: insert_new_lump_after:
 out of memory
 Jun 5 17:26:38 localhost SER[27772]: ERROR: insert_new_lump_before: 
 out of memory
 Jun 5 17:26:38 localhost SER[27772]: insert_RR(): Error while 
 inserting conditional lump Jun 5 17:26:38 localhost SER[27772]: 
 ERROR: subst_str: mem.
 allocation error (res->s)
 Jun 5 17:26:38 localhost SER[27772]: ERROR: do_action: memory 
 allocation failure Jun 5 17:26:38 localhost SER[27772]: ERROR:
 build_res_buf_from_sip_req: out of memory ; needs 375 Jun 5 
 17:26:38 localhost SER[27772]: ERROR: mk_proxy: memory allocation 
 failure Jun 5 17:26:38 localhost SER[27772]: 
 ERROR:tm:t_forward_nonack:
 failure to add branches

 At this point, similar errors continue to appear and the only way 
 to recover is to restart openser.
 Nothing unusual seems to cause the initial error. The output of 
 "top" appears to be fine after the "crash", and there is still 
 plenty free memory on the server.

 Could anybody give me some hints on how to solve the problem? I 
 suspect a portion of the config file where the record-route headers 
 are split on multiple lines (for compatibility issues with another 
 SIP proxy). It's the only place where variables are used. You can 
 find it here:

 http://switzernet.com/people/christian-lathion/files/070605-openser
<http://switzernet.com/people/christian-lathion/files/070605-openser> 
 -config-part.txt

 Otherwise the config file is quite usual, with basic processing and 
 routing of SIP packets.

 Regards,
 Christian

 _______________________________________________
 Users mailing list
 Users(a)openser.org
 http://openser.org/cgi-bin/mailman/listinfo/users
<http://openser.org/cgi-bin/mailman/listinfo/users>  
 _______________________________________________
 Users mailing list
 Users(a)openser.org
 http://openser.org/cgi-bin/mailman/listinfo/users
<http://openser.org/cgi-bin/mailman/listinfo/users> 

 _______________________________________________
 Users mailing list
 Users(a)openser.org
 http://openser.org/cgi-bin/mailman/listinfo/users
<http://openser.org/cgi-bin/mailman/listinfo/users>   

Disclaimer
The information in this e-mail and any attachments is confidential. It is intended solely
for the attention and use of the named addressee(s). If you are not the intended
recipient, or person responsible for delivering this information to the intended
recipient, please notify the sender immediately. Unless you are the intended recipient or
his/her representative you are not authorized to, and must not, read, copy, distribute,
use or retain this message or any part of it. E-mail transmission cannot be guaranteed to
be secure or error-free as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses.