I'm doing sipp performance measurements with openser 1.2 and I was surprised on how much memory the openser processes do allocate when confronted with a high call-per-second load.
I observed the following behavior:
One sipp instance creates SIP requests that get routed through openser and forwarded to another sipp instance running in UAS mode. The sipp scenario looks like
INVITE ----------> 100 <---------- 180 <---------- 200 <---------- ACK ----------> Pause [ 10.0s] BYE ----------> 200 <----------
Nothing special here. The openser instance record routes the SIP requests and forwards them in transaction stateful mode (t_relay) to the sipp UAS instance. The routing script looks like:
route{ if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1); }
route[1] { if (!t_relay()) sl_reply_error(); exit; }
( xx.xx.xx.xx is the IP of sipp UAS)
Running sipp with a call rate of 500 cps results in 45 MB of memory allocation per openser process after a few seconds. What could cause this behavior, message queuing maybe? But then again having a message queue of 40+ MB seems to be unrealistic.
And the memory consumption goes up the longer the test runs until openser dies because of out-of-memory.
I observed the same behavior with openser 1.1 and 1.2. Is this a memory leak or did I miss some settings? And what's the best method to debug memory issues with openser?
thanks in advance, Christian
Hello,
On 03/15/07 21:25, Christian Schlatter wrote:
I'm doing sipp performance measurements with openser 1.2 and I was surprised on how much memory the openser processes do allocate when confronted with a high call-per-second load.
I observed the following behavior:
One sipp instance creates SIP requests that get routed through openser and forwarded to another sipp instance running in UAS mode. The sipp scenario looks like
INVITE ----------> 100 <---------- 180 <---------- 200 <---------- ACK ----------> Pause [ 10.0s] BYE ----------> 200 <----------
Nothing special here. The openser instance record routes the SIP requests and forwards them in transaction stateful mode (t_relay) to the sipp UAS instance. The routing script looks like:
route{ if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1); }
route[1] { if (!t_relay()) sl_reply_error(); exit; }
( xx.xx.xx.xx is the IP of sipp UAS)
Running sipp with a call rate of 500 cps results in 45 MB of memory allocation per openser process after a few seconds. What could cause this behavior, message queuing maybe? But then again having a message queue of 40+ MB seems to be unrealistic.
It is not per process. It is the share memory, I guess. By default, each openser process can use maximum of 1MB. Share memory is 32MB by default, can be changed by -w command line parameter. It cannot go over it, it is the moment of out of memory. It is all the time visible as allocated by openser, but can be free. You can use "openserctl fifo get_statistics all" to see how much of memory is really used.
A transaction lasts a bit more in memory after a final response has been answered. The transaction keeps at least two times the size of initial request, plus a reply at a moment, so it demands some memory. If you get high load at a moment you need more memory. Try to set share memory size to higher values and run the tests again.
And the memory consumption goes up the longer the test runs until openser dies because of out-of-memory.
What do you mean by dies? It crashes or just prints lot of error messages in syslog? If you stop the sip flood and try a bit later, did you get same errors from first time?
I observed the same behavior with openser 1.1 and 1.2. Is this a memory leak or did I miss some settings?
We needed to set share memory to 768MB to be able to flood openser at full power of 2 sipp instances. The results were about 4000 cps per instance.
And what's the best method to debug memory issues with openser?
Not very easy right now, compiling without F_MALLOC and with DBG_QM_MALLOC is the best. Plans to improve in this aspects are scheduled for next release.
Cheers, Daniel
thanks in advance, Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Daniel,
Daniel-Constantin Mierla wrote: ...
Running sipp with a call rate of 500 cps results in 45 MB of memory allocation per openser process after a few seconds. What could cause this behavior, message queuing maybe? But then again having a message queue of 40+ MB seems to be unrealistic.
It is not per process. It is the share memory, I guess. By default, each openser process can use maximum of 1MB. Share memory is 32MB by default, can be changed by -w command line parameter. It cannot go over it, it is the moment of out of memory. It is all the time visible as allocated by openser, but can be free. You can use "openserctl fifo get_statistics all" to see how much of memory is really used.
Thanks for the clarification, I was always looking at 'top' output that shows total shared memory as per process reserved memory. The fifo statistics are very helpful.
A transaction lasts a bit more in memory after a final response has been answered. The transaction keeps at least two times the size of initial request, plus a reply at a moment, so it demands some memory. If you get high load at a moment you need more memory. Try to set share memory size to higher values and run the tests again.
Ok, so memory consumption is directly related to number of ongoing SIP transactions. Confusingly, 'top' only shows max used memory so I was thinking that openser didn't release memory after stopping the SIP flood.
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I could successfully achieve about 2000 cps over a long period with the simple config which is probably limited by the single sipp UAC instance and not by openser's performance.
I also tested userloc in db-only mode with a mysql cluster storing 30'000 permanent registration entries (location table). With this setup I could achieve about 800 cps (randomized user population), which is a very good number. With a BHCA of 5 this would be enough to support over 500'000 user agents.
thanks, Christian
And the memory consumption goes up the longer the test runs until openser dies because of out-of-memory.
What do you mean by dies? It crashes or just prints lot of error messages in syslog? If you stop the sip flood and try a bit later, did you get same errors from first time?
I observed the same behavior with openser 1.1 and 1.2. Is this a memory leak or did I miss some settings?
We needed to set share memory to 768MB to be able to flood openser at full power of 2 sipp instances. The results were about 4000 cps per instance.
And what's the best method to debug memory issues with openser?
Not very easy right now, compiling without F_MALLOC and with DBG_QM_MALLOC is the best. Plans to improve in this aspects are scheduled for next release.
Cheers, Daniel
thanks in advance, Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1);
if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1); }
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory' syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Hello,
that's the result of memory fragmentations, that should be the the capacity of the server, should stay at that rate. Memory manager allocates chunks of different sizes, but not merges then back by default. However, it tries to avoid allocating small sizes -- you can see in statistics used memory and real used sizes. The difference is not used but should ensure that shared variables will not fragment memory in small chunks.
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount of
shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory' syslog
messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Daniel-Constantin Mierla wrote:
Hello,
that's the result of memory fragmentations, that should be the the capacity of the server, should stay at that rate. Memory manager allocates chunks of different sizes, but not merges then back by default. However, it tries to avoid allocating small sizes -- you can see in statistics used memory and real used sizes. The difference is not used but should ensure that shared variables will not fragment memory in small chunks.
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is. Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount of
shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory' syslog
messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
On 03/18/07 23:09, Christian Schlatter wrote:
Daniel-Constantin Mierla wrote:
Hello,
that's the result of memory fragmentations, that should be the the capacity of the server, should stay at that rate. Memory manager allocates chunks of different sizes, but not merges then back by default. However, it tries to avoid allocating small sizes -- you can see in statistics used memory and real used sizes. The difference is not used but should ensure that shared variables will not fragment memory in small chunks.
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount
of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
FYI: There was a discussion on serdev recently - also with some performance comparisons: http://lists.iptel.org/pipermail/serdev/2007-February/009433.html and http://lists.iptel.org/pipermail/serdev/2007-March/date.html (Subject: Using system memory allocator as the main one in SER)
regards klaus
Daniel-Constantin Mierla wrote:
On 03/18/07 23:09, Christian Schlatter wrote:
Daniel-Constantin Mierla wrote:
Hello,
that's the result of memory fragmentations, that should be the the capacity of the server, should stay at that rate. Memory manager allocates chunks of different sizes, but not merges then back by default. However, it tries to avoid allocating small sizes -- you can see in statistics used memory and real used sizes. The difference is not used but should ensure that shared variables will not fragment memory in small chunks.
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount
of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the amount
of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
I will reproduce the tests and hope I can catch something ...
Cheers, Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the
amount of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Hello,
the supposed fragmentation turned out to be a mem leak in pkg. Please take the latest SVN version and try again to see if you got same results.
Thanks, Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ...
I always had 768MB shared memory configured though, so I still can't explain the memory allocation errors I got. Some more test runs revealed that I only get these errors when using a more production oriented config that loads more modules than the one posted in my earlier email. I now try to figure out what exactly causes these memory allocation errors that happen reproducibly after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the
amount of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Daniel-Constantin Mierla wrote:
Hello,
the supposed fragmentation turned out to be a mem leak in pkg. Please take the latest SVN version and try again to see if you got same results.
Works like a charm!
thanks a lot, Christian
Thanks, Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ... > > I always had 768MB shared memory configured though, so I still > can't explain the memory allocation errors I got. Some more test > runs revealed that I only get these errors when using a more > production oriented config that loads more modules than the one > posted in my earlier email. I now try to figure out what exactly > causes these memory allocation errors that happen reproducibly > after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the
amount of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Hi Daniel,
Can we backport this one to 1.2?
Regards, Ovidiu Sas
On 3/22/07, Daniel-Constantin Mierla daniel@voice-system.ro wrote:
Hello,
the supposed fragmentation turned out to be a mem leak in pkg. Please take the latest SVN version and try again to see if you got same results.
Thanks, Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote:
Christian Schlatter wrote: ... > > I always had 768MB shared memory configured though, so I still > can't explain the memory allocation errors I got. Some more test > runs revealed that I only get these errors when using a more > production oriented config that loads more modules than the one > posted in my earlier email. I now try to figure out what exactly > causes these memory allocation errors that happen reproducibly > after about 220s at 400 cps.
I think I found the cause for the memory allocation errors. As soon as I include an AVP write operation in the routing script, I get 'out of memory' messages after a certain number of calls generated with sipp.
The routing script to reproduce this behavior looks like (full config available at http://www.unc.edu/~cschlatt/openser/openser.cfg):
route{ $avp(s:ct) = $ct; # commenting this line solves # the memory problem
if (!method=="REGISTER") record_route(); if (loose_route()) route(1); if (uri==myself) rewritehost("xx.xx.xx.xx"); route(1);}
route[1] { if (!t_relay()) sl_reply_error(); exit; }
An example log file showing the 'out of memory' messages is available at http://www.unc.edu/~cschlatt/openser/openser.log .
Some observations:
- The 'out of memory' messages always appear after about 8000 test
calls per worker process. One call consists of two SIP transactions and six end-to-end SIP messages. An openser with 8 children handles about 64'000 calls, whereas 4 children only handle about 32'000 calls. The sipp call rate doesn't matter, only number of calls.
- The 8000 calls per worker process are independent from the
amount of shared memory available. Running openser with -m 128 or -m 768 does not make a difference.
- The more AVP writes are done in the script, the less calls go
through. It looks like each AVP write is leaking memory (unnoticed by the memory statistics).
- The fifo memory statistics do not reflect the 'out of memory'
syslog messages. Even if openser does not route a single SIP message because of memory issues, the statistics still show a lot of 'free' memory.
All tests were done with openser SVN 1.2 branch on Ubuntu dapper x86. I think the same is true for 1.1 version but I haven't tested that yet.
Christian
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Hello Ovidiu,
On 03/23/07 17:04, Ovidiu Sas wrote:
Hi Daniel,
Can we backport this one to 1.2?
already done, two minutes after the commit in trunk.
Cheers, Daniel
Regards, Ovidiu Sas
On 3/22/07, Daniel-Constantin Mierla daniel@voice-system.ro wrote:
Hello,
the supposed fragmentation turned out to be a mem leak in pkg. Please take the latest SVN version and try again to see if you got same results.
Thanks, Daniel
On 03/19/07 18:52, Christian Schlatter wrote:
...
The memory statistics indeed show a high number of memory fragments:
before 'out of memory':
shmem:total_size = 536870912 shmem:used_size = 59607040 shmem:real_used_size = 60106488 shmem:max_used_size = 68261536 shmem:free_size = 476764424 shmem:fragments = 9897
after 'out of memory' (about 8000 calls per process):
shmem:total_size = 536870912 shmem:used_size = 4171160 shmem:real_used_size = 4670744 shmem:max_used_size = 68261536 shmem:free_size = 532200168 shmem:fragments = 57902
You can try to compile openser with -DQM_JOIN_FREE (add it in DEFS variable of Makefile.defs) and test again. Free fragments should be merged and fragmentation should not occur -- processing will be slower. We will try for next release to provide a better solution for that.
Compiling openser with -DQM_JOIN_FREE did not help. I'm not sure how big of a problem this fragmentation issue is.
What is the number of fragments with QM_JOIN_FREE after flooding?
The numbers included above are with QM_JOIN_FREE enabled.
Do you think it would make sense to restart our production openser instances from time to time just to make sure they're not running into this memory fragmentation limits?
The issue will occur only when the call rate reaches the limits of the proxy's memory. Otherwise the chunks are reused. Transactions and avps are rounded up to be sure there will be minimized the number of different sizes for memory chunks. It wasn't reported too often, maybe that's why no big attention was paid to it. This memory system is in place since the beginning of ser. Alternative is to use sysv shared memory, but is much slower, along with libc private memory manager.
I've done some more testing and the same out-of-memory stuff happens when I run sipp with 10 calls per second only. I tested with 'children=1' and I only could get through about 8200 calls (again those 8000 calls / process). And this is with QM_JOIN_FREE enabled.
Memory statistics:
before: shmem:total_size = 536870912 shmem:used_size = 2311976 shmem:real_used_size = 2335720 shmem:max_used_size = 2465816 shmem:free_size = 534535192 shmem:fragments = 183
after: shmem:total_size = 536870912 shmem:used_size = 1853472 shmem:real_used_size = 1877224 shmem:max_used_size = 2465816 shmem:free_size = 534993688 shmem:fragments = 547
So I'm not sure if this is really a fragmentation issue. 10 cps surely doesn't reach the proxy's memory.
Thoughts?
Christian
Cheers, Daniel
thanks, Christian
Cheers, Daniel
On 03/18/07 01:21, Christian Schlatter wrote: > Christian Schlatter wrote: > ... >> >> I always had 768MB shared memory configured though, so I still >> can't explain the memory allocation errors I got. Some more test >> runs revealed that I only get these errors when using a more >> production oriented config that loads more modules than the one >> posted in my earlier email. I now try to figure out what exactly >> causes these memory allocation errors that happen reproducibly >> after about 220s at 400 cps. > > I think I found the cause for the memory allocation errors. As > soon as I include an AVP write operation in the routing script, I > get 'out of memory' messages after a certain number of calls > generated with sipp. > > The routing script to reproduce this behavior looks like (full > config available at > http://www.unc.edu/~cschlatt/openser/openser.cfg): > > route{ > $avp(s:ct) = $ct; # commenting this line solves > # the memory problem > > if (!method=="REGISTER") record_route(); > if (loose_route()) route(1); > > if (uri==myself) rewritehost("xx.xx.xx.xx"); > route(1); > } > > route[1] { > if (!t_relay()) sl_reply_error(); > exit; > } > > An example log file showing the 'out of memory' messages is > available at http://www.unc.edu/~cschlatt/openser/openser.log . > > Some observations: > > - The 'out of memory' messages always appear after about 8000 test > calls per worker process. One call consists of two SIP > transactions and six end-to-end SIP messages. An openser with 8 > children handles about 64'000 calls, whereas 4 children only > handle about 32'000 calls. The sipp call rate doesn't matter, only > number of calls. > > - The 8000 calls per worker process are independent from the > amount of shared memory available. Running openser with -m 128 or > -m 768 does not make a difference. > > - The more AVP writes are done in the script, the less calls go > through. It looks like each AVP write is leaking memory (unnoticed > by the memory statistics). > > - The fifo memory statistics do not reflect the 'out of memory' > syslog messages. Even if openser does not route a single SIP > message because of memory issues, the statistics still show a lot > of 'free' memory. > > > All tests were done with openser SVN 1.2 branch on Ubuntu dapper > x86. I think the same is true for 1.1 version but I haven't tested > that yet. > > > Christian >
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
Users mailing list Users@openser.org http://openser.org/cgi-bin/mailman/listinfo/users
-
Christian Schlatter -
Daniel-Constantin Mierla -
Klaus Darilion -
Ovidiu Sas