There is a new release of SER MediaProxy available. To update your MediaProxy installation you need to perform the following steps:
1. cvs update on your unstable SER tree (for stable copy modules/mediaproxy directory under your stable tree) and recompile 2. Download and install the latest server from http://www.ag-projects.com/SER_Media_Proxy.html
Among various bugs fixes the performance has been optimized and the billing features expanded. The live media sessions interface provides web access to media streams from multiple servers including CodecType, User Agent, Network traffic and Call status.
A snapshot can be seen at:
http://www.ag-projects.com/docs/MediaSessions.pdf
Regards, Adrian
Adrian Georgescu wrote:
There is a new release of SER MediaProxy available. To update your MediaProxy installation you need to perform the following steps:
Hi Adrian,
I am interested to know if you have done any sort of loadtest on a single server. Its usefull to have some sort of capacity benchmark
Thanks,
There is a load generator included with the distribution. If you need a benchmark that is 70 calls using G711 codec on a 1GHz PIII. This is pure orientative, use the load generator for testing you particular hardware. The idea (or the beauty of it) is that is possible to stack many boxes and distribute the load among them instead of trying to put many calls in one box (which would eventually become another single point of failure in your infrastructure).
Adrian
On Jun 29, 2004, at 9:45 PM, Andres wrote:
Adrian Georgescu wrote:
There is a new release of SER MediaProxy available. To update your MediaProxy installation you need to perform the following steps:
Hi Adrian,
I am interested to know if you have done any sort of loadtest on a single server. Its usefull to have some sort of capacity benchmark
Thanks,
-- Andres Network Admin http://www.telesip.net
Adrian Georgescu wrote:
There is a load generator included with the distribution. If you need a benchmark that is 70 calls using G711 codec on a 1GHz PIII. This is pure orientative, use the load generator for testing you particular hardware. The idea (or the beauty of it) is that is possible to stack many boxes and distribute the load among them instead of trying to put many calls in one box (which would eventually become another single point of failure in your infrastructure).
Adrian
Thanks Adrian,
I will have to try MediaProxy again sometime with the load distribution feature. I tried the plain MediaProxy when it first came out last year and it worked well.
Thanks,
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
TEST MEDIAPROXY.PY ------------------ 180 Simultaneous Calls rtpgenerator run on same server python-2.2.3-7 ------------------ SERVER: ------------------------------------------------------ vendor_id : GenuineIntel model name : Intel(R) Pentium(R) 4 CPU 2.60GHz cpu MHz : 2593.547 cache size : 512 KB ------------------------------------------------------ vendor_id : GenuineIntel model name : Intel(R) Pentium(R) 4 CPU 2.60GHz cpu MHz : 2593.547 cache size : 512 KB ------------------------------------------------------ ------------------------------------------------------ MemTotal: 1031500 kB SwapTotal: 2104312 kB ------------------------------------------------------ Linux 2.4.22-1.2115.nptlsmp
Ezequiel Colombo
----- Original Message ----- From: "Andres" andres@telesip.net To: "Adrian Georgescu" ag@ag-projects.com Cc: serusers@lists.iptel.org Sent: Tuesday, June 29, 2004 5:05 PM Subject: Re: [Serusers] new MediaProxy release 1.1.0
Adrian Georgescu wrote:
There is a load generator included with the distribution. If you need a benchmark that is 70 calls using G711 codec on a 1GHz PIII. This is pure orientative, use the load generator for testing you particular hardware. The idea (or the beauty of it) is that is possible to stack many boxes and distribute the load among them instead of trying to put many calls in one box (which would eventually become another single point of failure in your infrastructure).
Adrian
Thanks Adrian,
I will have to try MediaProxy again sometime with the load distribution feature. I tried the plain MediaProxy when it first came out last year and it worked well.
Thanks,
-- Andres Network Admin http://www.telesip.net
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
You can easily get 1,000 simultaneous calls on modern single-CPU box with rtpproxy.
-Maxim
Ezequiel Colombo wrote:
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
TEST MEDIAPROXY.PY
180 Simultaneous Calls rtpgenerator run on same server python-2.2.3-7
SERVER:
vendor_id : GenuineIntel model name : Intel(R) Pentium(R) 4 CPU 2.60GHz cpu MHz : 2593.547 cache size : 512 KB
vendor_id : GenuineIntel model name : Intel(R) Pentium(R) 4 CPU 2.60GHz cpu MHz : 2593.547 cache size : 512 KB
MemTotal: 1031500 kB SwapTotal: 2104312 kB
Linux 2.4.22-1.2115.nptlsmp
Ezequiel Colombo
----- Original Message ----- From: "Andres" andres@telesip.net To: "Adrian Georgescu" ag@ag-projects.com Cc: serusers@lists.iptel.org Sent: Tuesday, June 29, 2004 5:05 PM Subject: Re: [Serusers] new MediaProxy release 1.1.0
Adrian Georgescu wrote:
There is a load generator included with the distribution. If you need a benchmark that is 70 calls using G711 codec on a 1GHz PIII. This is pure orientative, use the load generator for testing you particular hardware. The idea (or the beauty of it) is that is possible to stack many boxes and distribute the load among them instead of trying to put many calls in one box (which would eventually become another single point of failure in your infrastructure).
Adrian
Thanks Adrian,
I will have to try MediaProxy again sometime with the load distribution feature. I tried the plain MediaProxy when it first came out last year and it worked well.
Thanks,
-- Andres Network Admin http://www.telesip.net
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
On Jun 30, 2004 at 08:53, Ezequiel Colombo ecolombo@arcotel.net wrote:
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
I tested it on a dual athlon mp2000. I've also modified rtpgenerator to work with nathelper (patch for nh version attached).
rtpgenerator has a few drawbacks: it eats more cpu than rtproxy and cannot create more than 510 calls (the python code seems to use internally select witch doesn't work on more than 1024 file descriptors).
Also mediaproxy cpu usage varies a lot (it doesn't stay constant it has a lot of peaks).
mediaproxy 1.0 eats all the cpu (100% ) at arround 90-120 simultaneous calls (it starts reaching 100% at 90 calls). rtpproxy (unstable, latest cvs) uses only 60-66% cpu for 500 simultaneous calls (rtpgenerator cannot generate more). rtpgenerator uses 70-81% during this time.
I've measured all this after 3-4 minute from starting rtpgenerator (because I was not interested on the initial session creation;, for example in the first minute rtpgenerator uses almost 100%).
Another interesting peformance benchmark is how many sessions can be handled per second, without any rtp traffic (the equivalent of a sip call: invite -> request, 200 ok -> lookup, bye -> delete). This shows how much is ser slowed down by sending commands to the rtp proxy.
Unfortunately my benchmark program works only with rtpproxy (different commands). Here are some results (this time on a pentiumM 1.6Ghz single cpu) maximum 100 simultaneous session : 5011 sessions/s 4000 simultaneous seessions : 46 sessions/S (this happens because poll scales very badly with the number of FDs used; 4000 sessions => 16000 open fds => rtpproxy spends the most time in the poll syscall).
Andrei
On Thursday 01 July 2004 14:11, you wrote:
On Jun 30, 2004 at 08:53, Ezequiel Colombo ecolombo@arcotel.net
wrote:
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
Ezequiel's test result seems to be in the expected range (given our own test results).
In our tests, on a 1GHz Athlon CPU running linux 2.4.26 we got the following results:
mediaproxy reaches 95% CPU load at around 60 simultaneous calls mediaproxy gets to 100% CPU load at 80-90 calls
rtpproxy reaches 95% CPU load at 120 calls rtpproxy gets to 100% CPU load at about 150 calls
Given that we discovered that rtpproxy is only at most twice as fast as mediaproxy, we never bothered to re-implement mediaproxy in C. It's much easier to stack another box (or even more), than to spend lots of time in optimizing a program and in the end to only get it to run twice as fast.
Given that Andrei's results are very different from what our tests showed, we suspect that there were some errors in the testing procedure. Also there is an error in his patch (see comments below for details). I attached a version of rtpgenerator that was modified to work with rtpproxy, but with this error fixed.
I tested it on a dual athlon mp2000. I've also modified rtpgenerator to work with nathelper (patch for nh version attached).
Patch has a problem. It sends packets to the same IP as the generator itself (the default is 127.0.0.1) and not to the rtpproxy returned IP. This means that in it's default configuration it sends from 127.0.0.1 to 127.0.0.1 where rtpproxy doesn't listen. We tried it and it gets flooded back with "udp port unreachable". This explains the high load on the generator you've seen and the much lower load on rtpproxy.
rtpgenerator has a few drawbacks: it eats more cpu than rtproxy and cannot create more than 510 calls (the python code seems to use internally select witch doesn't work on more than 1024 file descriptors).
You can call poll3 instead of poll, but it is not designed to generate that many calls. That's why we limited it to 30 max. If it generates more it loads the cpu too much and the results are not conclusive. Instead run multiple instances with 30 calls and they should do better. Or run it on a different host that the one that runs rtpproxy.
Another problem if you try to generate too many calls with it is that it won't be able to send the packets at exactly 20ms time intervals. In this case you will see a high load on the generator and a much smaller load on rtpproxy itself. (as we noticed the thing that generates the high load on both mediaproxy and rtpproxy is the combination of many calls with the small time interval that the packets arrive at the proxy (10-20 ms). if this time interval is increased because the generator cannot push that many data at every 20ms the load on rtpproxy will decrease very quickly while the load on the generator will sharply increase).
As a conclusion, I suggest the following to get results that are closer to reality:
1. Preferably run the generator on another machine. 2. Never use a count higher than 50 (at least for a 1GHz machine) for a running instance of the generator. The rule is to not see the generator eat more than 1% of CPU. If you need more calls run multiple instances of the generator to sum up the number of calls. (The attached generator accepts up to 50 calls - increased from 30) 3. Specify the commands you used to run both the generator and the proxy so it is easy to spot problems like data being sent to 127.0.0.1 or the fact that data is only sent in one direction but never answered (we've seen this sometimes if --ip is not specified) 4. Make sure the data is passing through the proxy. With mediaproxy you can use the sessions.py script of the media_sessions.phtml web page. With rtpproxy use iptraf or tcpdump.
In our test on a 1GHz Athlon linux 2.4.26 we got the results specified at the beginning of this email running the following commands:
./mediaproxy.py --no-fork --ip=10.0.0.1 --listen=10.0.0.1 --allow=any ./rtpgenerator.py --ip=10.0.0.1 --proxy=10.0.0.1 --g711 --count=30 ./rtpgenerator.py --ip=10.0.0.1 --proxy=10.0.0.1 --g711 --count=30
and for rtpproxy: ./rtpproxy -f -l 10.0.0.1 -s /tmp/rtpproxy.sock ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=50 ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=50 ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=20
In both cases, the CPU load of any generator was under 1%, while the proxies reached 95%
Also mediaproxy cpu usage varies a lot (it doesn't stay constant it has a lot of peaks).
mediaproxy 1.0 eats all the cpu (100% ) at arround 90-120 simultaneous calls (it starts reaching 100% at 90 calls). rtpproxy (unstable, latest cvs) uses only 60-66% cpu for 500 simultaneous calls (rtpgenerator cannot generate more). rtpgenerator uses 70-81% during this time.
I've measured all this after 3-4 minute from starting rtpgenerator (because I was not interested on the initial session creation;, for example in the first minute rtpgenerator uses almost 100%).
Another interesting peformance benchmark is how many sessions can be handled per second, without any rtp traffic (the equivalent of a sip call: invite -> request, 200 ok -> lookup, bye -> delete). This shows how much is ser slowed down by sending commands to the rtp proxy.
Unfortunately my benchmark program works only with rtpproxy (different commands). Here are some results (this time on a pentiumM 1.6Ghz single cpu) maximum 100 simultaneous session : 5011 sessions/s 4000 simultaneous seessions : 46 sessions/S (this happens because poll scales very badly with the number of FDs used; 4000 sessions => 16000 open fds => rtpproxy spends the most time in the poll syscall).
Andrei
Thanks Adrian for your notes. Maxim Sobolev are running a bad test ? jaja, so i cannot migrate to rtpproxy ! (1000 calls on a single pc like too much). I think mediaproxy in Python are a good idea. The speed difference with rtpproxy is low and the topological posibilities with mediaproxy are very interesting.
thanks, and sorry for my english Ezequiel (Argentina)
----- Original Message ----- From: "Adrian Georgescu" ag@ag-projects.com To: "Andrei Pelinescu-Onciul" pelinescu-onciul@fokus.fraunhofer.de Cc: "Ezequiel Colombo" ecolombo@arcotel.net; serusers@lists.iptel.org; andres@telesip.net Sent: Thursday, July 01, 2004 10:53 AM Subject: Re: [Serusers] new MediaProxy release 1.1.0
On Thursday 01 July 2004 14:11, you wrote:
On Jun 30, 2004 at 08:53, Ezequiel Colombo ecolombo@arcotel.net
wrote:
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
Ezequiel's test result seems to be in the expected range (given our own test results).
In our tests, on a 1GHz Athlon CPU running linux 2.4.26 we got the following results:
mediaproxy reaches 95% CPU load at around 60 simultaneous calls mediaproxy gets to 100% CPU load at 80-90 calls
rtpproxy reaches 95% CPU load at 120 calls rtpproxy gets to 100% CPU load at about 150 calls
Given that we discovered that rtpproxy is only at most twice as fast as mediaproxy, we never bothered to re-implement mediaproxy in C. It's much easier to stack another box (or even more), than to spend lots of time in optimizing a program and in the end to only get it to run twice as fast.
Given that Andrei's results are very different from what our tests showed, we suspect that there were some errors in the testing procedure. Also there is an error in his patch (see comments below for details). I attached a version of rtpgenerator that was modified to work with rtpproxy, but with this error fixed.
I tested it on a dual athlon mp2000. I've also modified rtpgenerator to work with nathelper (patch for nh version attached).
Patch has a problem. It sends packets to the same IP as the generator itself (the default is 127.0.0.1) and not to the rtpproxy returned IP. This means that in it's default configuration it sends from 127.0.0.1 to 127.0.0.1 where rtpproxy doesn't listen. We tried it and it gets flooded back with "udp port unreachable". This explains the high load on the generator you've seen and the much lower load on rtpproxy.
rtpgenerator has a few drawbacks: it eats more cpu than rtproxy and cannot create more than 510 calls (the python code seems to use internally select witch doesn't work on more than 1024 file descriptors).
You can call poll3 instead of poll, but it is not designed to generate that many calls. That's why we limited it to 30 max. If it generates more it loads the cpu too much and the results are not conclusive. Instead run multiple instances with 30 calls and they should do better. Or run it on a different host that the one that runs rtpproxy.
Another problem if you try to generate too many calls with it is that it won't be able to send the packets at exactly 20ms time intervals. In this case you will see a high load on the generator and a much smaller load on rtpproxy itself. (as we noticed the thing that generates the high load on both mediaproxy and rtpproxy is the combination of many calls with the small time interval that the packets arrive at the proxy (10-20 ms). if this time interval is increased because the generator cannot push that many data at every 20ms the load on rtpproxy will decrease very quickly while the load on the generator will sharply increase).
As a conclusion, I suggest the following to get results that are closer to reality:
- Preferably run the generator on another machine.
- Never use a count higher than 50 (at least for a 1GHz machine) for a running instance of the generator. The rule is to not see the
generator eat more than 1% of CPU. If you need more calls run multiple instances of the generator to sum up the number of calls. (The attached generator accepts up to 50 calls - increased from 30) 3. Specify the commands you used to run both the generator and the proxy so it is easy to spot problems like data being sent to 127.0.0.1 or the fact that data is only sent in one direction but never answered (we've seen this sometimes if --ip is not specified) 4. Make sure the data is passing through the proxy. With mediaproxy you can use the sessions.py script of the media_sessions.phtml web page. With rtpproxy use iptraf or tcpdump.
In our test on a 1GHz Athlon linux 2.4.26 we got the results specified at the beginning of this email running the following commands:
./mediaproxy.py --no-fork --ip=10.0.0.1 --listen=10.0.0.1 --allow=any ./rtpgenerator.py --ip=10.0.0.1 --proxy=10.0.0.1 --g711 --count=30 ./rtpgenerator.py --ip=10.0.0.1 --proxy=10.0.0.1 --g711 --count=30
and for rtpproxy: ./rtpproxy -f -l 10.0.0.1 -s /tmp/rtpproxy.sock ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=50 ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=50 ./rtp2generator.py --ip=10.0.0.1 --proxy=/tmp/rtpproxy.sock --g711 --count=20
In both cases, the CPU load of any generator was under 1%, while the proxies reached 95%
Also mediaproxy cpu usage varies a lot (it doesn't stay constant it has a lot of peaks).
mediaproxy 1.0 eats all the cpu (100% ) at arround 90-120 simultaneous calls (it starts reaching 100% at 90 calls). rtpproxy (unstable, latest cvs) uses only 60-66% cpu for 500 simultaneous calls (rtpgenerator cannot generate more). rtpgenerator uses 70-81% during this time.
I've measured all this after 3-4 minute from starting rtpgenerator (because I was not interested on the initial session creation;, for example in the first minute rtpgenerator uses almost 100%).
Another interesting peformance benchmark is how many sessions can be handled per second, without any rtp traffic (the equivalent of a sip call: invite -> request, 200 ok -> lookup, bye -> delete). This shows how much is ser slowed down by sending commands to the rtp proxy.
Unfortunately my benchmark program works only with rtpproxy (different commands). Here are some results (this time on a pentiumM 1.6Ghz single cpu) maximum 100 simultaneous session : 5011 sessions/s 4000 simultaneous seessions : 46 sessions/S (this happens because poll scales very badly with the number of FDs used; 4000 sessions => 16000 open fds => rtpproxy spends the most time in the poll syscall).
Andrei
---------------------------------------------------------------------------- ----
On Jul 01, 2004 at 15:53, Adrian Georgescu ag@ag-projects.com wrote:
On Thursday 01 July 2004 14:11, you wrote:
On Jun 30, 2004 at 08:53, Ezequiel Colombo ecolombo@arcotel.net
wrote:
Hi all, i am tested the mediaproxy 1.0.1 version on a dual CPU (2.6GHz) and get at least 180 simultaneous calls ! This is a good and very scalable solution to solve NAT related problems.
Ezequiel's test result seems to be in the expected range (given our own test results).
In our tests, on a 1GHz Athlon CPU running linux 2.4.26 we got the following results:
mediaproxy reaches 95% CPU load at around 60 simultaneous calls mediaproxy gets to 100% CPU load at 80-90 calls
rtpproxy reaches 95% CPU load at 120 calls rtpproxy gets to 100% CPU load at about 150 calls
Given that we discovered that rtpproxy is only at most twice as fast as mediaproxy, we never bothered to re-implement mediaproxy in C. It's much easier to stack another box (or even more), than to spend lots of time in optimizing a program and in the end to only get it to run twice as fast.
Given that Andrei's results are very different from what our tests showed, we suspect that there were some errors in the testing procedure.
Also there is an error in his patch (see comments below for details). I attached a version of rtpgenerator that was modified to work with rtpproxy, but with this error fixed.
I tested it on a dual athlon mp2000. I've also modified rtpgenerator to work with nathelper (patch for nh version attached).
Patch has a problem. It sends packets to the same IP as the generator itself (the default is 127.0.0.1) and not to the rtpproxy returned IP. This means that in it's default configuration it sends from 127.0.0.1 to 127.0.0.1 where rtpproxy doesn't listen. We tried it and it gets flooded back with "udp port unreachable". This explains the high load on the generator you've seen and the much lower load on rtpproxy.
No, rtpproxy was listening on 127.0.0.1 (I've started it with -l 127.0.0.1). If it would have not listened on 127.0.0.1 then there would have been no load on it (cpu 0%).
BTW: there is an error in your changes: command should be: "%(mode)s %(session)s %(ip)s %(port)d dummyfrom dummyto\n" % self.__dict_
and not "%(mode)s %(session)s %(port)d %(ip)s dummyfrom dummyto\n" ...
rtpgenerator has a few drawbacks: it eats more cpu than rtproxy and cannot create more than 510 calls (the python code seems to use internally select witch doesn't work on more than 1024 file descriptors).
You can call poll3 instead of poll, but it is not designed to generate that many calls. That's why we limited it to 30 max. If it generates more it loads the cpu too much and the results are not conclusive. Instead run multiple instances with 30 calls and they should do better. Or run it on a different host that the one that runs rtpproxy.
If you run it on a different host you end up with the same problem (it will still eat the cpu faster then rtpproxy, unless you have a much faster test host). In my tests I've used a dual machine. Both rtpproxy and rtpgenerator are single processes, so each of them got in fact a cpu for them (they were not competing for the same cpu). Moreover running rtpgenerator on a different host than rtpproxy will require more changes, since rtpproxy uses udp in this case.
Another problem if you try to generate too many calls with it is that it won't be able to send the packets at exactly 20ms time intervals. In this case you will see a high load on the generator and a much smaller load on rtpproxy itself. (as we noticed the thing that generates the high load on both mediaproxy and rtpproxy is the combination of many calls with the small time interval that the packets arrive at the proxy (10-20 ms). if this time interval is increased because the generator cannot push that many data at every 20ms the load on rtpproxy will decrease very quickly while the load on the generator will sharply increase).
As a conclusion, I suggest the following to get results that are closer to reality:
- Preferably run the generator on another machine.
As I said this won't solve much and requires deeper changes to rtpgenerator (udp support). OTOH I think a dual machine is pretty good for this kind of tests.
- Never use a count higher than 50 (at least for a 1GHz machine) for a running instance of the generator. The rule is to not see the
generator eat more than 1% of CPU. If you need more calls run multiple instances of the generator to sum up the number of calls.
In my case generator always use more than 1% of the CPU (even for count=30). A good test will run different instances on different machines. However you would need a lot of them.
(The attached generator accepts up to 50 calls - increased from 30) 3. Specify the commands you used to run both the generator and the proxy so it is easy to spot problems like data being sent to 127.0.0.1 or the fact that data is only sent in one direction but never answered (we've seen this sometimes if --ip is not specified)
Ok, I forgot to include them. I've re-run the tests with similar results: ulimit -n 10240 ./rtpproxy -f -l 127.0.0.1 -s /tmp/rtp.sock chmod 666 /tmp/rtp.sock
./rtp2generator.py --ip=127.0.0.1 --proxy=/tmp/rtp.sock --g711 --count=500
- Make sure the data is passing through the proxy. With mediaproxy you
can use the sessions.py script of the media_sessions.phtml web page. With rtpproxy use iptraf or tcpdump.
Or just watch the load and the udp socket queues.
[...]
You might have a point though about the large number of calls bad behaviour of rtpgenerator. Since it seems like it's using select it might scale very badly with the number of calls (select/poll performance drops much faster then linearly with the increasing number of files descriptors).
The same problem affects rtpproxy. I think its performance would improve dramatically if it would switch from poll to sigio (linux 2.4), epoll (2.6), kqueue (*bsd) or /dev/poll (solaris).
Andrei
-
Adrian Georgescu -
Andrei Pelinescu-Onciul -
Andres -
Ezequiel Colombo -
Maxim Sobolev