Re: [OpenSER-Users] Openser not processing requests at rate being sent to it

PS -- Henning, Bogdan, Daniel, etc -- email me off list and I'll gladly give you access to the x86_64 server if you want to take a look and\or test it with sipp. Michael -----Original Message----- From: users-bounces(a)lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Thursday, May 29, 2008 6:31 PM To: users(a)lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it More information on the performance problems I have been fighting: I replaced one of my OpenSER production boxes with a new server today. The old one was running x86_64 version of CentOS 5 (two dual core Opteron processors). The new one is running the i386 flavor of CentOS 5 (one quad core Xeon processor). I copied the configs from the old box to the new one. The boxes were built identically using the same process and applications. The new box is now matching the performance I was seeing on the 1.2.1 box of my provider. This new box is easily handling 50 new calls per second. So this indicates that something is wrong with the combination of Linux x86_64 and OpenSER 1.3.X. NOTE: I am NOT blaming OpenSER or saying that the problem lies with the OpenSER code. But something in the combination of those is causing a huge performance problem. For all I know the problem could be in an Ethernet driver in the x86_64 package. Those of you who have done benchmarking lately -- were your benchmarks run on x86_64? I would be curious to know if others who have reported performance problems lately were on x86_64 (I thought at least one other report indicated they were using that build). Michael Young -----Original Message----- From: users-bounces(a)lists.openser.org [mailto:users-bounces@lists.openser.org] On Behalf Of Michael Young Sent: Tuesday, May 27, 2008 8:16 AM To: 'Henning Westerholt'; users(a)lists.openser.org Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it Here is what I know (disclaimer: I am not a coder... as far as I am concerned C is for Cookie, and that's good enough for me): One of my providers is a company I used to work for. They have a primary OpenSER server (running 1.2.1) that at peak times receives 20 - 25 new INVITES per second. I have their ENUM configured to route half of those calls to my proxy1, and the other half to my proxy2. At 25 cps (new calls per second), the load on their SER box is .1 to .2 at most. The processor rarely gets to 1% utilized. OpenSER is using between 200 and 400 Mb of RAM. At 10 cps, my proxy1 shows a load of .9 to 1.2. The processor rarely gets to 1% utilized, and the server is using 700 Mb of RAM. At 20 cps, my proxy1 chokes and stops responding to invites. It is not the processor or the RAM that are limiting... there is something else involved that takes the load up to 2.0, at which point it stops responding. I have stripped down my config, trying to find out where the problem is. At this point I have wild theories, but few cold hard facts. Their OpenSER box is running ACC, and writing records to a MYSQL cluster that lives on a different box. Mine is not running ACC at all anymore. They use ENUM, I'm using dbaliases. They are using Ubuntu on an i386 processor, I am using CentOS x86_64. I actually have a Xeon-based server on order, wondering if the problem is only on x86_64 based systems. Increasing the UDP buffer size is the only thing that has gotten me to 10 cps, before I was dying at 3. My OpenSER proxies route calls to a server farm of ~70 Asterisk boxes. If one of those asterisk servers goes off line, my proxy's load will immediately go up to 2.0, the processor level will go to 100%, and it stops processing many (but not all) new calls. It seems to block some of the processes at this point, if I try to shut OpenSER down I have to manually kill 6 to 8 of the child processes. Here are the modules I'm loading: loadmodule "mysql.so" loadmodule "sl.so" loadmodule "tm.so" loadmodule "rr.so" loadmodule "maxfwd.so" loadmodule "textops.so" loadmodule "mi_fifo.so" loadmodule "uri_db.so" loadmodule "uri.so" loadmodule "xlog.so" loadmodule "permissions.so" loadmodule "alias_db.so" loadmodule "domain.so" Maybe this information can help someone else who is running into a similar problem. Michael -----Original Message----- From: Henning Westerholt [mailto:henning.westerholt@1und1.de] Sent: Tuesday, May 27, 2008 3:22 AM To: users(a)lists.openser.org Cc: Michael Young Subject: Re: [OpenSER-Users] Openser not processing requests at rate being sent to it On Saturday 24 May 2008, Michael Young wrote: Hi Michael, i can confirm a (albeit minor) slowdown from 0.9 branch to actual releases, probably caused e.g. from the added pseudo-variable stuff, more DNS lookups and further abstractions. But a slowdown like this sounds more like a bug to me. Do you can share more details about the problem? Cheers, Henning No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM _______________________________________________ Users mailing list Users(a)lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.1/1466 - Release Date: 5/25/2008 6:49 PM No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM _______________________________________________ Users mailing list Users(a)lists.openser.org http://lists.openser.org/cgi-bin/mailman/listinfo/users No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.24.3/1472 - Release Date: 5/29/2008 7:27 AM