we finally found what was causing the prob, at least the prob is gone after doing this:
In the syslog config file of the server where kamailio was working fine we found this entry:
local0.none /var/log/messages
we added it to the server with probs and problem solved.
Since we were using the facility 0 to log the messages from kamailio apparently we need to add that entry listed above as well to make the logging work better. this is how we setup kamailio's logging, using the example from the web:
# log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log
I do not know the reason why but it fixed our issue. I wanted to share with you. cheers and thank you a lot for your assistance
jp
________________________________ From: Juan Perez jperezsip2008@yahoo.com To: Henning Westerholt henning.westerholt@1und1.de; users@lists.kamailio.org Cc: myoung@redmonsters.net Sent: Thursday, September 17, 2009 8:49:00 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
than you a lot Henning for your suggestions, I will try to do all of that but the lab stuff because we already have a lab setup but it works superb too, only on the new machine is when we have found the prob. in all our other scenarios kamailio runs super fast, processing a lot of calls per seconds and the cpu is 99-100 % idle. fortunately we can do this in the same machine because it has an amount of traffic controllable by us so at any given time we can take it out or put more or use sipp, I will post the results as soon as I have something new, thank you guys a lot again
cheers jp
________________________________ From: Henning Westerholt henning.westerholt@1und1.de To: users@lists.kamailio.org Cc: Juan Perez jperezsip2008@yahoo.com; myoung@redmonsters.net Sent: Thursday, September 17, 2009 6:05:07 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
On Mittwoch, 16. September 2009, Juan Perez wrote:
In another scenario in our secondary site I am using another Dell Poweredge 1950, with dual quad core almost at the same speed and 64 bits , same version of CentOS and it is working like a charm. A lot of calls and CPU idle almost 100% all the time. That is we decided to buy a more powerful server to have a more robust platform in our main site but we haven't been able to move forward with this because of the bad performance. Any other ideas guys ?
Hello Juan,
as this looks more like a hardware issue to me i'd try to find issue that causes this particular server to be that slow. Somewhere it must spend this time. Some ideas:
- Start kamailio with only one worker children, and then use strace to see what its doing when its hangs - enable debugging (only for a few seconds if you've production load on the system) and then look to the timestamps where its spend the time - use the benchmark module as Daniel already suggested - try to replicate the setup in a lab environment and then use a load tester (e.g. sipp) to give some traffic to the system. then change certain parts of the configuration and setup to find the culprit
Cheers,
Henning
Hello,
On 29.09.2009 23:17 Uhr, Juan Perez wrote:
we finally found what was causing the prob, at least the prob is gone after doing this:
In the syslog config file of the server where kamailio was working fine we found this entry:
local0.none /var/log/messages
we added it to the server with probs and problem solved.
Since we were using the facility 0 to log the messages from kamailio apparently we need to add that entry listed above as well to make the logging work better. this is how we setup kamailio's logging, using the example from the web:
# log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log
I do not know the reason why but it fixed our issue. I wanted to share with you. cheers and thank you a lot for your assistance
before this local0.none line in syslog.conf, did you get the messages written by kamailio to /var/log/messages as well as in /var/log/proxy.log? If yes, that might caused delays since it was synchronous writing to /var/log/messages -- although pretty strange considering all local0 were diverted to proxy.log, but of course it depends to syslog default conf at the end.
Cheers, Daniel
I forgot to look for the precense of kamailio logs in the messages file before they expire. I do not kow for sure whether they were there or not. Also, that line was there in the default syslog.conf file of the proxy that was working fine, we did not put it manually. It was hard for us to find that fix. And both servers have CentOS 5.2, 64 bits, but it could be an update after the installation I guess. cheers
jp
----- Original Message ---- From: Daniel-Constantin Mierla miconda@gmail.com To: Juan Perez jperezsip2008@yahoo.com Cc: Henning Westerholt henning.westerholt@1und1.de; users@lists.kamailio.org Sent: Wednesday, September 30, 2009 3:20:16 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Hello,
On 29.09.2009 23:17 Uhr, Juan Perez wrote:
we finally found what was causing the prob, at least the prob is gone after doing this: In the syslog config file of the server where kamailio was working fine we found this entry: local0.none /var/log/messages we added it to the server with probs and problem solved. Since we were using the facility 0 to log the messages from kamailio apparently we need to add that entry listed above as well to make the logging work better. this is how we setup kamailio's logging, using the example from the web: # log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log I do not know the reason why but it fixed our issue. I wanted to share with you. cheers and thank you a lot for your assistance
before this local0.none line in syslog.conf, did you get the messages written by kamailio to /var/log/messages as well as in /var/log/proxy.log? If yes, that might caused delays since it was synchronous writing to /var/log/messages -- although pretty strange considering all local0 were diverted to proxy.log, but of course it depends to syslog default conf at the end.
Cheers, Daniel
-- Daniel-Constantin Mierla * Kamailio SIP Masterclass, Nov 9-13, 2009, Berlin * http://www.asipto.com/index.php/sip-router-masterclass/
On 30.09.2009 14:48 Uhr, Juan Perez wrote:
I forgot to look for the precense of kamailio logs in the messages file before they expire. I do not kow for sure whether they were there or not.
ok, no problem.
Also, that line was there in the default syslog.conf file of the proxy that was working fine, we did not put it manually. It was hard for us to find that fix. And both servers have CentOS 5.2, 64 bits, but it could be an update after the installation I guess.
interesting to see this happening, but this is life :-)
Cheers, Daniel
cheers
jp
----- Original Message ---- From: Daniel-Constantin Mierla miconda@gmail.com To: Juan Perez jperezsip2008@yahoo.com Cc: Henning Westerholt henning.westerholt@1und1.de; users@lists.kamailio.org Sent: Wednesday, September 30, 2009 3:20:16 AM Subject: Re: [Kamailio-Users] kamailio 1.4.3 slow
Hello,
On 29.09.2009 23:17 Uhr, Juan Perez wrote:
we finally found what was causing the prob, at least the prob is gone after doing this: In the syslog config file of the server where kamailio was working fine we found this entry: local0.none /var/log/messages we added it to the server with probs and problem solved. Since we were using the facility 0 to log the messages from kamailio apparently we need to add that entry listed above as well to make the logging work better. this is how we setup kamailio's logging, using the example from the web: # log messages with LOG_LOCAL0 in /var/log/ser.log local0.* -/var/log/proxy.log I do not know the reason why but it fixed our issue. I wanted to share with you. cheers and thank you a lot for your assistance
before this local0.none line in syslog.conf, did you get the messages written by kamailio to /var/log/messages as well as in /var/log/proxy.log? If yes, that might caused delays since it was synchronous writing to /var/log/messages -- although pretty strange considering all local0 were diverted to proxy.log, but of course it depends to syslog default conf at the end.
Cheers, Daniel
-- Daniel-Constantin Mierla
- Kamailio SIP Masterclass, Nov 9-13, 2009, Berlin
- http://www.asipto.com/index.php/sip-router-masterclass/
-
Daniel-Constantin Mierla -
Juan Perez