27 May
2009
27 May
'09
11:57 a.m.
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: poll error: flags 18
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110)
Connection timed out
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_send: connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send:
tcp_send failed
They are several emails in this mailing list explaining we should play with
tcp parameters. I will try to change my configuration and see if we can
solve this issue.
But when this kind of error happen we observe that the kamailio processes
are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ?
Is it normal to kamailio processes to overload CPU ?
Regards,
Pascal
27 May
27 May
12:07 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
2009/5/27 Pascal Maugeri <pascal.maugeri(a)gmail.com>om>:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: poll error: flags 18
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110)
Connection timed out
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_send: connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send:
tcp_send failed
They are several emails in this mailing list explaining we should play with
tcp parameters. I will try to change my configuration and see if we can
solve this issue.
But when this kind of error happen we observe that the kamailio processes
are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ?
Is it normal to kamailio processes to overload CPU ?
Unfortunatelly a kamailio TCP listeners blocks when initiates a
outgoing TCP connection. If it fails, you get a kamailio process
"stopped" waiting for that connection to end. If you have 8 kamailio
TCP listeners and have this issue in all of them, Kamailio cannot
handle TCP connection for a while.
This is of course something that should change radically (there have
been already threads about this issue). However I've ever seen CPU at
100% due to this issues. How many concurrent TCP connection does your
kamailio handle when it occurs?
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
12:19 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
On Wed, May 27, 2009 at 10:57 AM, Pascal Maugeri
<pascal.maugeri(a)gmail.com> wrote:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: poll error: flags 18
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110)
Connection timed out
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_send: connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]: ERROR:tm:msg_send:
tcp_send failed
They are several emails in this mailing list explaining we should play with
tcp parameters. I will try to change my configuration and see if we can
solve this issue.
But when this kind of error happen we observe that the kamailio processes
are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ?
Is it normal to kamailio processes to overload CPU ?
To my best knowledge, TCP support in Kamailio is blocking. This has
been addressed for sip-router (tcp_async=yes). Anyway, there are still
some issues for tls.
How many simultaneous TCP connections are you handling? You could try
to limit the max number of connections and check the impact on the CPU
consumption.
Cheers,
--
Victor Pascual Ávila
1:29 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
To my best knowledge, TCP support in Kamailio is blocking. This has
been addressed for sip-router (tcp_async=yes). Anyway, there are still
some issues for tls.
This is a good argument to switch to sip-router :-)
How many simultaneous TCP connections are you handling? You could try
to limit the max number of connections and check the impact on the CPU
consumption.
Theoretically we're using few simultaneous client connections (<10).
My first intention is to increase tcp_children and to decrease
tcp_send_timeout and tcp_connect_timeout.
Do you believe it is a good idea ?
Cheers
Pascal
Cheers,
--
Victor Pascual Ávila
28 May
28 May
1:36 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
FYI, I've changed the configuration to:
tcp_children=8
tcp_send_timeout=3
tcp_connect_timeout=5
and it looks we get less often the problem described here ( tcp_connect/send
time out error).
I confirm that the problem was triggered by the fact that our client is
using TCP only and that the tcp connections were unavailable after some
presence test (many NOTIFY messages sent to TCP).
I'm curious to know how many children do you use in a typical configuration
? 4, 8, more? Of course it depends on the traffic expected but let me know
that this is parameter you always increase in a production environment.
Regards,
Pascal
2009/5/27 Pascal Maugeri <pascal.maugeri(a)gmail.com>
To my best knowledge, TCP support in Kamailio is
blocking. This has
been addressed for sip-router (tcp_async=yes). Anyway, there are still
some issues for tls.
This is a good argument to switch to sip-router :-)
How many simultaneous TCP connections are you handling? You could try
to limit the max number of connections and check the impact on the CPU
consumption.
Theoretically we're using few simultaneous client connections (<10).
My first intention is to increase tcp_children and to decrease
tcp_send_timeout and tcp_connect_timeout.
Do you believe it is a good idea ?
Cheers
Pascal
Cheers,
--
Victor Pascual Ávila
1:59 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
2009/5/28 Pascal Maugeri <pascal.maugeri(a)gmail.com>om>:
FYI, I've changed the configuration to:
tcp_children=8
tcp_send_timeout=3
tcp_connect_timeout=5
and it looks we get less often the problem described here ( tcp_connect/send
time out error).
This is a good workaround, but the problem is that it replies on the
clients behaviour and it shouldn't occur (a server cannot get
"blocked" due to clients problems). I hope TCP handling will be
improved in SIP-Router.
I confirm that the problem was triggered by the fact
that our client is
using TCP only and that the tcp connections were unavailable after some
presence test (many NOTIFY messages sent to TCP).
I'm curious to know how many children do you use in a typical configuration
? 4, 8, more? Of course it depends on the traffic expected but let me know
that this is parameter you always increase in a production environment.
Theorically you can add so many TCP listeners as you desire (depending
on your CPU's and memory). But if your environment makes high usage of
TCP then increase it as much as possible.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
29 May
29 May
11:41 a.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
On Thu, May 28, 2009 at 12:59 PM, Iñaki Baz Castillo <ibc(a)aliax.net> wrote:
2009/5/28 Pascal Maugeri
<pascal.maugeri(a)gmail.com>om>:
Indeed the very same problem occured after a while :-(
It looks my new configuration only "delayed" it.
The symptoms are:
- errors in log file:
May 27 14:20:11 server2-operations /usr/local/sbin/kamailio[26227]:
WARNING:core:send2child: no free tcp receiver, connection passed to the
leastbusy one (1)
May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]:
ERROR:core:tcp_blocking_connect: timeout 10 s elapsed from 10 s
May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]:
ERROR:core:tcp_send: connect failed
May 27 14:20:13 server2-operations /usr/local/sbin/kamailio[26221]:
ERROR:sl:msg_send: tcp_send failed
- in top <http://unixhelp.ed.ac.uk/CGI/man-cgi?top> kamailio process have
high values in the %CPU column such as 50% or 99% !?
- when all kamailio processes have reached this top values, it is impossible
to connect a new client untill we reset kamailio :-/
I will try the parameter
modparam("registrar", "tcp_persistent_flag", 7)
from the registrar module.
-pascal
FYI, I've changed the configuration to:
tcp_children=8
tcp_send_timeout=3
tcp_connect_timeout=5
and it looks we get less often the problem described here (
tcp_connect/send
time out error).
This is a good workaround, but the problem is that it replies on the
clients behaviour and it shouldn't occur (a server cannot get
"blocked" due to clients problems). I hope TCP handling will be
improved in SIP-Router.
I confirm that the problem was triggered by the
fact that our client is
using TCP only and that the tcp connections were unavailable after some
presence test (many NOTIFY messages sent to TCP).
I'm curious to know how many children do you use in a typical
configuration
? 4, 8, more? Of course it depends on the traffic
expected but let me
know
that this is parameter you always increase in a
production environment.
Theorically you can add so many TCP listeners as you desire (depending
on your CPU's and memory). But if your environment makes high usage of
TCP then increase it as much as possible.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
12:54 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
2009/5/29 Pascal Maugeri <pascal.maugeri(a)gmail.com>om>:
May 27 14:20:11 server2-operations
/usr/local/sbin/kamailio[26227]:
WARNING:core:send2child: no free tcp receiver, connection passed to the
leastbusy one (1)
Yes, this is the warning I told about in other mail.
- in top kamailio process have high values in the %CPU
column such as 50% or
99% !?
The same occurs (kamailio takes 50-99% CPU) in case there is a problem
with MySQL and Kamailio cannot write into it.
It seems that when kamailio process must wait for something (TCP
connection, MysQL access) it consumes CPU resources (waiting).
- when all kamailio processes have reached this top
values, it is impossible
to connect a new client untill we reset kamailio :-/
Makes sense since no TCP listener is available :(
I will try the parameter
modparam("registrar", "tcp_persistent_flag", 7)
It will not solve the issue. The only it does it setting the TCP
connection duration for the registration expires time. It has nothing
to do with failed TCP connections.
--
Iñaki Baz Castillo
<ibc(a)aliax.net>
29 Jan
29 Jan
9:40 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
I have also found this problem with 1.5.3. Modified Kamailio's source code to
fix that problem. Now I don't have this problem.
Pascal Maugeri wrote:
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: poll error: flags 18
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110)
Connection timed out
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_send: connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:tm:msg_send:
tcp_send failed
They are several emails in this mailing list explaining we should play
with
tcp parameters. I will try to change my configuration and see if we can
solve this issue.
But when this kind of error happen we observe that the kamailio processes
are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ?
Is it normal to kamailio processes to overload CPU ?
Regards,
Pascal
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
--
View this message in context:
http://old.nabble.com/tcp_blocking_connect-related-errors-and-Kamailio-usin…
Sent from the OpenSER Users Mailing List mailing list archive at Nabble.com.
10:05 p.m.
New subject: [Kamailio-Users] tcp_blocking_connect related errors and Kamailio using 100% CPU
On 1/29/10 8:40 PM, Zunnun wrote:
I have also found this problem with 1.5.3. Modified
Kamailio's source code to
fix that problem. Now I don't have this problem.
can you send a patch? I will check it and commit it to svn if all ok.
Thanks,
Daniel
Pascal Maugeri wrote:
--
Daniel-Constantin Mierla
* http://www.asipto.com/
Hi
Using Kamailio 1.5 we periodically get the following errors:
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: poll error: flags 18
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_blocking_connect: failed to retrieve SO_ERROR (110)
Connection timed out
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcpconn_connect: tcp_blocking_connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:core:tcp_send: connect failed
Jul 1 09:54:41 localhost /usr/local/sbin/openser[3836]:
ERROR:tm:msg_send:
tcp_send failed
They are several emails in this mailing list explaining we should play
with
tcp parameters. I will try to change my configuration and see if we can
solve this issue.
But when this kind of error happen we observe that the kamailio processes
are using 100% of CPU (top).
Our client is using SIP over TCP only (no UDP).
Do you have any hint on how to configure Kamailio to improve TCP support ?
Is it normal to kamailio processes to overload CPU ?
Regards,
Pascal
_______________________________________________
Kamailio (OpenSER) - Users mailing list
Users(a)lists.kamailio.org
http://lists.kamailio.org/cgi-bin/mailman/listinfo/users
http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
5441
days inactive
5688
days old
9 comments
5 participants
participants (5)
-
Daniel-Constantin Mierla -
Iñaki Baz Castillo -
Pascal Maugeri -
Victor Pascual Ávila -
Zunnun