SR master: [tcp_main.c:700]: _wbufq_add(): ERROR: wbufq_add(371 bytes): write queue full or timeout
Hi, I'm testing TCP support in Kamailio master branch.
The test is a bit special because I open a TCP connection (client and proxy in localhost) and send a TCP data containing 5000 SIP requests together (which is valid). Then wait for responses and parses them. Kamailio just replies stateless for all those requests (no routing, no forwarding, no more processing). There are two TCP workers.
Sometimes it works ok and all the replies are received, but sometimes Kamailio gives an error after receiving N requests (i.e. 1072 requests):
kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1071#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:700]: _wbufq_add(): ERROR: wbufq_add(372 bytes): write queue full or timeout (32455, total 32455, last write 0 s ago) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1072#015#012 kamailio[18350]: ERROR: <core> [tcp_main.c:3594]: handle_ser_child(): handle_ser_child: ERROR: received CON_ERROR for 0x7fc0dd172880 (id 65), refcnt 3, flags 0x601c kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1073#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1074#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1075#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1076#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1077#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed
And finally Kamailio closes the TCP connection:
kamailio[18349]: WARNING: <core> [tcp_read.c:1195]: handle_io(): WARNING: tcp_receive: handle_io: F_TCPCONN connection marked as bad: 0x7fc0dd172880 id 65 refcnt 1
I've no special parameters for TCP in the kamailio.cfg, just the default config file. Is it a bug? can it be improved by setting some parameter? However it's strange the fact that the error just occurs sometimes (but very often).
Thanks.
2010/12/10 Iñaki Baz Castillo ibc@aliax.net:
kamailio[18349]: WARNING: <core> [tcp_read.c:1195]: handle_io(): WARNING: tcp_receive: handle_io: F_TCPCONN connection marked as bad: 0x7fc0dd172880 id 65 refcnt 1
I've no special parameters for TCP in the kamailio.cfg, just the default config file. Is it a bug? can it be improved by setting some parameter? However it's strange the fact that the error just occurs sometimes (but very often).
In Kamailio 1.5.4 this issue doesn't occur. Same test has *never* failed in 1.5.4.
Hi, some comment about it please? :)
2010/12/10 Iñaki Baz Castillo ibc@aliax.net:
Hi, I'm testing TCP support in Kamailio master branch.
The test is a bit special because I open a TCP connection (client and proxy in localhost) and send a TCP data containing 5000 SIP requests together (which is valid). Then wait for responses and parses them. Kamailio just replies stateless for all those requests (no routing, no forwarding, no more processing). There are two TCP workers.
Sometimes it works ok and all the replies are received, but sometimes Kamailio gives an error after receiving N requests (i.e. 1072 requests):
kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1071#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:700]: _wbufq_add(): ERROR: wbufq_add(372 bytes): write queue full or timeout (32455, total 32455, last write 0 s ago) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1072#015#012 kamailio[18350]: ERROR: <core> [tcp_main.c:3594]: handle_ser_child(): handle_ser_child: ERROR: received CON_ERROR for 0x7fc0dd172880 (id 65), refcnt 3, flags 0x601c kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1073#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1074#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1075#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1076#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO: <script>: Branch = z9hG4bK1---num-request-1077#015#012 kamailio[18349]: ERROR: <core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR: <core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect & send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed
And finally Kamailio closes the TCP connection:
kamailio[18349]: WARNING: <core> [tcp_read.c:1195]: handle_io(): WARNING: tcp_receive: handle_io: F_TCPCONN connection marked as bad: 0x7fc0dd172880 id 65 refcnt 1
I've no special parameters for TCP in the kamailio.cfg, just the default config file. Is it a bug? can it be improved by setting some parameter? However it's strange the fact that the error just occurs sometimes (but very often).
Thanks.
-- Iñaki Baz Castillo ibc@aliax.net
Hello,
On 12/16/10 12:36 PM, Iñaki Baz Castillo wrote:
Hi, some comment about it please? :)
try to increase the value of parameter: http://www.kamailio.org/dokuwiki/doku.php/core-cookbook:3.1.x#tcp_conn_wq_ma...
Since there is a connection under heavy traffic, you have to adjust the setting for tcp a bit.
Cheers, Daniel
2010/12/10 Iñaki Baz Castilloibc@aliax.net:
Hi, I'm testing TCP support in Kamailio master branch.
The test is a bit special because I open a TCP connection (client and proxy in localhost) and send a TCP data containing 5000 SIP requests together (which is valid). Then wait for responses and parses them. Kamailio just replies stateless for all those requests (no routing, no forwarding, no more processing). There are two TCP workers.
Sometimes it works ok and all the replies are received, but sometimes Kamailio gives an error after receiving N requests (i.e. 1072 requests):
kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1071#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:700]: _wbufq_add(): ERROR: wbufq_add(372 bytes): write queue full or timeout (32455, total 32455, last write 0 s ago) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1072#015#012 kamailio[18350]: ERROR:<core> [tcp_main.c:3594]: handle_ser_child(): handle_ser_child: ERROR: received CON_ERROR for 0x7fc0dd172880 (id 65), refcnt 3, flags 0x601c kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1073#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1074#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1075#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1076#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed kamailio[18349]: INFO:<script>: Branch = z9hG4bK1---num-request-1077#015#012 kamailio[18349]: ERROR:<core> [tcp_main.c:2743]: tcpconn_1st_send(): connect 127.0.0.1:51901 failed (RST) Connection refused kamailio[18349]: ERROR:<core> [tcp_main.c:2753]: tcpconn_1st_send(): 127.0.0.1:51901: connect& send for 0x7fc0dd17c648 failed: Connection refused (111) kamailio[18349]: ERROR: sl [../../forward.h:169]: msg_send(): msg_send: ERROR: tcp_send failed
And finally Kamailio closes the TCP connection:
kamailio[18349]: WARNING:<core> [tcp_read.c:1195]: handle_io(): WARNING: tcp_receive: handle_io: F_TCPCONN connection marked as bad: 0x7fc0dd172880 id 65 refcnt 1
I've no special parameters for TCP in the kamailio.cfg, just the default config file. Is it a bug? can it be improved by setting some parameter? However it's strange the fact that the error just occurs sometimes (but very often).
Thanks.
-- Iñaki Baz Castillo ibc@aliax.net
2010/12/17 Daniel-Constantin Mierla miconda@gmail.com:
try to increase the value of parameter: http://www.kamailio.org/dokuwiki/doku.php/core-cookbook:3.1.x#tcp_conn_wq_ma...
Since there is a connection under heavy traffic, you have to adjust the setting for tcp a bit.
100% right :) Due to the nature of my test, setting that parameter makes the work :)
Thanks a lot.
-
Daniel-Constantin Mierla -
Iñaki Baz Castillo