[kamailio/kamailio] Kamailio-Master tcp request relaying problem (EAGAIN) (#1836)

List overview All Threads
Download

newer

older

git:master:94a51c88: core: main.c...

Re: [sr-dev] [SR-Users] Sending...

Yasin CANER

5 Feb 2019 5 Feb '19

5:48 a.m.

### Description Hello, Kamailio -master version couldnt relay tcp request to other sides. it gives EAGAIN error. When i test scenario on kamailio-5.2 version , it can relay.

### Troubleshooting

#### Reproduction ``` route[TCPTEST]{ remove_hf("Route"); #loose_route(); if (!is_method("INVITE")) return;

$du="sip: mykamailio1.test:5060;transport=tcp";

if(!t_newtran()){ xlog("L_WARN"," Couldnt create new transaction \r\n"); sl_send_reply("503", "Service not available"); exit; }

if(!t_relay()){ send_reply("503", "Service not available"); exit; } exit; }

```

#### Debugging Data

#### Log Messages

``` 7(15781) DEBUG: <core> [core/tcp_main.c:1229]: tcpconn_new(): on port 5060, type 2 7(15781) DEBUG: <core> [core/tcp_main.c:1539]: tcpconn_add(): hashes: 507:2382:0, 2 7(15781) DEBUG: <core> [core/tcp_main.c:2873]: tcpconn_1st_send(): pending write on new connection 0x7fd15b5e7c98 (-1/1483 bytes written) (err: 11 - Resource temporarily unavailable) 7(15781) ERROR: <core> [core/tcp_main.c:2891]: tcpconn_1st_send(): [FD61:C23C:B3B6:0:0:0:0:5]:5060: EAGAIN and write queue full or failed for 0x7fd15b5e7c98 7(15781) DEBUG: <core> [core/tcp_main.c:3153]: tcpconn_chld_put(): destroying connection 0x7fd15b5e7c98 (2, -1) flags 0060

7(15781) ERROR: tm [../../core/forward.h:251]: msg_send_buffer(): tcp_send failed ```

#### SIP Traffic

``` UAC: IPv6: TCP ---> Kamailio (5.2) :IPv6 :TCP ---> Client or Server IPv6:TCP ```

### Possible Solutions

### Additional Information

``` Kamailio master from git ```

* **Operating System**:

``` NAME="CentOS Linux" VERSION="7 (Core)" 3.10.0-957.1.3.el7.x86_64 ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836

Attachments:

attachment.html (text/html — 3.5 KB)

Show replies by date

Yasin CANER

5 Feb 5 Feb

5:53 a.m.

here my [mail](http://sip-router.1086192.n5.nabble.com/ERROR-EAGAIN-and-write-queue-full-or...) about issue.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-460522092

Henning Westerholt

9:17 p.m.

Just thinking about it - one major addition was the PROXY functionality in the TCP area. Did you enabled this functionality by any chance?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-460807106

Yasin CANER

6 Feb 6 Feb

4:40 a.m.

it is nearly default cfg .i just removed auth and add my route block. after downgrading kamailio version to 5.2 , it works fine. i didnt touch` tcp_accept_haproxy=yes` parameter

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-460899138

sergey-safarov

10 Feb 10 Feb

9:54 p.m.

I have same issue over TCP trasport. Also same config on 5.2 and on 5.1 branch is works my debug logs ``` 14(22) DEBUG: dispatcher [dispatch.c:2043]: ds_manage_routes(): set [1] 14(22) DEBUG: dispatcher [dispatch.c:2147]: ds_manage_routes(): using alg [4] hash [0] 14(22) DEBUG: dispatcher [dispatch.c:2186]: ds_manage_routes(): selected [4-1-0/0] sip:pbx.rcsnet.ru:7000;transport=tcp 14(22) DEBUG: dispatcher [dispatch.c:2207]: ds_manage_routes(): using first entry [1/0] 14(22) DEBUG: dispatcher [dispatch.c:2005]: ds_select_dst_limit(): selected target destinations: 1 14(22) exec: *** cfgtrace:request_route=[DISPATCH] c=[/etc/kamailio/kamailio.cfg] l=334 a=25 n=xdbg 14(22) DEBUG: <script>: --- SCRIPT: going to sip:rcsnet.ru via sip:pbx.rcsnet.ru:7000;transport=tcp (attrs: ) 14(22) exec: *** cfgtrace:request_route=[DISPATCH] c=[/etc/kamailio/kamailio.cfg] l=335 a=25 n=t_on_failure 14(22) exec: *** cfgtrace:request_route=[DISPATCH] c=[/etc/kamailio/kamailio.cfg] l=336 a=5 n=route 14(22) exec: *** cfgtrace:request_route=[RELAY] c=[/etc/kamailio/kamailio.cfg] l=252 a=16 n=if 14(22) exec: *** cfgtrace:request_route=[RELAY] c=[/etc/kamailio/kamailio.cfg] l=249 a=24 n=t_relay 14(22) DEBUG: tm [t_lookup.c:1320]: t_newtran(): msg id=2 , global msg id=2 , T on entrance=0 14(22) DEBUG: tm [t_lookup.c:477]: t_lookup_request(): start searching: hash=58321, isACK=0 14(22) DEBUG: tm [t_lookup.c:436]: matching_3261(): RFC3261 transaction matching failed 14(22) DEBUG: tm [t_lookup.c:660]: t_lookup_request(): no transaction found 14(22) DEBUG: tm [t_hooks.c:348]: run_reqin_callbacks_internal(): trans=0x7fc67c6698a8, callback type 1, id 0 entered 14(22) DEBUG: <core> [core/md5utils.c:67]: MD5StringArray(): MD5 calculated: 304be5a21ec5fd4786116faca8adf918 14(22) DEBUG: <core> [core/dns_cache.c:527]: _dns_hash_find(): (pbx.rcsnet.ru(13), 1), h=1005 14(22) DEBUG: <core> [core/tcp_main.c:1977]: tcp_send(): no open tcp connection found, opening new one 14(22) DEBUG: <core> [core/ip_addr.c:229]: print_ip(): tcpconn_new: new tcp connection: 217.12.247.98 14(22) DEBUG: <core> [core/tcp_main.c:1229]: tcpconn_new(): on port 7000, type 2 14(22) DEBUG: <core> [core/tcp_main.c:1535]: tcpconn_add(): hashes: 2231:0:0, 3 14(22) DEBUG: <core> [core/tcp_main.c:2871]: tcpconn_1st_send(): pending write on new connection 0x7fc67c692180 (-1/811 bytes written) (err: 11 - Resource temporarily unavailable) 14(22) ERROR: <core> [core/tcp_main.c:2890]: tcpconn_1st_send(): 217.12.247.98:7000: EAGAIN and write queue full or failed for 0x7fc67c692180 14(22) DEBUG: <core> [core/tcp_main.c:3152]: tcpconn_chld_put(): destroying connection 0x7fc67c692180 (3, -1) flags 0060 14(22) ERROR: tm [../../core/forward.h:251]: msg_send_buffer(): tcp_send failed 14(22) DEBUG: tm [t_fwd.c:1547]: t_send_branch(): send to 217.12.247.98:7000 (2) failed 14(22) WARNING: tm [t_fwd.c:1567]: t_send_branch(): sending request on branch 0 failed 14(22) DEBUG: tm [t_funcs.c:334]: t_relay_to(): t_forward_nonack returned error -1 (-477) 14(22) DEBUG: tm [t_funcs.c:352]: t_relay_to(): -477 error reply generation delayed 14(22) exec: *** cfgtrace:request_route=[RELAY] c=[/etc/kamailio/kamailio.cfg] l=250 a=24 n=sl_reply_error 14(22) DEBUG: websocket [ws_conn.c:452]: wsconn_get(): wsconn_get for id [2] 14(22) DEBUG: websocket [ws_conn.c:458]: wsconn_get(): wsconn_get returns wsc [0x7fc67c682058] refcnt [2] 14(22) DEBUG: websocket [ws_frame.c:738]: ws_frame_transmit(): Tx message: SIP/2.0 477 Unfortunately error on sending to next hop occurred (477/SL) Via: SIP/2.0/WSS df7jal23ls0d.invalid;branch=z9hG4bK1T8IU9SPqhTVtPkfa8ZcmNATMyZhZoXS;rport=51976;received=192.168.100.31 From: "Sergey Safarov"sip:2496@rcsnet.ru;tag=4zUV2R25QizRCe3FWVXG To: "Sergey Safarov"sip:2496@rcsnet.ru;tag=52bcac7a84e52c398793b50c505689c5.7db8 Call-ID: 94ce71ed-0e53-d627-7cc2-39133bd30985 CSeq: 51999 REGISTER Server: kamailio (5.3.0-dev2 (x86_64/linux)) Content-Length: 0

14(22) DEBUG: websocket [ws_frame.c:144]: encode_and_send_ws_frame(): encoding WebSocket frame 14(22) DEBUG: websocket [ws_frame.c:170]: encode_and_send_ws_frame(): supported non-control frame: 0x1 14(22) DEBUG: <core> [core/tcp_main.c:2455]: tcpconn_send_put(): send from reader (22 (14)), reusing fd 14(22) DEBUG: <core> [core/tcp_main.c:2690]: tcpconn_do_send(): sending... 14(22) DEBUG: <core> [core/tcp_main.c:2724]: tcpconn_do_send(): after real write: c= 0x7fc67c663c88 n=517 fd=7 14(22) DEBUG: <core> [core/tcp_main.c:2725]: tcpconn_do_send(): buf=

14(22) DEBUG: websocket [ws_conn.c:416]: wsconn_put_mode(): wsconn_put start for [0x7fc67c682058] refcnt [2] 14(22) DEBUG: websocket [ws_conn.c:430]: wsconn_put_mode(): wsconn_put end for [0x7fc67c682058] refcnt [1] 14(22) ERROR: sl [sl_funcs.c:372]: sl_reply_error(): stateless error reply used: Unfortunately error on sending to next hop occurred (477/SL) 14(22) exec: *** cfgtrace:request_route=[RELAY] c=[/etc/kamailio/kamailio.cfg] l=252 a=2 n=exit 14(22) DEBUG: <core> [core/receive.c:353]: receive_msg(): request-route executed in: 957 usec 14(22) DEBUG: tm [t_lookup.c:1484]: t_unref(): delayed error reply generation(-477) 14(22) DEBUG: tm [t_reply.c:1650]: cleanup_uac_timers(): RETR/FR timers reset 14(22) DEBUG: websocket [ws_conn.c:452]: wsconn_get(): wsconn_get for id [2] 14(22) DEBUG: websocket [ws_conn.c:458]: wsconn_get(): wsconn_get returns wsc [0x7fc67c682058] refcnt [2] 14(22) DEBUG: websocket [ws_frame.c:738]: ws_frame_transmit(): Tx message: SIP/2.0 477 Unfortunately error on sending to next hop occurred (477/TM) Via: SIP/2.0/WSS df7jal23ls0d.invalid;branch=z9hG4bK1T8IU9SPqhTVtPkfa8ZcmNATMyZhZoXS;rport=51976;received=192.168.100.31 From: "Sergey Safarov"sip:2496@rcsnet.ru;tag=4zUV2R25QizRCe3FWVXG To: "Sergey Safarov"sip:2496@rcsnet.ru;tag=1bb77df00a25769725961c2385ab007f-7db8 Call-ID: 94ce71ed-0e53-d627-7cc2-39133bd30985 CSeq: 51999 REGISTER Server: kamailio (5.3.0-dev2 (x86_64/linux)) Content-Length: 0

14(22) DEBUG: websocket [ws_conn.c:416]: wsconn_put_mode(): wsconn_put start for [0x7fc67c682058] refcnt [2] 14(22) DEBUG: websocket [ws_conn.c:430]: wsconn_put_mode(): wsconn_put end for [0x7fc67c682058] refcnt [1] 14(22) DEBUG: tm [t_reply.c:596]: _reply_light(): reply sent out. buf=0x7fc6806cb328: SIP/2.0 477 Unfortun..., shmem=0x7fc67c680348: SIP/2.0 477 Unfortun 14(22) DEBUG: tm [t_reply.c:607]: _reply_light(): finished 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/usr_avp.c:636]: destroy_avp_list(): destroying list 0 14(22) DEBUG: <core> [core/xavp.c:497]: xavp_destroy_list(): destroying xavp list 0 14(22) DEBUG: <core> [core/receive.c:457]: receive_msg(): cleaning up ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462178067

sergey-safarov

11 Feb 11 Feb

9:37 a.m.

Using `git bisect` found that bug is introduced in e3ecad34bb13e72b16a4213e7ce785194e88c534

Tested this commit with path from 9d38aa06d645bc44e37052d9b315474a99503359

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462265342

sergey-safarov

9:52 a.m.

After reverting this commit on current master issue is fixed.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462269516

Sebastian Denz

10:14 a.m.

Same issue here while testing with the current master...

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462275754

Daniel-Constantin Mierla

10:18 a.m.

Thanks for digging into it.

Maybe @henningw can take a look, it seems to be his commit. Normally, that should just replace some log messages.

I will also check it once I get a bit of time for it.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462276920

sergey-safarov

12:23 p.m.

I reverted small amount of code and this fixed issue. Diff follows. ```diff --- a/src/core/tcp_main.c +++ b/src/core/tcp_main.c @@ -643,7 +643,6 @@ inline static int _wbufq_add(struct tcp_connection* c, const char* data, wb_size=MAX_unsigned(cfg_get(tcp, tcp_cfg, wq_blk_size), size); wb=shm_malloc(sizeof(*wb)+wb_size-1); if (unlikely(wb==0)) - SHM_MEM_ERROR; goto error; wb->b_size=wb_size; wb->next=0; ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462309365

Daniel-Constantin Mierla

12:24 p.m.

Looks like you catch it. Please make a pull request and it will be merged. Thanks!

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462309709

lazedo

12:35 p.m.

looks like the change should be

``` - if (unlikely(wb==0)) + if (unlikely(wb==0)) { SHM_MEM_ERROR; goto error; + } ```

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462312285

ng-voice GmbH

1:37 p.m.

Closed #1836 via 130f5cd0fca0f387bb63d0bdaa773b98089a2168.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#event-2130571753

Henning Westerholt

7:55 p.m.

@lazedo Thank you for the bisect, great that you found it. I reviewed the patch one more time, there is another error like this in the xavp and blacklist file. I will fix it asap. There is a reason why I don't like the if cases without brackets. :-(

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462471019

lazedo

8:07 p.m.

@henningw not me, i only commented on the way to fix it.credits should go to @sergey-safarov

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462475029

Henning Westerholt

8:19 p.m.

@sergey-safarov - then thank you as well. :-) The error in the blacklisting support was fixed in ffc10c8a, the xavp one was already fixed from me some weeks ago.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-462479030

IvanKalashnikov

10 May 10 May

10:44 a.m.

Hello, will your merge this PR to 5.2 branch? @henningw

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-491245510

Henning Westerholt

5:39 p.m.

The error should not be present in 5.2, have a look also to the original description.

henning@linux:~/repositories/5.2/src> grep "SHM_MEM_ERROR" core/tcp_* henning@linux:~/repositories/5.2/src>

Are you referring to another pull request?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-491371595

IvanKalashnikov

6:06 p.m.

Sorry, my mistake.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/1836#issuecomment-491380890

2174

Age (days ago)

2268

Last active (days ago)

sr-dev@lists.kamailio.org

18 comments

8 participants

tags (0)

participants (8)

Daniel-Constantin Mierla
Henning Westerholt
IvanKalashnikov
lazedo
ng-voice GmbH
Sebastian Denz
sergey-safarov
Yasin CANER