TCP/TLS CRITICAL error. send_fd(): sendmsg failed sending : Too many references: cannot splice - sr-users

15 Aug 2020


      While executing this piece of config script, after a few days of run
time, Kamailio starts experiences some issue that leads to instability
and unresponsiveness of the proxy via TCP with TLS. UDP workers seems to
be working fine... TCP clients can't connect.
route[RELAY] {
        ...
        if (!t_relay()) {
                sl_reply_error();
        }
        exit;
}
Please advise how we can troubleshoot this issue.
Are there any immediate (or obvious) TCP/TLS/FD Kamailio/Linux
parameters that should be increased to allocate more resources for the
Proxy?
Or may be any other parameters that may lead to a quicker resources release.
Please see the errors bellow...
Environment: Ubuntu 16 x86_64, Kamailio 5.3.5.
Thanks,
Dmytro
Aug 15 04:48:19  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 164 on 16: Too many references: cannot splice (109)
Aug 15 04:48:19  ERROR: <core> [core/tcp_main.c:3908]:
handle_ser_child(): CONN_GET_FD: send_fd failed
Aug 15 04:48:19  ERROR: tm [../../core/forward.h:292]:
msg_send_buffer(): tcp_send failed
Aug 15 04:48:19  WARNING: tm [t_fwd.c:1573]: t_send_branch(): sending
request on branch 0 failed
Aug 15 04:48:19  ERROR: sl [sl_funcs.c:392]: sl_reply_error(): stateless
error reply used: Unfortunately error on sending to next hop occurred
(477/SL)
....
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e91c31e0 (flags 0x4018), fd 74
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 75 on 43: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e9d727b0 (flags 0x4018), fd 75
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 75 on 73: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e9d727b0 (flags 0x4018), fd 75
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 75 on 45: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e9d727b0 (flags 0x4018), fd 75
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 74 on 43: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e91c31e0 (flags 0x4018), fd 74
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 76 on 43: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e7856fa8 (flags 0x4018), fd 76
Aug 15 05:52:04  CRITICAL: <core> [core/pass_fd.c:191]: send_fd():
sendmsg failed sending 74 on 47: Too many references: cannot splice (109)
Aug 15 05:52:04  ERROR: <core> [core/tcp_main.c:4225]: send2child():
send_fd failed for 0x7f92e7856fa8 (flags 0x4018), fd 74
.....