5 Jul
2021
5 Jul
'21
8:25 a.m.
we are getting the following errors in Kamailio logs
**kamailio.log**
Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core>
[core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f524f7862d0,
fd 30406
Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core>
[core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f525a6c5f90,
fd 30407
Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core>
[core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f52449e0978,
fd 30415
Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core>
[core/io_wait.h:596]: io_watch_del(): invalid fd 30355, not in [0, 4607)
Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core>
[core/io_wait.h:596]: io_watch_del(): invalid fd 30356, not in [0, 4607)
Jun 29 08:06:27 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: CRITICAL: <core>
[core/io_wait.h:596]: io_watch_del(): invalid fd 30358, not in [0, 4607)
**Description**
On checking this issue we found that Kamailio was not accepting TLS connections. We were
getting an error which is mentioned above.
We double-checked TLS and TCP connections and they were normal. We have 5000+ TLS
connections and the max connection limit is 30,000. So we were not exceeding the TLS
connection limit.
We also verified open_files_limits. It is around 1 million which is more than enough.
**Kamailio Version** - output of kamailio -v
version: kamailio 5.4.2 (x86_64/linux) f8885c
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: f8885c
**Operating System:**
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796
5 Jul
5 Jul
9:43 a.m.
What is the output of:
```
kamctl rpc core.tcp_info
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-873847993
9:53 a.m.
actually, we don't have jsonrpcs module loaded so command throwing following error
/usr/local/fep-kamailio/sbin/kamctl rpc core.tcp_info
ERROR: Error opening Kamailio's FIFO /run/kamailio/kamailio_rpc.fifo
ERROR: Make sure you have loaded the jsonrpcs module and set FIFO transport parameters
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-873852888
10:11 a.m.
Is your config making other tcp connections from modules? What is the list of loaded
modules?
```
grep loadmodule kamailio.cfg
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-873863814
10:46 a.m.
list of modules
loadmodule "cfgutils.so"
loadmodule "db_mysql.so"
loadmodule "xhttp.so"
loadmodule "jsonrpcs.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "maxfwd.so"
loadmodule "textops.so"
loadmodule "siputils.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "ctl.so"
loadmodule "counters.so"
loadmodule "sqlops.so"
loadmodule "http_client.so"
loadmodule "permissions.so"
#loadmodule "alias_db.so"
loadmodule "nathelper.so"
loadmodule "rtpproxy.so"
loadmodule "htable.so"
loadmodule "pike.so"
loadmodule "sdpops.so"
#loadmodule "debugger.so"
loadmodule "ipops.so"
loadmodule "tcpops.so"
loadmodule "avpops.so"
loadmodule "tls.so"
loadmodule "websocket.so"
loadmodule "textopsx.so"
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-873884191
6 Jul
6 Jul
11:10 a.m.
Only http_client does own connections, is it configure to reuse connections or creates one
each time?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-874554276
11:21 a.m.
no, we don't do that if you are talking about the following param
modparam("http_client", "keep_connections", 1)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-874562129
12:29 p.m.
It is strange that the file descriptor value in the error messages is so high (e.g.,
30406), given the open connections is not close to the limit:
```
Jun 29 08:08:14 fep-1 /usr/local/fep-kamailio/sbin/kamailio[14111]: ERROR: <core>
[core/tcp_main.c:4451]: handle_tcpconn_ev(): io_watch_del(3) failed: for 0x7f524f7862d0,
fd 30406
```
The POSIX requires to use always the lowest available file descriptor value - `File
Descriptor Allocation`:
* https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#t…
Maybe the implementation in your OS has bugs or doesn't follow the POSIX. You can try
to investigate the list of file descriptors listed in the proc for Kamailio processes,
should be in paths like:
```
/proc/_PID_/fd/
```
And you have to replace `_PID_` with the values corresponding to Kamailio processes,
interesting would be for `tcp main process` (usually the last process in `kamctl ps`) and
for the `tcp receiver ...`.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-874608571
26 Jul
26 Jul
7:07 p.m.
Were you able to gather more details for file descriptors based on my remarks above?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-886834042
14 Sep
14 Sep
5:14 p.m.
Closing it -- more than two months without providing requested data. Issue can be reopened
if troubleshooting data is posted here as requested above.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#issuecomment-919193110
5:14 p.m.
Closed #2796.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/2796#event-5299955354
1165
days inactive
1236
days old
10 comments
2 participants
participants (2)
-
aatif218 -
Daniel-Constantin Mierla