Hello! It often happens to me that the service begins to refuse service (503) and so far I have not been able to understand the exact reason. Typically, before clients start receiving 503, many before them receive a 408 code. As soon as the service starts responding with 503, it does not recover on its own, you have to restart. There is no way to understand from the logs what led to this (nothing unusual). My server is powerful enough for the number of users of my service (what I mean is that there is no problem with resources). For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the database, 10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers. Please help me figure out what causes a service failure!?
OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15;
kamailio -v version: kamailio 5.5.3 (x86_64/linux) 473cef flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64. Incoming calls are sent via push notifications( Federico Cabiddu method: https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico%...).
NetBridging(for SIP and RTPEngine). ims_charging for billing (integration with our billing system using the Diameter protocol).
In the logs exactly at the moment when the service began to refuse, the recording disappears completely for 7 minutes, absolutely no entries, after which the recording of the TCP queue full begins.
Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE 032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285 requests queued (total handled 2899418) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests queued (total handled 2964063) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests queued (total handled 2923484) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests queued (total handled 2911633) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests queued (total handled 2907132) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests queued (total handled 2903281) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests queued (total handled 2903438) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286 requests queued (total handled 2899419) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286 requests queued (total handled 2897842) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286 requests queued (total handled 2895429) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286 requests queued (total handled 2895303) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286 requests queued (total handled 2891873) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286 requests queued (total handled 2892619) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 27, socket 256: queue full, 286 requests queued (total handled 2891769) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 0, retries 98, connection 0x7f0b6d974348, tcp so Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 30, socket 262: queue full, 286 requests queued (total handled 2891614) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 31, socket 264: queue full, 286 requests queued (total handled 2890778) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 33, socket 268: queue full, 286 requests queued (total handled 2890852) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 0, socket 202: queue full, 287 requests queued (total handled 3071438) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 287 requests queued (total handled 2964064) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 2, socket 206: queue full, 287 requests queued (total handled 2944928) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 287 requests queued (total handled 2923485) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 5, socket 212: queue full, 287 requests queued (total handled 2915778) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 287 requests queued (total handled 2911634) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 287 requests queued (total handled 2907133) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 287 requests queued (total handled 2903282) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 287 requests queued (total handled 2903439) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 10, socket 222: queue full, 287 requests queued (total handled 2900227) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 287 requests queued (total handled 2899420) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 287 requests queued (total handled 2897843) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 204 , queue entry 0, retries 88, connection 0x7f0b742a1968, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 13, socket 228: queue full, 287 requests queued (total handled 2896210) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 210 , queue entry 0, retries 86, connection 0x7f0b74ffaa08, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 287 requests queued (total handled 2895430) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 214 , queue entry 0, retries 88, connection 0x7f0b75003218, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 216 , queue entry 1, retries 86, connection 0x7f0b75007620, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 218 , queue entry 2, retries 84, connection 0x7f0b74772f70, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 287 requests queued (total handled 2895304) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 220 , queue entry 0, retries 87, connection 0x7f0b74777378, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 1, retries 83, connection 0x7f0b7477fb88, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 21, socket 244: queue full, 287 requests queued (total handled 2893042)
Hello,
Kamailio 5.5.x series are no longer maintained, you should upgrade to a supported version, for example 5.7.x series.
Then, it seems there is a problem with transmission, either the targets of tcp/tls connections do not accept them, or the corresponding connections get stuck.
If you look at kamailio core cookbook, there are some parameters to tune the size of outgoing tcp buffers, but probably getting the queue filled up is the effect of another cause and trying to handle better the effect is not going to help much in long term, just prolong a bit more. You should figure out the cause, maybe you have end points behind the nat and try to open connections towards them?
You can also look at the output generated by "kamctl trap" (it requires gdb) to see what each kamailio process is doing. You can also increase the debug level when such log messages appear to get more details in the syslog (there is a rpc command to change debug level at runtime without restart).
Cheers, Daniel
On 11.10.23 11:59, Masud via sr-users wrote:
Hello! It often happens to me that the service begins to refuse service (503) and so far I have not been able to understand the exact reason. Typically, before clients start receiving 503, many before them receive a 408 code. As soon as the service starts responding with 503, it does not recover on its own, you have to restart. There is no way to understand from the logs what led to this (nothing unusual). My server is powerful enough for the number of users of my service (what I mean is that there is no problem with resources). For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the database, 10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers. Please help me figure out what causes a service failure!?
OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15;
kamailio -v version: kamailio 5.5.3 (x86_64/linux) 473cef flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64. Incoming calls are sent via push notifications( Federico Cabiddu method: https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico%...).
NetBridging(for SIP and RTPEngine). ims_charging for billing (integration with our billing system using the Diameter protocol).
In the logs exactly at the moment when the service began to refuse, the recording disappears completely for 7 minutes, absolutely no entries, after which the recording of the TCP queue full begins.
Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE 032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285 requests queued (total handled 2899418) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests queued (total handled 2964063) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests queued (total handled 2923484) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests queued (total handled 2911633) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests queued (total handled 2907132) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests queued (total handled 2903281) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests queued (total handled 2903438) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286 requests queued (total handled 2899419) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286 requests queued (total handled 2897842) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286 requests queued (total handled 2895429) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286 requests queued (total handled 2895303) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286 requests queued (total handled 2891873) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286 requests queued (total handled 2892619) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 27, socket 256: queue full, 286 requests queued (total handled 2891769) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 0, retries 98, connection 0x7f0b6d974348, tcp so Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 30, socket 262: queue full, 286 requests queued (total handled 2891614) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 31, socket 264: queue full, 286 requests queued (total handled 2890778) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 33, socket 268: queue full, 286 requests queued (total handled 2890852) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 0, socket 202: queue full, 287 requests queued (total handled 3071438) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 287 requests queued (total handled 2964064) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 2, socket 206: queue full, 287 requests queued (total handled 2944928) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 287 requests queued (total handled 2923485) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 5, socket 212: queue full, 287 requests queued (total handled 2915778) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 287 requests queued (total handled 2911634) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 287 requests queued (total handled 2907133) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 287 requests queued (total handled 2903282) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 287 requests queued (total handled 2903439) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 10, socket 222: queue full, 287 requests queued (total handled 2900227) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 287 requests queued (total handled 2899420) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 287 requests queued (total handled 2897843) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 204 , queue entry 0, retries 88, connection 0x7f0b742a1968, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 13, socket 228: queue full, 287 requests queued (total handled 2896210) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 210 , queue entry 0, retries 86, connection 0x7f0b74ffaa08, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 287 requests queued (total handled 2895430) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 214 , queue entry 0, retries 88, connection 0x7f0b75003218, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 216 , queue entry 1, retries 86, connection 0x7f0b75007620, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 218 , queue entry 2, retries 84, connection 0x7f0b74772f70, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 287 requests queued (total handled 2895304) Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 220 , queue entry 0, retries 87, connection 0x7f0b74777378, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: ERROR: [core/tcp_main.c:3458]: send_fd_queue_run(): send_fd failed on socket 224 , queue entry 1, retries 83, connection 0x7f0b7477fb88, tcp so Oct 3 18:45:17 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 21, socket 244: queue full, 287 requests queued (total handled 2893042) __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Thank you very much for your quick response!
I am considering the option of switching to 5.7, but I would like to understand this error now and fix it.
Most of the clients are behind NAT, but Kamailio and RTPEngiene have a white IP, and billing to whom Kamailio contacts to receive a quota and further billing are located in the same subnet. The database is on the same server as Kamailio.
There is one more point that I did not mention above, before the 2nd accident occurs: cdp[api_process.c:110] api_callback(): Recived diameter response outside of threshold (500) - [520-800]. CDP config: Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="0" DropUnknownOnDisconnect="0" Tc="30" Workers="10" QueueLength="2" ConnectTimeout="5" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="60" MaxAuthSessionTimeout="300"
Are these parameters enough for cps 70-100?
I tried "kamctl trap" several times but couldn't figure it out. The main question that worries me is why, when the workers are full, the service freezes and does not recover on its own? Only restarting the service helps.
Hi,
I would go with the advise from Daniel to upgrade your Kamailio and if the error stays then come back or fill a issue on Github.
And your server is really, really good sized for your use. I use VMs with much less specs and a higher CPS.
Kind regards
Masud via sr-users sr-users@lists.kamailio.org schrieb am Fr., 13. Okt. 2023, 09:10:
Hello. Daniel can you help with these questions?) __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Hi Karsten, I will definitely switch to the current version later, but I am wondering if I have configured the Kamailio and CDP module configs correctly enough for my load?
Hello,
Try to check the limits to kamailio processes in OS
cat /proc/pid/limits
On Fri, 13 Oct 2023 at 13:10 Masud via sr-users sr-users@lists.kamailio.org wrote:
Hello. Daniel can you help with these questions?) __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
Hello, cat /proc/3461653/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 514004 514004 processes Max open files 65000 65000 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 514004 514004 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
-
Daniel-Constantin Mierla -
Karsten Horsmann -
masoud96＠mail.ru -
Nick F