very frequent 503 error code and complete service failure - sr-dev

5 Oct 2023

Hello!
It often happens to me that the service begins to refuse service (503) and so far I have
not been able to understand the exact reason. Typically, before clients start receiving
503, many before them receive a 408 code. As soon as the service starts responding with
503, it does not recover on its own, you have to restart. There is no way to understand
from the logs what led to this (nothing unusual). My server is powerful enough for the
number of users of my service (what I mean is that there is no problem with resources).
For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the database,
10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers.
Please help me figure out what causes a service failure!?

OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15;

kamailio -v
version: kamailio 5.5.3 (x86_64/linux) 473cef
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.

udp_children: 30; tcp_children: 34;  TLS: YES; async_workers: 64.
Incoming calls are sent via push notifications( Federico Cabiddu method:
https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…).

NetBridging(for SIP and RTPEngine).
ims_charging for billing (integration with our billing system using the Diameter
protocol).