Description

We have 9 SIP proxy servers. 5 of them are running kamailio 5.5.4, and 4 of them are running 5.8.3.

The 4 kamailio instances running 5.8.3 are experiencing a periodic crash, seemingly related to TLS.

Troubleshooting

Reproduction

Unknown at this time, it appears to be random so far.

Log Messages


ERROR: http_client [functions.c:352]: curL_request_url(): failed to perform curl (52) (url: http://127.0.0.1:8082/flushchain)
tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED}
ALERT: <core> [main.c:809]: handle_sigs(): core was not generated
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*:  {IP_REDACTED}:*
NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback handler for socket [0.0.0.0:0], server_name='<default>' ...
tcp: 127.0.0.1 [127.0.0.1]:8080
ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:29749
Listening on
udp:  {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback handler for socket [0.0.0.0:0], server_name='<default>' ...
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579785 (state: 0) - closing
ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:030000D6:digital envelope routines::generate error (sni: unknown)
ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:19625
CRITICAL: <core> [core/mem/q_malloc.c:136]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fd8721da060 (address 0x7fd8721da0a0) end overwritten (3438203331736c74, 0)! Memory allocator was called from tls: tls_init.c:397. Fragment marked by (null):0. Exec from core/mem/q_malloc.c:546.
Listening on
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
WARNING: {f96acebe-204b-405f-88c2-e58732f5317d 1 2 BYE} dialog [dlg_handlers.c:1343]: dlg_onroute(): unable to find dialog for BYE with route param '195.3a31' [1425:5027] and call-id 'f96acebe-204b-405f-88c2-e58732f5317d'
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e531d080), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e512edd0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error
ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr:  {IP_REDACTED}:{TLS_PORT_REDACTED}
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0800006B:elliptic curve routines::point is not on curve (sni: {FQDN_REDACTED})
udp:  {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e6489600), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
Aliases:
*: r{FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A00012D:SSL routines::ssl session id callback failed (sni: unknown)
tcp:  {IP_REDACTED} [ {IP_REDACTED}]:{TLS_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
Aliases:
*: {FQDN_REDACTED}:*
WARNING: {0_3602603144@192.168.0.173 1 3 BYE} dialog [dlg_handlers.c:1343]: dlg_onroute(): unable to find dialog for BYE with route param 'e17.a901' [1822:4250] and call-id '0_3602603144@192.168.0.173'
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579807 (state: 0) - closing
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A080010:SSL routines::EC lib (sni: mobile-srv.primevox.net)
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e52d9200), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e4f878a0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 69
CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e46597b0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring
tcp:  {IP_REDACTED} [{IP_REDACTED}]: {TCP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
tcp: 127.0.0.1 [127.0.0.1]:8080
*: {FQDN_REDACTED}:*
tcp:  {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED}
udp:  {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
udp:  {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED}
tcp:  {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ALERT: <core> [main.c:805]: handle_sigs(): child process 2969636 exited by a signal 6
*: {FQDN_REDACTED}:*
CRITICAL: <core> [core/pass_fd.c:85]: recv_all(): 1st recv on 68 failed: Connection reset by peer
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
WARNING: http_client [functions.c:318]: curL_request_url(): failed to connect() to host (url: http://127.0.0.1:8082/flushchain)
WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 19922944 and 9961472 bytes
ALERT: <core> [main.c:805]: handle_sigs(): child process 262943 exited by a signal 11
ALERT: <core> [main.c:809]: handle_sigs(): core was not generated
CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579806 (state: 0) - closing
tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED}
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
*: {IP_REDACTED}:*
WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 19922944 and 9961472 bytes
*: {FQDN_REDACTED}:*
*: {FQDN_REDACTED}:*
ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:1C8000C5:Provider routines::reseed error (sni: unknown)
ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr: 104.225.96.168:8647

Additional Information

version: kamailio 5.8.3 (x86_64/linux) 
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled on 17:00:40 Nov 11 2024 with gcc 12.2.0

Debian GNU/Linux 12 (bookworm)
Linux {FQDN} 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linux

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <kamailio/kamailio/issues/4055@github.com>