[sr-dev] [kamailio/kamailio] Recurring crash: freeing already freed pointer called from tls: tls_init.c: ser_free (Issue #4055)

### Description We have 9 SIP proxy servers. 5 of them are running kamailio 5.5.4, and 4 of them are running 5.8.3. The 4 kamailio instances running 5.8.3 are experiencing a periodic crash, seemingly related to TLS. ### Troubleshooting #### Reproduction Unknown at this time, it appears to be random so far. #### Log Messages ``` ERROR: http_client [functions.c:352]: curL_request_url(): failed to perform curl (52) (url: http://127.0.0.1:8082/flushchain) tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED} ALERT: <core> [main.c:809]: handle_sigs(): core was not generated *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {IP_REDACTED}:* NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback handler for socket [0.0.0.0:0], server_name='<default>' ... tcp: 127.0.0.1 [127.0.0.1]:8080 ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:29749 Listening on udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED} *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* NOTICE: tls [tls_domain.c:1176]: ksr_tls_fix_domain(): registered server_name callback handler for socket [0.0.0.0:0], server_name='<default>' ... CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579785 (state: 0) - closing ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:030000D6:digital envelope routines::generate error (sni: unknown) ERROR: tls [tls_server.c:1316]: tls_h_read_f(): src addr: 159.65.167.207:19625 CRITICAL: <core> [core/mem/q_malloc.c:136]: qm_debug_check_frag(): BUG: qm: fragm. 0x7fd8721da060 (address 0x7fd8721da0a0) end overwritten (3438203331736c74, 0)! Memory allocator was called from tls: tls_init.c:397. Fragment marked by (null):0. Exec from core/mem/q_malloc.c:546. Listening on *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* WARNING: {f96acebe-204b-405f-88c2-e58732f5317d 1 2 BYE} dialog [dlg_handlers.c:1343]: dlg_onroute(): unable to find dialog for BYE with route param '195.3a31' [1425:5027] and call-id 'f96acebe-204b-405f-88c2-e58732f5317d' CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e531d080), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e512edd0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring ERROR: tls [tls_server.c:1312]: tls_h_read_f(): protocol level error ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr: {IP_REDACTED}:{TLS_PORT_REDACTED} ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0800006B:elliptic curve routines::point is not on curve (sni: {FQDN_REDACTED}) udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED} CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e6489600), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring Aliases: *: r{FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A00012D:SSL routines::ssl session id callback failed (sni: unknown) tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TLS_PORT_REDACTED} *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* Aliases: *: {FQDN_REDACTED}:* WARNING: {0_3602603144(a)192.168.0.173 1 3 BYE} dialog [dlg_handlers.c:1343]: dlg_onroute(): unable to find dialog for BYE with route param 'e17.a901' [1822:4250] and call-id &#39;0_3602603144(a)192.168.0.173&#39; CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579807 (state: 0) - closing ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:0A080010:SSL routines::EC lib (sni: mobile-srv.primevox.net) CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e52d9200), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e4f878a0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring CRITICAL: <core> [core/pass_fd.c:281]: receive_fd(): EOF on 69 CRITICAL: <core> [core/mem/q_malloc.c:555]: qm_free(): BUG: freeing already freed pointer (0x7f19e46597b0), called from tls: tls_init.c: ser_free(397), first free tls: tls_init.c: ser_free(397) - ignoring tcp: {IP_REDACTED} [{IP_REDACTED}]: {TCP_PORT_REDACTED} *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* tcp: 127.0.0.1 [127.0.0.1]:8080 *: {FQDN_REDACTED}:* tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED} udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED} udp: {IP_REDACTED} [ {IP_REDACTED}]:{UDP_PORT_REDACTED} tcp: {IP_REDACTED} [ {IP_REDACTED}]:{TCP_PORT_REDACTED} *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* ALERT: <core> [main.c:805]: handle_sigs(): child process 2969636 exited by a signal 6 *: {FQDN_REDACTED}:* CRITICAL: <core> [core/pass_fd.c:85]: recv_all(): 1st recv on 68 failed: Connection reset by peer *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* WARNING: http_client [functions.c:318]: curL_request_url(): failed to connect() to host (url: http://127.0.0.1:8082/flushchain) WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 19922944 and 9961472 bytes ALERT: <core> [main.c:805]: handle_sigs(): child process 262943 exited by a signal 11 ALERT: <core> [main.c:809]: handle_sigs(): core was not generated CRITICAL: <core> [core/tcp_main.c:5544]: tcp_timer_check_connections(): message processing timeout on connection id: 579806 (state: 0) - closing tls: {IP_REDACTED} [{IP_REDACTED}]:{TLS_PORT_REDACTED} *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* *: {IP_REDACTED}:* WARNING: tls [tls_init.c:980]: tls_h_mod_init_f(): openssl bug #1491 (crash/mem leaks on low memory) workaround enabled (on low memory tls operations will fail preemptively) with free memory thresholds 19922944 and 9961472 bytes *: {FQDN_REDACTED}:* *: {FQDN_REDACTED}:* ERROR: tls [tls_util.h:49]: tls_err_ret(): TLS accept:error:1C8000C5:Provider routines::reseed error (sni: unknown) ERROR: tls [tls_server.c:1319]: tls_h_read_f(): dst addr: 104.225.96.168:8647 ``` ### Additional Information * **Kamailio Version** - output of `kamailio -v` ``` version: kamailio 5.8.3 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 17:00:40 Nov 11 2024 with gcc 12.2.0 ``` * **Operating System**: <!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) --> ``` Debian GNU/Linux 12 (bookworm) Linux {FQDN} 6.1.0-27-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.115-1 (2024-11-01) x86_64 GNU/Linux ``` -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/issues/4055 You are receiving this because you are subscribed to this thread. Message ID: &lt;kamailio/kamailio/issues/4055(a)github.com&gt;