### Description
Kamailio running with mix of TLS/WebSocket/UDP transports.
### Troubleshooting
#### Reproduction
Really random issues, can't really catch it
#### Debugging Data
``` [root@pbxphone-0866-fe-1 kamailio_crash]# gdb kamailio core.kamailio.222460.pbxphone-0866-fe-1.company.domain.1646903834 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /root/kamailio_crash/kamailio...done.
warning: exec file is newer than core file. [New LWP 222460] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f04af578716 in tls_complete_init (c=0x7f042fe11c98) at tls_server.c:236 236 data->ssl = SSL_new(dom->ctx[process_no]); Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-5.el7.x86_64 glibc-2.17-325.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-51.el7_9.x86_64 libacl-2.2.51-15.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-11.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 libunistring-0.9.3-9.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libxml2-2.9.1-6.el7_9.6.x86_64 lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x86_64 lua-5.1.4-15.el7.x86_64 net-snmp-agent-libs-5.7.2-49.el7_9.1.x86_64 net-snmp-libs-5.7.2-49.el7_9.1.x86_64 nspr-4.32.0-1.el7_9.x86_64 nss-3.67.0-4.el7_9.x86_64 nss-softokn-freebl-3.67.0-3.el7_9.x86_64 nss-util-3.67.0-1.el7_9.x86_64 openssl-libs-1.0.2k-24.el7_9.x86_64 pcre-8.32-17.el7.x86_64 perl-libs-5.16.3-299.el7_9.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-90.el7.x86_64 rpm-libs-4.11.3-48.el7_9.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-19.el7_9.x86_64 (gdb) bt full #0 0x00007f04af578716 in tls_complete_init (c=0x7f042fe11c98) at tls_server.c:236 dom = 0x7f042fbcccf8 data = 0x7f042f96dec0 cfg = 0x7f042f96dec0 state = S_TLS_CONNECTING sname = 0x0 srvid = 0x0 __FUNCTION__ = "tls_complete_init" #1 0x00007f04af578d07 in tls_fix_connection_unsafe (c=0x7f042fe11c98) at tls_server.c:297 __FUNCTION__ = "tls_fix_connection_unsafe" #2 0x00007f04af580784 in tls_h_encode_f (c=0x7f042fe11c98, pbuf=0x7ffd8be909e0, plen=0x7ffd8be909d8, rest_buf=0x7ffd8be909e8, rest_len=0x7ffd8be909dc, send_flags=0x7ffd8be909c0) at tls_server.c:772 n = 0 offs = 0 ssl = 0x1 tls_c = 0x7f042fe11ca0 wr_buf = "\026\003\001\002\000\001\000\001\374\003\003\276,'P^\252j\027\021S\351\252\021\342(~\347\216\351\332Hh!\vb\024\361\324+\003\177\a\000\000\254\300\060\300,\300(\300$\300\024\300\n\000\245\000\243\000\241\000\237\000k\000j\000i\000h\000\071\000\070\000\067\000\066\000\210\000\207\000\206\000\205\300\062\300.\300*\300&\300\017\300\005\000\235\000=\000\065\000\204\300/\300+\300'\300#\300\023\300\t\000\244\000\242\000\240\000\236\000g\000@\000?\000>\000\063\000\062\000\061\000\060\000\232\000\231\000\230\000\227\000E\000D\000C\000B\300\061\300-\300)\300%\300\016\300\004\000\234\000<\000/\000\226\000A\300\022\300\b\000\026\000\023\000\020\000\r\300\r\300\003"... rd = {buf = 0x18 <Address 0x18 out of bounds>, pos = 22930096, used = 1771061403, size = -1947662592} wr = {buf = 0x4b6e74a40 <Address 0x4b6e74a40 out of bounds>, pos = -1226359400, used = 32516, size = -1562640382} ssl_error = 0 err_src = 0x7f04af5afc09 "TLS write:" ip_buf = "\000\000\000\000\000\000\000\000\230\034\341/\004\177\000\000\000\000\000\000\000\000\000\000\370@\347\266\001\000\000\000\340\b\351\213\375\177\000\000f\211Q\000\000\000\000\000\000\000\000\000\001\000\000\000\060\000\000\000\060\000\000" buf = 0x7ffd8be90fd0 "OPTIONS sip:73054@82.66.66.105:46050;transport=tls SIP/2.0\r\nVia: SIP/2.0/TLS tone.company.domain:5061;branch=z9hG4bKx.2854.1.0\r\nFrom: sip:pbxphone@tone.company.domain;tag=uloc-6228658a-43655-2c-31dd201c-6229c21a"... len = 308 x = 4 __FUNCTION__ = "tls_h_encode_f" #3 0x000000000051e729 in tcp_send (dst=0x7ffd8be90d60, from=0x0, buf=0x7ffd8be90fd0 "OPTIONS sip:73054@82.66.66.105:46050;transport=tls SIP/2.0\r\nVia: SIP/2.0/TLS tone.company.domain:5061;branch=z9hG4bKx.2854.1.0\r\nFrom: sip:pbxphone@tone.company.domain;tag=uloc-6228658a-43655-2c-31dd201c-6229c21a"..., len=308) at core/tcp_main.c:2093 c = 0x7f042fe11c98 ip = {af = 2, len = 4, u = {addrl = {1765950034, 4294967295}, addr32 = {1765950034, 0, 4294967295, 0}, addr16 = {16978, 26946, 0, 0, 65535, 65535, 0, 0}, addr = "RBBi\000\000\000\000\377\377\377\377\000\000\000"}} port = 46050 fd = 22 response = {139655966192064, 0} n = 0 con_lifetime = 9600 try_local_port = 5061 rest_buf = 0x0 t_buf = 0x7ffd8be90fd0 "OPTIONS sip:73054@82.66.66.105:46050;transport=tls SIP/2.0\r\nVia: SIP/2.0/TLS tone.company.domain:5061;branch=z9hG4bKx.2854.1.0\r\nFrom: sip:pbxphone@tone.company.domain;tag=uloc-6228658a-43655-2c-31dd201c-6229c21a"... rest_len = 0 t_len = 308 resp = 140726950756352 t_send_flags = {f = 0, blst_imask = 0} __FUNCTION__ = "tcp_send" #4 0x00007f04b2aa0ceb in ul_ka_send (kamsg=0x7ffd8be90fc0, kadst=0x7ffd8be90d60) at ul_keepalive.c:271 __FUNCTION__ = "ul_ka_send" #5 0x00007f04b2aa0b58 in ul_ka_urecord (ur=0x7f042fca3340) at ul_keepalive.c:239 ---Type <return> to continue, or q <return> to quit--- uc = 0x7f042fca3458 kabuf = "OPTIONS sip:73054@82.66.66.105:46050;transport=tls SIP/2.0\r\nVia: SIP/2.0/TLS tone.company.domain:5061;branch=z9hG4bKx.2854.1.0\r\nFrom: sip:pbxphone@tone.company.domain;tag=uloc-6228658a-43655-2c-31dd201c-6229c21a"... kabuf_len = 308 kamsg = { s = 0x7ffd8be90fd0 "OPTIONS sip:73054@82.66.66.105:46050;transport=tls SIP/2.0\r\nVia: SIP/2.0/TLS tone.company.domain:5061;branch=z9hG4bKx.2854.1.0\r\nFrom: sip:pbxphone@tone.company.domain;tag=uloc-6228658a-43655-2c-31dd201c-6229c21a"..., len = 308} vaddr = {s = 0x7f04b6e74008 "tone.company.domain", len = 12} vport = {s = 0x7f04b6e740f8 "5061", len = 4} sdst = {s = 0x7f042fca3770 "sip:82.66.66.105:46050;transport=tls", len = 36} sproto = {s = 0x81bb4b "TLS", len = 3} duri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x7f042fca3774 "82.66.66.105:46050;transport=tls", len = 12}, port = {s = 0x7f042fca3781 "46050;transport=tls", len = 5}, params = {s = 0x7f042fca3787 "transport=tls", len = 13}, sip_params = {s = 0x7f042fca3787 "transport=tls", len = 13}, headers = {s = 0x0, len = 0}, port_no = 46050, proto = 3, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x7f042fca3787 "transport=tls", len = 13}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x7f042fca3791 "tls", len = 3}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} dproto = 3 '\003' he = 0xb88fa0 <he.5288> ssock = 0x7f04b6e73d98 idst = {send_sock = 0x7f04b6e73d98, to = {s = {sa_family = 2, sa_data = "\263\342RBBi\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 58035, sin_addr = {s_addr = 1765950034}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 58035, sin6_flowinfo = 1765950034, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = { 0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\263\342RBBi", '\000' <repeats 111 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 3 '\003', proto_pad0 = 0 '\000', proto_pad1 = 0} bcnt = 1 aortype = 0 i = 5 tv = {tv_sec = 1646903834, tv_usec = 680904} tnow = 1646903834 __FUNCTION__ = "ul_ka_urecord" #6 0x00007f04b2a6bb99 in mem_timer_udomain (_d=0x7f042f9c5388, istart=0, istep=1) at udomain.c:1234 ptr = 0x7f042fca3340 t = 0x7f0429d60470 <str_record_route_col> i = 28 #7 0x00007f04b2a9dd0e in synchronize_all_udomains (istart=0, istep=1) at dlist.c:966 res = 0 ptr = 0x7f042f9c5288 #8 0x00007f04b2a7b019 in ul_core_timer (ticks=1791845, param=0x0) at usrloc_mod.c:527 __FUNCTION__ = "ul_core_timer" #9 0x00000000004f1e69 in compat_old_handler (ti=28669522, tl=0x7f042f82e060, data=0x7f042f82e060) at core/timer.c:980 t = 0x7f042f82e060 #10 0x00000000004f29a5 in slow_timer_main () at core/timer.c:1105 n = 12 ret = 4294967295 tl = 0x7f042f82e060 i = 308 __FUNCTION__ = "slow_timer_main" #11 0x000000000042a707 in main_loop () at main.c:1765 i = 4 pid = 0 si = 0x0 si_desc = "udp receiver child=3 sock=128.141.18.117:5060 (tone.company.domain:5060)\000\000\000\000\000\000\000\000\200S/\004\177\000\000\000\000\000\000\000\000\000\000\220Q\234/\004\177\000\000\200\034\351\213\375\177\000\000\000R\234/\004\177\000\000B\227Z\257\004\177\000\000\300\023\006\267\004\177\000" nrprocs = 4 ---Type <return> to continue, or q <return> to quit--- woneinit = 1 __FUNCTION__ = "main_loop" #12 0x000000000043423e in main (argc=9, argv=0x7ffd8be921f8) at main.c:2942 cfg_stream = 0xf6a040 c = -1 r = 0 tmp = 0x7ffd8be935cc "" tmp_len = 1472 port = 960 proto = 32516 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1413124693 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffd8be920b0 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 16553497, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1646819986, tv_nsec = 228520360}, st_mtim = {tv_sec = 1646903531, tv_nsec = 836398702}, st_ctim = {tv_sec = 1646903531, tv_nsec = 836398702}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, { name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) (gdb) info locals cfg_stream = 0xf6a040 c = -1 r = 0 tmp = 0x7ffd8be935cc "" tmp_len = 1472 port = 960 proto = 32516 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1413124693 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffd8be920b0 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 16553497, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1646819986, tv_nsec = 228520360}, st_mtim = {tv_sec = 1646903531, tv_nsec = 836398702}, st_ctim = {tv_sec = 1646903531, tv_nsec = 836398702}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, { name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, { name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, { name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) cfg_stream = 0xf6a040 c = -1 r = 0 tmp = 0x7ffd8be935cc "" tmp_len = 1472 port = 960 proto = 32516 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1413124693 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffd8be920b0 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 16553497, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1646819986, tv_nsec = 228520360}, st_mtim = {tv_sec = 1646903531, tv_nsec = 836398702}, st_ctim = {tv_sec = 1646903531, tv_nsec = 836398702}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, { name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, { name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, { name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) cfg_stream = 0xf6a040 c = -1 r = 0 tmp = 0x7ffd8be935cc "" tmp_len = 1472 port = 960 proto = 32516 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1413124693 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffd8be920b0 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 16553497, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1646819986, tv_nsec = 228520360}, st_mtim = {tv_sec = 1646903531, tv_nsec = 836398702}, st_ctim = {tv_sec = 1646903531, tv_nsec = 836398702}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, { name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, { name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, { name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) cfg_stream = 0xf6a040 c = -1 r = 0 tmp = 0x7ffd8be935cc "" tmp_len = 1472 port = 960 proto = 32516 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 1413124693 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffd8be920b0 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 16553497, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1646819986, tv_nsec = 228520360}, st_mtim = {tv_sec = 1646903531, tv_nsec = 836398702}, st_ctim = {tv_sec = 1646903531, tv_nsec = 836398702}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, { name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, { name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, { name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, { name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" (gdb) list ... here goes a code of tls_server.c but if I'll leave it - Comment is too long (maximum is 65536 characters) (gdb) Line number 1513 out of range; tls_server.c has 1512 lines. ```
#### Log Messages
``` 2022-03-10T10:17:16.838905+01:00 pbxphone-0866-fe-1.company.domain /usr/local/sbin/kamailio[222489]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 27 ```
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
``` kamailio -V version: kamailio 5.4.7 (x86_64/linux) a19fe1 flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: a19fe1 compiled on 20:40:47 Nov 22 2021 with gcc 4.8.5 ```
* **Operating System**:
<!-- Details about the operating system, the type: Linux (e.g.,: Debian 8.4, Ubuntu 16.04, CentOS 7.1, ...), MacOS, xBSD, Solaris, ...; Kernel details (output of `lsb_release -a` and `uname -a`) -->
``` Linux pbxphone-0866-fe-1.company.domain 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ```
[full_crash_report.txt](https://github.com/kamailio/kamailio/files/8224804/full_crash_report.txt)
Just got the same again, attaching logs if they would be useful [kamailio_crash_report.tar.gz](https://github.com/kamailio/kamailio/files/8700004/kamailio_crash_report.tar...)
could you show gdb output for the following commands as well? ``` p process_no p dom->ctx p dom->ctx[process_no] ```
``` (gdb) p process_no $1 = 19 (gdb) p dom->ctx No symbol "dom" in current context. (gdb) ```
was it really in frame 0 within `tls_complete_init` ? your `bt all` shows `dom` local variable in the frame.
another strangeness is that pointers to the `cfg` and `data` are equal. `data` is freshly allocated and could get this address if `cfg` is freed in another place so at the first glance looks like issues with reference in `cfg->ref_count`.
do you use TLS configuration reloads in runtime?
Yes, it shows, but I'm not an expert in `gdb` ( No, I don't reload TLS in runtime.
If any, I can provide full crash dumps with kamailio binary, but they are ~ 2,5Gb each
Thanks, but the dumps are usually not useful if you do not have the exact same environment available.
Got a several more crashes: This time 2 core files were generated: ``` gdb /usr/local/sbin/kamailio core.kamailio.59874.mycompanyphone-0058-fe-1.mycompany.domain.1655841238 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 59874] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f909edee478 in atomic_cmpxchg_int (var=0x0, old=0, new_v=1) at ../../core/atomic/atomic_x86.h:224 224 ATOMIC_FUNC_CMPXCHG(cmpxchg, "cmpxchgl %2, %1", int , int) Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-5.el7.x86_64 glibc-2.17-326.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-51.el7_9.x86_64 libacl-2.2.51-15.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-11.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 libunistring-0.9.3-9.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libxml2-2.9.1-6.el7_9.6.x86_64 lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x86_64 lua-5.1.4-15.el7.x86_64 net-snmp-agent-libs-5.7.2-49.el7_9.2.x86_64 net-snmp-libs-5.7.2-49.el7_9.2.x86_64 nspr-4.32.0-1.el7_9.x86_64 nss-3.67.0-4.el7_9.x86_64 nss-softokn-freebl-3.67.0-3.el7_9.x86_64 nss-util-3.67.0-1.el7_9.x86_64 openssl-libs-1.0.2k-25.el7_9.x86_64 pcre-8.32-17.el7.x86_64 perl-libs-5.16.3-299.el7_9.x86_64 popt-1.13-16.el7.x86_64 python-libs-2.7.5-90.el7.x86_64 rpm-libs-4.11.3-48.el7_9.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-20.el7_9.x86_64 (gdb) bt full #0 0x00007f909edee478 in atomic_cmpxchg_int (var=0x0, old=0, new_v=1) at ../../core/atomic/atomic_x86.h:224 ret = 0 #1 0x00007f909edee4c4 in futex_get (lock=0x0) at ../../core/futexlock.h:99 v = 32656 i = 1024 #2 0x00007f909edf5461 in wsconn_get (id=251) at ws_conn.c:452 id_hash = 251 wsc = 0x100000001 __FUNCTION__ = "wsconn_get" #3 0x00007f909ee09d71 in ws_frame_transmit (evp=0x7fff600c67e0) at ws_frame.c:728 wsev = 0x7fff600c6820 frame = {fin = 1, rsv1 = 0, rsv2 = 0, rsv3 = 0, opcode = 1, mask = 0, payload_len = 300, masking_key = "\000\000\000", payload_data = 0x7fff600c6e50 "OPTIONS sip:73935@ndf4oomn6lvr.invalid;transport=wss SIP/2.0\r\nVia: SIP/2.0/WS pbx.mycompany.domain:8089;branch=z9hG4bKx.1.1.0\r\nFrom: sip:mycompanyphone@pbx.mycompany.domain;tag=uloc-6297d7d7-ea08-81-31dd2800-62b221d6-ac"..., wsc = 0x0} __FUNCTION__ = "ws_frame_transmit" #4 0x00000000004f3762 in sr_event_exec (type=11, evp=0x7fff600c67e0) at core/events.c:296 ret = 0 i = 12927 #5 0x0000000000544edf in wss_send (dst=0x7fff600c6be0, buf=0x7fff600c6e50 "OPTIONS sip:73935@ndf4oomn6lvr.invalid;transport=wss SIP/2.0\r\nVia: SIP/2.0/WS pbx.mycompany.domain:8089;branch=z9hG4bKx.1.1.0\r\nFrom: sip:mycompanyphone@pbx.mycompany.domain;tag=uloc-6297d7d7-ea08-81-31dd2800-62b221d6-ac"..., len=300) at core/tcp_main.c:5209 port = 12927 ip = {af = 2, len = 4, u = {addrl = {2213784257, 140734804814288}, addr32 = {2213784257, 0, 1611426256, 32767}, addr16 = {43713, 33779, 0, 0, 27088, 24588, 32767, 0}, addr = "\301\252\363\203\000\000\000\000\320i\f`\377\177\000"}} from = 0x0 local_addr = {s = {sa_family = 19200, sa_data = "\264\000\000\000\000\000\240l\f`\377\177\000"}, sin = {sin_family = 19200, sin_port = 180, sin_addr = {s_addr = 0}, sin_zero = "\240l\f`\377\177\000"}, sin6 = {sin6_family = 19200, sin6_port = 180, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\240l\f`\377\177\000\000\000\000g\244\220\177\000", __u6_addr16 = {27808, 24588, 32767, 0, 0, 42087, 32656, 0}, __u6_addr32 = {1611426976, 32767, 2758213632, 32656}}}, sin6_scope_id = 0}, sas = {ss_family = 19200, __ss_padding = "\264\000\000\000\000\000\240l\f`\377\177\000\000\000\000g\244\220\177\000\000\000\000\000\000\000\000\000\000\240h\f`\377\177\000\000\310\320i\000\000\000\000\000\020@f\244\220\177\000\000\000\000\000\000\000\000\000\000XR}\000\004\000\000\000\360\362\200\000\000\000\000\000\377\377\377\377", '\000' <repeats 12 times>, "\320h\f`\377\177\000\000\200\320\303K\221\177\000\000\000\000\000\000\000\000\000", __ss_align = 140734804815008}} con = 0x7f90a5167000 wsev = {type = 11, buf = 0x7fff600c6e50 "OPTIONS sip:73935@ndf4oomn6lvr.invalid;transport=wss SIP/2.0\r\nVia: SIP/2.0/WS pbx.mycompany.domain:8089;branch=z9hG4bKx.1.1.0\r\nFrom: sip:mycompanyphone@pbx.mycompany.domain;tag=uloc-6297d7d7-ea08-81-31dd2800-62b221d6-ac"..., len = 300, id = 251} evp = {data = 0x7fff600c6820, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} ret = 0 __FUNCTION__ = "wss_send" #6 0x00007f9127539c47 in ul_ka_send (kamsg=0x7fff600c6e40, kadst=0x7fff600c6be0) at ul_keepalive.c:259 __FUNCTION__ = "ul_ka_send" #7 0x00007f9127539b58 in ul_ka_urecord (ur=0x7f90a59b26f0) at ul_keepalive.c:239 uc = 0x7f90a5778e10 kabuf = "OPTIONS sip:73935@ndf4oomn6lvr.invalid;transport=wss SIP/2.0\r\nVia: SIP/2.0/WS pbx.mycompany.domain:8089;branch=z9hG4bKx.1.1.0\r\nFrom: sip:mycompanyphone@pbx.mycompany.domain;tag=uloc-6297d7d7-ea08-81-31dd2800-62b221d6-ac"... kabuf_len = 300 kamsg = {s = 0x7fff600c6e50 "OPTIONS sip:73935@ndf4oomn6lvr.invalid;transport=wss SIP/2.0\r\nVia: SIP/2.0/WS pbx.mycompany.domain:8089;branch=z9hG4bKx.1.1.0\r\nFrom: sip:mycompanyphone@pbx.mycompany.domain;tag=uloc-6297d7d7-ea08-81-31dd2800-62b221d6-ac"..., len = 300} vaddr = {s = 0x7f912b90da40 "pbx.mycompany.domain", len = 12} vport = {s = 0x7f912b90db30 "8089", len = 4} sdst = {s = 0x7f90a5779108 "sip:193.170.243.131:12927;transport=ws", len = 38} sproto = {s = 0x81bb5d "WS", len = 2} duri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x7f90a577910c "193.170.243.131:12927;transport=ws", len = 15}, port = {s = 0x7f90a577911c "12927;transport=ws", len = 5}, params = {s = 0x7f90a5779122 "transport=ws", len = 12}, sip_params = { s = 0x7f90a5779122 "transport=ws", len = 12}, headers = {s = 0x0, len = 0}, port_no = 12927, proto = 5, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x7f90a5779122 "transport=ws", len = 12}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x7f90a577912c "ws", len = 2}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, maddr_val = { s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}} dproto = 5 '\005' he = 0xb88fa0 <he.5288> ssock = 0x7f912b90d7d0 idst = {send_sock = 0x7f912b90d7d0, to = {s = {sa_family = 2, sa_data = "2\177\301\252\363\203\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 32562, sin_addr = {s_addr = 2213784257}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 32562, sin6_flowinfo = 2213784257, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "2\177\301\252\363\203", '\000' <repeats 111 times>, __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 5 '\005', proto_pad0 = 0 '\000', proto_pad1 = 0} bcnt = 1 aortype = 0 i = 5 tv = {tv_sec = 1655841238, tv_usec = 705013} tnow = 1655841238 __FUNCTION__ = "ul_ka_urecord" #8 0x00007f9127504b99 in mem_timer_udomain (_d=0x7f90a445c6b0, istart=0, istep=1) at udomain.c:1234 ptr = 0x7f90a59b26f0 t = 0x1ea00000000 i = 0 #9 0x00007f9127536d0e in synchronize_all_udomains (istart=0, istep=1) at dlist.c:966 res = 0 ptr = 0x7f90a445c5b0 #10 0x00007f9127513c5a in destroy () at usrloc_mod.c:498 __FUNCTION__ = "destroy" #11 0x00000000005934ee in destroy_modules () at core/sr_module.c:772 t = 0x7f912b91c828 foo = 0x7f912b91c338 __FUNCTION__ = "destroy_modules" #12 0x000000000041cda7 in cleanup (show_status=1) at main.c:569 memlog = 0 __FUNCTION__ = "cleanup" #13 0x000000000041e682 in shutdown_children (sig=15, show_status=1) at main.c:712 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "shutdown_children" #14 0x000000000042172a in handle_sigs () at main.c:812 chld = 0 chld_status = 139 any_chld_stopped = 1 memlog = -1538881128 __FUNCTION__ = "handle_sigs" #15 0x000000000042babb in main_loop () at main.c:1845 i = 4 pid = 59914 si = 0x0 si_desc = "udp receiver child=3 sock=128.141.18.117:5060 (pbx.mycompany.domain:5060)\000\f`\377\177\000\000XR}\000\000\000\000\000\030\000\000\000\000\000\000\000\060\033\227K\221\177\000\000\003\246\201\000\000\000\000\000\277\033\227K\221\177\000\000B'\004$\221\177\000\000Н\257+\221\177\000" nrprocs = 4 woneinit = 1 __FUNCTION__ = "main_loop" #16 0x000000000043423e in main (argc=9, argv=0x7fff600c83b8) at main.c:2942 cfg_stream = 0x1866040 c = -1 r = 0 tmp = 0x7fff600c96ab "" tmp_len = 1472 port = 960 proto = 32657 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2652679689 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7fff600c8270 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 46446, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1654114502, tv_nsec = 69899782}, st_mtim = {tv_sec = 1654118357, tv_nsec = 685932486}, st_ctim = {tv_sec = 1654118357, tv_nsec = 685932486}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" ``` and second one ```
gdb /usr/local/sbin/kamailio core.kamailio.59908.mycompanyphone-0058-fe-1.mycompany.domain.1655841236 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-120.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/... Reading symbols from /usr/local/sbin/kamailio...done. [New LWP 59908] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/local/sbin/kamailio -P /run/kamailio/kamailio.pid -f /usr/local/etc/kamail'. Program terminated with signal 11, Segmentation fault. #0 0x00007f912aa55c09 in freelist_extract.isra.0 () from /lib64/libssl.so.10 Missing separate debuginfos, use: debuginfo-install audit-libs-2.8.5-4.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.176-5.el7.x86_64 glibc-2.17-326.el7_9.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-51.el7_9.x86_64 libacl-2.2.51-15.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libcap-2.22-11.el7.x86_64 libcap-ng-0.7.5-4.el7.x86_64 libcom_err-1.42.9-19.el7.x86_64 libdb-5.3.21-25.el7.x86_64 libffi-3.0.13-19.el7.x86_64 libgcc-4.8.5-44.el7.x86_64 libselinux-2.5-15.el7.x86_64 libstdc++-4.8.5-44.el7.x86_64 libunistring-0.9.3-9.el7.x86_64 libuuid-2.23.2-65.el7_9.1.x86_64 libxml2-2.9.1-6.el7_9.6.x86_64 lm_sensors-libs-3.4.0-8.20160601gitf9185e5.el7.x86_64 lua-5.1.4-15.el7.x86_64 net-snmp-agent-libs-5.7.2-49.el7_9.2.x86_64 net-snmp-libs-5.7.2-49.el7_9.2.x86_64 nspr-4.32.0-1.el7_9.x86_64 nss-3.67.0-4.el7_9.x86_64 nss-softokn-freebl-3.67.0-3.el7_9.x86_64 nss-util-3.67.0-1.el7_9.x86_64 openssl-libs-1.0.2k-25.el7_9.x86_64 pcre-8.32-17.el7.x86_64 perl-libs-5.16.3-299.el7_9.x86_64 popt-1.13-16.el7.x86_64 python-cffi-1.6.0-5.el7.x86_64 python-libs-2.7.5-90.el7.x86_64 python2-cryptography-1.7.2-2.el7.x86_64 rpm-libs-4.11.3-48.el7_9.x86_64 tcp_wrappers-libs-7.6-77.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-20.el7_9.x86_64 (gdb) bt full #0 0x00007f912aa55c09 in freelist_extract.isra.0 () from /lib64/libssl.so.10 No symbol table info available. #1 0x00007f912aa56735 in ssl3_setup_read_buffer () from /lib64/libssl.so.10 No symbol table info available. #2 0x00007f912aa568a9 in ssl3_setup_buffers () from /lib64/libssl.so.10 No symbol table info available. #3 0x00007f912aa48dba in ssl3_accept () from /lib64/libssl.so.10 No symbol table info available. #4 0x00007f91240136c8 in tls_accept (c=0x7f90afaed4a8, error=0x7fff600c7330) at tls_server.c:422 ret = 32656 ssl = 0x7f90a6cf34a8 cert = 0x0 tls_c = 0x7f90a787c2c8 tls_log = 0 __FUNCTION__ = "tls_accept" pkey = 0x0 #5 0x00007f912401e049 in tls_h_read_f (c=0x7f90afaed4a8, flags=0x7fff600c789c) at tls_server.c:1126 r = 0x7f90afaed5d0 bytes_free = 16383 bytes_read = 307 read_size = 16383 ssl_error = 0 ssl_read = 0 ssl = 0x7f90a6cf34a8 rd_buf = "\026\003\001\001.\001\000\001*\003\003b\262!\324n\250\260,O\237\331B\233\024\347\354\242 !\363l\035w\227\310b.\005s\343\320\304\000\000\240̨̩\300,\300\060\000\237\300\255\300\237\300$\300(\000k\300\n\300\024\000\071\300\257\300\243\300\207\300\213\300}\300s\300w\000\304\000\210\300+\300/\000\236\300\254\300\236\300#\300'\000g\300\t\300\023\000\063\300\256\300\242\300\206\300\212\300|\300r\300v\000\276\000E\000\235\300\235\000=\000\065\300\062\300*\300\017\300.\300&\300\005\300\241\300{\000\300\000\204\300\215\300y\300\211\300u\000\234\300\234\000<\000/\300\061\300)\300\016\300-\300%\300\004\300\240\300z\000\272\000A\300\214\300x\300\210\300"... wr_buf = "\026\003\003\000\312\004\000\000\306\000\000\034 \000\300\262\344M\206\372eBƃ\002\204-\230\205\031\246\065\303\332H&\244s\330\002\027\315\326\061\030\002\353\060~\221QL\254\366\204;\f\257_\241\214i'\366:\223\273\316G\257\252E\001\304^\237\244\266\200\275\342\273\355r-{\272b\225\206\034\324y\265\203\202\211\335ܡ\245R9c\370ؼ\237\273\003\001\273\213\000\205\304\343\064\224{\343tI4]\212\235\273`Z,\224\030ӯ\030t\030\207\377eh\352|ѯ\311E&K\201\271\v\001\350\214$\025eV\263E\313ɻs\242\340\277\b0\031r?\315X\b\264\341N\320E#\215S\316\315N\341\265\001\067m>\354\027,2\335y\000\326G\347\063\254"... rd = {buf = 0x7fff600b7330 "\026\003\001\001.\001", pos = 0, used = 307, size = 65536} wr = {buf = 0x7fff600a7330 "\026\003\003", pos = 0, used = 0, size = 65536} tls_c = 0x7f90a787c2c8 enc_rd_buf = 0x0 n = 0 flush_flags = 0 err_src = 0x7f9124049650 "TLS read:" ip_buf = "128.141.18.117", '\000' <repeats 49 times> x = 0 tls_dbg = 1611429200 __FUNCTION__ = "tls_h_read_f" #6 0x0000000000679bae in tcp_read_headers (c=0x7f90afaed4a8, read_flags=0x7fff600c789c) at core/tcp_read.c:477 bytes = 32656 remaining = -1398425128 p = 0x1faca5bd7c <Address 0x1faca5bd7c out of bounds> r = 0x7f90afaed5d0 mc = 2771329336 body_len = 0 mfline = 0x0 mtransid = {s = 0x7f90a52f1d98 "o\003", len = -1523638864} __FUNCTION__ = "tcp_read_headers" #7 0x00000000006816e0 in tcp_read_req (con=0x7f90afaed4a8, bytes_read=0x7fff600c78a0, read_flags=0x7fff600c789c) at core/tcp_read.c:1492 bytes = -1 total_bytes = 0 resp = 1 size = 24 req = 0x7f90afaed5d0 dst = {send_sock = 0x7fff600c7890, to = {s = {sa_family = 8, sa_data = "\000\000(\000\000\000ع\245\254\220\177\000"}, sin = {sin_family = 8, sin_port = 0, sin_addr = {s_addr = 40}, sin_zero = "ع\245\254\220\177\000"}, sin6 = {sin6_family = 8, sin6_port = 0, sin6_flowinfo = 40, sin6_addr = {__in6_u = {__u6_addr8 = "ع\245\254\220\177\000\000݅\177\000\002\000\000", __u6_addr16 = {47576, 44197, 32656, 0, 34269, 127, 2, 0}, __u6_addr32 = {2896542168, 32656, 8357341, 2}}}, sin6_scope_id = 1611399169}, sas = { ss_family = 8, __ss_padding = "\000\000(\000\000\000ع\245\254\220\177\000\000݅\177\000\002\000\000\000\001\000\f`!\000\000\000\340K\264\000\000\000\000\000 ͱ\000\377\377\377\377h;\254\255\220\177\000\000\002\000\000\000\000\000\000\000\377\377\377\377\001\000\000\000\024\000\000\000\000\000\000\000\001\000\000\000\001\000\000\000%\000\000\000\000\000\000\000\340\357\265+\221\177\000\000\220x\f`\377\177\000\000\b\000\000\000\000\000\000", __ss_align = 0}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = -96 '\240', proto_pad0 = 118 'v', proto_pad1 = 24588} c = 50 '2' ret = -1 __FUNCTION__ = "tcp_read_req" #8 0x0000000000686622 in handle_io (fm=0x7f912bb5cb20, events=1, idx=-1) at core/tcp_read.c:1802 ret = 8 n = 8 read_flags = 1 con = 0x7f90afaed4a8 s = 37 resp = 1 t = 733026646 ee = 0x0 ---Type <return> to continue, or q <return> to quit--- __FUNCTION__ = "handle_io" #9 0x00000000006743a3 in io_wait_loop_epoll (h=0xb44be0 <io_w>, t=2, repeat=0) at core/io_wait.h:1070 n = 1 r = 0 fm = 0x7f912bb5cb20 revents = 1 __FUNCTION__ = "io_wait_loop_epoll" #10 0x00000000006893d2 in tcp_receive_loop (unix_sock=40) at core/tcp_read.c:1998 __FUNCTION__ = "tcp_receive_loop" #11 0x0000000000544147 in tcp_init_children () at core/tcp_main.c:5115 r = 0 i = 3 reader_fd_1 = 40 pid = 0 si_desc = "tcp receiver (generic)\000\000\000\000\000\000\000\000\000\000p|\f`\377\177\000\000\251Ub\000\000\000\000\000\240J\337\001\000\000\000\000\003\000\000\000\000\000\000\000XŐ+\221\177\000\000(7\263+\000\000\000\000 |\f`\377\177\000\000v\253w\000\000\000\000\000\034\000\000\000\035\000\000\000\230\263\000\244\220\177\000\000\060\033\227K\000\000\000\000\230\211F\244\001\000\000" si = 0x0 __FUNCTION__ = "tcp_init_children" #12 0x000000000042b03a in main_loop () at main.c:1799 i = 4 pid = 59888 si = 0x0 si_desc = "udp receiver child=3 sock=128.141.18.117:5060 (pbx.mycompany.domain:5060)\000\f`\377\177\000\000XR}\000\000\000\000\000\030\000\000\000\000\000\000\000\060\033\227K\221\177\000\000\003\246\201\000\000\000\000\000\277\033\227K\221\177\000\000B'\004$\221\177\000\000Н\257+\221\177\000" nrprocs = 4 woneinit = 1 __FUNCTION__ = "main_loop" #13 0x000000000043423e in main (argc=9, argv=0x7fff600c83b8) at main.c:2942 cfg_stream = 0x1866040 c = -1 r = 0 tmp = 0x7fff600c96ab "" tmp_len = 1472 port = 960 proto = 32657 ahost = 0x0 aport = 0 options = 0x7d7ef8 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2652679689 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7fff600c8270 p = 0xf0b5ff <Address 0xf0b5ff out of bounds> st = {st_dev = 20, st_ino = 46446, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1654114502, tv_nsec = 69899782}, st_mtim = {tv_sec = 1654118357, tv_nsec = 685932486}, st_ctim = {tv_sec = 1654118357, tv_nsec = 685932486}, __unused = {0, 0, 0}} tbuf = '\000' <repeats 392 times>... option_index = 0 long_options = {{name = 0x7da22f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7d56d4 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7da234 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7da23a "subst", has_arg = 1, flag = 0x0, val = 1025}, { name = 0x7da240 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7da249 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7da253 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7da25d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7da268 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7da271 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7da27c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7da282 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" ``` Not sure if it will help, but just in a case.
Just to add, it's more often case now (due to increase number of TLS clients) ``` 2022-10-12T12:46:52.905533+02:00 mycompanyphone-0058-fe-1.mycompany.domain /usr/local/sbin/kamailio[53316]: CRITICAL: <core> [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer (0x7f51947f3470), called from tls: tls_init.c: ser_free(316), first free tls: tls_init.c: ser_free(316) - ignoring 2022-10-12T12:46:52.905969+02:00 mycompanyphone-0058-fe-1.mycompany.domain /usr/local/sbin/kamailio[53316]: CRITICAL: <core> [core/mem/q_malloc.c:521]: qm_free(): BUG: freeing already freed pointer (0x7f51947fba20), called from tls: tls_init.c: ser_free(316), first free tls: tls_init.c: ser_malloc(282) - ignoring ``` Centos 7, ``` openssl version OpenSSL 1.0.2k-fips 26 Jan 2017 ```
I think this issue could be closed as I can confirm it's related to OpenSSL library as I can't reproduce it on Kamailio 5.6.2 + tlsa (see https://marc.info/?l=sr-users&m=166609531117092&w=2)
Closed #3052 as completed.
Thanks for posting the conclusions, should be useful for others facing similar cases.
-
Daniel-Constantin Mierla -
Henning Westerholt
-
Igor Olhovskiy
-
Ihor Olkhovskyi
-
Michael Furmur