### Description After the upgrade of our system to Debian Buster, kamailio started crashing due to TLS module. The issue look similar to the one described in https://github.com/kamailio/kamailio/issues/1860
In Debian Stretch everything was working fine because we compiled kamailio using openssl-1.0 as suggested in the linked issue. Unfortunately Debian Buster doesn't support that old version of the package so we compiled it with openssl-1.1 and we put in place the workaround suggested here https://github.com/kamailio/kamailio/commit/efdc141ecb5ff72e3224e47deaaa79fe... but this didn't solved the issue.
#### Debugging Data
At the moment I don have full access to the system so I can provide only the following backtrace:
(gdb) bt full #0 aesni_ecb_encrypt () at crypto/aes/aesni-x86_64.s:624 No locals. 0000001 0x00007fe7b2159917 in aesni_ecb_cipher (len=16, in=0x7fe7ae3052d0 "\271a\321\064vGKiB\337\344\070\353\220\005\245\020O", <incomplete sequence \323>, out=0x7fe7ae58d068 "", ctx=0x7fe7ae3053b8) at ../crypto/evp/e_aes.c:319 bl = <optimized out> bl = <optimized out> #2 aesni_ecb_cipher (ctx=0x7fe7ae3053b8, out=0x7fe7ae58d068 "", in=0x7fe7ae3052d0 "\271a\321\064vGKiB\337\344\070\353\220\005\245\020O", <incomplete sequence \323>, len=16) at ../crypto/evp/e_aes.c:311 bl = <optimized out> 0000003 0x00007fe7b2165533 in evp_EncryptDecryptUpdate (ctx=0x7fe7ae3053b8, out=0x7fe7ae58d068 "", outl=0x7fff207dc534, in=0x7fe7ae3052d0 "\271a\321\064vGKiB\337\344\070\353\220\005\245\020O", <incomplete sequence \323>, inl=16) at ../crypto/evp/evp_enc.c:333 i = <optimized out> j = <optimized out> bl = 16 cmpl = <optimized out> #4 0x00007fe7b219830f in drbg_ctr_generate (drbg=0x7fe7ae3051e8, out=0x7fe7ae58d068 "", outlen=32, adin=0x0, adinlen=0) at ../crypto/rand/drbg_ctr.c:340 outl = 16 ctr = 0x7fe7ae305290 #5 0x00007fe7b21991fb in RAND_DRBG_generate (drbg=drbg@entry=0x7fe7ae3051e8, out=out@entry=0x7fe7ae58d068 "", outlen=outlen@entry=32, prediction_resistance=prediction_resistance@entry=0, adin=0x0, adinlen=adinlen@entry=0) at ../crypto/rand/drbg_lib.c:638 reseed_required = <optimized out> #6 0x00007fe7b2199481 in RAND_DRBG_bytes (drbg=0x7fe7ae3051e8, out=0x7fe7ae58d068 "", outlen=32) at ../crypto/rand/drbg_lib.c:679 additional = 0x0 additional_len = 0 chunk = 32 ret = <optimized out> 0000007 0x00007fe7b22f96fd in ssl_fill_hello_random (s=s@entry=0x7fe7ae588de0, server=server@entry=0, result=0x7fe7ae58d068 "", len=len@entry=32, dgrd=dgrd@entry=DOWNGRADE_NONE) at ../ssl/s3_lib.c:4589 send_time = <optimized out> ret = <optimized out> #8 0x00007fe7b231b06e in tls_construct_client_hello (s=0x7fe7ae588de0, pkt=0x7fff207dc700) at ../ssl/statem/statem_clnt.c:1153 p = <optimized out> sess_id_len = <optimized out> i = <optimized out> protverr = 0 comp = <optimized out> sess = 0x0 session_id = <optimized out> 0000009 0x00007fe7b231a33f in write_state_machine (s=0x7fe7ae588de0) at ../ssl/statem/statem.c:843 post_work = 0x7fe7b231f5a0 <ossl_statem_client_post_work> mt = 1 pkt = {buf = 0x7fe7ae59fc90, staticbuf = 0x0, curr = 4, written = 4, maxsize = 18446744073709551615, subs = 0x7fe7ae58be30} ret = <optimized out> pre_work = 0x7fe7b231d180 <ossl_statem_client_pre_work> --Type <RET> for more, q to quit, c to continue without paging-- get_construct_message_f = 0x7fe7b231d250 <ossl_statem_client_construct_message> confunc = 0x7fe7b231ad20 <tls_construct_client_hello> st = 0x7fe7ae588e28 transition = 0x7fe7b231cde0 <ossl_statem_client_write_transition> cb = 0x7fe7b2375fb0 st = <optimized out> ret = <optimized out> transition = <optimized out> pre_work = <optimized out> post_work = <optimized out> get_construct_message_f = <optimized out> cb = <optimized out> confunc = <optimized out> mt = <optimized out> pkt = <optimized out> #10 state_machine (s=0x7fe7ae588de0, server=0) at ../ssl/statem/statem.c:443 buf = 0x0 cb = 0x7fe7b2375fb0 st = <optimized out> ret = <optimized out> ssret = <optimized out> 0000011 0x00007fe7b2306264 in SSL_do_handshake (s=0x7fe7ae588de0) at ../ssl/ssl_lib.c:3599 ret = 1 #12 0x00007fe7b23a40b4 in tls_connect () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so No symbol table info available. 0000013 0x00007fe7b23a568d in tls_encode_f () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so No symbol table info available. #14 0x000055f60cceaf7e in tcp_send () No symbol table info available. 0000015 0x00007fe7b4e65920 in send_pr_buffer () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #16 0x00007fe7b4e826e8 in t_send_branch () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. 0000017 0x00007fe7b4e85adf in t_forward_nonack () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #18 0x00007fe7b4e69452 in t_relay_to () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #19 0x00007fe7b4e340ea in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/tm.so No symbol table info available. #20 0x000055f60cc50f29 in do_action () No symbol table info available. 0000021 0x000055f60cc4fa1a in run_actions () No symbol table info available. #22 0x000055f60cc517e2 in do_action () --Type <RET> for more, q to quit, c to continue without paging-- No symbol table info available. 0000023 0x000055f60cc4fa1a in run_actions () No symbol table info available. #24 0x000055f60cc5154f in do_action () No symbol table info available. 0000025 0x000055f60cc4fa1a in run_actions () No symbol table info available. #26 0x000055f60cc5d46f in run_top_route () No symbol table info available. 0000027 0x000055f60cd594cf in receive_msg () No symbol table info available. #28 0x000055f60cc7ab45 in udp_rcv_loop () No symbol table info available. 0000029 0x000055f60cc0febb in main_loop () No symbol table info available. #30 0x000055f60cc07415 in main () No symbol table info available. (gdb)
### Additional Information Kamailio version 5.2.3 Debian Buster 10.1
Please find below the full backtrace including kamailio debug symbols:
(gdb) bt full #0 aesni_ecb_encrypt () at crypto/aes/aesni-x86_64.s:624 No locals. #1 0x00007f16bf3b4917 in aesni_ecb_cipher (len=16, in=0x7f16bb55fe30 "\272", out=0x7f16bba89288 "", ctx=0x7f16bb55ff18) at ../crypto/evp/e_aes.c:319 bl = <optimized out> bl = <optimized out> #2 aesni_ecb_cipher (ctx=0x7f16bb55ff18, out=0x7f16bba89288 "", in=0x7f16bb55fe30 "\272", len=16) at ../crypto/evp/e_aes.c:311 bl = <optimized out> #3 0x00007f16bf3c0533 in evp_EncryptDecryptUpdate (ctx=0x7f16bb55ff18, out=0x7f16bba89288 "", outl=0x7ffd991be074, in=0x7f16bb55fe30 "\272", inl=16) at ../crypto/evp/evp_enc.c:333 i = <optimized out> j = <optimized out> bl = 16 cmpl = <optimized out> #4 0x00007f16bf3f330f in drbg_ctr_generate (drbg=0x7f16bb55fd48, out=0x7f16bba89288 "", outlen=32, adin=0x0, adinlen=0) at ../crypto/rand/drbg_ctr.c:340 outl = 16 ctr = 0x7f16bb55fdf0 #5 0x00007f16bf3f41fb in RAND_DRBG_generate (drbg=drbg@entry=0x7f16bb55fd48, out=out@entry=0x7f16bba89288 "", outlen=outlen@entry=32, prediction_resistance=prediction_resistance@entry=0, adin=0x0, adinlen=adinlen@entry=0) at ../crypto/rand/drbg_lib.c:638 reseed_required = <optimized out> #6 0x00007f16bf3f4481 in RAND_DRBG_bytes (drbg=0x7f16bb55fd48, out=0x7f16bba89288 "", outlen=32) at ../crypto/rand/drbg_lib.c:679 additional = 0x0 additional_len = 0 chunk = 32 ret = <optimized out> #7 0x00007f16bf5546fd in ssl_fill_hello_random (s=s@entry=0x7f16bba70bf0, server=server@entry=0, result=0x7f16bba89288 "", len=len@entry=32, dgrd=dgrd@entry=DOWNGRADE_NONE) at ../ssl/s3_lib.c:4589 send_time = <optimized out> ret = <optimized out> #8 0x00007f16bf57606e in tls_construct_client_hello (s=0x7f16bba70bf0, pkt=0x7ffd991be240) at ../ssl/statem/statem_clnt.c:1153 p = <optimized out> sess_id_len = <optimized out> i = <optimized out> protverr = 0 comp = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- sess = 0x0 session_id = <optimized out> #9 0x00007f16bf57533f in write_state_machine (s=0x7f16bba70bf0) at ../ssl/statem/statem.c:843 post_work = 0x7f16bf57a5a0 <ossl_statem_client_post_work> mt = 1 pkt = {buf = 0x7f16bba9a2b8, staticbuf = 0x0, curr = 4, written = 4, maxsize = 18446744073709551615, subs = 0x7f16bbaaa418} ret = <optimized out> pre_work = 0x7f16bf578180 <ossl_statem_client_pre_work> get_construct_message_f = 0x7f16bf578250 <ossl_statem_client_construct_message> confunc = 0x7f16bf575d20 <tls_construct_client_hello> st = 0x7f16bba70c38 transition = 0x7f16bf577de0 <ossl_statem_client_write_transition> cb = 0x7f16bf5d0fb0 <sr_ssl_ctx_info_callback> st = <optimized out> ret = <optimized out> transition = <optimized out> pre_work = <optimized out> post_work = <optimized out> get_construct_message_f = <optimized out> cb = <optimized out> confunc = <optimized out> mt = <optimized out> pkt = <optimized out> #10 state_machine (s=0x7f16bba70bf0, server=0) at ../ssl/statem/statem.c:443 buf = 0x0 cb = 0x7f16bf5d0fb0 <sr_ssl_ctx_info_callback> st = <optimized out> ret = <optimized out> ssret = <optimized out> #11 0x00007f16bf561264 in SSL_do_handshake (s=0x7f16bba70bf0) at ../ssl/ssl_lib.c:3599 ret = 1 #12 0x00007f16bf5ff0b4 in tls_connect (c=c@entry=0x7f16bba42da0, error=error@entry=0x7ffd991be3cc) at tls_server.c:494 ssl = 0x7f16bba70bf0 ret = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- cert = <optimized out> tls_c = 0x7f16bba70760 tls_log = <optimized out> __func__ = "tls_connect" pkey = <optimized out> #13 0x00007f16bf60068d in tls_encode_f (c=0x7f16bba42da0, pbuf=0x7ffd991be4b0, plen=0x7ffd991be49c, rest_buf=0x7ffd991be4a8, rest_len=0x7ffd991be498, send_flags=0x7ffd991be4a0) at tls_server.c:802 n = <optimized out> offs = 0 ssl = 0x7f16bba70bf0 tls_c = <optimized out>
wr_buf = '\000' <repeats 65535 times> rd = {buf = 0x0, pos = 0, used = 0, size = 0} wr = {buf = 0x7f16bf632560 <wr_buf> "", pos = 0, used = 0, size = 65536} ssl_error = 0 err_src = 0x7f16bf61ccbb "TLS write:" buf = 0x7f16bba6c770 "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d84e236-69f6-68-0-6b37d392;ngcplb=yes;socket="... len = 708 x = <optimized out> __func__ = "tls_encode_f" #14 0x00005590572e3f7e in tcp_send (dst=dst@entry=0x7f16bba7c510, from=<optimized out>, buf=<optimized out>, len=<optimized out>) at core/tcp_main.c:1847 c = 0x7f16bba42da0 ip = {af = 2, len = 4, u = {addrl = {3925074265, 0}, addr32 = {3925074265, 0, 0, 0}, addr16 = {57689, 59891, 0, 0, 0, 0, 0, 0}, addr = "Y\341\363\351", '\000' <repeats 11 times>}} port = <optimized out> fd = 4 response = {0, 0} n = <optimized out> con_lifetime = <optimized out> rest_buf = 0x0 t_buf = 0x7f16bba6c770 "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d84e236-69f6-68-0-6b37d392;ngcplb=yes;socket="... rest_len = 0 --Type <RET> for more, q to quit, c to continue without paging-- t_len = 708 resp = 3 t_send_flags = {f = 4, blst_imask = 0} __func__ = "tcp_send" #15 0x00007f16c20c0920 in msg_send_buffer (flags=0, len=<optimized out>, buf=0x7f16bba6c770 "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d84e236-69f6-68-0-6b37d392;ngcplb=yes;socket="..., dst=0x7f16bba7c510) at ../../core/forward.h:282 outb = { s = 0x7f16bba6c770 "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d84e236-69f6-68-0-6b37d392;ngcplb=yes;socket="..., len = 708} from = <optimized out> con = 0x0 netinfo = {data = {s = 0x16c2a71f045e6d57 <error: Cannot access memory at address 0x16c2a71f045e6d57>, len = -585208489}, rcv = 0x559000000000, dst = 0x55905751a660} ip = {af = 3148334224, len = 32534, u = {addrl = {140727172196272, 4294977025}, addr32 = {2568742832, 32765, 9729, 1}, addr16 = {59312, 39195, 32765, 0, 9729, 0, 1, 0}, addr = "\260\347\033\231\375\177\000\000\001&\000\000\001\000\000"}} local_addr = {s = {sa_family = 2, sa_data = "\000\000\005\207ac\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 0, sin_addr = { s_addr = 1667335941}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 0, sin6_flowinfo = 1667335941, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}} wsev = {type = 4, buf = 0x0, len = 3148334352, id = 32534} ret = <optimized out> new_dst = {send_sock = 0x0, to = {s = {sa_family = 49816, sa_data = "\247\273\026\177\000\000\000\000\000\000\000\000\000"}, sin = { sin_family = 49816, sin_port = 48039, sin_addr = {s_addr = 32534}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 49816, sin6_port = 48039, sin6_flowinfo = 32534, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 3258773256}}, id = 32534, proto = -100 '\234', send_flags = {f = 49677, blst_imask = 32534}} evp = {data = 0x7ffd991be570, rcv = 0x0, dst = 0x7f16bba7c510} port = <optimized out> new_dst = <optimized out> outb = <optimized out> netinfo = <optimized out> evp = <optimized out> port = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- ip = <optimized out> from = <optimized out> local_addr = <optimized out> con = <optimized out> wsev = <optimized out> ret = <optimized out> __func__ = "msg_send_buffer" __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> #16 send_pr_buffer (rb=rb@entry=0x7f16bba7c4c0, buf=0x7f16bba6c770, len=<optimized out>) at t_funcs.c:70 __func__ = "send_pr_buffer" #17 0x00007f16c20dd6e8 in t_send_branch (t=t@entry=0x7f16bba7c298, branch=branch@entry=0, p_msg=p_msg@entry=0x7f16c23cef08, proxy=proxy@entry=0x0, lock_replies=lock_replies@entry=1) at t_fwd.c:1539 ip = {af = 3148333720, len = 32534, u = {addrl = {94078425791499, 140727172196296}, addr32 = {1462139915, 21904, 2568742856, 32765}, addr16 = { 31755, 22310, 21904, 0, 59336, 39195, 32765, 0}, addr = "\v|&W\220U\000\000\310\347\033\231\375\177\000"}} ret = 0 uac = 0x7f16bba7c4b0 __func__ = "t_send_branch" #18 0x00007f16c20e0adf in t_forward_nonack (t=0x7f16bba7c298, p_msg=p_msg@entry=0x7f16c23cef08, proxy=proxy@entry=0x0, proto=proto@entry=0) at t_fwd.c:1750 branch_ret = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- lowest_ret = -1 current_uri = {s = 0x0, len = 0} added_branches = 1 first_branch = 0 i = 0 q = -1 t_invite = <optimized out> success_branch = 0 try_new = <optimized out> lock_replies = 1 dst_uri = {s = 0x0, len = 0} path = {s = 0x0, len = 0} instance = {s = 0x0, len = 0} ruid = {s = 0x0, len = 0} location_ua = {s = 0x0, len = 0} si = 0x0 backup_bflags = 0 bflags = 0 __func__ = "t_forward_nonack" #19 0x00007f16c20c4452 in t_relay_to (p_msg=0x7f16c23cef08, proxy=0x0, proto=0, replicate=replicate@entry=0) at t_funcs.c:334 ret = 0 new_tran = <optimized out> t = 0x7f16bba7c298 dst = {send_sock = 0x0, to = {s = {sa_family = 47744, sa_data = ".\302\026\177\000\000\b\357<\302\026\177\000"}, sin = {sin_family = 47744, sin_port = 49710, sin_addr = {s_addr = 32534}, sin_zero = "\b\357<\302\026\177\000"}, sin6 = {sin6_family = 47744, sin6_port = 49710, sin6_flowinfo = 32534, sin6_addr = {__in6_u = {__u6_addr8 = "\b\357<\302\026\177\000\000Ⱦ.\302\026\177\000", __u6_addr16 = {61192, 49724, 32534, 0, 48840, 49710, 32534, 0}, __u6_addr32 = {3258773256, 32534, 3257843400, 32534}}}, sin6_scope_id = 3257843408}}, id = 32534, proto = 1 '\001', send_flags = {f = 0, blst_imask = 0}} port = <optimized out> host = {s = 0x208730 <error: Cannot access memory at address 0x208730>, len = 2910936} comp = <optimized out> __func__ = "t_relay_to" #20 0x00007f16c208b365 in _w_t_relay_to (p_msg=p_msg@entry=0x7f16c23cef08, proxy=proxy@entry=0x0, force_proto=force_proto@entry=0) at tm.c:1628 t = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- res = <optimized out> __func__ = "_w_t_relay_to" #21 0x00007f16c208f0ea in w_t_relay_to (msg=0x7f16c23cef08, proxy=0x0, flags=<optimized out>) at tm.c:2549 fl = <optimized out> px = 0x0 param = {orig = 0x0, type = 0, v = {asciiz = 0x0, str = {s = 0x0, len = 28}, i = 0, regex = 0x0, avp = {flags = 0, name = {n = 28, s = { s = 0x1c <error: Cannot access memory at address 0x1c>, len = 1466053571}, re = 0x1c}, index = 6400}, select = 0x0, subst = 0x0, pvs = 0x0, pve = 0x0}, fixed = 0x0} #22 0x0000559057249f29 in do_action (h=h@entry=0x7ffd991bfcc0, a=a@entry=0x7f16c2345fa8, msg=msg@entry=0x7f16c23cef08) at core/action.c:1073 ret = <optimized out> v = <optimized out> dst = {send_sock = 0x7f16c239f320, to = {s = {sa_family = 3, sa_data = "\000\000\000\000\000\000x1:\302\026\177\000"}, sin = {sin_family = 3, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "x1:\302\026\177\000"}, sin6 = {sin6_family = 3, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "x1:\302\026\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {12664, 49722, 32534, 0, 0, 0, 0, 0}, __u6_addr32 = {3258593656, 32534, 0, 0}}}, sin6_scope_id = 3258593504}}, id = 32534, proto = 23 '\027', send_flags = {f = 22326, blst_imask = 21904}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = 0x7f16c21bfc78 len = <optimized out> user = <optimized out> uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = { s = 0x0, len = 0}, ttl = {s = 0x0, len = -1214441216}, user_param = { s = 0x5590576233d3 <buf+147> "lr;received=sip:89.225.243.233:52406%3Btransport%3Dtls;socket=sip:5.135.97.99:5061;ngcpdevid=9990110000055210001>\r\nFrom: sip:pinger@localhost;tag=uloc-1-5d84e236-69f6-68-0-6b37d392\r\nTo: sip:521000@172"..., len = -1036194040}, maddr = {s = 0x7f16c22ea888 "\b\003", len = 1466050240}, method = {s = 0x559057503200 "corefunc", len = 1465227358}, lr = {s = 0x7ffd991bee20 "", len = 1462012442}, r2 = { s = 0x1 <error: Cannot access memory at address 0x1>, len = -1726222816}, gr = {s = 0x7f16c22ea888 "\b\003", len = -1036194040}, transport_val = {s = 0x7ffd00000000 <error: Cannot access memory at address 0x7ffd00000000>, len = -1726222936}, ttl_val = { s = 0x7ffd991bfcc0 "\003", len = 1462068183}, user_param_val = {s = 0x0, len = -1}, maddr_val = {s = 0x7ffd991bfcc0 "\003", len = -1019318953}, method_val = {s = 0x7f16c23cef08 "\002", len = -1726222364}, lr_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1726219072}, r2_val = {s = 0x16c2a71f15be6d57 <error: Cannot access memory at address 0x16c2a71f15be6d57>, len = -585208489}, gr_val = { --Type <RET> for more, q to quit, c to continue without paging-- s = 0x7ffd00000000 <error: Cannot access memory at address 0x7ffd00000000>, len = -1036194040}} next_hop = {user = {s = 0x7f16c23cef08 "\002", len = -1}, passwd = {s = 0x7ffd991bfcc0 "\003", len = -1214441216}, host = { s = 0x7f16c229d670 "\001", len = -1036181960}, port = {s = 0x0, len = 1466050240}, params = {s = 0x7f16c21ca0b8 "\021G\333\301\026\177", len = 1465227358}, sip_params = {s = 0x7ffd991bf400 "", len = 1464170047}, headers = {s = 0x7f16c239d178 "\276\063bW\220U", len = -1214441216}, port_no = 61192, proto = 49724, type = 32534, flags = (unknown: 3080526080), transport = {s = 0x5590576226c0 <log_level_info> "&\033SW\220U", len = -1726222364}, ttl = {s = 0x7ffd991bfcc0 "\003", len = 1461852364}, user_param = { s = 0x207e00 <error: Cannot access memory at address 0x207e00>, len = -1214441216}, maddr = {s = 0x5590574fc1c6 "core: core/rvalue.c", len = -1726219072}, method = {s = 0x0, len = -1036194040}, lr = {s = 0x7ffd991bf150 "\001", len = -1}, r2 = {s = 0x7ffd991bfcc0 "\003", len = 1462068183}, gr = {s = 0x0, len = 1}, transport_val = {s = 0x7ffd991bfcc0 "\003", len = -1019318953}, ttl_val = { s = 0x7f16c23cef08 "\002", len = -1726222000}, user_param_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1726219072}, maddr_val = {s = 0x16c2a71f1a1e6d57 <error: Cannot access memory at address 0x16c2a71f1a1e6d57>, len = -585208489}, method_val = {s = 0x7f1600000000 <error: Cannot access memory at address 0x7f1600000000>, len = 1463999779}, lr_val = { s = 0x2084e8 <error: Cannot access memory at address 0x2084e8>, len = 1464815620}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = -1214441216}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = 0 avp = <optimized out> st = {flags = 3258773256, id = 32534, name = {n = -1, s = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1726219072}, re = 0xffffffff}, avp = 0x96c4632cb79d1900} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x0 rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -1214441216, s = { s = 0x96c4632cb79d1900 <error: Cannot access memory at address 0x96c4632cb79d1900>, len = 0}, re = 0x96c4632cb79d1900}, pval = {rs = { s = 0x96c4632cb79d1900 <error: Cannot access memory at address 0x96c4632cb79d1900>, len = 0}, ri = -1726222000, flags = 32765}}, i2s = "\300\374\033\231\375\177\000\000\314\030"W\220U\000\000\340\060:\302\026\177"} s = {s = 0x0, len = 0} srevp = {0x7f1600000001, 0xb} --Type <RET> for more, q to quit, c to continue without paging-- evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #23 0x0000559057248a1a in run_actions (h=h@entry=0x7ffd991bfcc0, a=0x7f16c2342060, msg=msg@entry=0x7f16c23cef08) at core/action.c:1569 t = 0x7f16c2345fa8 ret = <optimized out> tvb = {tv_sec = 140727172201664, tv_usec = 139735718325485} tve = {tv_sec = 139735722440664, tv_usec = 94076963651595} tz = {tz_minuteswest = 1465227358, tz_dsttime = 21904} tdiff = <optimized out> __func__ = "run_actions" #24 0x000055905724a7e2 in do_action (h=h@entry=0x7ffd991bfcc0, a=a@entry=0x7f16c22ed0c0, msg=msg@entry=0x7f16c23cef08) at core/action.c:691 ret = <optimized out> v = <optimized out> dst = {send_sock = 0xfffffdff, to = {s = {sa_family = 32918, sa_data = "k\277\026\177\000\000\300\365\033\231\375\177\000"}, sin = { sin_family = 32918, sin_port = 49003, sin_addr = {s_addr = 32534}, sin_zero = "\300\365\033\231\375\177\000"}, sin6 = {sin6_family = 32918, sin6_port = 49003, sin6_flowinfo = 32534, sin6_addr = {__in6_u = { __u6_addr8 = "\300\365\033\231\375\177\000\000\340\265\323\001\000\000\000", __u6_addr16 = {62912, 39195, 32765, 0, 46560, 467, 0, 0}, __u6_addr32 = {2568746432, 32765, 30651872, 0}}}, sin6_scope_id = 2568746580}}, id = 32765, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1726219072}, passwd = { --Type <RET> for more, q to quit, c to continue without paging-- s = 0x16c2a71f1c9e6d57 <error: Cannot access memory at address 0x16c2a71f1c9e6d57>, len = -585208489}, host = { s = 0x559000000000 <error: Cannot access memory at address 0x559000000000>, len = 6}, port = { s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1214441216}, params = {s = 0x7f16c2374348 "\001", len = -1}, sip_params = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1036492816}, headers = { s = 0x7f16c1d8c848 <_xlog_buf> "\260\313?\302\026\177", len = -1214441216}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (URI_SIP_USER_PHONE | unknown: 3252164460), transport = {s = 0x7f1600000000 <error: Cannot access memory at address 0x7f1600000000>, len = 134}, ttl = { s = 0x7f16c23fcbb0 "Relaying request, fs='«tls:5.135.97.99:5061»' du='«sip:89.225.243.233:52406;transport=tls»' - R=«sip:521000@172.26.46.229:52406;transport=tls» ID=«77995d42-caf71fc7-fb1f4e5@10.10.91.23» UA='<n"..., len = -1214441216}, user_param = { s = 0x7f16bb9496b8 "\370\177a\277\026\177", len = -1036194040}, maddr = {s = 0x7f16c22e6378 "\374\002", len = 1466050240}, method = { s = 0x559057503200 "corefunc", len = 1465227358}, lr = {s = 0x7ffd991bf400 "", len = 1462012442}, r2 = {s = 0x0, len = -1726221312}, gr = { s = 0x7f16c22e6378 "\374\002", len = -1036194040}, transport_val = {s = 0x7ffd00000000 <error: Cannot access memory at address 0x7ffd00000000>, len = -1726221432}, ttl_val = {s = 0x7ffd991bfcc0 "\003", len = -1214441216}, user_param_val = {s = 0x0, len = -1726219072}, maddr_val = { s = 0x7f16c22a9b90 "\001", len = -1036194040}, method_val = {s = 0x7ffd991bf474 "\001", len = -1}, lr_val = {s = 0x7ffd991bfcc0 "\003", len = 1461886711}, r2_val = {s = 0x16c2a71f217e6d57 <error: Cannot access memory at address 0x16c2a71f217e6d57>, len = -585208489}, gr_val = { s = 0x7f1600000000 <error: Cannot access memory at address 0x7f1600000000>, len = 1465227358}} next_hop = {user = {s = 0x7ffd991bfb00 "\320\324.\302\026\177", len = 1462017833}, passwd = {s = 0x1 <error: Cannot access memory at address 0x1>, len = -1087425871}, host = {s = 0x0, len = -1726218720}, port = {s = 0x3 <error: Cannot access memory at address 0x3>, len = -1085168472}, params = {s = 0x1 <error: Cannot access memory at address 0x1>, len = -1155280896}, sip_params = { s = 0x8 <error: Cannot access memory at address 0x8>, len = 293}, headers = { s = 0x7f1600000001 <error: Cannot access memory at address 0x7f1600000001>, len = -1214441216}, port_no = 49000, proto = 48020, type = 32534, flags = (unknown: 3080526080), transport = {s = 0x0, len = -1726220860}, ttl = {s = 0x7ffd991bfcc0 "\003", len = 1461852364}, user_param = { s = 0x18 <error: Cannot access memory at address 0x18>, len = -1214441216}, maddr = { s = 0x393132343261705f <error: Cannot access memory at address 0x393132343261705f>, len = -1726219072}, method = {s = 0x0, len = -1036194040}, lr = {s = 0x7ffd991bf730 "\001", len = -1}, r2 = {s = 0x7ffd991bfcc0 "\003", len = 1462068183}, gr = {s = 0x0, len = 0}, transport_val = { s = 0x7ffd991bfcc0 "\003", len = -1019318953}, ttl_val = {s = 0x7f16c23cef08 "\002", len = -1726220496}, user_param_val = { s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = -1726219072}, maddr_val = { s = 0x16c2a71f27de6d57 <error: Cannot access memory at address 0x16c2a71f27de6d57>, len = -585208489}, method_val = { s = 0x7f1600000000 <error: Cannot access memory at address 0x7f1600000000>, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = { s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = -1214441216}, gr_val = { s = 0x4bb23d000 <error: Cannot access memory at address 0x4bb23d000>, len = -1039967502}} u = <optimized out> port = <optimized out> dst_host = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- i = <optimized out> flags = 32765 avp = <optimized out> st = {flags = 2568747776, id = 32765, name = {n = 1462017833, s = { s = 0x559057249f29 <do_action+2569> "E1҅\300\211C\bA\017\224\302\211\005\255\215=", len = 1}, re = 0x559057249f29 <do_action+2569>}, avp = 0x7f16bf2f32b1 <asn1_template_noexp_d2i+673>} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x5590572563d7 <run_actions_safe+71> rv1 = <optimized out> c1 = {cache_type = 3750576, val_type = RV_NONE, c = {avp_val = {n = -1214441216, s = { s = 0x96c4632cb79d1900 <error: Cannot access memory at address 0x96c4632cb79d1900>, len = 53}, re = 0x96c4632cb79d1900}, pval = {rs = { s = 0x96c4632cb79d1900 <error: Cannot access memory at address 0x96c4632cb79d1900>, len = 53}, ri = -1726220496, flags = 32765}}, i2s = "\300\374\033\231\375\177\000\000\314\030"W\220U\000\000\017\065bW\220U"} s = {s = 0x0, len = 0} srevp = {0x0, 0x200} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #25 0x0000559057248a1a in run_actions (h=h@entry=0x7ffd991bfcc0, a=0x7f16c22e8840, msg=msg@entry=0x7f16c23cef08) at core/action.c:1569 t = 0x7f16c22ed0c0 ret = <optimized out> tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 103079215104, tv_usec = 94078428878942} tz = {tz_minuteswest = -1036194040, tz_dsttime = 32534} tdiff = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- __func__ = "run_actions" #26 0x000055905724a54f in do_action (h=h@entry=0x7ffd991bfcc0, a=a@entry=0x7f16c22ed4d0, msg=msg@entry=0x7f16c23cef08) at core/action.c:1062 ret = <optimized out> v = <optimized out> dst = {send_sock = 0x0, to = {s = {sa_family = 42675, sa_data = "1W\220U\000\000\b䙻\026\177\000"}, sin = {sin_family = 42675, sin_port = 22321, sin_addr = {s_addr = 21904}, sin_zero = "\b䙻\026\177\000"}, sin6 = {sin6_family = 42675, sin6_port = 22321, sin6_flowinfo = 21904, sin6_addr = {__in6_u = {__u6_addr8 = "\b䙻\026\177\000\000\252\004\000\000\000\000\000", __u6_addr16 = {58376, 48025, 32534, 0, 1194, 0, 0, 0}, __u6_addr32 = {3147424776, 32534, 1194, 0}}}, sin6_scope_id = 3147150992}}, id = 32534, proto = 24 '\030', send_flags = {f = 48993, blst_imask = 32534}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0xa <error: Cannot access memory at address 0xa>, len = -1726219520}, passwd = {s = 0x7f16c23736e8 "N\004", len = -1214441216}, host = {s = 0x7ffd00000000 <error: Cannot access memory at address 0x7ffd00000000>, len = -1147490344}, port = {s = 0x7f16c23cef08 "\002", len = -1726219664}, params = {s = 0x5590576e3720 <onreply_rt> "\200\376=\302\026\177", len = -1039686414}, sip_params = {s = 0x0, len = -1039668392}, headers = {s = 0x7f16c23cef08 "\002", len = -1039667007}, port_no = 5, proto = 0, type = 32534, flags = (unknown: 0), transport = {s = 0x7f1600000064 <error: Cannot access memory at address 0x7f1600000064>, len = -1036194040}, ttl = { s = 0x125 <error: Cannot access memory at address 0x125>, len = -1083473770}, user_param = {s = 0x7f16bb9ab0a8 <incomplete sequence \310>, len = -1726219664}, maddr = {s = 0x206cf8 <error: Cannot access memory at address 0x206cf8>, len = 2897984}, method = { s = 0x393ab0 <error: Cannot access memory at address 0x393ab0>, len = 8}, lr = {s = 0x38 <error: Cannot access memory at address 0x38>, len = -1214441216}, r2 = {s = 0x7f16c212e010 "\001", len = -1214441216}, gr = {s = 0x7ffd991bfa40 "", len = -1043263833}, transport_val = { s = 0xffffffffc1d281e0 <error: Cannot access memory at address 0xffffffffc1d281e0>, len = 0}, ttl_val = { s = 0xa <error: Cannot access memory at address 0xa>, len = 10}, user_param_val = {s = 0xa <error: Cannot access memory at address 0xa>, len = -1036182752}, maddr_val = {s = 0x7f16c21dab90 "text:///etc/kamailio/lb/db", len = 10}, method_val = { s = 0xa <error: Cannot access memory at address 0xa>, len = -1214441216}, lr_val = {s = 0x7f16bb5575f8 "\233^", len = -1726219520}, r2_val = { s = 0x0, len = -1036568856}, gr_val = {s = 0x7f16c23cef08 "\002", len = -1147489808}} next_hop = {user = {s = 0x64 <error: Cannot access memory at address 0x64>, len = 1462068345}, passwd = {s = 0x7f16bb5575f8 "\233^", len = 1462871493}, host = {s = 0x7ffd991bfa40 "", len = 0}, port = {s = 0x38 <error: Cannot access memory at address 0x38>, len = 536871205}, params = {s = 0x7f16bb99e8b2 "", len = -1038307920}, sip_params = {s = 0xa <error: Cannot access memory at address 0xa>, len = 10}, headers = { s = 0xa <error: Cannot access memory at address 0xa>, len = -1214441216}, port_no = 30200, proto = 47957, type = 32534, --Type <RET> for more, q to quit, c to continue without paging-- flags = (unknown: 3255033656), transport = {s = 0x7f16bb23d000 "", len = -1084133281}, ttl = { s = 0x143 <error: Cannot access memory at address 0x143>, len = -1084145664}, user_param = { s = 0x4b0 <error: Cannot access memory at address 0x4b0>, len = -1038305808}, maddr = {s = 0xa <error: Cannot access memory at address 0xa>, len = 10}, method = {s = 0xa <error: Cannot access memory at address 0xa>, len = -1039926069}, lr = {s = 0x7f16bb5575f8 "\233^", len = -1214441216}, r2 = {s = 0x7f16c23cef08 "\002", len = -1147490344}, gr = {s = 0x7f16c23cef08 "\002", len = -1726219664}, transport_val = { s = 0x5 <error: Cannot access memory at address 0x5>, len = -1147489808}, ttl_val = {s = 0x64 <error: Cannot access memory at address 0x64>, len = -1039645902}, user_param_val = {s = 0xa <error: Cannot access memory at address 0xa>, len = 0}, maddr_val = { s = 0x5590576d7ce0 <def_list> "", len = 1466793192}, method_val = {s = 0x5590576d7cf0 <def_list+16> "", len = 1466793208}, lr_val = { s = 0x5590576d7d00 <def_list+32> "", len = 1466793224}, r2_val = {s = 0x559057622b60 <_xavp_list_head> "", len = -1038300312}, gr_val = { s = 0xa <error: Cannot access memory at address 0xa>, len = 10}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = 32765 avp = <optimized out> st = {flags = 100, id = 0, name = {n = 1462068345, s = {s = 0x559057256479 <run_top_route+105> "H\213\214", <incomplete sequence \330>, len = -1152027144}, re = 0x559057256479 <run_top_route+105>}, avp = 0x55905731a5c5 <init_mod_child+37>} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x7f16c207634e <t_should_relay_response+3134> rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -1038299152, s = {s = 0x7f16c21ccff0 "P\317\034\302\026\177", len = 10}, re = 0x7f16c21ccff0}, pval = {rs = {s = 0x7f16c21ccff0 "P\317\034\302\026\177", len = 10}, ri = 10, flags = 0}}, i2s = "\n\000\000\000\000\000\000\000\000\031\235\267,cĖ\370uU\273\026\177"} s = {s = 0x7ffd991bf874 "\377\377\377\377", len = -1036194040} srevp = {0x0, 0x0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = { number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, --Type <RET> for more, q to quit, c to continue without paging-- data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, { type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #27 0x0000559057248a1a in run_actions (h=h@entry=0x7ffd991bfcc0, a=a@entry=0x7f16c21e5d68, msg=msg@entry=0x7f16c23cef08) at core/action.c:1569 t = 0x7f16c22ed4d0 ret = <optimized out> tvb = {tv_sec = 7, tv_usec = 94078429705489} tve = {tv_sec = 139735724774536, tv_usec = 94078429705465} tz = {tz_minuteswest = 1466053882, tz_dsttime = 21904} tdiff = <optimized out> __func__ = "run_actions" #28 0x000055905725646f in run_top_route (a=0x7f16c21e5d68, msg=msg@entry=0x7f16c23cef08, c=c@entry=0x0) at core/action.c:1654 ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140727172201664, -1641116586892956329, 139735722777960, 139735724781320, 0, 139735724792760, 1640056966797487447, 4762903372428635479}, __mask_was_saved = 0, __saved_mask = {__val = { 10863917244834715904, 2126992, 139735724774368, 139735724781320, 10863917244834715904, 94078429705489, 1, 1073741825, 139735721671992, 1, 139735721348117, 139735724792760, 94078426423542, 140727172201920, 140727172398714, 2125136}}}}} p = 0x0 ret = <optimized out> sfbk = 0 #29 0x00005590573524cf in receive_msg ( buf=buf@entry=0x559057623340 <buf> "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3010589\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:89.225.243.233:52406%3Btransport%3Dtl"..., len=<optimized out>, rcv_info=rcv_info@entry=0x7ffd991c0030) at core/receive.c:341 msg = 0x7f16c23cef08 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140727172202128, -1641116586892956329, 139735724869216, 139735724781320, 0, 0, 1640056966797487447, 4762903372428635479}, __mask_was_saved = 0, __saved_mask = {__val = {139735677501590, 33554432, 30656312, 2125080, 2898120, 3750576, 8, 56, 10863917244834715904, 139735722024976, 94078427651363, 10, 94078428467204, 94078428548920, 94078428543483, 139735608948216}}}}} bctx = <optimized out> ret = <optimized out> tvb = {tv_sec = 10, tv_usec = 10} tve = {tv_sec = 10, tv_usec = 10} tz = {tz_minuteswest = -1083795557, tz_dsttime = 32534} --Type <RET> for more, q to quit, c to continue without paging-- diff = 0 inb = { s = 0x559057623340 <buf> "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3010589\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:89.225.243.233:52406%3Btransport%3Dtl"..., len = 465} netinfo = {data = {s = 0x393ab0 <error: Cannot access memory at address 0x393ab0>, len = 8}, rcv = 0x38, dst = 0x96c4632cb79d1900} keng = 0x0 evp = {data = 0x7ffd991bfe40, rcv = 0x7ffd991c0030, dst = 0x0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 __func__ = "receive_msg" #30 0x0000559057273b45 in udp_rcv_loop () at core/udp_server.c:541 len = 465 buf = "OPTIONS sip:521000@172.26.46.229:52406;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3010589\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:89.225.243.233:52406%3Btransport%3Dtl"... tmp = <optimized out> from = <optimized out> fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {10863917242146032138, 140727172202624}, addr32 = {391842314, 2529452844, 2568749184, 32765}, addr16 = {2570, 5979, 25388, 38596, 128, 39196, 32765, 0}, addr = "\n\n[\027,cĖ\200\000\034\231\375\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1734019594, 0}, addr32 = {1734019594, 0, 0, 0}, addr16 = {2570, 26459, 0, 0, 0, 0, 0, 0}, addr = "\n\n[g", '\000' <repeats 11 times>}}, src_port = 5062, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = { sa_family = 2, sa_data = "\023\306\n\n[\027\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50707, sin_addr = { s_addr = 391842314}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50707, sin6_flowinfo = 391842314, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f16c21be5a0, proto = 1 '\001'} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\000\003\034\231\375\177\000\000+\003\034\231\375\177\000\000\177\003\034\231\375\177\000\000\330\177\035\302\026\177\000\000\n\000\000\000\000\000\000\000\n\000\000\000\000\000\000\000\n", '\000' <repeats 15 times>, "\370uU\273\026\177\000\000\263\246\061W\220U\000\000\000\000\000\000\004", '\000' <repeats 27 times>, "\263%PW\220U\000\000\230\216\035\302\026\177\000\000\n\000\000\000\000\000\000\000\n\000\000\000\000\000\000\000\n", '\000' <repeats 15 times>, "\370uU\273\026\177\000\000ť1W\220U\000\000\002\000\000\000\000\000\000\000\240\237\060\304\026\177\000\000\000\000\000\000\000\000\000\000\020"... i = <optimized out> j = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- l = <optimized out> __func__ = "udp_rcv_loop" __llevel = <optimized out> #31 0x0000559057208ebb in main_loop () at main.c:1655 i = 0 pid = <optimized out> si = 0x7f16c21be5a0 si_desc = "udp receiver child=1 sock=10.10.91.103:5060\000\001\000\000\000\210\067RW\220U\000\000\000\000\000\000\000\000\000\000N\000 \000\000\000\000\000$\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\000\031\235\267,cĖ\000@a\277\026\177\000\000\000\031\235\267,cĖH7nW\220U\000\000\001\000\000\000\000\000\000" nrprocs = <optimized out> woneinit = <optimized out> __func__ = "main_loop" #32 0x0000559057200415 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2696 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffd991c1e4b "" tmp_len = -1726217152 port = 32765 proto = -1726217136 options = 0x5590574f7788 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 4273330862 rfd = <optimized out> debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = <optimized out> p = <optimized out> st = {st_dev = 21, st_ino = 5142882, st_nlink = 2, st_mode = 16872, st_uid = 113, st_gid = 118, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1569317956, tv_nsec = 464789223}, st_mtim = {tv_sec = 1569375047, tv_nsec = 381719437}, st_ctim = {tv_sec = 1569375047, tv_nsec = 381719437}, __glibc_reserved = {0, 0, 0}} __func__ = "main" (gdb)
Can you try with latest version in git branch 5.2? The pre-loaded library is no longer needed there. Just start kamailio as usual.
Hi Daniel,
I recompiled kamailio with the latest commits included in branch 5.2 and I deployed it on our test system, but the issue looks not solved.
Here a new backtrace obtained with the new packages:
`(gdb) bt full #0 aesni_ecb_encrypt () at crypto/aes/aesni-x86_64.s:524 No locals. #1 0x00007f6d1b5d5917 in aesni_ecb_cipher (len=16, in=0x7f6d17780fc0 "\036\234\364\354\062\311\005\316\036۫\023\065N\235\305l\001\306-", out=0x7f6d17d97aa8 "", ctx=0x7f6d177810a8) at ../crypto/evp/e_aes.c:319 bl = <optimized out> bl = <optimized out> #2 aesni_ecb_cipher (ctx=0x7f6d177810a8, out=0x7f6d17d97aa8 "", in=0x7f6d17780fc0 "\036\234\364\354\062\311\005\316\036۫\023\065N\235\305l\001\306-", len=16) at ../crypto/evp/e_aes.c:311 bl = <optimized out> #3 0x00007f6d1b5e1533 in evp_EncryptDecryptUpdate (ctx=0x7f6d177810a8, out=0x7f6d17d97aa8 "", outl=0x7ffe569bebf4, in=0x7f6d17780fc0 "\036\234\364\354\062\311\005\316\036۫\023\065N\235\305l\001\306-", inl=16) at ../crypto/evp/evp_enc.c:333 i = <optimized out> j = <optimized out> bl = 16 cmpl = <optimized out> #4 0x00007f6d1b61430f in drbg_ctr_generate (drbg=0x7f6d17780ed8, out=0x7f6d17d97aa8 "", outlen=32, adin=0x0, adinlen=0) at ../crypto/rand/drbg_ctr.c:340 outl = 16 ctr = 0x7f6d17780f80 #5 0x00007f6d1b6151fb in RAND_DRBG_generate (drbg=drbg@entry=0x7f6d17780ed8, out=out@entry=0x7f6d17d97aa8 "", outlen=outlen@entry=32, prediction_resistance=prediction_resistance@entry=0, adin=0x0, adinlen=adinlen@entry=0) at ../crypto/rand/drbg_lib.c:638 reseed_required = <optimized out> #6 0x00007f6d1b615481 in RAND_DRBG_bytes (drbg=0x7f6d17780ed8, out=0x7f6d17d97aa8 "", outlen=32) at ../crypto/rand/drbg_lib.c:679 additional = 0x0 additional_len = 0 chunk = 32 ret = <optimized out> #7 0x00007f6d1b7756fd in ssl_fill_hello_random (s=s@entry=0x7f6d17d00e40, server=server@entry=0, result=0x7f6d17d97aa8 "", len=len@entry=32, dgrd=dgrd@entry=DOWNGRADE_NONE) at ../ssl/s3_lib.c:4589 send_time = <optimized out> ret = <optimized out> #8 0x00007f6d1b79706e in tls_construct_client_hello (s=0x7f6d17d00e40, pkt=0x7ffe569bedc0) at ../ssl/statem/statem_clnt.c:1153 p = <optimized out> sess_id_len = <optimized out> i = <optimized out> protverr = 0 comp = <optimized out> sess = 0x0 session_id = <optimized out> #9 0x00007f6d1b79633f in write_state_machine (s=0x7f6d17d00e40) at ../ssl/statem/statem.c:843 post_work = 0x7f6d1b79b5a0 <ossl_statem_client_post_work> mt = 1 pkt = {buf = 0x7f6d17d80270, staticbuf = 0x0, curr = 4, written = 4, maxsize = 18446744073709551615, subs = 0x7f6d17d9f918} ret = <optimized out> pre_work = 0x7f6d1b799180 <ossl_statem_client_pre_work> get_construct_message_f = 0x7f6d1b799250 <ossl_statem_client_construct_message> confunc = 0x7f6d1b796d20 <tls_construct_client_hello> st = 0x7f6d17d00e88 transition = 0x7f6d1b798de0 <ossl_statem_client_write_transition> cb = 0x7f6d1b7f1fb0 <sr_ssl_ctx_info_callback> st = <optimized out> ret = <optimized out> transition = <optimized out> pre_work = <optimized out> post_work = <optimized out> get_construct_message_f = <optimized out> cb = <optimized out> confunc = <optimized out> mt = <optimized out> pkt = <optimized out> --Type <RET> for more, q to quit, c to continue without paging-- #10 state_machine (s=0x7f6d17d00e40, server=0) at ../ssl/statem/statem.c:443 buf = 0x0 cb = 0x7f6d1b7f1fb0 <sr_ssl_ctx_info_callback> st = <optimized out> ret = <optimized out> ssret = <optimized out> #11 0x00007f6d1b782264 in SSL_do_handshake (s=0x7f6d17d00e40) at ../ssl/ssl_lib.c:3599 ret = 1 #12 0x00007f6d1b81ad14 in tls_connect (c=c@entry=0x7f6d17d0f750, error=error@entry=0x7ffe569bef4c) at tls_server.c:494 ssl = 0x7f6d17d00e40 ret = <optimized out> cert = <optimized out> tls_c = 0x7f6d17d95460 tls_log = <optimized out> __func__ = "tls_connect" pkey = <optimized out> #13 0x00007f6d1b81c2ed in tls_encode_f (c=0x7f6d17d0f750, pbuf=0x7ffe569bf030, plen=0x7ffe569bf01c, rest_buf=0x7ffe569bf028, rest_len=0x7ffe569bf018, send_flags=0x7ffe569bf020) at tls_server.c:802 n = <optimized out> offs = 0 ssl = 0x7f6d17d00e40 tls_c = <optimized out> wr_buf = "\027\003\003\003\r\352\374cwYd\354j\002\333\301\365\375\376\355\262\376\v\327SP\230\034g\036 [\221\017\233\255s\245\343\061\350\363\236"\a\033g4\377\347\234Y\364\224\273\235\177u\356>6\026\217\272\303ŕx\213\031+sư\000i1\242\321\064\230\271\302\303\016\306单\271\302\063u\024qv\322P\204\261\063MV\246RoDq\207\245\303\323\t\037=\323\321ݣp\363\356\351\020#խH\tq\342\205K\350\004\261\204\375fh\360B\350O\027n\351\352\302\331\021\351\275\310J\273+\275\307N\020\244\312\361\317\353\307\332\356=։yc\v\027TWS3\244#\347\030\254\267\342\366A\252\024\322\034Ar\326J\354\066\326\346\062\002\247\352h\312"... rd = {buf = 0x0, pos = 0, used = 0, size = 0} wr = { buf = 0x7f6d1b853560 <wr_buf> "\027\003\003\003\r\352\374cwYd\354j\002\333\301\365\375\376\355\262\376\v\327SP\230\034g\036 [\221\017\233\255s\245\343\061\350\363\236"\a\033g4\377\347\234Y\364\224\273\235\177u\356>6\026\217\272\303ŕx\213\031+sư", pos = 0, used = 0, size = 65536} ssl_error = 0 err_src = 0x7f6d1b83c1a9 "TLS write:" buf = 0x7f6d17d808a8 "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d8ca5aa-5b09-c11-0-f9bc5695;ngcplb=yes;socket="... len = 709 x = <optimized out> __func__ = "tls_encode_f" #14 0x0000558a272e77ae in tcp_send (dst=dst@entry=0x7f6d17d9a510, from=<optimized out>, buf=<optimized out>, len=<optimized out>) at core/tcp_main.c:1860 c = 0x7f6d17d0f750 ip = {af = 2, len = 4, u = {addrl = {3264381123, 0}, addr32 = {3264381123, 0, 0, 0}, addr16 = {32963, 49810, 0, 0, 0, 0, 0, 0}, addr = "À\222\302", '\000' <repeats 11 times>}} port = <optimized out> fd = 4 response = {0, 0} n = <optimized out> con_lifetime = <optimized out> rest_buf = 0x0 t_buf = 0x7f6d17d808a8 "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d8ca5aa-5b09-c11-0-f9bc5695;ngcplb=yes;socket="... rest_len = 0 t_len = 709 resp = 3 t_send_flags = {f = 4, blst_imask = 0} __func__ = "tcp_send" #15 0x00007f6d1e3052ff in msg_send_buffer (flags=0, len=<optimized out>, buf=0x7f6d17d808a8 "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d8ca5aa-5b09-c11-0-f9bc5695;ngcplb=yes;socket="..., dst=0x7f6d17d9a510) at ../../core/forward.h:291 outb = { s = 0x7f6d17d808a8 "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nMax-Forwards: 70\r\nRecord-Route: <sip:5.135.97.99:5061;transport=tls;r2=on;lr=on;ftag=uloc-1-5d8ca5aa-5b09-c11-0-f9bc5695;ngcplb=yes;socke--Type <RET> for more, q to quit, c to continue without paging-- t="..., len = 709} from = <optimized out> con = 0x0 netinfo = {data = {s = 0x2dd194deacaac81c <error: Cannot access memory at address 0x2dd194deacaac81c>, len = 413845532}, rcv = 0x558a00000000, dst = 0x558a2753c7a0} ip = {af = 400139408, len = 32621, u = {addrl = {140730351481648, 4294977025}, addr32 = {1453060912, 32766, 9729, 1}, addr16 = {62256, 22171, 32766, 0, 9729, 0, 1, 0}, addr = "0\363\233V\376\177\000\000\001&\000\000\001\000\000"}} local_addr = {s = {sa_family = 2, sa_data = "\000\000\005\207ac\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 0, sin_addr = {s_addr = 1667335941}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = { sin6_family = 2, sin6_port = 0, sin6_flowinfo = 1667335941, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}} wsev = {type = 4, buf = 0x0, len = 400139536, id = 32621} ret = <optimized out> new_dst = {send_sock = 0x0, to = {s = {sa_family = 41624, sa_data = "\331\027m\177\000\000\000\000\000\000\000\000\000"}, sin = {sin_family = 41624, sin_port = 6105, sin_addr = {s_addr = 32621}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 41624, sin6_port = 6105, sin6_flowinfo = 32621, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 509685512}}, id = 32621, proto = 12 '\f', send_flags = {f = 7729, blst_imask = 32621}} evp = {data = 0x0, rcv = 0x0, dst = 0x0} port = <optimized out> new_dst = <optimized out> outb = <optimized out> netinfo = <optimized out> evp = <optimized out> port = <optimized out> ip = <optimized out> from = <optimized out> local_addr = <optimized out> con = <optimized out> wsev = <optimized out> ret = <optimized out> __func__ = "msg_send_buffer" __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> __llevel = <optimized out> #16 send_pr_buffer (rb=rb@entry=0x7f6d17d9a4c0, buf=0x7f6d17d808a8, len=<optimized out>) at t_funcs.c:70 __func__ = "send_pr_buffer" #17 0x00007f6d1e322258 in t_send_branch (t=t@entry=0x7f6d17d9a298, branch=branch@entry=0, p_msg=p_msg@entry=0x7f6d1e612f08, proxy=proxy@entry=0x0, lock_replies=lock_replies@entry=1) at t_fwd.c:1542 ip = {af = 400138904, len = 32621, u = {addrl = {94051850765019, 140730351481672}, addr32 = {656917211, 21898, 1453060936, 32766}, addr16 = {49883, 10023, 21898, 0, 62280, 22171, 32766, 0}, addr = "\333\302''\212U\000\000H\363\233V\376\177\000"}} ret = 0 uac = 0x7f6d17d9a4b0 __func__ = "t_send_branch" #18 0x00007f6d1e32564f in t_forward_nonack (t=0x7f6d17d9a298, p_msg=p_msg@entry=0x7f6d1e612f08, proxy=proxy@entry=0x0, proto=proto@entry=0) at t_fwd.c:1753 branch_ret = <optimized out> lowest_ret = -1 current_uri = {s = 0x0, len = 0} --Type <RET> for more, q to quit, c to continue without paging-- added_branches = 1 first_branch = 0 i = 0 q = -1 t_invite = <optimized out> success_branch = 0 try_new = <optimized out> lock_replies = 1 dst_uri = {s = 0x0, len = 0} path = {s = 0x0, len = 0} instance = {s = 0x0, len = 0} ruid = {s = 0x0, len = 0} location_ua = {s = 0x0, len = 0} si = 0x0 backup_bflags = 0 bflags = 0 __func__ = "t_forward_nonack" #19 0x00007f6d1e308fc2 in t_relay_to (p_msg=0x7f6d1e612f08, proxy=0x0, proto=0, replicate=replicate@entry=0) at t_funcs.c:334 ret = 0 new_tran = <optimized out> t = 0x7f6d17d9a298 dst = {send_sock = 0x0, to = {s = {sa_family = 64128, sa_data = "R\036m\177\000\000\b/a\036m\177\000"}, sin = {sin_family = 64128, sin_port = 7762, sin_addr = {s_addr = 32621}, sin_zero = "\b/a\036m\177\000"}, sin6 = { sin6_family = 64128, sin6_port = 7762, sin6_flowinfo = 32621, sin6_addr = {__in6_u = {__u6_addr8 = "\b/a\036m\177\000\000\310\376R\036m\177\000", __u6_addr16 = {12040, 7777, 32621, 0, 65224, 7762, 32621, 0}, __u6_addr32 = {509685512, 32621, 508755656, 32621}}}, sin6_scope_id = 508755664}}, id = 32621, proto = 1 '\001', send_flags = {f = 0, blst_imask = 0}} port = <optimized out> host = {s = 0x2086c8 <error: Cannot access memory at address 0x2086c8>, len = 2910936} comp = <optimized out> __func__ = "t_relay_to" #20 0x00007f6d1e2cf705 in _w_t_relay_to (p_msg=p_msg@entry=0x7f6d1e612f08, proxy=proxy@entry=0x0, force_proto=force_proto@entry=0) at tm.c:1628 t = <optimized out> res = <optimized out> __func__ = "_w_t_relay_to" #21 0x00007f6d1e2d348a in w_t_relay_to (msg=0x7f6d1e612f08, proxy=0x0, flags=<optimized out>) at tm.c:2549 fl = <optimized out> px = 0x0 param = {orig = 0x0, type = 0, v = {asciiz = 0x0, str = {s = 0x0, len = 28}, i = 0, regex = 0x0, avp = {flags = 0, name = {n = 28, s = {s = 0x1c <error: Cannot access memory at address 0x1c>, len = 660902914}, re = 0x1c}, index = 14336}, select = 0x0, subst = 0x0, pvs = 0x0, pve = 0x0}, fixed = 0x0} #22 0x0000558a2726c691 in do_action (h=h@entry=0x7ffe569c0840, a=a@entry=0x7f6d1e589fa8, msg=msg@entry=0x7f6d1e612f08) at core/action.c:1073 ret = <optimized out> v = <optimized out> dst = {send_sock = 0x7f6d1e5e3390, to = {s = {sa_family = 3, sa_data = "\000\000\000\000\000\000\220H^\036m\177\000"}, sin = {sin_family = 3, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\220H^\036m\177\000"}, sin6 = {sin6_family = 3, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\220H^\036m\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {18576, 7774, 32621, 0, 0, 0, 0, 0}, __u6_addr32 = { 509495440, 32621, 0, 0}}}, sin6_scope_id = 509495288}}, id = 32621, proto = 103 'g', send_flags = {f = 10039, blst_imask = 21898}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = 0x7f6d1e403c78 len = <optimized out> user = <optimized out> uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = -854181888}, user_param = { s = 0x558a27649412 <buf+146> "lr;received=sip:195.128.146.194:47626%3Btransport%3Dtls;socket=sip:5.135.97.99:5061;ngcpdevid=XXXXX%20XXXXX>\r\nFrom: sip:pinger@localhost;tag=uloc-1-5d8ca5aa-5b09-c11-0-f9bc5695\r\nTo: sip:1001@192.168"..., len = 509685512}, maddr = {s = 0x7f6d1e52e888 "\b\003", len = 660899552}, method = {s = 0x558a27528bc0 "corefunc", len = 660077054}, lr = {s = 0x7ffe569bf9a0 "", len = 656846906}, r2 = { s = 0x1 <error: Cannot access memory at address 0x1>, len = 1453062560}, gr = {s = 0x7f6d1e52e888 "\b\003", len = 509685512}, transport_val = {s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 1453062440}, ttl_val = {s = 0x7ffe569c0840 "\003", len = 656903303}, user_param_val = {s = 0x0, len = -1}, maddr_val = {s = 0x7ffe569c0840 "\003", len = 1925892124}, method_val = {s = 0x7f6d1e612f08 "\b", --Type <RET> for more, q to quit, c to continue without paging-- len = 1453063012}, lr_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1453066304}, r2_val = {s = 0x2dd194de9f4ac81c <error: Cannot access memory at address 0x2dd194de9f4ac81c>, len = 413845532}, gr_val = {s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 509685512}} next_hop = {user = {s = 0x7f6d1e612f08 "\b", len = -1}, passwd = {s = 0x7ffe569c0840 "\003", len = -854181888}, host = {s = 0x7f6d1e4e1670 "\001", len = 509677984}, port = {s = 0x0, len = 660899552}, params = { s = 0x7f6d1e40e0b8 "9w\377\035m\177", len = 660077054}, sip_params = {s = 0x7ffe569bff80 "", len = 659017151}, headers = {s = 0x7f6d1e5e2370 "\375\223d'\212U", len = -854181888}, port_no = 12040, proto = 7777, type = 32621, flags = (unknown: 3440785408), transport = {s = 0x558a276486e0 <log_level_info> "F\214U'\212U", len = 1453063012}, ttl = {s = 0x7ffe569c0840 "\003", len = 656688300}, user_param = { s = 0x8ffffffff <error: Cannot access memory at address 0x8ffffffff>, len = -854181888}, maddr = {s = 0x558a27521ca6 "core: core/rvalue.c", len = 1453066304}, method = {s = 0x0, len = 509685512}, lr = { s = 0x7ffe569bfcd0 "\001", len = -1}, r2 = {s = 0x7ffe569c0840 "\003", len = 656903303}, gr = {s = 0x0, len = 1}, transport_val = {s = 0x7ffe569c0840 "\003", len = 1925892124}, ttl_val = {s = 0x7f6d1e612f08 "\b", len = 1453063376}, user_param_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1453066304}, maddr_val = { s = 0x2dd194de82eac81c <error: Cannot access memory at address 0x2dd194de82eac81c>, len = 413845532}, method_val = {s = 0x7f6d00000000 <error: Cannot access memory at address 0x7f6d00000000>, len = 658846883}, lr_val = {s = 0x2084f0 <error: Cannot access memory at address 0x2084f0>, len = 659664900}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = -854181888}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = 0 avp = <optimized out> st = {flags = 509685512, id = 32621, name = {n = -1, s = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1453066304}, re = 0xffffffff}, avp = 0x433c55b8cd163800} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x0 rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = -854181888, s = {s = 0x433c55b8cd163800 <error: Cannot access memory at address 0x433c55b8cd163800>, len = 0}, re = 0x433c55b8cd163800}, pval = { rs = {s = 0x433c55b8cd163800 <error: Cannot access memory at address 0x433c55b8cd163800>, len = 0}, ri = 1453063376, flags = 32766}}, i2s = "@\b\234V\376\177\000\000\254D$'\212U\000\000\370G^\036m\177"} s = {s = 0x0, len = 0} srevp = {0x7f6d00000001, 0xb} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #23 0x0000558a2726b03a in run_actions (h=h@entry=0x7ffe569c0840, a=0x7f6d1e586060, msg=msg@entry=0x7f6d1e612f08) at core/action.c:1572 t = 0x7f6d1e589fa8 ret = <optimized out> tvb = {tv_sec = 140730351487040, tv_usec = 140106631389469} tve = {tv_sec = 140106635540856, tv_usec = 94051193847819} tz = {tz_minuteswest = 660077054, tz_dsttime = 21898} tdiff = <optimized out> __func__ = "run_actions" #24 0x0000558a2726dd52 in do_action (h=h@entry=0x7ffe569c0840, a=a@entry=0x7f6d1e5310c0, msg=msg@entry=0x7f6d1e612f08) at core/action.c:691 ret = <optimized out> v = <optimized out> dst = {send_sock = 0xfffffdff, to = {s = {sa_family = 7704, sa_data = "a\036m\177\000\000@\001\234V\376\177\000"}, sin = {sin_family = 7704, sin_port = 7777, sin_addr = {s_addr = 32621}, sin_zero = "@\001\234V\376\177\000"}, sin6 = {sin6_family = 7704, sin6_port = 7777, sin6_flowinfo = 32621, sin6_addr = {__in6_u = {__u6_addr8 = "@\001\234V\376\177\000\000\321\001!\036m\177\000", __u6_addr16 = {320, 22172, 32766, 0, 465, 7713, 32621, 0}, __u6_addr32 = {1453064512, 32766, 505479633, 32621}}}, sin6_scope_id = 1453064660}}, id = 32766, proto = 0 '\000', send_flags = {f = 0, blst_imask = 0}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1453066304}, passwd = {s = 0x2dd194de846ac81c <error: Cannot access memory at address 0x2dd194de846ac81c>, len = 413845532}, host = { --Type <RET> for more, q to quit, c to continue without paging-- s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 460773692}, port = {s = 0x0, len = 509489192}, params = {s = 0x7ffe569bfd98 "\365\002", len = 1}, sip_params = { s = 0x4000 <error: Cannot access memory at address 0x4000>, len = 1453064064}, headers = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 660077054}, port_no = 1664, proto = 22172, type = 32766, flags = (URI_SIP_USER_PHONE | unknown: 503080964), transport = {s = 0x0, len = 757}, ttl = {s = 0x1800000000 <error: Cannot access memory at address 0x1800000000>, len = -854181888}, user_param = {s = 0x7f6d17b73040 "", len = 509685512}, maddr = {s = 0x7f6d1e52a378 "\374\002", len = 660899552}, method = {s = 0x558a27528bc0 "corefunc", len = 660077054}, lr = {s = 0x7ffe569bff80 "", len = 656846906}, r2 = { s = 0x7ffe569c0680 "\320\024S\036m\177", len = 1453064064}, gr = {s = 0x7f6d1e52a378 "\374\002", len = 509685512}, transport_val = {s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 1453063944}, ttl_val = {s = 0x7ffe569c0840 "\003", len = -854181888}, user_param_val = {s = 0x0, len = 1453066304}, maddr_val = {s = 0x7f6d1e4edb90 "\001", len = 509685512}, method_val = { s = 0x7ffe569bfff4 "\001", len = -1}, lr_val = {s = 0x7ffe569c0840 "\003", len = 656722647}, r2_val = {s = 0x2dd194de8b8ac81c <error: Cannot access memory at address 0x2dd194de8b8ac81c>, len = 413845532}, gr_val = { s = 0x0, len = 1453064240}} next_hop = {user = {s = 0x7ffe569c09a0 "\r", len = 1453064240}, passwd = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 458310321}, host = {s = 0x0, len = 1453066656}, port = { s = 0x3 <error: Cannot access memory at address 0x3>, len = 460567720}, params = {s = 0x1 <error: Cannot access memory at address 0x1>, len = -854181888}, sip_params = { s = 0x8 <error: Cannot access memory at address 0x8>, len = 399956376}, headers = {s = 0x0, len = 509489192}, port_no = 757, proto = 0, type = ERROR_URI_T, flags = (unknown: 3440785408), transport = {s = 0x0, len = 1453064516}, ttl = {s = 0x7ffe569c0840 "\003", len = 656688300}, user_param = {s = 0x7ffe569c00b8 "\376\371W'\212U", len = -854181888}, maddr = {s = 0x7f6d17d4d2d8 "B", len = 1453066304}, method = {s = 0x0, len = 509685512}, lr = {s = 0x7ffe569c02b0 "\001", len = -1}, r2 = {s = 0x7ffe569c0840 "\003", len = 656903303}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x7ffe569c0840 "\003", len = 1925892124}, ttl_val = { s = 0x7f6d1e612f08 "\b", len = 1453064880}, user_param_val = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1453066304}, maddr_val = { s = 0x2dd194de892ac81c <error: Cannot access memory at address 0x2dd194de892ac81c>, len = 413845532}, method_val = {s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 657238994}, lr_val = { s = 0x7f6d1b853560 <wr_buf> "\027\003\003\003\r\352\374cwYd\354j\002\333\301\365\375\376\355\262\376\v\327SP\230\034g\036 [\221\017\233\255s\245\343\061\350\363\236"\a\033g4\377\347\234Y\364\224\273\235\177u\356>6\026\217\272\303ŕx\213\031+sư", len = 11}, r2_val = {s = 0x100000312 <error: Cannot access memory at address 0x100000312>, len = -854181888}, gr_val = {s = 0x7ffe00000000 <error: Cannot access memory at address 0x7ffe00000000>, len = 505912082}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = 32766 avp = <optimized out> st = {flags = 1453066656, id = 32766, name = {n = 1453064240, s = {s = 0x7ffe569c0030 "@\001\234V\376\177", len = 1}, re = 0x7ffe569c0030}, avp = 0x7f6d1b5142b1 <asn1_template_noexp_d2i+673>} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x558a27278c87 <run_actions_safe+71> rv1 = <optimized out> c1 = {cache_type = 1408, val_type = RV_NONE, c = {avp_val = {n = -854181888, s = {s = 0x433c55b8cd163800 <error: Cannot access memory at address 0x433c55b8cd163800>, len = 786}, re = 0x433c55b8cd163800}, pval = {rs = { s = 0x433c55b8cd163800 <error: Cannot access memory at address 0x433c55b8cd163800>, len = 786}, ri = 1453064880, flags = 32766}}, i2s = "@\b\234V\376\177\000\000\254D$'\212U\000\000K\225d'\212U"} s = {s = 0x0, len = 0} srevp = {0x7f6d00000000, 0x31200000200} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #25 0x0000558a2726b03a in run_actions (h=h@entry=0x7ffe569c0840, a=0x7f6d1e52c840, msg=msg@entry=0x7f6d1e612f08) at core/action.c:1572 t = 0x7f6d1e5310c0 ret = <optimized out> tvb = {tv_sec = 0, tv_usec = 0} tve = {tv_sec = 103079215104, tv_usec = 94051853924862} tz = {tz_minuteswest = 509685512, tz_dsttime = 32621} tdiff = <optimized out> __func__ = "run_actions" #26 0x0000558a2726ccb7 in do_action (h=h@entry=0x7ffe569c0840, a=a@entry=0x7f6d1e5314d0, msg=msg@entry=0x7f6d1e612f08) at core/action.c:1062 ret = <optimized out> v = <optimized out> dst = {send_sock = 0x7f6d17778944, to = {s = {sa_family = 60931, sa_data = "1'\212U\000\000\220\216\274\027m\177\000"}, sin = {sin_family = 60931, sin_port = 10033, sin_addr = {s_addr = 21898}, sin_zero = "\220\216\274\027m\177\000"}, sin6 = {sin6_family = 60931, sin6_port = 10033, sin6_flowinfo = 21898, sin6_addr = {__in6_u = {__u6_addr8 = "\220\216\274\027m\177\000\000\252\004\000\000\000\000\000", --Type <RET> for more, q to quit, c to continue without paging-- __u6_addr16 = {36496, 6076, 32621, 0, 1194, 0, 0, 0}, __u6_addr32 = {398233232, 32621, 1194, 0}}}, sin6_scope_id = 397958776}}, id = 32621, proto = 24 '\030', send_flags = {f = 7043, blst_imask = 32621}} tmp = <optimized out> new_uri = <optimized out> end = <optimized out> crt = <optimized out> cmd = <optimized out> len = <optimized out> user = <optimized out> uri = {user = {s = 0x7f6d17d70a08 <incomplete sequence \310>, len = 1453065648}, passwd = {s = 0x7f6d17778944 "\331\061", len = 657583619}, host = {s = 0x7ffe569c0570 "\b\006\234V\376\177", len = 1453065392}, port = { s = 0x8 <error: Cannot access memory at address 0x8>, len = 458310321}, params = {s = 0x0, len = 507534400}, sip_params = {s = 0xd <error: Cannot access memory at address 0xd>, len = 13}, headers = { s = 0xd <error: Cannot access memory at address 0xd>, len = 0}, port_no = 35140, proto = 6007, type = 32621, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 657583616), transport = {s = 0x7f6d17bc88cc "", len = 398234426}, ttl = {s = 0x125 <error: Cannot access memory at address 0x125>, len = 462262422}, user_param = {s = 0x2000000 <error: Cannot access memory at address 0x2000000>, len = -854181888}, maddr = { s = 0x206cf8 <error: Cannot access memory at address 0x206cf8>, len = 1453065856}, method = {s = 0x0, len = 509310696}, lr = {s = 0x7f6d1e612f08 "\b", len = 399969104}, r2 = {s = 0x7f6d1e612f08 "\b", len = 656903465}, gr = {s = 0x7ffe569c05c0 "", len = 502611623}, transport_val = {s = 0x7f6d1df6b1e0 <__func__.5354> "dbt_init", len = 502702009}, ttl_val = {s = 0xd <error: Cannot access memory at address 0xd>, len = 13}, user_param_val = {s = 0xd <error: Cannot access memory at address 0xd>, len = 509696800}, maddr_val = {s = 0x7f6d1e41eb90 "text:///etc/kamailio/lb/db", len = 13}, method_val = { s = 0xd <error: Cannot access memory at address 0xd>, len = -854181888}, lr_val = {s = 0x7f6d17778944 "\331\061", len = 505945944}, r2_val = {s = 0x7f6d1b72ee00 <RSAPrivateKey_it> "\001", len = 507570360}, gr_val = { s = 0xd <error: Cannot access memory at address 0xd>, len = 13}} next_hop = {user = {s = 0x7f6d1e41eb97 "/etc/kamailio/lb/db", len = 19}, passwd = {s = 0x7f6d17778944 "\331\061", len = 657583381}, host = {s = 0x7ffe569c05c0 "", len = 505953515}, port = { s = 0x38 <error: Cannot access memory at address 0x38>, len = 506177921}, params = {s = 0x1 <error: Cannot access memory at address 0x1>, len = 1453065648}, sip_params = { s = 0x4 <error: Cannot access memory at address 0x4>, len = 399968568}, headers = {s = 0xc8 <error: Cannot access memory at address 0xc8>, len = 399969104}, port_no = 12040, proto = 7777, type = 32621, flags = (URI_SIP_USER_PHONE | unknown: 506222992), transport = {s = 0x7f6d00000064 <error: Cannot access memory at address 0x7f6d00000064>, len = 461570049}, ttl = {s = 0x558a276b5180 <def_list> "", len = 661344648}, user_param = {s = 0x558a276b5190 <def_list+16> "", len = 661344664}, maddr = {s = 0x558a276b51a0 <def_list+32> "", len = 660900736}, method = {s = 0xd <error: Cannot access memory at address 0xd>, len = 0}, lr = { s = 0x7f6d17778944 "\331\061", len = 657583381}, r2 = {s = 0x0, len = 0}, gr = {s = 0x7f6d17bc8778 "\006", len = 398231752}, transport_val = {s = 0x7ffe569c0608 "\003\356\061'\212U", len = 507578304}, ttl_val = { s = 0xd <error: Cannot access memory at address 0xd>, len = -854181888}, user_param_val = {s = 0xd <error: Cannot access memory at address 0xd>, len = 507579240}, maddr_val = { s = 0xd <error: Cannot access memory at address 0xd>, len = 13}, method_val = {s = 0x0, len = 200}, lr_val = {s = 0x0, len = 502316172}, r2_val = {s = 0x0, len = 507579240}, gr_val = { s = 0xd <error: Cannot access memory at address 0xd>, len = 13}} u = <optimized out> port = <optimized out> dst_host = <optimized out> i = <optimized out> flags = 21898 avp = <optimized out> st = {flags = 507636631, id = 32621, name = {n = 19, s = {s = 0x13 <error: Cannot access memory at address 0x13>, len = 393709892}, re = 0x13}, avp = 0x558a2731ed15 <init_mod_child+37>} sct = <optimized out> sjt = <optimized out> rve = <optimized out> mct = <optimized out> rv = 0x7f6d1e2c03f4 <relay_reply+1188> rv1 = <optimized out> c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 507580400, s = {s = 0x7f6d1e410ff0 "P\017A\036m\177", len = 13}, re = 0x7f6d1e410ff0}, pval = {rs = {s = 0x7f6d1e410ff0 "P\017A\036m\177", len = 13}, ri = 13, flags = 0}}, i2s = "\r\000\000\000\000\000\000\000\000\070\026\315\270U<CD\211w\027m\177"} s = {s = 0x4 <error: Cannot access memory at address 0x4>, len = 0} srevp = {0xd, 0x0} evp = {data = 0x0, rcv = 0x0, dst = 0x0} mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}} __func__ = "do_action" #27 0x0000558a2726b03a in run_actions (h=h@entry=0x7ffe569c0840, a=a@entry=0x7f6d1e429d68, msg=msg@entry=0x7f6d1e612f08) at core/action.c:1572 t = 0x7f6d1e5314d0 ret = <optimized out> tvb = {tv_sec = 7, tv_usec = 94051854751053} tve = {tv_sec = 140106637644632, tv_usec = 94051854751029} tz = {tz_minuteswest = 660903222, tz_dsttime = 21898} --Type <RET> for more, q to quit, c to continue without paging-- tdiff = <optimized out> __func__ = "run_actions" #28 0x0000558a27278d1f in run_top_route (a=0x7f6d1e429d68, msg=msg@entry=0x7f6d1e612f08, c=c@entry=0x0) at core/action.c:1657 ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140730351487040, -3301919276976650212, 140106635844968, 140106637848328, 0, 140106637859768, 3301583629350258716, 8735144498933975068}, __mask_was_saved = 0, __saved_mask = {__val = {4844841551346939904, 2127816, 140106637860800, 140106637848328, 4844841551346939904, 94051854751053, 1, 1073741825, 140106634741928, 1, 140106634416053, 140106637859768, 94051851331334, 140730351487296, 140730351697530, 2125960}}}}} p = 0x0 ret = <optimized out> sfbk = 0 #29 0x0000558a27356d0f in receive_msg ( buf=buf@entry=0x558a27649380 <buf> "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3855084\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:195.128.146.194:47626%3Btransport%3Dtl"..., len=<optimized out>, rcv_info=rcv_info@entry=0x7ffe569c0bb0) at core/receive.c:341 msg = 0x7f6d1e612f08 ctx = {rec_lev = 0, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140730351487504, -3301919276976650212, 140106637936224, 140106637848328, 0, 0, 3301583629350258716, 8735144498933975068}, __mask_was_saved = 0, __saved_mask = {__val = {140106590425238, 33554432, 30656312, 2125080, 2898120, 3750576, 8, 56, 4844841551346939904, 140106635091984, 94051852694691, 13, 94051853512708, 94051853589272, 94051853583827, 140106521872708}}}}} bctx = <optimized out> ret = <optimized out> tvb = {tv_sec = 13, tv_usec = 13} tve = {tv_sec = 13, tv_usec = 13} tz = {tz_minuteswest = 461944123, tz_dsttime = 32621} diff = 0 inb = { s = 0x558a27649380 <buf> "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3855084\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:195.128.146.194:47626%3Btransport%3Dtl"..., len = 461} netinfo = {data = {s = 0x393ab0 <error: Cannot access memory at address 0x393ab0>, len = 8}, rcv = 0x38, dst = 0x433c55b8cd163800} keng = 0x0 evp = {data = 0x7ffe569c09c0, rcv = 0x7ffe569c0bb0, dst = 0x0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 __func__ = "receive_msg" #30 0x0000558a2728a7f5 in udp_rcv_loop () at core/udp_server.c:541 len = 461 buf = "OPTIONS sip:1001@192.168.101.38:47626;transport=tls SIP/2.0\r\nVia: SIP/2.0/UDP 10.10.91.23:5062;branch=z9hG4bK3855084\r\nRoute: <sip:lb@10.10.91.103;lr;received=sip:195.128.146.194:47626%3Btransport%3Dtl"... tmp = <optimized out> from = <optimized out> fromlen = 16 ri = {src_ip = {af = 2, len = 4, u = {addrl = {4844841548297996810, 140730351488000}, addr32 = {391842314, 1128027576, 1453067264, 32766}, addr16 = {2570, 5979, 21944, 17212, 3072, 22172, 32766, 0}, addr = "\n\n[\027\270U<C\000\f\234V\376\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {1734019594, 0}, addr32 = {1734019594, 0, 0, 0}, addr16 = {2570, 26459, 0, 0, 0, 0, 0, 0}, addr = "\n\n[g", '\000' <repeats 11 times>}}, src_port = 5062, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\306\n\n[\027\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50707, sin_addr = {s_addr = 391842314}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50707, sin6_flowinfo = 391842314, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, bind_address = 0x7f6d1e4025a0, proto = 1 '\001'} evp = {data = 0x0, rcv = 0x0, dst = 0x0} printbuf = "\200\016\234V\376\177\000\000\253\016\234V\376\177\000\000\377\016\234V\376\177\000\000ؿA\036m\177\000\000\r\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\r", '\000' <repeats 15 times>, "D\211w\027m\177\000\000\003\356\061'\212U\000\000\000\000\000\000\004", '\000' <repeats 27 times>, "\223\200R'\212U\000\000\230\316A\036m\177\000\000\r\000\000\000\000\000\000\000\r\000\000\000\000\000\000\000\r", '\000' <repeats 15 times>, "D\211w\027m\177\000\000\025\355\061'\212U\000\000\002\000\000\000\000\000\000\000\240\337R m\177\000\000\000\000\000\000\000\000\000\000\220"... i = <optimized out> j = <optimized out> l = <optimized out> __func__ = "udp_rcv_loop" __llevel = <optimized out> #31 0x0000558a2722deec in main_loop () at main.c:1669 i = 0 pid = <optimized out> si = 0x7f6d1e4025a0 --Type <RET> for more, q to quit, c to continue without paging-- si_desc = "udp receiver child=4 sock=10.10.91.103:5060\000\001\000\000\000\350YT'\212U\000\000\000\000\000\000\000\000\000\000N\000 \000\000\000\000\000$\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\000\070\026\315\270U<C\000P\203\033m\177\000\000\000\070\026\315\270U<Ch\271q'\212U\000\000\001\000\000\000\000\000\000" nrprocs = <optimized out> woneinit = <optimized out> __func__ = "main_loop" #32 0x0000558a27225415 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2710 cfg_stream = <optimized out> c = <optimized out> r = <optimized out> tmp = 0x7ffe569c2ea6 "" tmp_len = 1453068224 port = 32766 proto = 1453068240 options = 0x558a2751d788 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2451291985 rfd = <optimized out> debug_save = <optimized out> debug_flag = <optimized out> dont_fork_cnt = <optimized out> n_lst = <optimized out> p = <optimized out> st = {st_dev = 21, st_ino = 131708, st_nlink = 2, st_mode = 16872, st_uid = 113, st_gid = 118, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1569430338, tv_nsec = 687617842}, st_mtim = {tv_sec = 1569855057, tv_nsec = 674161167}, st_ctim = {tv_sec = 1569855057, tv_nsec = 674161167}, __glibc_reserved = {0, 0, 0}} __func__ = "main" (gdb) quit`
So it is during tls connect, no much done by kamailio there (I mean, no content specifically set by kamailio for sending out). Can you list the code at frames 0 and 1?
``` frame 0 list frame 1 list ```
Is it easy to reproduce? Or, eventually any convenient way to get access to the system that exposes the crash so I can investigate the core file?
Here the the code you required:
``` (gdb) frame 0 #0 aesni_ecb_encrypt () at crypto/aes/aesni-x86_64.s:524 524 in crypto/aes/aesni-x86_64.s (gdb) list 519 in crypto/aes/aesni-x86_64.s
(gdb) frame 1 #1 0x00007f6d1b5d5917 in aesni_ecb_cipher (len=16, in=0x7f6d17780fc0 "\036\234\364\354\062\311\005\316\036۫\023\065N\235\305l\001\306-", out=0x7f6d17d97aa8 "", ctx=0x7f6d177810a8) at ../crypto/evp/e_aes.c:319 319 ../crypto/evp/e_aes.c: No such file or directory. (gdb) list 314 in ../crypto/evp/e_aes.c (gdb) ```
The system is not accessible by outside. I'm checking if I can find a patter to reproduce the problem on other systems different from that particular one.
Thanks
Can you install source code packages for libssl/libcrypto? The list commands above don't show the code that ended up crashing.
here the result with the source code packages. I hope I did it right.
``` (gdb) frame 0 #0 aesni_ecb_encrypt () at crypto/aes/aesni-x86_64.s:524 warning: Source file is more recent than executable. 524 movl 240(%rcx),%eax (gdb) list 519 aesni_ecb_encrypt: 520 .cfi_startproc 521 andq $-16,%rdx 522 jz .Lecb_ret 523 524 movl 240(%rcx),%eax 525 movups (%rcx),%xmm0 526 movq %rcx,%r11 527 movl %eax,%r10d 528 testl %r8d,%r8d (gdb) frame 1 #1 0x00007f6d1b5d5917 in aesni_ecb_cipher (len=16, in=0x7f6d17780fc0 "\036\234\364\354\062\311\005\316\036۫\023\065N\235\305l\001\306-", out=0x7f6d17d97aa8 "", ctx=0x7f6d177810a8) at ../crypto/evp/e_aes.c:319 319 aesni_ecb_encrypt(in, out, len, &EVP_C_DATA(EVP_AES_KEY,ctx)->ks.ks, (gdb) list 314 size_t bl = EVP_CIPHER_CTX_block_size(ctx); 315 316 if (len < bl) 317 return 1; 318 319 aesni_ecb_encrypt(in, out, len, &EVP_C_DATA(EVP_AES_KEY,ctx)->ks.ks, 320 EVP_CIPHER_CTX_encrypting(ctx)); 321 322 return 1; 323 } ```
The crash seems to be in the default random bytes generator shipped with libssl 1.1.x, which can be caused by the changes for thread-safe-only-approach of libssl 1.1.x.
I pushed a commit in master branch to be able to set a different random bytes generator - a basic one for now, but should help to confirm if that is the issue.
Can you test with master branch? You must set the following modparam for tls module:
modparam("tls", "rand_engine", "krand")
The parameter is not yet documented in the readme of tls, the above is the only valid option to set a custom random bytes generator.
Good work Daniel, I hope that this OpenSSL issues stops now soon.. One suggestion about the random number generator - I think the core/fastrand/ one has a better quality than the one provided from glibc (kam_rand, which points to rand). If this fix is confirmed, I would suggest to use this one for now.
I looked already quite a bit into this random number generator topic. Most servers and also OS use the "Fortuna" PRNG. I was thinking of using fastrand for all "normal" random numbers (to replace the one from glibc) and then using fortuna for "secure" random numbers. I planned to work on this when 5.3. has been released.
Thank Daniel for you quick fix.
I backported the commit to my 5.2 branch because I cannot use the master one. I deployed it on the affected system and at the moment we didn't noticed any new crash.
I'll leave the system running for a couple of days and I'll keep you updated about the status.
@marcocapetta - thanks for reporting back, it is good sign for the moment that it was fixed if it is no longer crashing as fast as it happened before. Let's see how it goes further.
I will try to see if the default number generator for libssl 1.1 can be reused protected by an additional mutex.
@henningw - fastrand can be used here, either in place of kam_rand() or as an additional option -- kamrand was the quick variant I coded to see if goes around the crash. If you find a BSD-licensed Fortuna-PRNG implementation, it can be added even now in the tls module, this is part of the work on fixing a bug. It is not really a new feature because we have to keep tls as secure as possible with this custom random bytes generator.
Ok, I found one suitable implementation with MIT license, which is liberal (comparable to BSD) to be included in the core. I will strip it to the necessary parts and include it for now in lib/srutils (as it has already some other hash functions). After opening the master branch again it makes probably sense to split it into an own library (e.g. srcrypto). This could be also used for example from other modules like rtpengine, to get rid of the crypto/openssl dependency (for SHA1 hashing) again.
Maybe you can just add it in tls module right now, could be inside tls_rand.{c,h}, which are compiled only for libssl 1.1. After 5.3 is released, it can be moved in the core or other place.
Regarding the internal libraries: they are not that much used/active (from the perspective of new code), they also seem to create some problems on some packaging systems (i.e., different packages with various modules need same internal lib). We probably should revise what to do with them: package them separately so they are a clear dependency or figure out something else.
Anyhow, right now I won't add a dependency of an internal library for tls module.
Well, unfortunately I need the SHA265 in srutils - of course I can also copy it to tls module. This would be about seven files to add to the module.
About the general library question - I see three options: - either package them as dedicated dependencies - or move them to the core if they are used from at least two modules - or place them in the module if just used from one module
Sorry, I haven't read all of this, but I know in FreeSWITCH, openSSL had to be compiled in: https://freeswitch.org/jira/browse/FS-12000
@henningw - it seems to be quite a lot of code for that random generator. Eventually it can be added as a subfolder in tls module, with sha256 included there. app_sqlang module has also code in a subfolder. Maybe this should be discussed on a separate issue or pull request, not to divert the discussion on this one from the real subject.
@jungle-boogie - importing libssl and patching it should be the last solution, because it will add a lot of maintenance overhead to take care of sync'ing with upstream for security patches, new features, etc...
@miconda - the core implementation is not that big, but it needs SHA256 and AES128 as building blocks. I think I will just create a subfolder in core for now for the crypto code, and then add it to the existing rand folder. Then we can discuss on the list or in another issue how to integrate it.
@henningw - so it includes the code for full implementation of AES128 as well? Maybe it should be placed in crypto module to use libcrypto for AES128 and register a callback to core in a similar way that is done for generating call-id with better randomness.
We have already MD5 in the core, and SHA256 in the lib srutils. I also plan to use SHA256 instead of MD5 in different places and modules. Therefore I think it is better to have it in the core as well.
Moving sha256 in the core is fine, my remark was about fortuna-prng. If there are many files I would look for a solution with a module that register to some core API that we can define for PRNG functions. In this way, other PRNG engines can be added in the future, even when they have external dependencies. A similar approach was done also for dnssec.
Anyhow, for 5.3 I would do a self-contained PRNG solution for tls. Relocating sha256 code will require touching other parts of code, so I would avoid it. Or if is too complex, then we can skip it and use existing random variants.
Everything should be now commited to core git master branch. For the master TLS module it should be probably work easily with the API in core/rand/fortuna/random.c - but I will look into that.
For 5.3 branch I will tomorrow collect the necessary files and add them to a sub-folder in the TLS module.
@minconda - I have added the local copy to 5.3 branch as well. Maybe you can have a look to it as well. There is probably one or two things that needs to be fixed related to the Makefile (I've compiled it as a static lib) and it also needs to be included in the tls_rand.* implementation.
From Thursday afternoon, the platform where I installed the latest fixes about the rend_engine looks stable.
I think we can confirm that the issue is fixed.
Thank Daniel and Henning for all your support.
From Thursday afternoon, the platform where I installed the latest fixes about the rend_engine looks stable.
I think we can confirm that the issue is fixed.
Thank Daniel and Henning for all your support.
@marcocapetta - thank you for the confirmation, great to hear that. The fix you've backported is not recommended for production use. I will keep the issue open until the new safe random number generator has been added for 5.3 as well.
Closed #2077.
@marcocapetta - ok, good to hear it was working. I will try to add a variant for using the internal randon number generator guarded by some mutexes, would you be able to test it? I will write you once is done.
@henningw - there is no reason to wait for backporting, because the reported issue was identified and fixed.
Anyhow, probably its' time to release a new version in 5.2.x series soon, including the fix.
@miconda - I've just pushed the cryptorand version to git master. For 5.3 branch this is not finished yet.
@henningw - ok, new random generators are going to be done to 5.3 and then backported to 5.2 and 5.1.
But they are a different matter than the crash reported here and discussion here is too long to follow it over the time, if the issue needs to be revised for various purposes. If you really want to discuss more about PRNGs, better use sr-dev or create a dedicated issue.
I think it is all fine - I have one question regarding your implementation about the wrapper functions - just write you a quick e-mail about it.