[kamailio/kamailio] Kamailio 5.6.4 crashing when using http_connect with libcrypto.so.1.1 on bullseye (Issue #3539)
14 Aug
2023
14 Aug
'23
2:47 p.m.
### Description
Hi,
we are experiencing Kamailio 5.6.4 Loadbalancers nodes crashing from time to time (about
10 crashes in last 6 weeks) in our production environment. The nodes are running on Debian
Bullseye on virtual machines. The Core Dumps written by Kamailio are always pointing to a
curl request (http_connect function) we are performing for each call to retrieve necessary
data from external node. The curl request is quite simple, we are sending a requst to
something like https://example.com/<SIPusername> and are only interested in the
result code (200 or 404) for our next decisions in kamailio configuration.
Some month ago after introducing the curl request into our kamailio configuration we were
experiencing several crashes per node and day and found the hint to start kamailio with
extra startup parameter --atexit=no. This improved situation a lot, but problem still
occurs on lower frequency.
Maybe interesting for this issue: For historical reasons we are using different kamailio
nodes for our customers using SIP via UDP and TCP. Both Kamailios used for UDP and TCP SIP
Traffic are running same Kamailio version and nearly identical Kamailio configuration.
Only TCP Kamailios are affected. But of course it is not clear if TCP usage in SIP message
from customer is the main reason why it chrashes or maybe one / or some users are
triggering the problem due to special device behavior?
We have seen similar issues for openssl3 being resolved with
https://github.com/kamailio/kamailio/pull/3482, but here we are still using openssl
version 1.1.1n on Bullseye.
Below anonymized output from gdb (IPs, URLs, usernames removed):
### Troubleshooting
#### Reproduction
#### Debugging Data
```
# gdb /usr/sbin/kamailio /var/cores/core_kamailio_1357620_11_111_1691754786
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
Reading symbols from
/usr/lib/debug/.build-id/2f/4a56865957849818b0f870fc3f29d01216a3e3.debug...
warning: Can't open file /dev/zero (deleted) during file-backed mapping note
processing
[New LWP 1357620]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio_sip_lb_tcp_de.pid
-f /etc/kama'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f3ccd37903a in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
(gdb) bt full
#0 0x00007f3ccd37903a in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#1 0x00007f3ccd37922c in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#2 0x00007f3ccd379b49 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#3 0x00007f3ccd3794d7 in ASN1_item_ex_i2d () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#4 0x00007f3ccd379bb6 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#5 0x00007f3ccd4dd43c in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#6 0x00007f3ccd4dd91d in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#7 0x00007f3ccd4ddef7 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#8 0x00007f3ccd37818c in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#9 0x00007f3ccd378b56 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#10 0x00007f3ccd378dc1 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#11 0x00007f3ccd378468 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#12 0x00007f3ccd378b56 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#13 0x00007f3ccd378dc1 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#14 0x00007f3ccd378468 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#15 0x00007f3ccd378eee in ASN1_item_ex_d2i () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#16 0x00007f3ccd378f6b in ASN1_item_d2i () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#17 0x00007f3ccd46cbac in PEM_X509_INFO_read_bio () from
/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#18 0x00007f3ccd4ce093 in X509_load_cert_crl_file () from
/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#19 0x00007f3ccd4ce1e2 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#20 0x00007f3ccd4d0faf in X509_STORE_load_locations () from
/lib/x86_64-linux-gnu/libcrypto.so.1.1
No symbol table info available.
#21 0x00007f3c4c526893 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#22 0x00007f3c4c527ddf in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#23 0x00007f3c4c528ea1 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#24 0x00007f3c4c4e8805 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#25 0x00007f3c4c4fe0c9 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#26 0x00007f3c4c4ff286 in curl_multi_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
No symbol table info available.
#27 0x00007f3c4c4d5a93 in curl_easy_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
--Type <RET> for more, q to quit, c to continue without paging--c
No symbol table info available.
#28 0x00007f3c4c56ce92 in curL_request_url (_m=0x7f3ccfb31208, _met=0x0,
_url=0x7f3ccf9f4c90 "https://<someurl>/<username>",
_dst=0x7fff5ab37eb0, params=0x7fff5ab37d50) at functions.c:274
totaltime = 1.657809211691619e-316
connecttime = 5.2824500700663664e-290
curl = 0x5579509e1210
res = CURLE_OK
at = 0x0
stream = {buf = 0x0, curr_size = 0, pos = 0, max_size = 0}
stat = 0
rval = {s = 0x0, len = 0}
download_size = 0
headerlist = 0x0
__func__ = "curL_request_url"
#29 0x00007f3c4c577964 in curl_con_query_url_f (_m=0x7f3ccfb31208,
connection=0x7fff5ab37fb0, url=0x7fff5ab37fc0, result=0x7fff5ab37eb0, contenttype=0x0,
post=0x0, failover=0) at functions.c:601
conn = 0x7f3c4d2c2bc8
pconn = 0x7f3ccf8dafd8
urlbuf = 0x7f3ccf9f4c90 "https://<someurl>/<username>"
postdata = 0x0
failovercon = 0x0
query_params = {username = 0x0, secret = 0x0, contenttype = 0x7f3c4c5949cb
"text/plain", post = 0x0, clientcert = 0x0, clientkey = 0x0, cacert = 0x0,
ciphersuites = 0x0, http_proxy = 0x0, failovercon = 0x0, useragent = 0x7f3ccf8db468
"kamailio (5.6.4 (x86_64/linux))", hdrs = 0x0, netinterface = 0x0, authmethod =
3, http_proxy_port = 0, tlsversion = 0, verify_peer = 1, verify_host = 2, timeout = 4,
http_follow_redirect = 0, oneline = 0, maxdatasize = 0, keep_connections = 0, pconn =
0x7f3ccf8dafd8}
maxdatasize = 0
res = 32767
__func__ = "curl_con_query_url_f"
#30 0x00007f3c4c578291 in curl_con_query_url (_m=0x7f3ccfb31208,
connection=0x7fff5ab37fb0, url=0x7fff5ab37fc0, result=0x7fff5ab37eb0, contenttype=0x0,
post=0x0) at functions.c:633
No locals.
#31 0x00007f3c4c582a6e in ki_curl_connect_helper (_m=0x7f3ccfb31208, con=0x7fff5ab37fb0,
url=0x7fff5ab37fc0, dst=0x7f3ccf99c558) at http_client.c:614
result = {s = 0x0, len = 0}
val = {rs = {s = 0x7fff5ab38020 "P\200\263Z\377\177", len = 0}, ri =
944, flags = 32572}
ret = 0
__func__ = "ki_curl_connect_helper"
#32 0x00007f3c4c584257 in w_curl_connect (_m=0x7f3ccfb31208, _con=0x7f3ccf9aa750
"http_endpoint_primary", _url=0x7f3ccf99c410 "\260Ù\317<\177",
_result=0x7f3ccf99c558 "\004") at http_client.c:674
con = {s = 0x7f3ccf9aa750 "http_endpoint_primary", len = 12}
url = {s = 0x7f3ccf865060 "/routing/<username>", len = 22}
dst = 0x7f3ccf99c558
__func__ = "w_curl_connect"
#33 0x000055794f7c3522 in do_action (h=0x7fff5ab388c0, a=0x7f3ccf9ad758,
msg=0x7f3ccfb31208) at core/action.c:1097
ret = -5
v = 0
dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data =
"\000\000\000\000\000\000`\206\263Z\377\177\000"}, sin = {sin_family = 0,
sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "`\206\263Z\377\177\000"},
sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u =
{__u6_addr8 = "`\206\263Z\377\177\000\000(\361\232\317<\177\000", __u6_addr16
= {34400, 23219, 32767, 0, 61736, 53146, 32572, 0}, __u6_addr32 = {1521714784, 32767,
3483037992, 32572}}}, sin6_scope_id = 3484619272}, sas = {ss_family = 0, __ss_padding =
"\000\000\000\000\000\000`\206\263Z\377\177\000\000(\361\232\317<\177\000\000\b\022\263\317<\177\000\000д\263Z\377\177\000\000\000\000\000\000\n\000\000\000p\365\244\321<\177\000\000(q\247\317<\177\000\000\t\004\000\000\000\000\000\000PN\270`<\177\000\000\t\004\000\000\000\000\000\000\060\067\200[<\177\000\000\000\000\000\000\020\000\000\000\200\207\263Z\377\177\000\000\345;\251OyU\000",
__ss_align = 139899159386304}}, id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0
'\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
tmp = 0x55794fabae7c <fm_info+46>
"H\213E\370H\213P\bH\213E\340H\211\020H\213E\340H\307@(\b"
new_uri = 0x7f3c4c74b3d4 <pkg_proc_update_stats+130>
"H\213\r\025s"
end = 0x7f3c5b803728 "\310\030\213\317<\177"
crt = 0x7fff5ab38210 "p\202\263Z\377\177"
cmd = 0x7f3ccf8d0038
len = -811585760
user = 32572
uri = {user = {s = 0x8b900000005 <error: Cannot access memory at address
0x8b900000005>, len = 1338089616}, passwd = {s = 0x55794fc17ec1 "core:
core/msg_translator.c", len = 1040}, host = {s = 0x7f3c60b84e08 "@\004",
len = 1294704640}, port = {s = 0x400000030 <error: Cannot access memory at address
0x400000030>, len = 1338365855}, params = {s = 0x5ab38400 <error: Cannot access
memory at address 0x5ab38400>, len = -1277437696}, sip_params = {s = 0x7fff5ab38400
"", len = 1336626667}, headers = {s = 0x55794fc1786b "core", len =
-810346320}, port_no = 41104, proto = 20417, type = 21881, flags = (URI_USER_NORMALIZE |
unknown: 0x4fc17ec0), transport = {s = 0x410 <error: Cannot access memory at address
0x410>, len = 1294704640}, ttl = {s = 0x8800000000 <error: Cannot access memory at
address 0x8800000000>, len = -1277437696}, user_param = {s = 0x0, len = 1622691336},
maddr = {s = 0x7f3c60b84e08 "@\004", len = 1294704640}, method = {s = 0x0, len =
0}, lr = {s = 0x7f3ccf8b18f8 "耱\317<\177", len = -812967672}, r2 = {s =
0x7f3ccf8b1240 "\300\227\261\317<\177", len = -812969392}, gr = {s =
0x7f3ccf8b18c8 "\006", len = 1336651388}, transport_val = {s = 0x7fff5ab38400
"", len = -813359088}, ttl_val = {s = 0x7fff5ab38400 "", len =
-813359088}, user_param_val = {s = 0x7fff5ab38440 "\320/\240\317<\177", len =
1282716628}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x2000000 <error: Cannot
access memory at address 0x2000000>, len = 28927392}, lr_val = {s = 0x3fad18 <error:
Cannot access memory at address 0x3fad18>, len = 4627040}, r2_val = {s = 0x7f3ccfa03028
"\270", len = -813359088}, gr_val = {s = 0x65 <error: Cannot access memory at
address 0x65>, len = -1277437696}}
next_hop = {user = {s = 0x7f3ccfa02fd0 "\020\001", len = -813359088},
passwd = {s = 0x7f3ccfa02fd0 "\020\001", len = -813359088}, host = {s = 0x0, len
= 1336651388}, port = {s = 0x7fff5ab384b0 "", len = -813359088}, params = {s =
0xb00000018 <error: Cannot access memory at address 0xb00000018>, len = -813359088},
sip_params = {s = 0x7fff5ab384f0 "\340p\247\317<\177", len = 1282716628},
headers = {s = 0x55794fc033a8 <__func__.7> "free_lump", len = 0}, port_no
= 0, proto = 512, type = ERROR_URI_T, flags = (unknown: 0x1b969a0), transport = {s =
0x3faac8 <error: Cannot access memory at address 0x3faac8>, len = 4626016}, ttl = {s
= 0x7f3ccfa77160 "8", len = -813359088}, user_param = {s = 0x6a <error:
Cannot access memory at address 0x6a>, len = -1277437696}, maddr = {s = 0x7f3ccfa770e0
"\270", len = -813359088}, method = {s = 0x7f3ccfa770e0 "\270", len =
-813359088}, lr = {s = 0x7f3ccfa75100 "\270", len = 0}, r2 = {s = 0x7fff5ab38620
"`\206\263Z\377\177", len = 1336633822}, gr = {s = 0x0, len = -1277437696},
transport_val = {s = 0x55794fc02ce0 "core", len = -810346320}, ttl_val = {s =
0x55794fc03910 <__func__.1> "free_duped_lump_list", len = 1337994469},
user_param_val = {s = 0x7f3ccfa77128 "", len = -813359088}, maddr_val = {s =
0x7fff5ab38620 "`\206\263Z\377\177", len = 1333836165}, method_val = {s = 0x0,
len = -811585168}, lr_val = {s = 0x7f3ccf852010 "\001", len = -811110176},
r2_val = {s = 0x46da90 <error: Cannot access memory at address 0x46da90>, len =
-811119128}, gr_val = {s = 0x7fff5ab3861c "", len = 1521714720}}
u = 0x7f3ccf852010
port = 0
dst_host = 0x7fff5ab38550
i = -811585760
flags = 0
avp = 0x7fff5ab38220
st = {flags = 0, id = 0, name = {n = 1338365855, s = {s = 0x54fc5d79f <error:
Cannot access memory at address 0x54fc5d79f>, len = 1521714064}, re = 0x54fc5d79f}, avp
= 0x3e0792b00000000}
sct = 0x55794f8cea0d <process_lumps+49737>
sjt = 0x7fff5ab38270
rve = 0x7f3ccf852010
mct = 0x7fff5ab38260
rv = 0x2000000
rv1 = 0x55794fabae7c <fm_info+46>
c1 = {cache_type = 1521714736, val_type = 32767, c = {avp_val = {n = 1334728331, s
= {s = 0x55794f8e568b <branch_builder+358> "\203\370\377u\020\353\a\270",
len = 1521714736}, re = 0x55794f8e568b <branch_builder+358>}, pval = {rs = {s =
0x55794f8e568b <branch_builder+358> "\203\370\377u\020\353\a\270", len =
1521714736}, ri = -1277437696, flags = 65042731}}, i2s =
"0\030\263\317<\177\000\000\260\030\263\317<\177\000\000\000\000\000\000\000"}
s = {s = 0x7fff5ab3848c "<\177", len = 1622691408}
srevp = {0x55794f79ac10 <_start>, 0xf000000000000}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#34 0x000055794f7d2391 in run_actions (h=0x7fff5ab388c0, a=0x7f3ccf9ad758,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccf9ad758
ret = -1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = -810346320, tz_dsttime = 32572}
tdiff = 3017529600
__func__ = "run_actions"
#35 0x000055794f7d2aff in run_actions_safe (h=0x7fff5ab3b4d0, a=0x7f3ccf9ad758,
msg=0x7f3ccfb31208) at core/action.c:1648
ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf =
{12884901888, -6152238486944205147, 93979512777744, 0, 0, 0, -6152238486866610523,
-30698175615977819}, __mask_was_saved = 0, __saved_mask = {__val = {139896960643982, 1,
7779588272, 139899158388928, 140734715104520, 139899157788368, 139899159384584,
140734715103616, 139899159384584, 139897209894648, 139896969507664, 0, 139899158389000,
140734715103808, 93979515819061, 0}}}}}
ret = 32572
ign_on_break = 0
#36 0x000055794f8a1329 in lval_pvar_assign (h=0x7fff5ab3b4d0, msg=0x7f3ccfb31208,
lv=0x7f3ccf9ad620, rv=0x7f3ccf9af130) at core/lvalue.c:284
pvar = 0x7f3ccf9ad490
pval = {rs = {s = 0x0, len = 0}, ri = 0, flags = 24}
r_avp = 0x7f3ccf852010
avp_val = {n = 1521715904, s = {s = 0x7fff5ab38ac0 "\370\355\302OyU",
len = -813359088}, re = 0x7fff5ab38ac0}
ret = 0
v = 3
destroy_pval = 0
__func__ = "lval_pvar_assign"
#37 0x000055794f8a28f9 in lval_assign (h=0x7fff5ab3b4d0, msg=0x7f3ccfb31208,
lv=0x7f3ccf9ad620, rve=0x7f3ccf9af128) at core/lvalue.c:400
rv = 0x7f3ccf9af130
ret = -1
__func__ = "lval_assign"
#38 0x000055794f7d01e3 in do_action (h=0x7fff5ab3b4d0, a=0x7f3ccf9af838,
msg=0x7f3ccfb31208) at core/action.c:1458
ret = -5
v = 0
dst = {send_sock = 0x7fff5ab395a0, to = {s = {sa_family = 46768, sa_data =
"\232\317<\177\000\000\210\276\232\317<\177\000"}, sin = {sin_family =
46768, sin_port = 53146, sin_addr = {s_addr = 32572}, sin_zero =
"\210\276\232\317<\177\000"}, sin6 = {sin6_family = 46768, sin6_port = 53146,
sin6_flowinfo = 32572, sin6_addr = {__in6_u = {__u6_addr8 =
"\210\276\232\317<\177\000\000\b\022\263\317<\177\000", __u6_addr16 =
{48776, 53146, 32572, 0, 4616, 53171, 32572, 0}, __u6_addr32 = {3483025032, 32572,
3484619272, 32572}}}, sin6_scope_id = 0}, sas = {ss_family = 46768, __ss_padding =
"\232\317<\177\000\000\210\276\232\317<\177\000\000\b\022\263\317<\177\000\000\000\000\000\000\r\000\000\000\000\000\000\000\016\000\000\000\016\000\000\000\016\000\000\000\000\331۳+y\340\003໋\317<\177\000\000\000\000\000\000\003\000\000\000\020\254yOyU",
'\000' <repeats 18 times>,
"\256v\225OyU\000\000\340\222\263Z\377\177\000\000д\263Z\377\177\000",
__ss_align = 139899159384584}}, id = -811965496, send_flags = {f = 32572, blst_imask = 0},
proto = 48 '0', proto_pad0 = 18 '\022', proto_pad1 = -12365}
tmp = 0x0
new_uri = 0x557950744900 ""
end = 0x600000000 <error: Cannot access memory at address 0x600000000>
crt = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E"
cmd = 0x557950743ba0
len = 0
user = 0
uri = {user = {s = 0x0, len = -813359088}, passwd = {s = 0x7fff5ab38e80
"\360\034\237\317<\177", len = 1282716628}, host = {s = 0x900000009
<error: Cannot access memory at address 0x900000009>, len = 0}, port = {s =
0x2000000 <error: Cannot access memory at address 0x2000000>, len = 28929864},
params = {s = 0x3fa910 <error: Cannot access memory at address 0x3fa910>, len =
4624568}, sip_params = {s = 0x46da90 <error: Cannot access memory at address
0x46da90>, len = 8}, headers = {s = 0x70 <error: Cannot access memory at address
0x70>, len = -1277437696}, port_no = 7408, proto = 53151, type = 32572, flags =
(unknown: 0xcf852010), transport = {s = 0x7f3ccf9f1cf0 "\360\005", len =
-813359088}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x7fff5ab38fb0
"`\220\263Z\377\177", len = 1336633822}, maddr = {s = 0x7fff5ab38f50
"\200\220\263Z\377\177", len = 0}, method = {s = 0x55794fc2e97b
"core", len = 1349794720}, lr = {s = 0x55794fc2ec00 <__func__.27>
"rval_destroy_helper", len = 1338173953}, r2 = {s = 0x7f3ccf9f1d38
"\002", len = -813359088}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0,
len = -811655880}, ttl_val = {s = 0x7f3ccf852010 "\001", len = -811655952},
user_param_val = {s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s =
0x7fff5ab39080 "\340\220\263Z\377\177", len = 1333373968}, lr_val = {s = 0x0,
len = 0}, r2_val = {s = 0x0, len = -1277437696}, gr_val = {s = 0x7f3ccf9abe88
"\002", len = 0}}
next_hop = {user = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, passwd = {s
= 0x0, len = 0}, host = {s = 0x7fff5ab39060 "\200\220\263Z\377\177", len =
1335338733}, port = {s = 0x1cfb31208 <error: Cannot access memory at address
0x1cfb31208>, len = -811942264}, params = {s = 0x7fff5ab39000
"\220\226\263Z\377\177", len = 0}, sip_params = {s = 0x557900000000 <error:
Cannot access memory at address 0x557900000000>, len = 0}, headers = {s = 0x0, len =
0}, port_no = 38544, proto = 23219, type = 32767, flags = (URI_USER_NORMALIZE |
URI_SIP_USER_PHONE | unknown: 0x4f7c33f0), transport = {s = 0xd <error: Cannot access
memory at address 0xd>, len = -1277437696}, ttl = {s = 0x7fff5ab39150 "", len
= -1277437696}, user_param = {s = 0x1cf99b270 <error: Cannot access memory at address
0x1cf99b270>, len = -811942264}, maddr = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, method = {s
= 0x0, len = 0}, lr = {s = 0x7fff5ab39080 "\340\220\263Z\377\177", len =
1335338805}, r2 = {s = 0x0, len = -811942264}, gr = {s = 0x7fff5ab390e0
"`\222\263Z\377\177", len = 1335370774}, transport_val = {s = 0x0, len =
-1277437696}, ttl_val = {s = 0x2000000000 <error: Cannot access memory at address
0x2000000000>, len = 0}, user_param_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, maddr_val =
{s = 0x0, len = 0}, method_val = {s = 0x7fff5ab390f0 "`\222\263Z\377\177", len =
1335338805}, lr_val = {s = 0x7fff5ab39260 "Г\263Z\377\177", len = -811942264},
r2_val = {s = 0x7fff5ab39260 "Г\263Z\377\177", len = 1335377740}, gr_val = {s =
0x7fff5ab39120 "\220Ś\317<\177", len = 1286204524}}
u = 0x0
port = 53171
dst_host = 0x9
i = 0
flags = 0
avp = 0x7fff5ab38db8
st = {flags = 3483201536, id = 2, name = {n = 0, s = {s = 0x300000000 <error:
Cannot access memory at address 0x300000000>, len = 1333373968}, re = 0x300000000}, avp
= 0x0}
sct = 0x0
sjt = 0x7fff5ab38db4
rve = 0x7f3ccf99c328
mct = 0x7fff5ab38e00
rv = 0x0
rv1 = 0x900000000
c1 = {cache_type = 3483026832, val_type = 32572, c = {avp_val = {n = 1521717948, s
= {s = 0x7fff5ab392bc "", len = -810348024}, re = 0x7fff5ab392bc}, pval = {rs =
{s = 0x7fff5ab392bc "", len = -810348024}, ri = 1521726672, flags = 32767}}, i2s
=
"\350k\251\317<\177\000\000\000\000\000\000<\177\000\000\000\000\000\000yU"}
s = {s = 0x7f3ccf9f2086 "<username>", len = 1}
srevp = {0x7fff5ab395a8, 0x0}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#39 0x000055794f7d2391 in run_actions (h=0x7fff5ab3b4d0, a=0x7f3ccf9af838,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccf9af838
ret = -1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 1521726672, tz_dsttime = 32767}
tdiff = 1521718304
__func__ = "run_actions"
#40 0x000055794f7c32d3 in do_action (h=0x7fff5ab3b4d0, a=0x7f3ccf9c2550,
msg=0x7f3ccfb31208) at core/action.c:1070
ret = 1
v = 1
dst = {send_sock = 0x7f3ccfb31208, to = {s = {sa_family = 46288, sa_data =
"\263Z\377\177\000\000(\253\252\317<\177\000"}, sin = {sin_family = 46288,
sin_port = 23219, sin_addr = {s_addr = 32767}, sin_zero =
"(\253\252\317<\177\000"}, sin6 = {sin6_family = 46288, sin6_port = 23219,
sin6_flowinfo = 32767, sin6_addr = {__in6_u = {__u6_addr8 =
"(\253\252\317<\177\000\000\000\000\000\000\000\000\000", __u6_addr16 =
{43816, 53162, 32572, 0, 0, 0, 0, 0}, __u6_addr32 = {3484068648, 32572, 0, 0}}},
sin6_scope_id = 1333373968}, sas = {ss_family = 46288, __ss_padding =
"\263Z\377\177\000\000(\253\252\317<\177\000\000\000\000\000\000\000\000\000\000\020\254yOyU\000\000\372Q\221OyU",
'\000' <repeats 11 times>,
"\331۳+y\340\003໋\317<\177\000\000P1\252\317<\177\000\000\020\254yOyU",
'\000' <repeats 26 times>, "
\232\263Z\377\177\000\000\065\247\227OyU\000", __ss_align = 0}}, id = -810929840,
send_flags = {f = 32572, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0
'\000', proto_pad1 = 0}
tmp = 0x7fff5ab3b4d0 "\005"
new_uri = 0x0
end = 0x0
crt = 0x7f3c000003a0 <error: Cannot access memory at address
0x7f3c000003a0>
cmd = 0x0
len = 0
user = 0
uri = {user = {s = 0x7fff5ab397b0 " \231\263Z\377\177", len = 0}, passwd
= {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, host = {s =
0x0, len = 0}, port = {s = 0x7fff5ab397b0 " \231\263Z\377\177", len =
1335349767}, params = {s = 0x0, len = 0}, sip_params = {s = 0x7fff5ab39830
"P\231\263Z\377\177", len = 0}, headers = {s = 0x7f3ccf9a7cf8 "\003",
len = 1521719676}, port_no = 4616, proto = 53171, type = 32572, flags = (unknown:
0x5ab3b4d0), transport = {s = 0x7fff5ab397e0 "\360|\232\317<\177", len = 0},
ttl = {s = 0x7f3ccf99b270 "\325\001", len = 1521719408}, user_param = {s = 0x0,
len = -1277437696}, maddr = {s = 0x7f3ccf986ff8 "\245\001", len = 0}, method =
{s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, lr = {s =
0x0, len = 0}, r2 = {s = 0x7fff5ab397e0 "\360|\232\317<\177", len =
1335227121}, gr = {s = 0x0, len = -810348024}, transport_val = {s = 0x7f3ccf9a1af8
"\002", len = 1521726672}, ttl_val = {s = 0x0, len = -811655952}, user_param_val
= {s = 0x0, len = 0}, maddr_val = {s = 0x7f3ccfa3e108 "\336̌PyU", len =
-1277437696}, method_val = {s = 0x7fff5ab398b0
"\245\066t\020\261۞\252\245\066(\003,\360\222\377", len = 0}, lr_val = {s =
0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, r2_val = {s
= 0x0, len = 0}, gr_val = {s = 0x7fff5ab39920 "\220\232\263Z\377\177", len =
1335371841}}
next_hop = {user = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, passwd = {s
= 0x0, len = 0}, host = {s = 0x7f3ccf9a7cf0 "\001", len = 1521719676}, port = {s
= 0x7f3ccfb31208 "\311)\f", len = 1521726672}, params = {s = 0x46da90 <error:
Cannot access memory at address 0x46da90>, len = 0}, sip_params = {s = 0x7f3c00000000
<error: Cannot access memory at address 0x7f3c00000000>, len = 1521720044}, headers
= {s = 0x7f3ccfb31208 "\311)\f", len = 1521726672}, port_no = 39248, proto =
23219, type = 32767, flags = 0, transport = {s = 0x0, len = -810348024}, ttl = {s = 0x0,
len = 1521726672}, user_param = {s = 0x7f3ccf98b048 "\257\001", len = 0}, maddr
= {s = 0x0, len = -1}, method = {s = 0x300000000 <error: Cannot access memory at
address 0x300000000>, len = 223622821}, lr = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, r2 = {s =
0x0, len = 0}, gr = {s = 0xaa9edbb1107436a5 <error: Cannot access memory at address
0xaa9edbb1107436a5>, len = 52967077}, transport_val = {s = 0x0, len = -1277437696},
ttl_val = {s = 0x2000000000 <error: Cannot access memory at address 0x2000000000>,
len = 0}, user_param_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -1277437696},
maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, lr_val = {s
= 0x0, len = 0}, r2_val = {s = 0x7fff5ab39a90 "", len = 1335373694}, gr_val = {s
= 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}}
u = 0x7f3ccfb31208
port = 0
dst_host = 0x0
i = 0
flags = 21881
avp = 0x7f3ccfab95f8
st = {flags = 0, id = 0, name = {n = -1277437696, s = {s = 0x3e0792bb3dbd900
<error: Cannot access memory at address 0x3e0792bb3dbd900>, len = 1521719840}, re =
0x3e0792bb3dbd900}, avp = 0x300000000}
sct = 0x300000000
sjt = 0x55794f79ac10 <_start>
rve = 0x7f3ccf9acca0
mct = 0x7fff5ab395b0
rv = 0x7f3ccfb31208
rv1 = 0x7fff5ab39700
c1 = {cache_type = 3483010048, val_type = 32572, c = {avp_val = {n = 1521720040, s
= {s = 0x7fff5ab39ae8 "", len = -810348024}, re = 0x7fff5ab39ae8}, pval = {rs =
{s = 0x7fff5ab39ae8 "", len = -810348024}, ri = 1521726672, flags = 32767}}, i2s
=
"\000\000\000\000yU\000\000\000\000\000\000\001\000\000\000\000\000\000\000<\177"}
s = {s = 0x7fff5ab39670 "\260\227\263Z\377\177", len = 1335227121}
srevp = {0x0, 0x3e0792bb3dbd900}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#41 0x000055794f7d2391 in run_actions (h=0x7fff5ab3b4d0, a=0x7f3ccf9c2550,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccf9c2550
ret = -1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 1521726672, tz_dsttime = 32767}
tdiff = 1521720400
__func__ = "run_actions"
#42 0x000055794f7c32d3 in do_action (h=0x7fff5ab3b4d0, a=0x7f3ccf9c2680,
msg=0x7f3ccfb31208) at core/action.c:1070
ret = 1
v = 1
dst = {send_sock = 0x7f3ccfb31208, to = {s = {sa_family = 46288, sa_data =
"\263Z\377\177\000\000\000\000\000\000\000\000\000"}, sin = {sin_family = 46288,
sin_port = 23219, sin_addr = {s_addr = 32767}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 46288, sin6_port = 23219,
sin6_flowinfo = 32767, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15
times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}},
sin6_scope_id = 0}, sas = {ss_family = 46288, __ss_padding = "\263Z\377\177",
'\000' <repeats 43 times>,
"\331۳+y\340\003H>\230\317\001\000\000\000\000\000\000\000\003\000\000\000\020\254yOyU\000\000\372Q\221OyU\000\000\000\000\000\000\004\000\000\000\200\244\263Z\377\177\000\000@\243\263Z\377\177\000\000\341\037\212OyU\000",
__ss_align = 139899157634776}}, id = -812099152, send_flags = {f = 32572, blst_imask = 0},
proto = 8 '\b', proto_pad0 = 18 '\022', proto_pad1 = -12365}
tmp = 0x5b5ab3a380 <error: Cannot access memory at address 0x5b5ab3a380>
new_uri = 0x0
end = 0x0
crt = 0x7f3c000003b0 <error: Cannot access memory at address
0x7f3c000003b0>
cmd = 0x7f3ccf8bb7b8
len = 0
user = 0
uri = {user = {s = 0x7f3c00000000 <error: Cannot access memory at address
0x7f3c00000000>, len = 1521726672}, passwd = {s = 0x0, len = -1277437696}, host = {s =
0x7f3ccfa74f8a "someprovider.com", len = 0}, port = {s = 0x55794f79ac10
<_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len =
0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x7fff5ab3a060
"\200\241\263Z\377\177", len = 1333601169}, headers = {s = 0x55794f79ac10
<_start> "1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len =
0}, port_no = 0, proto = 0, type = ERROR_URI_T, flags = 0, transport = {s = 0x7fff5ab3a010
"P\241\263Z\377\177", len = -810348024}, ttl = {s = 0x7f3ccf98b048
"\257\001", len = 1521721504}, user_param = {s = 0x7f3ccf986ff8
"\245\001", len = 1521726672}, maddr = {s = 0x7f3ccfb31208 "\311)\f",
len = 0}, method = {s = 0x0, len = -1}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0},
gr = {s = 0x0, len = 0}, transport_val = {s = 0xaa9edbb107f436a5 <error: Cannot access
memory at address 0xaa9edbb107f436a5>, len = 52967077}, ttl_val = {s = 0x7fff00000000
<error: Cannot access memory at address 0x7fff00000000>, len = 0}, user_param_val =
{s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, maddr_val =
{s = 0x0, len = 0}, method_val = {s = 0x7fff5ab3a230 "@\243\263Z\377\177", len =
1286017678}, lr_val = {s = 0x7fff5ab3a2a0 "\020\254yOyU", len = 1336651388},
r2_val = {s = 0x7f3ccf92b300 "", len = -810348024}, gr_val = {s = 0x7fff00b3a150
<error: Cannot access memory at address 0x7fff00b3a150>, len = -813359088}}
next_hop = {user = {s = 0x7fff5ab3a050 "", len = 1282716628}, passwd =
{s = 0x0, len = -1277437696}, host = {s = 0x7fff5ab3a150
"\300\242\263Z\377\177", len = 1335349869}, port = {s = 0x3fa358 <error:
Cannot access memory at address 0x3fa358>, len = -1277437696}, params = {s = 0x0, len =
0}, sip_params = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, headers =
{s = 0x0, len = 0}, port_no = 41344, proto = 23219, type = 32767, flags =
(URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 0x4f7d2afc), transport = {s = 0x0, len
= -810348024}, ttl = {s = 0x7f3ccf98b048 "\257\001", len = -1277437696},
user_param = {s = 0x7fff5ab3a380 "\b\022\263\317<\177", len = 0}, maddr = {s
= 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, method = {s
= 0x0, len = 0}, lr = {s = 0x7fff5ab3a180 "\300\242\263Z\377\177", len =
1335227121}, r2 = {s = 0x0, len = -810348024}, gr = {s = 0x7f3ccf992270 "\002",
len = 1521726672}, transport_val = {s = 0x0, len = -811585240}, ttl_val = {s =
0x55794fd0c098 <_ksr_ipaddr_list+120> "\n", len = 1521722096},
user_param_val = {s = 0x0, len = -1277437696}, maddr_val = {s = 0x0, len = 0}, method_val
= {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, lr_val = {s
= 0x0, len = -1277437696}, r2_val = {s = 0x7fff5ab3a2c0 "0\244\263Z\377\177",
len = 0}, gr_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}}
u = 0x7f3ccfb31208
port = 0
dst_host = 0x0
i = 0
flags = 0
avp = 0x7f3ccf986ff8
st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 1521721536}, re =
0x0}, avp = 0x55794f95bbbd <eval_elem+1084>}
sct = 0x7f3c00000001
sjt = 0x0
rve = 0x7f3ccf9a9930
mct = 0x13c400000023
rv = 0x7fff5ab39ed0
rv1 = 0x7fff5ab39f30
c1 = {cache_type = 1521722048, val_type = 32767, c = {avp_val = {n = 1335349767, s
= {s = 0x55794f97d207 <rval_get_int+239>
"H\213\225\370\376\377\377\211\002H\213\205\370\376\377\377\213", len =
-810348024}, re = 0x55794f97d207 <rval_get_int+239>}, pval = {rs = {s =
0x55794f97d207 <rval_get_int+239>
"H\213\225\370\376\377\377\211\002H\213\205\370\376\377\377\213", len =
-810348024}, ri = 1521726672, flags = 32767}}, i2s = '\000' <repeats 16
times>, "\360\"\231\317<\177"}
s = {s = 0x0, len = 0}
srevp = {0x0, 0x0}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#43 0x000055794f7d2391 in run_actions (h=0x7fff5ab3b4d0, a=0x7f3ccf9a1718,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccf9c2680
ret = 1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 1521726672, tz_dsttime = 32767}
tdiff = 1521722496
__func__ = "run_actions"
#44 0x000055794f7c3329 in do_action (h=0x7fff5ab3b4d0, a=0x7f3ccfa02210,
msg=0x7f3ccfb31208) at core/action.c:1074
ret = 1
v = 0
dst = {send_sock = 0x7f3ccfb31208, to = {s = {sa_family = 46288, sa_data =
"\263Z\377\177\000\000\370\062\220\317<\177\000"}, sin = {sin_family = 46288,
sin_port = 23219, sin_addr = {s_addr = 32767}, sin_zero =
"\370\062\220\317<\177\000"}, sin6 = {sin6_family = 46288, sin6_port = 23219,
sin6_flowinfo = 32767, sin6_addr = {__in6_u = {__u6_addr8 =
"\370\062\220\317<\177\000\000\000\000\000\000\000\000\000", __u6_addr16 =
{13048, 53136, 32572, 0, 0, 0, 0, 0}, __u6_addr32 = {3482333944, 32572, 0, 0}}},
sin6_scope_id = 0}, sas = {ss_family = 46288, __ss_padding =
"\263Z\377\177\000\000\370\062\220\317<\177", '\000' <repeats 35
times>, "\331۳+y\340\003", '\000' <repeats 12 times>,
"\003\000\000\000\020\254yOyU\000\000\000\000\000\000\000\000\000\000\001\000\000\000\004\000\000\000\260\254\263Z\377\177\000\000p\253\263Z\377\177\000\000\242\034\212OyU\000",
__ss_align = 139899156965896}}, id = -812767032, send_flags = {f = 32572, blst_imask = 0},
proto = 8 '\b', proto_pad0 = 18 '\022', proto_pad1 = -12365}
tmp = 0x7fff5ab3b4d0 "\005"
new_uri = 0x0
end = 0x400000000 <error: Cannot access memory at address 0x400000000>
crt = 0x7fff5ab3a6d0 "`ܔ\317<\177"
cmd = 0x0
len = 0
user = 0
uri = {user = {s = 0x7fff00000000 <error: Cannot access memory at address
0x7fff00000000>, len = 1335371841}, passwd = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -1277437696},
host = {s = 0x0, len = 0}, port = {s = 0x7f3ccf94dc60 "\001", len = 1521723500},
params = {s = 0x7f3ccfb31208 "\311)\f", len = 1521726672}, sip_params = {s =
0x7fff5ab3a890 "\260\251\263Z\377\177", len = 0}, headers = {s = 0x7f3c00000000
<error: Cannot access memory at address 0x7f3c00000000>, len = 1521723864}, port_no
= 4616, proto = 53171, type = 32572, flags = (unknown: 0x5ab3b4d0), transport = {s =
0x7f3ccfa03128 "\270\004", len = -810348024}, ttl = {s = 0x7f3ccf9199c0
"\265", len = 1521723600}, user_param = {s = 0x0, len = -1277437696}, maddr = {s
= 0x15ab3a780 <error: Cannot access memory at address 0x15ab3a780>, len = 0}, method
= {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, lr = {s =
0x0, len = 0}, r2 = {s = 0x7fff5ab3a840 "\200\251\263Z\377\177", len =
1335227121}, gr = {s = 0x0, len = -810348024}, transport_val = {s = 0x7f3ccf978600
"\002", len = 1521726672}, ttl_val = {s = 0x1cf852010 <error: Cannot access
memory at address 0x1cf852010>, len = -811585240}, user_param_val = {s = 0x0, len = 0},
maddr_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -1277437696},
method_val = {s = 0x7fff5ab3a910 "\320r\227\317<\177", len = 0}, lr_val = {s
= 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, r2_val = {s
= 0x0, len = -1277437696}, gr_val = {s = 0x7fff5ab3a980 "\360\252\263Z\377\177",
len = 0}}
next_hop = {user = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, passwd = {s
= 0x0, len = 0}, host = {s = 0x7fff5ab3a980 "\360\252\263Z\377\177", len =
1335349767}, port = {s = 0x7f3ccfb31208 "\311)\f", len = 1521726672}, params =
{s = 0x0, len = 0}, sip_params = {s = 0x7f3ccf978680 "\003", len = 1521724232},
headers = {s = 0x7f3ccfb31208 "\311)\f", len = 1521726672}, port_no = 43440,
proto = 23219, type = 32767, flags = 0, transport = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -810348024},
ttl = {s = 0x0, len = -1277437696}, user_param = {s = 0x15ab3a910 <error: Cannot access
memory at address 0x15ab3a910>, len = 0}, maddr = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, method = {s
= 0x0, len = 0}, lr = {s = 0x7fff5ab3a9b0 "x\206\227\317<\177", len =
1335227121}, r2 = {s = 0x0, len = -810348024}, gr = {s = 0x7f3ccf9772d0 "\002",
len = 1521726672}, transport_val = {s = 0x0, len = -1277437696}, ttl_val = {s =
0x20cf8e1c38 <error: Cannot access memory at address 0x20cf8e1c38>, len = 0},
user_param_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -1277437696},
maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}, lr_val = {s
= 0x0, len = 0}, r2_val = {s = 0x7fff5ab3aaf0 "`\254\263Z\377\177", len =
1335371841}, gr_val = {s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = 0}}
u = 0x0
port = 0
dst_host = 0x7fff5ab3a86c
i = 13
flags = 21881
avp = 0x7f3ccf8fef90
st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 1521723632}, re =
0x0}, avp = 0x55794f95bbbd <eval_elem+1084>}
sct = 0x300000000
sjt = 0x55794f79ac10 <_start>
rve = 0x7f3ccf9922e8
mct = 0x0
rv = 0x7f3ccfb31208
rv1 = 0x7f3ccfb31208
c1 = {cache_type = 3482814072, val_type = 32572, c = {avp_val = {n = 1521724232, s
= {s = 0x7fff5ab3ab48 "", len = -810348024}, re = 0x7fff5ab3ab48}, pval = {rs =
{s = 0x7fff5ab3ab48 "", len = -810348024}, ri = 1521726672, flags = 32767}}, i2s
= "\020\061\216\317<\177", '\000' <repeats 14 times>,
"<\177"}
s = {s = 0x7fff5ab3a6d0 "`ܔ\317<\177", len = 1335227121}
srevp = {0x0, 0x0}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#45 0x000055794f7d2391 in run_actions (h=0x7fff5ab3b4d0, a=0x7f3ccf983bd8,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccfa02210
ret = 1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = 1521726672, tz_dsttime = 32767}
tdiff = 1521724592
__func__ = "run_actions"
#46 0x000055794f7c3329 in do_action (h=0x7fff5ab3b4d0, a=0x7f3ccfa04f70,
msg=0x7f3ccfb31208) at core/action.c:1074
ret = 1
v = 0
dst = {send_sock = 0x7fff5ab3b250, to = {s = {sa_family = 55552, sa_data =
"۳+y\340\003\000\000\000\000\000\000\000"}, sin = {sin_family = 55552, sin_port
= 46043, sin_addr = {s_addr = 65042731}, sin_zero =
"\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 55552, sin6_port = 46043,
sin6_flowinfo = 65042731, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats
12 times>, "\003\000\000", __u6_addr16 = {0, 0, 0, 0, 0, 0, 3, 0},
__u6_addr32 = {0, 0, 0, 3}}}, sin6_scope_id = 1333373968}, sas = {ss_family = 55552,
__ss_padding = "۳+y\340\003", '\000' <repeats 12 times>,
"\003\000\000\000\020\254yOyU\000\000\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\b\022\263\317<\177\000\000@\263\263Z\377\177\000\000\323N\260OyU\000\000\330f\306OyU\000\000\220\374\254\317<\177\000\000\354͌PyU\000\000\235͌PyU\000\000\000\000\000\000\a\000\000\000\250͌PyU\000",
__ss_align = 0}}, id = 1351405036, send_flags = {f = 21881, blst_imask = 0}, proto = -93
'\243', proto_pad0 = -51 '\315', proto_pad1 = 20620}
tmp = 0x7f3ccd5f2719 "\366\205\331\005"
new_uri = 0x14b734 <error: Cannot access memory at address 0x14b734>
end = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E"
crt = 0xffffffff508ccc94 <error: Cannot access memory at address
0xffffffff508ccc94>
cmd = 0x7f3ccf8bdb00
len = 4
user = 0
uri = {user = {s = 0x7fff5ab3af10 "\020\260\263Z\377\177", len =
1336651388}, passwd = {s = 0x7fff5ab3af20 "\200\257\263Z\377\177", len =
-813359088}, host = {s = 0x4547dcbf8 <error: Cannot access memory at address
0x4547dcbf8>, len = 1336651388}, port = {s = 0x7fff5ab3af40 "", len =
-813359088}, params = {s = 0x7fff5ab3b010 "\310E\237\317<\177", len =
-813359088}, sip_params = {s = 0x7fff5ab3af80 "\260\257\263Z\377\177", len =
1282716628}, headers = {s = 0x3f9538 <error: Cannot access memory at address
0x3f9538>, len = 0}, port_no = 0, proto = 512, type = ERROR_URI_T, flags = (unknown:
0x1b97fe8), transport = {s = 0x3f9828 <error: Cannot access memory at address
0x3f9828>, len = 4620312}, ttl = {s = 0x46da90 <error: Cannot access memory at
address 0x46da90>, len = 8}, user_param = {s = 0x7c <error: Cannot access memory at
address 0x7c>, len = -1277437696}, maddr = {s = 0x7fff5ab3afb0
"\240\260\263Z\377\177", len = 1334098835}, method = {s = 0x0, len =
1338365855}, lr = {s = 0x7fff5ab3b0a0 "", len = 0}, r2 = {s = 0x7fff5ab3b0a0
"", len = 1336627661}, gr = {s = 0x55794fc68dcf "core", len =
-813359088}, transport_val = {s = 0x55794fc68dc0 <__func__.3>
"parse_to_param", len = 1338413720}, ttl_val = {s = 0x30 <error: Cannot
access memory at address 0x30>, len = -813359088}, user_param_val = {s = 0x7cfaf8718
<error: Cannot access memory at address 0x7cfaf8718>, len = 1351404736}, maddr_val =
{s = 0x55794f79ac10 <_start>
"1\355I\211\321^H\211\342H\203\344\360PTL\215\005\332\\E", len = -811645496},
method_val = {s = 0x7f3ccf9f45c8 "8", len = -813359088}, lr_val = {s =
0x7fff5ab3b170 "", len = 1336092425}, r2_val = {s = 0x7fff5ab3b2a0 "",
len = 56896}, gr_val = {s = 0x400000386 <error: Cannot access memory at address
0x400000386>, len = 1521725620}}
next_hop = {user = {s = 0x5547dcbf8 <error: Cannot access memory at address
0x5547dcbf8>, len = 1351404666}, passwd = {s = 0x44f79ac10 <error: Cannot access
memory at address 0x44f79ac10>, len = -1277437696}, host = {s = 0x5579508ccc7f
";rport;branch=z9hG4bKPjdffcf0f7-8b17-4b9b-971d-6509002eb588;alias\r\nMax-Forwards:
69\r\nFrom:
sip:<username>@someprovider.com;tag=bcb22e86-01c0-4163-a7c9-a7435220505c\r\nTo:
sip:<phonenumber>@someprovider.com;tag=0c9a7"..., len = 0}, port = {s =
0x44f79ac10 <error: Cannot access memory at address 0x44f79ac10>, len = 1338412495},
params = {s = 0x0, len = -1277437696}, sip_params = {s = 0x7fff5ab3b200
"\220\332F", len = 1336874015}, headers = {s = 0x505b0e0 <error: Cannot
access memory at address 0x505b0e0>, len = 0}, port_no = 45344, proto = 23219, type =
32767, flags = (unknown: 0x5ab3b248), transport = {s = 0x0, len = -810581096}, ttl = {s =
0x5579508ccdec "", len = 1351404861}, user_param = {s = 0xd003f9478 <error:
Cannot access memory at address 0xd003f9478>, len = 29}, maddr = {s = 0x46da90
<error: Cannot access memory at address 0x46da90>, len = 8}, method = {s = 0x84
<error: Cannot access memory at address 0x84>, len = -1277437696}, lr = {s =
0x7fff5ab3b150 "@\262\263Z\377\177", len = 1334098835}, r2 = {s = 0x0, len =
1338365855}, gr = {s = 0x7fff5ab3b240 "\020\254yOyU", len = 0}, transport_val =
{s = 0x7fff5ab3b240 "\020\254yOyU", len = 1336627661}, ttl_val = {s =
0x55794fc6664b "core", len = -811645424}, user_param_val = {s = 0x0, len =
1351404904}, maddr_val = {s = 0xf0 <error: Cannot access memory at address 0xf0>,
len = -813359088}, method_val = {s = 0x1f003f8b50 <error: Cannot access memory at
address 0x1f003f8b50>, len = 1336651388}, lr_val = {s = 0x7fff5ab3b1e0 "",
len = -813359088}, r2_val = {s = 0x7f3ccfad0098 "\001", len = 1337032407},
gr_val = {s = 0x7f3ccfacfcb8 "\004", len = -810746712}}
u = 0x0
port = 20404
dst_host = 0x0
i = 7
flags = 0
avp = 0x7fff5ab3af50
st = {flags = 1521725640, id = 32767, name = {n = 1417530360, s = {s =
0x7f3c547dcbf8 "\254\b", len = 1351404691}, re = 0x7f3c547dcbf8}, avp =
0x7f3cd1a4f4a7 <send+119>}
sct = 0x3e0792bb3dbd900
sjt = 0x0
rve = 0x7f3ccf978d88
mct = 0x7fff5ab3b020
rv = 0x25ab3adf0
rv1 = 0x3e0792bb3dbd900
c1 = {cache_type = 33554432, val_type = RV_NONE, c = {avp_val = {n = 28933880, s =
{s = 0x1b97ef8 <error: Cannot access memory at address 0x1b97ef8>, len = 4167960},
re = 0x1b97ef8}, pval = {rs = {s = 0x1b97ef8 <error: Cannot access memory at address
0x1b97ef8>, len = 4167960}, ri = 4620552, flags = 0}}, i2s =
"\220\332F\000\000\000\000\000\b\000\000\000\000\000\000\000y\000\000\000\000"}
s = {s = 0x0, len = 0}
srevp = {0x7f3ccfacfa30, 0x0}
evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl
= 0x0, rplcode = 0, mode = 0}
mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0,
len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 1,
string = 0x1 <error: Cannot access memory at address 0x1>, str = {s = 0x1 <error:
Cannot access memory at address 0x1>, len = 0}, data = 0x1, attr = 0x1, select = 0x1}},
{type = RVE_FREE_FIXUP_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 14}, data
= 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str =
{s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number
= 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type
= NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr =
0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len
= 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string =
0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
__func__ = "do_action"
#47 0x000055794f7d2391 in run_actions (h=0x7fff5ab3b4d0, a=0x7f3ccf8de910,
msg=0x7f3ccfb31208) at core/action.c:1584
t = 0x7f3ccfa04f70
ret = 1
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
tz = {tz_minuteswest = -810348024, tz_dsttime = 32572}
tdiff = 0
__func__ = "run_actions"
#48 0x000055794f7d2c0b in run_top_route (a=0x7f3ccf8de910, msg=0x7f3ccfb31208, c=0x0) at
core/action.c:1669
ctx = {rec_lev = 5, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf =
{12884901888, -6152238486275213659, 93979512777744, 0, 0, 0, -6152238486155675995,
-30698175615977819}, __mask_was_saved = 0, __saved_mask = {__val = {0, 139897030523508,
279356405505054976, 140734715115120, 93979516405432, 140734715114896, 93979513240322,
140734715114896, 140734715115216, 1906965479424, 279356405505054976, 12884901888,
12884901888, 93979512777744, 0, 0}}}}}
p = 0x7fff5ab3b4d0
ret = 0
sfbk = 0
#49 0x000055794f9301b2 in receive_msg (buf=0x5579508ccc30 "ACK
sip:<phonenumber>@someprovider.com SIP/2.0\r\nVia: SIP/2.0/TLS
<customer_IP>:34365;rport;branch=z9hG4bKPjdffcf0f7-8b17-4b9b-971d-6509002eb588;alias\r\nMax-Forwards:
69\r\nFrom: sip:<username>@someprovider.com;tag=b"..., len=444,
rcv_info=0x7f3c50cf3530) at core/receive.c:515
msg = 0x7f3ccfb31208
ctx = {rec_lev = 0, run_flags = 2, last_retcode = 0, jmp_env = {{__jmpbuf =
{140734715115664, 93979515746255, 139897092295672, 139897092295696, 42949672960,
93979530808368, 1530148517, -30698175615977819}, __mask_was_saved = 1521727504,
__saved_mask = {__val = {0, 0, 279356405505054976, 0, 2, 93979512777744, 0, 0, 0,
140734715115856, 93979515723953, 139897092295672, 0, 0, 0, 140734715116428}}}}}
bctx = 0x0
ret = 0
tvb = {tv_sec = 0, tv_usec = 0}
tve = {tv_sec = 0, tv_usec = 0}
diff = 0
inb = {s = 0x5579508ccc30 "ACK sip:<phonenumber>@someprovider.com
SIP/2.0\r\nVia: SIP/2.0/TLS
<customer_IP>:34365;rport;branch=z9hG4bKPjdffcf0f7-8b17-4b9b-971d-6509002eb588;alias\r\nMax-Forwards:
69\r\nFrom: sip:<username>@someprovider.com;tag=b"..., len = 444}
netinfo = {data = {s = 0x0, len = 0}, bufsize = 0, rcv = 0x0, dst = 0x0}
keng = 0x0
evp = {data = 0x7fff5ab3b6c0, obuf = {s = 0x0, len = 0}, rcv = 0x7f3c50cf3530, dst
= 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0}
cidlockidx = 0
cidlockset = 0
errsipmsg = 0
exectime = 0
__func__ = "receive_msg"
#50 0x000055794fa7274b in receive_tcp_msg (tcpbuf=0x7f3c50cf38b8 "ACK
sip:<phonenumber>@someprovider.com SIP/2.0\r\nVia: SIP/2.0/TLS
<customer_IP>:34365;rport;branch=z9hG4bKPjdffcf0f7-8b17-4b9b-971d-6509002eb588;alias\r\nMax-Forwards:
70\r\nFrom: sip:<username>@someprovider.com;tag=b"..., len=444,
rcv_info=0x7f3c50cf3530, con=0x7f3c50cf3518) at core/tcp_read.c:1413
ret = 0
buf = 0x5579508ccc30 "ACK sip:<phonenumber>@someprovider.com
SIP/2.0\r\nVia: SIP/2.0/TLS
<customer_IP>:34365;rport;branch=z9hG4bKPjdffcf0f7-8b17-4b9b-971d-6509002eb588;alias\r\nMax-Forwards:
69\r\nFrom: sip:<username>@someprovider.com;tag=b"...
bsize = 65535
blen = 65535
__func__ = "receive_tcp_msg"
#51 0x000055794fa74dac in tcp_read_req (con=0x7f3c50cf3518, bytes_read=0x7fff5ab3bb84,
read_flags=0x7fff5ab3bb8c) at core/tcp_read.c:1604
bytes = 444
total_bytes = 444
resp = 1
size = 0
req = 0x7f3c50cf3640
dst = {send_sock = 0x7fff5ab3bb00, to = {s = {sa_family = 56539, sa_data =
"\211OyU\000\000\260\272\263Z\377\177\000"}, sin = {sin_family = 56539, sin_port
= 20361, sin_addr = {s_addr = 21881}, sin_zero = "\260\272\263Z\377\177\000"},
sin6 = {sin6_family = 56539, sin6_port = 20361, sin6_flowinfo = 21881, sin6_addr =
{__in6_u = {__u6_addr8 =
"\260\272\263Z\377\177\000\000\030\000\000\000\000\000\000", __u6_addr16 =
{47792, 23219, 32767, 0, 24, 0, 0, 0}, __u6_addr32 = {1521728176, 32767, 24, 0}}},
sin6_scope_id = 0}, sas = {ss_family = 56539, __ss_padding =
"\211OyU\000\000\260\272\263Z\377\177\000\000\030", '\000' <repeats
15 times>,
"m\025\205mP\000\000\000\310Q\263\317<\177\000\000\200\060\325OyU\000\000\031\267\024\000\001
\000\000\310Q\263\317<\177\000\000\260+\232\317<\177\000\000\000\000\000\000,\000\000\000\001\000\000\000\000\000\000\000\300\217\345R<\177\000\000л\263Z\377\177\000\000\b\000\000\000\000\000\000",
__ss_align = 20}}, id = 1, send_flags = {f = 1, blst_imask = 0}, proto = 71 'G',
proto_pad0 = 0 '\000', proto_pad1 = 0}
c = 103 'g'
ret = 0
__func__ = "tcp_read_req"
#52 0x000055794fa79774 in handle_io (fm=0x7f3ccfb34cb8, events=1, idx=-1) at
core/tcp_read.c:1855
ret = 0
n = 10
read_flags = RD_CONN_SHORT_READ
con = 0x7f3c50cf3518
s = 71
resp = 1
t = 1837438317
ee = 0x0
__func__ = "handle_io"
#53 0x000055794fa63bef in io_wait_loop_epoll (h=0x55794fd52ec0 <io_w>, t=2,
repeat=0) at core/io_wait.h:1070
n = 1
r = 0
fm = 0x7f3ccfb34cb8
revents = 1
__func__ = "io_wait_loop_epoll"
#54 0x000055794fa7b252 in tcp_receive_loop (unix_sock=44) at core/tcp_read.c:1976
__func__ = "tcp_receive_loop"
#55 0x000055794fa57e17 in tcp_init_children (woneinit=0x7fff5ab3bf78) at
core/tcp_main.c:5229
r = 2
i = 15
reader_fd_1 = 44
pid = 0
si_desc = "tcp receiver
(generic)\000\000͎\236OyU\000\000P\277\263Z\377\177\000\000\212`\250O\000\000\000\000\240\032\213\317\000\000\000\000\273\355\303OyU\000\000\241\000\263Z\377\177\000\000\221\243\272OyU\000\000\020\254yOyU\000\000\070\061,M<\177\000\000\000\000\000\000\000\000\000\000\210\372lM\001\000\000\000@\277\263Z\377\177\000\000\227\245\272OyU\000"
si = 0x0
__func__ = "tcp_init_children"
#56 0x000055794f7ae63a in main_loop () at main.c:1849
i = 16
pid = 1357613
si = 0x0
si_desc = "udp receiver child=15
sock=<kamailio_IP>:5060\000\000\000\002\000\000\000(\000\000\000\000\331۳+y\340\003\000\000\000\000\000\000\000\000\031\267\024\000\000\000\000\000\020\254yOyU",
'\000' <repeats 26 times>,
"\020\301\263Z\377\177\000\000s\377\225OyU\000"
nrprocs = 16
woneinit = 1
__func__ = "main_loop"
#57 0x000055794f7b920c in main (argc=18, argv=0x7fff5ab3c668) at main.c:3078
cfg_stream = 0x5579506392a0
c = -1
r = 0
tmp = 0x7fff5ab3ce4b ""
tmp_len = 0
port = 0
proto = 7
ahost = 0x0
aport = 0
options = 0x55794fbf4258
":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
ret = -1
seed = 1093004594
rfd = 4
debug_save = 0
debug_flag = 0
dont_fork_cnt = 0
n_lst = 0x0
p = 0x0
st = {st_dev = 22, st_ino = 687, st_nlink = 2, st_mode = 16832, st_uid = 111,
st_gid = 301, __pad0 = 0, st_rdev = 0, st_size = 80, st_blksize = 4096, st_blocks = 0,
st_atim = {tv_sec = 1690436755, tv_nsec = 157127840}, st_mtim = {tv_sec = 1691487362,
tv_nsec = 342809961}, st_ctim = {tv_sec = 1691487362, tv_nsec = 342809961},
__glibc_reserved = {0, 0, 0}}
tbuf =
"\000\000\000\000\000\000\000\000\030R\265Z\377\177\000\000\000\000\000\000 ",
'\000' <repeats 27 times>,
"\001\000\000\000\000\000\000\000\366u\256\003\001", '\000' <repeats
19 times>, "\002", '\000' <repeats 319 times>...
option_index = 12
long_options = {{name = 0x55794fbf66c6 "help", has_arg = 0, flag = 0x0,
val = 104}, {name = 0x55794fbf150c "version", has_arg = 0, flag = 0x0, val =
118}, {name = 0x55794fbf66cb "alias", has_arg = 1, flag = 0x0, val = 1024},
{name = 0x55794fbf66d1 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name =
0x55794fbf66d7 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name =
0x55794fbf66e0 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name =
0x55794fbf66ea "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name =
0x55794fbf66f4 "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name =
0x55794fbf66ff "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name =
0x55794fbf6708 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name =
0x55794fbf6713 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name =
0x55794fbf6719 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name =
0x55794fbf6723 "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0,
has_arg = 0, flag = 0x0, val = 0}}
__func__ = "main"
```
#### Log Messages
```
Aug 11 13:53:11 sip-tcploadbalancer01.live.sipgate.net /usr/sbin/kamailio[1357595]: ALERT:
<core> [main.c:774]: handle_sigs(): child process 1357620 exited by a signal 11
Aug 11 13:53:11 sip-tcploadbalancer01.live.sipgate.net /usr/sbin/kamailio[1357595]: ALERT:
<core> [main.c:777]: handle_sigs(): core was generated
Aug 11 13:53:12 sip-tcploadbalancer01.live.sipgate.net systemd[1]:
kamailio_sip_lb_tcp_de.service: Main process exited, code=exited, status=1/FAILURE
Aug 11 13:53:12 sip-tcploadbalancer01.live.sipgate.net systemd[1]:
kamailio_sip_lb_tcp_de.service: Failed with result 'exit-code'.
Aug 11 13:53:12 sip-tcploadbalancer01.live.sipgate.net systemd[1]:
kamailio_sip_lb_tcp_de.service: Consumed 13h 18min 12.966s CPU time.
Aug 11 13:53:14 sip-tcploadbalancer01.live.sipgate.net systemd[1]:
kamailio_sip_lb_tcp_de.service: Scheduled restart job, restart counter is at 4.
Aug 11 13:53:14 sip-tcploadbalancer01.live.sipgate.net systemd[1]: Stopped Kamailio - the
sipgate SIP Server.
Aug 11 13:53:14 sip-tcploadbalancer01.live.sipgate.net systemd[1]:
kamailio_sip_lb_tcp_de.service: Consumed 13h 18min 12.966s CPU time.```
#### SIP Traffic
### Possible Solutions
### Additional Information
* **Kamailio Version** - output of `kamailio -v`
```
$ sudo kamailio -v
[sudo] password for <user>:
version: kamailio 5.6.4 (x86_64/linux)
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY,
USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR,
USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535,
DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 10.2.1
```
* **Operating System**:
```
# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 11 (bullseye)
Release: 11
Codename: bullseye
# uname -a
Linux sip-tcploadbalancer01.<domain> 5.10.0-23-amd64 #1 SMP Debian 5.10.179-2
(2023-07-14) x86_64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3539
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3539(a)github.com>
21 Aug
21 Aug
2:50 p.m.
It might not be related to Kamailio per se, maybe an incompatible/buggy version -- in the
past I met a situation when crash happened at tls handshake of opening the https
connection, before kamailio specific code got to do anything. A later upgrade made it go
away.
However, you should install the packages with the debugging symbols for libcurl, libssl
and libcrypto in order to see what are the executed functions from those libs. Then retake
the full backtrace and paste it here, we may be able to provide further hints.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3539#issuecomment-1686184077
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3539/1686184077(a)github.com>
25 Aug
25 Aug
12:07 p.m.
Thanks @miconda for your hints. We will try this if kamailio crashes again.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3539#issuecomment-1693033259
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3539/1693033259(a)github.com>
28 Aug
28 Aug
noon
closing issue until problems occurs again and we have additional debug information. Thanks
a lot
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3539#issuecomment-1695311735
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/3539/1695311735(a)github.com>
noon
Closed #3539 as completed.
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/3539#event-10206635579
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issue/3539/issue_event/10206635579(a)github.com>
456
days inactive
470
days old
4 comments
3 participants
participants (3)
-
AndreasTarp -
Daniel-Constantin Mierla
-
Jonas