Hi Daniel,
I tried to increase shared memory by setting a parameter in kamctlrc:
STARTOPTIONS="-m 64"
When recreating the crash I counted pv_get_ruri_attr in the logfile just to see if the shared memory changed anything:
> grep pv_get_ruri_attr /var/log/kamailio.log | wc -l
75472 with default settings
75436 with shared memory set to 64
75440 with shared memory set to 256
But in top I saw a massive increase in memory usage.
I don’t know if this is any helpful for you.
Kind regards
Timo Klecker
Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Montag, 8. Dezember 2014 16:25
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: [SR-Users] Kamailio Crash when modifying username of Request Uri
Hello,
I noticed that there was an issue with the result stored in r-uri and it should be fixed in the latest git branches -- you can fetch them and try, practically, if the resulting user is empty, then the password is removed as well.
However, I wanted to hunt the real reason of the crash, as kamailio can handle cases when the r-uri is broken (could be an incoming r-uri).
Cheers,
Daniel
On 08/12/14 13:50, Timo Klecker wrote:
Hi Daniel,
in the configuration I sent last week is a very simplified version of our loop:
route[START_ROUTING] {
if ($rU == "123") {
xlog("L_NOTICE", "123 reached\n");
exit;
}
route(MODIFY_NUMBER);
}
route[MODIFY_NUMBER] {
$rU = $(rU{s.strip,6});
xlog("L_NOTICE", "rU set to: $rU\n");
xlog("L_NOTICE", "ru set to: $ru\n");
$rU = '123' + $rU;
xlog("L_NOTICE", "rU set to: $rU\n");
xlog("L_NOTICE", "ru set to: $ru\n");
route(START_ROUTING);
}
Due to the parsing Error in $rU the break of the loop is never triggered. So you are right, there is an infinite loop. When breaking this loop, no crash happens.
But still one problem remains:
Request Uri: sip:123456:pass@sip.custom.com
$rU = $(rU{s.strip,6});
Afterwards the Request Uri is set to
There is an @ missing.
Dec 4 13:27:50 /sbin/kamailio[8043]: ERROR: <core> [action.c:805]: do_action(): ERROR: do_action: bad uri <sip::pass10.10.207.20>, dropping packet
And $rU and $ru are null due to parsing-errors.
Kind regards
Timo Klecker
Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Freitag, 5. Dezember 2014 18:04
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri
Hello,
according to the trace, the crash trace is:
#0 0x00007f544c119801 in _IO_default_xsputn_internal () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f544c0e93a9 in vfprintf () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f544c0f4368 in fprintf () from /lib64/libc.so.6
No symbol table info available.
#3 0x00007f544c189bc8 in __vsyslog_chk () from /lib64/libc.so.6
No symbol table info available.
#4 0x00007f544c18a210 in syslog () from /lib64/libc.so.6
No symbol table info available.
#5 0x00007f54479f2341 in pv_get_ruri_attr (msg=0x7f544bed9c30, param=0x7f544beb2a50, res=0x7fffe1341e88) at pv_core.c:304
__FUNCTION__ = "pv_get_ruri_attr"
Line 304 in pv_core is:
LM_ERR("failed to parse the R-URI\n");
Which is a bit strange as a point of crash. I will try to see if I can spot something else...
I noticed that you seem to have somehow kind of infinite loop with calling route blocks - like calling route B from A and route A from B...
Cheers,
DanielOn 05/12/14 17:45, Timo Klecker wrote:
Hi Daniel,
yes, Kamailio 4.1 using syslog:
log_facility=LOG_LOCAL4
# modules
loadmodule "xlog.so"
# kamailio -v
version: kamailio 4.1.6 (x86_64/linux) 010d57
flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 010d57
compiled on 17:35:09 Dec 4 2014 with gcc 4.4.6
Compiled yesterday because I needed the right debug-rpm and had to recompile.
Kind regards
Timo Klecker
Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Freitag, 5. Dezember 2014 17:40
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri
I looked at the last gdb trace you sent and the crash is inside syslog() function, which prints a message with static text, if I matched it with the right version of the sources.
Is the trace from running kamailio 4.1? What is the output of:
kamailio -v
Cheers,
DanielOn 04/12/14 17:23, Daniel-Constantin Mierla wrote:
Btw, you need to install dbg symbols package, in case you installed from some deb/rpm.
DanielOn 04/12/14 17:22, Daniel-Constantin Mierla wrote:
Hello,
I need the output of the 'bt full' -- I guess it will require a few 'more' paginations, because it is printing the values for local variables which are important to troubleshoot.
Cheers,
danielOn 04/12/14 17:13, Timo Klecker wrote:
Hi Daniel,
gdb /path/to/kamailio /path/to/corefile
sent me into interactive mode. Typing
bt full
now sent me into something like “more” so I had to copy everything by hand, like 46k lines.
So I searched the web for a non interactive method and came up with this. If it is not ok, please tell me how to get the “bt full” string out of this interactive gdb shell.
I will provide you with the output of gdb itself, I missed that in the last mail.
Greetings
Timo
Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Donnerstag, 4. Dezember 2014 16:47
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri
Hello,
wondering why not simply sending what I asked for, respectively output for:
gdb /path/to/kamailio /path/to/corefile
bt full
I guess I wanted to get some useful information out of those commands...
Cheers,
Daniel
On 04/12/14 16:41, Timo Klecker wrote:Hi Daniel,
sorry for sending the core files, I now sent the output of:
gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" /sbin/kamailio /core.8933
Hope this is what you are looking for.
Kind regards,
Timo
Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com]
Gesendet: Donnerstag, 4. Dezember 2014 16:16
An: Timo Klecker; 'Kamailio (SER) - Users Mailing List'
Betreff: Re: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri
Hello,
corefiles themselves are useless. I need the backtrace from them, respectively the output of:
gdb /path/to/kamailio /path/to/corefile
bt full
Cheers,
DanielOn 04/12/14 15:56, Timo Klecker wrote:
Hello Daniel,
yes, with the production config I still get a core, not with the testing config, though. The last line in log with testing config is:
Dec 4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: ERROR: <core> [action.c:1572]: run_actions(): WARNING: too many recursive routing table lookups (101) giving up!
Dec 4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: WARNING: <core> [receive.c:214]: receive_msg(): WARNING: receive_msg: error while trying script
But I have 2 cores, each 52M in size from production config. I will bzip them and send them via email.
Kind regards,
Timo Klecker
--Daniel-Constantin Mierlahttp://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda