Re: [SR-Users] Kamailio Crash when modifying username of Request Uri

Hi Daniel, I tried to increase shared memory by setting a parameter in kamctlrc: STARTOPTIONS="-m 64" When recreating the crash I counted pv_get_ruri_attr in the logfile just to see if the shared memory changed anything: 75472 with default settings 75436 with shared memory set to 64 75440 with shared memory set to 256 But in top I saw a massive increase in memory usage. I don't know if this is any helpful for you. Kind regards Timo Klecker Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Gesendet: Montag, 8. Dezember 2014 16:25 An: Timo Klecker; 'Kamailio (SER) - Users Mailing List' Betreff: Re: [SR-Users] Kamailio Crash when modifying username of Request Uri Hello, I noticed that there was an issue with the result stored in r-uri and it should be fixed in the latest git branches -- you can fetch them and try, practically, if the resulting user is empty, then the password is removed as well. However, I wanted to hunt the real reason of the crash, as kamailio can handle cases when the r-uri is broken (could be an incoming r-uri). Cheers, Daniel On 08/12/14 13:50, Timo Klecker wrote: Hi Daniel, in the configuration I sent last week is a very simplified version of our loop: route[START_ROUTING] { if ($rU == "123") { xlog("L_NOTICE", "123 reached\n"); exit; } route(MODIFY_NUMBER); } route[MODIFY_NUMBER] { $rU = $(rU{s.strip,6}); xlog("L_NOTICE", "rU set to: $rU\n"); xlog("L_NOTICE", "ru set to: $ru\n"); $rU = '123' + $rU; xlog("L_NOTICE", "rU set to: $rU\n"); xlog("L_NOTICE", "ru set to: $ru\n"); route(START_ROUTING); } Due to the parsing Error in $rU the break of the loop is never triggered. So you are right, there is an infinite loop. When breaking this loop, no crash happens. But still one problem remains: Request Uri: sip:123456:pass@sip.custom.com $rU = $(rU{s.strip,6}); Afterwards the Request Uri is set to sip::passsip.custom.com There is an @ missing. Dec 4 13:27:50 /sbin/kamailio[8043]: ERROR: <core> [action.c:805]: do_action(): ERROR: do_action: bad uri <sip::pass10.10.207.20>, dropping packet And $rU and $ru are null due to parsing-errors. Kind regards Timo Klecker Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Gesendet: Freitag, 5. Dezember 2014 18:04 An: Timo Klecker; 'Kamailio (SER) - Users Mailing List' Betreff: Re: AW: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri Hello, according to the trace, the crash trace is: #0 0x00007f544c119801 in _IO_default_xsputn_internal () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f544c0e93a9 in vfprintf () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f544c0f4368 in fprintf () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f544c189bc8 in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f544c18a210 in syslog () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f54479f2341 in pv_get_ruri_attr (msg=0x7f544bed9c30, param=0x7f544beb2a50, res=0x7fffe1341e88) at pv_core.c:304 __FUNCTION__ = "pv_get_ruri_attr" Line 304 in pv_core is: LM_ERR("failed to parse the R-URI\n"); Which is a bit strange as a point of crash. I will try to see if I can spot something else... I noticed that you seem to have somehow kind of infinite loop with calling route blocks - like calling route B from A and route A from B... Cheers, Daniel On 05/12/14 17:45, Timo Klecker wrote: Hi Daniel, yes, Kamailio 4.1 using syslog: log_facility=LOG_LOCAL4 # modules loadmodule "xlog.so" # kamailio -v version: kamailio 4.1.6 (x86_64/linux) 010d57 flags: STATS: Off, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: 010d57 compiled on 17:35:09 Dec 4 2014 with gcc 4.4.6 Compiled yesterday because I needed the right debug-rpm and had to recompile. Kind regards Timo Klecker Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Gesendet: Freitag, 5. Dezember 2014 17:40 An: Timo Klecker; 'Kamailio (SER) - Users Mailing List' Betreff: Re: AW: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri I looked at the last gdb trace you sent and the crash is inside syslog() function, which prints a message with static text, if I matched it with the right version of the sources. Is the trace from running kamailio 4.1? What is the output of: kamailio -v Cheers, Daniel On 04/12/14 17:23, Daniel-Constantin Mierla wrote: Btw, you need to install dbg symbols package, in case you installed from some deb/rpm. Daniel On 04/12/14 17:22, Daniel-Constantin Mierla wrote: Hello, I need the output of the 'bt full' -- I guess it will require a few 'more' paginations, because it is printing the values for local variables which are important to troubleshoot. Cheers, daniel On 04/12/14 17:13, Timo Klecker wrote: Hi Daniel, gdb /path/to/kamailio /path/to/corefile sent me into interactive mode. Typing bt full now sent me into something like "more" so I had to copy everything by hand, like 46k lines. So I searched the web for a non interactive method and came up with this. If it is not ok, please tell me how to get the "bt full" string out of this interactive gdb shell. I will provide you with the output of gdb itself, I missed that in the last mail. Greetings Timo Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Gesendet: Donnerstag, 4. Dezember 2014 16:47 An: Timo Klecker; 'Kamailio (SER) - Users Mailing List' Betreff: Re: AW: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri Hello, wondering why not simply sending what I asked for, respectively output for: gdb /path/to/kamailio /path/to/corefile bt full I guess I wanted to get some useful information out of those commands... Cheers, Daniel On 04/12/14 16:41, Timo Klecker wrote: Hi Daniel, sorry for sending the core files, I now sent the output of: gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" /sbin/kamailio /core.8933 Hope this is what you are looking for. Kind regards, Timo Von: Daniel-Constantin Mierla [mailto:miconda@gmail.com] Gesendet: Donnerstag, 4. Dezember 2014 16:16 An: Timo Klecker; 'Kamailio (SER) - Users Mailing List' Betreff: Re: AW: AW: [SR-Users] Kamailio Crash when modifying username of Request Uri Hello, corefiles themselves are useless. I need the backtrace from them, respectively the output of: gdb /path/to/kamailio /path/to/corefile bt full Cheers, Daniel On 04/12/14 15:56, Timo Klecker wrote: Hello Daniel, yes, with the production config I still get a core, not with the testing config, though. The last line in log with testing config is: Dec 4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: ERROR: <core> [action.c:1572]: run_actions(): WARNING: too many recursive routing table lookups (101) giving up! Dec 4 13:28:06 lvm-centos-kamailio1 /sbin/kamailio[8045]: WARNING: <core> [receive.c:214]: receive_msg(): WARNING: receive_msg: error while trying script But I have 2 cores, each 52M in size from production config. I will bzip them and send them via email. Kind regards, Timo Klecker -- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda