Hi folks,
I am doing some experiments with Kamailio and Cgrates. To be precise I am testing evapi module. When in testing environment I do not have any problem. After redirecting some production traffic to this kamailio, It crashes in at random times (few minutes to 20). I didn't find any pattern here.
CGRateS doesn't write anything in logs.
I'm on the last 4.4 Kamailio (compiled recently). CGRateS is on the last version.
Can you please point to how to continue with debugging?
Thanks
Efelin
Here is the problem in syslog:
kamailio[9594]: : <core> [mem/f_malloc.c:599]: fm_free(): BUG: fm_free: bad pointer 0x97894818 (out of memory block!), called from tm: h_table.c: free_cell_helper(185) - abortingkamailio[9608]: CRITICAL: <core> [pass_fd.c:277]: receive_fd(): EOF on 16 kamailio[9588]: ALERT: <core> [main.c:743]: handle_sigs(): child process 9594 exited by a signal 6 kamailio[9588]: ALERT: <core> [main.c:746]: handle_sigs(): core was not generated kamailio[9588]: INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD kamailio[9604]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9606]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9603]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received
bt full
#0 0xb77d1424 in __kernel_vsyscall () No symbol table info available. #1 0xb7673661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #2 0xb7676a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #3 0x082611a3 in fm_free (qmp=0x56779000, p=0x2d4e4f49, file=0x970932ee "tm: h_table.c", func=0x970935b7 "free_cell_helper", line=185, mname=0x97092d20 "tm") at mem/f_malloc.c:601 qm = 0x56779000 f = 0x56df046c __FUNCTION__ = "fm_free" #4 0x96fd0559 in free_cell_helper (dead_cell=0x56d935c4, silent=0, fname=0x970b90ca "timer.c", fline=648) at h_table.c:185 b = 0x56df0490 "CANCEL sip:00444123456@192.168.1.3:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.1.2;branch=z9hG4bKb6ae.5d03d7e94e5ec25f34930629f434a006.1\r\nMax-Forwards: 66\r\nFrom: "0044454321" <sip:0044454321@i"... i = 1 rpl = 0x9797e818 tt = 0xbfe44c68 foo = 0x970ccb14 cbs = 0x0 cbs_tmp = 0x56d12b68 __FUNCTION__ = "free_cell_helper" #5 0x9706884f in wait_handler (ti=31715179, wait_tl=0x56d9360c, data=0x56d935c4) at timer.c:648 p_cell = 0x56d935c4 ret = 0 #6 0x08241e20 in timer_list_expire (t=31715179, h=0x567a3b68, slow_l=0x567a5394, slow_mark=1772) at timer.c:873 tl = 0x56d9360c ret = 0 #7 0x08242211 in timer_handler () at timer.c:938 saved_ticks = 31715179 run_slow_timer = 0 i = 748 __FUNCTION__ = "timer_handler" #8 0x0824266b in timer_main () at timer.c:977 No locals. #9 0x080f0005 in main_loop () at main.c:1699 i = 3 pid = 0 si = 0x0 si_desc = "udp receiver child=2 sock=192.168.1.2:5060\000\270\301\310V\270\301\310V\000\000\000\000\000\000\000\020c\000\000\000\001\000\000\000\000\000\000\000\374Q\344\277HO\344\277\341`&\b\230\071\220\227\004\000\000\000٧5\b$\271\065\bn\000\000\000\324\247\065\b\n\000\000\000\363\320\372\060", '\000' <repeats 11 times> nrprocs = 3 woneinit = 1 __FUNCTION__ = "main_loop" #10 0x080f6b3a in main (argc=13, argv=0xbfe45374) at main.c:2644 cfg_stream = 0x8491008 c = -1 r = 0 tmp = 0xbfe4577c "" tmp_len = 1 port = 0 proto = 0 options = 0x8350e74 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 1759582648 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfe452a0 p = 0x805cc8c "[\201Ä4;" st = {st_dev = 14, __pad1 = 0, st_ino = 6161, st_mode = 16877, st_nlink = 2, st_uid = 109, st_gid = 111, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1560160293, tv_nsec = 983219454}, st_mtim = {tv_sec = 1560165082, tv_nsec = 861178334}, st_ctim = {tv_sec = 1560165082, tv_nsec = 861178334}, __unused4 = 0, __unused5 = 0} __FUNCTION__ = "main"
Hello,
kamailio 4.4 is no longer maintained. Upgrade to latest kamailio 5.2.x and try again. The backtrace indicates the race in tm wait handler that was fixed a while ago, but likely after 4.4 series.
If you still get the issue with 5.2.x, then send the new backtrace you get with that version.
Cheers, Daniel
On 18.06.19 16:29, Efelin Novak wrote:
Hi folks,
I am doing some experiments with Kamailio and Cgrates. To be precise I am testing evapi module. When in testing environment I do not have any problem. After redirecting some production traffic to this kamailio, It crashes in at random times (few minutes to 20). I didn't find any pattern here.
CGRateS doesn't write anything in logs.
I'm on the last 4.4 Kamailio (compiled recently). CGRateS is on the last version.
Can you please point to how to continue with debugging?
Thanks
Efelin
Here is the problem in syslog:
kamailio[9594]: : <core> [mem/f_malloc.c:599]: fm_free(): BUG: fm_free: bad pointer 0x97894818 (out of memory block!), called from tm: h_table.c: free_cell_helper(185) - abortingkamailio[9608]: CRITICAL: <core> [pass_fd.c:277]: receive_fd(): EOF on 16 kamailio[9588]: ALERT: <core> [main.c:743]: handle_sigs(): child process 9594 exited by a signal 6 kamailio[9588]: ALERT: <core> [main.c:746]: handle_sigs(): core was not generated kamailio[9588]: INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD kamailio[9604]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9606]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9603]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received
bt full
#0 0xb77d1424 in __kernel_vsyscall () No symbol table info available. #1 0xb7673661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #2 0xb7676a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #3 0x082611a3 in fm_free (qmp=0x56779000, p=0x2d4e4f49, file=0x970932ee "tm: h_table.c", func=0x970935b7 "free_cell_helper", line=185, mname=0x97092d20 "tm") at mem/f_malloc.c:601 qm = 0x56779000 f = 0x56df046c __FUNCTION__ = "fm_free" #4 0x96fd0559 in free_cell_helper (dead_cell=0x56d935c4, silent=0, fname=0x970b90ca "timer.c", fline=648) at h_table.c:185 b = 0x56df0490 "CANCEL sip:00444123456@192.168.1.3:5060 http://sip:00444123456@192.168.1.3:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.1.2;branch=z9hG4bKb6ae.5d03d7e94e5ec25f34930629f434a006.1\r\nMax-Forwards: 66\r\nFrom: "0044454321" <sip:0044454321@i"... i = 1 rpl = 0x9797e818 tt = 0xbfe44c68 foo = 0x970ccb14 cbs = 0x0 cbs_tmp = 0x56d12b68 __FUNCTION__ = "free_cell_helper" #5 0x9706884f in wait_handler (ti=31715179, wait_tl=0x56d9360c, data=0x56d935c4) at timer.c:648 p_cell = 0x56d935c4 ret = 0 #6 0x08241e20 in timer_list_expire (t=31715179, h=0x567a3b68, slow_l=0x567a5394, slow_mark=1772) at timer.c:873 tl = 0x56d9360c ret = 0 #7 0x08242211 in timer_handler () at timer.c:938 saved_ticks = 31715179 run_slow_timer = 0 i = 748 __FUNCTION__ = "timer_handler" #8 0x0824266b in timer_main () at timer.c:977 No locals. #9 0x080f0005 in main_loop () at main.c:1699 i = 3 pid = 0 si = 0x0 si_desc = "udp receiver child=2 sock=192.168.1.2:5060 http://192.168.1.2:5060\000\270\301\310V\270\301\310V\000\000\000\000\000\000\000\020c\000\000\000\001\000\000\000\000\000\000\000\374Q\344\277HO\344\277\341`&\b\230\071\220\227\004\000\000\000٧5\b$\271\065\bn\000\000\000\324\247\065\b\n\000\000\000\363\320\372\060", '\000' <repeats 11 times> nrprocs = 3 woneinit = 1 __FUNCTION__ = "main_loop" #10 0x080f6b3a in main (argc=13, argv=0xbfe45374) at main.c:2644 cfg_stream = 0x8491008 c = -1 r = 0 tmp = 0xbfe4577c "" tmp_len = 1 port = 0 proto = 0 options = 0x8350e74 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 1759582648 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfe452a0 p = 0x805cc8c "[\201Ä4;" st = {st_dev = 14, __pad1 = 0, st_ino = 6161, st_mode = 16877, st_nlink = 2, st_uid = 109, st_gid = 111, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1560160293, tv_nsec = 983219454}, st_mtim = {tv_sec = 1560165082, tv_nsec = 861178334}, st_ctim = {tv_sec = 1560165082, tv_nsec = 861178334}, __unused4 = 0, __unused5 = 0} __FUNCTION__ = "main"
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
I was afraid of such answer :) I'll have a look what I can do here.
Thanks anyway.
Regards
Efelin
ut 18. 6. 2019 o 16:41 Daniel-Constantin Mierla miconda@gmail.com napísal(a):
Hello,
kamailio 4.4 is no longer maintained. Upgrade to latest kamailio 5.2.x and try again. The backtrace indicates the race in tm wait handler that was fixed a while ago, but likely after 4.4 series.
If you still get the issue with 5.2.x, then send the new backtrace you get with that version.
Cheers, Daniel On 18.06.19 16:29, Efelin Novak wrote:
Hi folks,
I am doing some experiments with Kamailio and Cgrates. To be precise I am testing evapi module. When in testing environment I do not have any problem. After redirecting some production traffic to this kamailio, It crashes in at random times (few minutes to 20). I didn't find any pattern here.
CGRateS doesn't write anything in logs.
I'm on the last 4.4 Kamailio (compiled recently). CGRateS is on the last version.
Can you please point to how to continue with debugging?
Thanks
Efelin
Here is the problem in syslog:
kamailio[9594]: : <core> [mem/f_malloc.c:599]: fm_free(): BUG: fm_free: bad pointer 0x97894818 (out of memory block!), called from tm: h_table.c: free_cell_helper(185) - abortingkamailio[9608]: CRITICAL: <core> [pass_fd.c:277]: receive_fd(): EOF on 16 kamailio[9588]: ALERT: <core> [main.c:743]: handle_sigs(): child process 9594 exited by a signal 6 kamailio[9588]: ALERT: <core> [main.c:746]: handle_sigs(): core was not generated kamailio[9588]: INFO: <core> [main.c:768]: handle_sigs(): terminating due to SIGCHLD kamailio[9604]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9606]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received kamailio[9603]: INFO: <core> [main.c:823]: sig_usr(): signal 15 received
bt full
#0 0xb77d1424 in __kernel_vsyscall () No symbol table info available. #1 0xb7673661 in raise () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #2 0xb7676a92 in abort () from /lib/i386-linux-gnu/i686/cmov/libc.so.6 No symbol table info available. #3 0x082611a3 in fm_free (qmp=0x56779000, p=0x2d4e4f49, file=0x970932ee "tm: h_table.c", func=0x970935b7 "free_cell_helper", line=185, mname=0x97092d20 "tm") at mem/f_malloc.c:601 qm = 0x56779000 f = 0x56df046c __FUNCTION__ = "fm_free" #4 0x96fd0559 in free_cell_helper (dead_cell=0x56d935c4, silent=0, fname=0x970b90ca "timer.c", fline=648) at h_table.c:185 b = 0x56df0490 "CANCEL sip:00444123456@192.168.1.3:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.1.2;branch=z9hG4bKb6ae.5d03d7e94e5ec25f34930629f434a006.1\r\nMax-Forwards: 66\r\nFrom: "0044454321" <sip:0044454321@i"... i = 1 rpl = 0x9797e818 tt = 0xbfe44c68 foo = 0x970ccb14 cbs = 0x0 cbs_tmp = 0x56d12b68 __FUNCTION__ = "free_cell_helper" #5 0x9706884f in wait_handler (ti=31715179, wait_tl=0x56d9360c, data=0x56d935c4) at timer.c:648 p_cell = 0x56d935c4 ret = 0 #6 0x08241e20 in timer_list_expire (t=31715179, h=0x567a3b68, slow_l=0x567a5394, slow_mark=1772) at timer.c:873 tl = 0x56d9360c ret = 0 #7 0x08242211 in timer_handler () at timer.c:938 saved_ticks = 31715179 run_slow_timer = 0 i = 748 __FUNCTION__ = "timer_handler" #8 0x0824266b in timer_main () at timer.c:977 No locals. #9 0x080f0005 in main_loop () at main.c:1699 i = 3 pid = 0 si = 0x0 si_desc = "udp receiver child=2 sock=192.168.1.2:5060\000\270\301\310V\270\301\310V\000\000\000\000\000\000\000\020c\000\000\000\001\000\000\000\000\000\000\000\374Q\344\277HO\344\277\341`&\b\230\071\220\227\004\000\000\000٧5\b$\271\065\bn\000\000\000\324\247\065\b\n\000\000\000\363\320\372\060", '\000' <repeats 11 times> nrprocs = 3 woneinit = 1 __FUNCTION__ = "main_loop" #10 0x080f6b3a in main (argc=13, argv=0xbfe45374) at main.c:2644 cfg_stream = 0x8491008 c = -1 r = 0 tmp = 0xbfe4577c "" tmp_len = 1 port = 0 proto = 0 options = 0x8350e74 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:" ret = -1 seed = 1759582648 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0xbfe452a0 p = 0x805cc8c "[\201Ä4;" st = {st_dev = 14, __pad1 = 0, st_ino = 6161, st_mode = 16877, st_nlink = 2, st_uid = 109, st_gid = 111, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1560160293, tv_nsec = 983219454}, st_mtim = {tv_sec = 1560165082, tv_nsec = 861178334}, st_ctim = {tv_sec = 1560165082, tv_nsec = 861178334}, __unused4 = 0, __unused5 = 0} __FUNCTION__ = "main"
Kamailio (SER) - Users Mailing Listsr-users@lists.kamailio.orghttps://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda
-
Daniel-Constantin Mierla -
Efelin Novak