[SR-Users] Re: Kamailio stops processing the calls - restart fixes it.

Hello, Just to sort out some easier reasons: do you increased the shared memory pool and private memory pool in your configuration? The defaults are pretty low. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com -----Original Message----- From: Muhammad Danish Moosa &lt;danishmoosa(a)gmail.com&gt; Sent: Mittwoch, 8. März 2023 02:21 To: Henning Westerholt &lt;hw(a)gilawa.com&gt; Cc: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio stops processing the calls - restart fixes it. Hi Guys, While I am still testing v 5.6.4 on my lab, Production server (v 5.5.3) became unresponsive again. I took "bt full" to all processes and have been analysing that. It looks like a memory allocation issue. I see frequent occurrences of "No symbol table info available". This is a purpose specific server with plenty of resources available including 8G RAM and hardly a fraction of that is used. Only main services are redis and Kamailio. I am not using mysql as well. It has only 30-40 concurrent calls. . I have seen selinux is permissive and uimits has liberated values. Do I need to do any other special OS level settings? Or is it really a version upgrade required? Should I get rid of KEMI or do config directly in the cfg file? I had seen another user reported that issue https://github.com/kamailio/kamailio/issues/2380 and was advised to use v 5.3.5, This is pretty strange because I had experience with opensips before , even with the very old version it used to handle 3k calls and literally several years of non-interruptive function without a single restart or fine tuning etc. Not sure I can attach the logs here , please see the excerpts below. Excerpts from bt full: Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 #0 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x000000000042dec6 in main_loop () at main.c:1904 i = 8 pid = 22833 si = 0x0 si_desc = "udp receiver child=7 sock=10.10.16.240:5060\000:\000\000\000\001\000\000\000\376\177\000\000\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300\351\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@\300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #2 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\312\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 long_options = {{name = 0x7e361f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7de694 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7e3624 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7e362a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7e3630 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7e3639 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7e3643 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7e364d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7e3658 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7e3661 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7e366c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7e3672 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x7e367c "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" [Inferior 1 (process 22811) detached] x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b9094 in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f20a78, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22679472} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f20a78, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f20a78, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK04B64e74bc7ae4d17c8\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bK2280db05\r\nFrom: \"Danish Queue\""..., len=1366, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f20a78 Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b908a in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f26158, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22689984} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f26158, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f26158, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len=1372, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f26158 ctx = {rec_lev = 259, run_flags = 0, last_retcode = 1677719528, jmp_env = {{__jmpbuf = {8717649, 0, 2, 0, 22808048, 0, 0, 140045002453120}, __mask_was_saved = 8, __saved_mask = {__val = {8589934604, 528280977410, 257698037764, 1, 39600, 21349824, 0, 0, 0, 140732587295568, 8311117, 4309056, 8573144, 1372, 140044999523263, 0}}}}} bctx = 0x0 ret = 0 tvb = {tv_sec = 1677719528, tv_usec = 404766} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = {s = 0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len = 1372} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0xae6060 <_sr_kemi_eng_list> evp = {data = 0x7ffedbdfbd30, obuf = {s = 0x0, len = 0}, rcv = 0x7ffedbdfc200, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 1 __FUNCTION__ = "receive_msg" #27 0x000000000047e32e in udp_rcv_loop () at core/udp_server.c:543 len = 1372 buf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""... tmp = 0x7f5ebd6c48f0 "" fromaddr = 0x7f5ec3eec738 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {3088058890, 0}, addr32 = {3088058890, 0, 0, 0}, addr16 = {2570, 47120, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\270", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {4027582986, 0}, addr32 = {4027582986, 0, 0, 0}, addr16 = {2570, 61456, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\360", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\n\020\270\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 3088058890}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3088058890, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\023\304\n\n\020\270", '\000' <repeats 111 times>, __ss_align = 0}}, bind_address = 0x7f5ec3ec6308, rflags = (unknown: 0), proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0 0D 0A Via: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK0D \000A \000\300\337\333\376\177\000\000\351\305h\301^\177\000\000\330Ђ\000\000\000\000\000\a\000\000\000\004\000\000\000P\301\337\333\376\177\000\000\267\343T\000\000\000\000\000\360\016\201\000\000\000\000\000\360׀\000\000\000\000\000\004\000\000\000\000\000\000\000\004\000\000\000\004\000\000\000\230\337h\301^\177\000\000\334\001"... i = 100 j = 106 l = 4 __FUNCTION__ = "udp_rcv_loop" #28 0x000000000042b3a5 in main_loop () at main.c:1730 i = 3 pid = 0 si = 0x7f5ec3ec6308 si_desc = "udp receiver child=3 sock=192.168.10.240:5060\000:\000\000\000\001\000\000\000\376\177\000\000\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300\351\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@\300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #29 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\312\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 On Fri, Mar 3, 2023 at 12:50 AM Henning Westerholt &lt;hw(a)gilawa.com&gt; wrote: -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless