[SR-Users] Re: Kamailio stops processing the calls - restart fixes it.

Hello, you did install the kamailio debugging symbols from the package management, right? Cheers, Henning From: Muhammad Danish Moosa &lt;danishmoosa(a)gmail.com&gt; Sent: Mittwoch, 8. März 2023 09:50 To: Henning Westerholt &lt;hw(a)gilawa.com&gt; Cc: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio stops processing the calls - restart fixes it. Hi , I will apply memory config and will monitor. Strangely I dont see any errors related to memory in logs and nagios graph does not show any spikes.There are only 30 concurrent calls. But in bt full suggests those ‘ no symbol table’ messages. Hopefully memory optimisation will solve it. On Wed, 8 Mar 2023 at 7:23 pm, Henning Westerholt &lt;hw(a)gilawa.com&gt; wrote: Hello, you can check with „ps” to see the configured values for Kamailio. On Debian you can adapt it in /etc/default/kamailio For PKG pool, using 16 MB should be usually fine. For SHM pool, you could use 1GB, depending on the server memory. Much more is usually not needed. You should see some memory related errors in case you run into a low memory situation in the log files. Cheers, Henning From: Muhammad Danish Moosa &lt;danishmoosa(a)gmail.com&gt; Sent: Mittwoch, 8. März 2023 09:11 To: Henning Westerholt &lt;hw(a)gilawa.com&gt; Cc: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio stops processing the calls - restart fixes it. Hi Henning, Thanks for your email. Oh , I have not done the explicit config for shared and private memory pool. Where can i see the related documentation? For 8G RAM , I am happy to allocate 5G for Kamailio. What config do you recommend? On Wed, 8 Mar 2023 at 6:01 pm, Henning Westerholt &lt;hw(a)gilawa.com&gt; wrote: Hello, Just to sort out some easier reasons: do you increased the shared memory pool and private memory pool in your configuration? The defaults are pretty low. Cheers, Henning -- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.com -----Original Message----- From: Muhammad Danish Moosa &lt;danishmoosa(a)gmail.com&gt; Sent: Mittwoch, 8. März 2023 02:21 To: Henning Westerholt &lt;hw(a)gilawa.com&gt; Cc: Kamailio (SER) - Users Mailing List &lt;sr-users(a)lists.kamailio.org&gt; Subject: Re: [SR-Users] Kamailio stops processing the calls - restart fixes it. Hi Guys, While I am still testing v 5.6.4 on my lab, Production server (v 5.5.3) became unresponsive again. I took "bt full" to all processes and have been analysing that. It looks like a memory allocation issue. I see frequent occurrences of "No symbol table info available". This is a purpose specific server with plenty of resources available including 8G RAM and hardly a fraction of that is used. Only main services are redis and Kamailio. I am not using mysql as well. It has only 30-40 concurrent calls. . I have seen selinux is permissive and uimits has liberated values. Do I need to do any other special OS level settings? Or is it really a version upgrade required? Should I get rid of KEMI or do config directly in the cfg file? I had seen another user reported that issue https://github.com/kamailio/kamailio/issues/2380 and was advised to use v 5.3.5, This is pretty strange because I had experience with opensips before , even with the very old version it used to handle 3k calls and literally several years of non-interruptive function without a single restart or fine tuning etc. Not sure I can attach the logs here , please see the excerpts below. Excerpts from bt full: Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 #0 0x00007f5ec46fc980 in __pause_nocancel () from /lib64/libc.so.6 No symbol table info available. #1 0x000000000042dec6 in main_loop () at main.c:1904 i = 8 pid = 22833 si = 0x0 si_desc = "udp receiver child=7 sock=10.10.16.240:5060\000:\000\000\000\001\000\000\000\376\177\000\0 00\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300\3 51\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@\300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #2 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\31 2\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 long_options = {{name = 0x7e361f "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x7de694 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x7e3624 "alias", has_arg = 1, flag = 0x0, val = 1024}, {name = 0x7e362a "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x7e3630 "substdef", has_arg = 1, flag = 0x0, val = 1026}, {name = 0x7e3639 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x7e3643 "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x7e364d "loadmodule", has_arg = 1, flag = 0x0, val = 1029}, {name = 0x7e3658 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x7e3661 "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {name = 0x7e366c "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x7e3672 "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x7e367c "atexit", has_arg = 1, flag = 0x0, val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}} __FUNCTION__ = "main" [Inferior 1 (process 22811) detached] x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b9094 in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f20a78, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22679472} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f20a78, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f20a78, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK04B64e74bc7ae4d17c8\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bK2280db05\r\nFrom: \"Danish Queue\""..., len=1366, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f20a78 Using host libthread_db library "/lib64/libthread_db.so.1". 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 #0 0x00007f5ec47437fc in __lll_lock_wait_private () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5ec46bfba2 in _L_lock_16654 () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5ec46bc7e3 in malloc () from /lib64/libc.so.6 No symbol table info available. #3 0x00007f5ec46aea6a in open_memstream () from /lib64/libc.so.6 No symbol table info available. #4 0x00007f5ec472f65a in __vsyslog_chk () from /lib64/libc.so.6 No symbol table info available. #5 0x00007f5ec472fbbf in syslog () from /lib64/libc.so.6 No symbol table info available. #6 0x0000000000424070 in sig_usr (signo=17) at main.c:920 __llevel = 3 memlog = 0 __FUNCTION__ = "sig_usr" #7 <signal handler called> No symbol table info available. #8 0x00007f5ec46b908a in _int_malloc () from /lib64/libc.so.6 No symbol table info available. #9 0x00007f5ec46bc78c in malloc () from /lib64/libc.so.6 No symbol table info available. #10 0x00007f5ec241fd0e in luaM_realloc_ () from /lib64/liblua-5.1.so No symbol table info available. #11 0x00007f5ec24239b8 in luaS_newlstr () from /lib64/liblua-5.1.so No symbol table info available. #12 0x00007f5ec2417b5a in lua_pushlstring () from /lib64/liblua-5.1.so No symbol table info available. #13 0x00007f5ec2427ad3 in emptybuffer () from /lib64/liblua-5.1.so No symbol table info available. #14 0x00007f5ec24288b9 in luaL_pushresult () from /lib64/liblua-5.1.so No symbol table info available. #15 0x00007f5ec242bb0b in read_chars () from /lib64/liblua-5.1.so No symbol table info available. #16 0x00007f5ec242bd32 in g_read () from /lib64/liblua-5.1.so No symbol table info available. #17 0x00007f5ec241c324 in luaD_precall () from /lib64/liblua-5.1.so No symbol table info available. #18 0x00007f5ec2426e57 in luaV_execute () from /lib64/liblua-5.1.so No symbol table info available. #19 0x00007f5ec241c74d in luaD_call () from /lib64/liblua-5.1.so No symbol table info available. #20 0x00007f5ec241ba6e in luaD_rawrunprotected () from /lib64/liblua-5.1.so No symbol table info available. #21 0x00007f5ec241c8da in luaD_pcall () from /lib64/liblua-5.1.so No symbol table info available. #22 0x00007f5ec241844d in lua_pcall () from /lib64/liblua-5.1.so No symbol table info available. #23 0x00007f5ec2660cbc in app_lua_run_ex (msg=0x7f5ec3f26158, func=0x7f5ec26804ae "ksr_request_route", p1=0x0, p2=0x0, p3=0x0, emode=1) at app_lua_api.c:713 n = 0 ret = 0 txt = {s = 0x7ffedbdfbb38 "Z\336P", len = 22689984} bmsg = 0x0 ltop = 17 __FUNCTION__ = "app_lua_run_ex" #24 0x00007f5ec264a6c9 in sr_kemi_config_engine_lua (msg=0x7f5ec3f26158, rtype=1, rname=0x0, rparam=0x0) at app_lua_mod.c:119 ret = -1 __FUNCTION__ = "sr_kemi_config_engine_lua" #25 0x00000000004a2592 in sr_kemi_route (keng=0xae6060 <_sr_kemi_eng_list>, msg=0x7f5ec3f26158, rtype=1, ename=0x0, edata=0x0) at core/kemi.c:3471 sfbk = 0 ret = 7 #26 0x00000000005dc9aa in receive_msg (buf=0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len=1372, rcv_info=0x7ffedbdfc200) at core/receive.c:488 msg = 0x7f5ec3f26158 ctx = {rec_lev = 259, run_flags = 0, last_retcode = 1677719528, jmp_env = {{__jmpbuf = {8717649, 0, 2, 0, 22808048, 0, 0, 140045002453120}, __mask_was_saved = 8, __saved_mask = {__val = {8589934604, 528280977410, 257698037764, 1, 39600, 21349824, 0, 0, 0, 140732587295568, 8311117, 4309056, 8573144, 1372, 140044999523263, 0}}}}} bctx = 0x0 ret = 0 tvb = {tv_sec = 1677719528, tv_usec = 404766} tve = {tv_sec = 0, tv_usec = 0} diff = 0 inb = {s = 0xad0020 <buf.7142> "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""..., len = 1372} netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0} keng = 0xae6060 <_sr_kemi_eng_list> evp = {data = 0x7ffedbdfbd30, obuf = {s = 0x0, len = 0}, rcv = 0x7ffedbdfc200, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} cidlockidx = 0 cidlockset = 0 errsipmsg = 0 exectime = 1 __FUNCTION__ = "receive_msg" #27 0x000000000047e32e in udp_rcv_loop () at core/udp_server.c:543 len = 1372 buf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK08B977c11fc3b3ee498\r\nVIA: SIP/2.0/TLS 52.114.20.29:5061;branch=z9hG4bKed64125d\r\nFrom: \"Danish Queue\""... tmp = 0x7f5ebd6c48f0 "" fromaddr = 0x7f5ec3eec738 fromaddrlen = 16 rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {3088058890, 0}, addr32 = {3088058890, 0, 0, 0}, addr16 = {2570, 47120, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\270", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {4027582986, 0}, addr32 = {4027582986, 0, 0, 0}, addr16 = {2570, 61456, 0, 0, 0, 0, 0, 0}, addr = "\n\n\020\360", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, sa_data = "\023\304\n\n\020\270\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 3088058890}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 3088058890, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, __ss_padding = "\023\304\n\n\020\270", '\000' <repeats 111 times>, __ss_align = 0}}, bind_address = 0x7f5ec3ec6308, rflags = (unknown: 0), proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0} evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0} printbuf = "REFER sip:192.168.10.100:5060;transport=tcp SIP/2.0 0D 0A Via: SIP/2.0/UDP 192.168.10.184:5060;branch=z9hG4bK0D \000A \000\300\337\333\376\177\000\000\351\305h\301^\177\000\000\330Ђ\000\000\000\000\000\a\000\000\000\004\000\000\000P\301\337\333\376\177\000\000\267\343T\000\000\000\000\000\360\016\201\000\000\000\000\000\360׀\000\000\000\000\000\004\000\000\000\000\000\000\000\004\000\000\000\004\000\000\000\230\337h\301^\177\000\000\334\001"... i = 100 j = 106 l = 4 __FUNCTION__ = "udp_rcv_loop" #28 0x000000000042b3a5 in main_loop () at main.c:1730 i = 3 pid = 0 si = 0x7f5ec3ec6308 si_desc = "udp receiver child=3 sock=192.168.10.240:5060\000:\000\000\000\001\000\000\000\376\177\000 \000\260\232\000\000\000\000\000\000\300\305E\001\000\000\000\000\300 \351\202\000k\000\000\000[̀", '\000' <repeats 13 times>, "\340\311\337\333\376\177\000\000\030\342}\000\000\000\000\000\027\000\000\000\000\000\000\000@\300A\000\000\000\000" nrprocs = 8 woneinit = 1 __FUNCTION__ = "main_loop" #29 0x000000000043684b in main (argc=5, argv=0x7ffedbdfcac8) at main.c:3053 cfg_stream = 0x1456040 c = -1 r = 0 tmp = 0x0 tmp_len = 0 port = 0 proto = 0 ahost = 0x0 aport = 0 options = 0x7e1208 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:" ret = -1 seed = 2616789248 rfd = 4 debug_save = 0 debug_flag = 0 dont_fork_cnt = 0 n_lst = 0x7ffedbdfc980 p = 0x0 st = {st_dev = 19, st_ino = 21232, st_nlink = 2, st_mode = 16832, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, st_atim = {tv_sec = 1675551785, tv_nsec = 549443593}, st_mtim = {tv_sec = 1677547400, tv_nsec = 664002717}, st_ctim = {tv_sec = 1677547400, tv_nsec = 664002717}, __unused = {0, 0, 0}} tbuf = "\377\377\377\377", '\000' <repeats 12 times>, "\350sd\304^\177\000\000\310d3\305^\177", '\000' <repeats 90 times>, "\200\022\254\000\000\000\000\000\200\304A\000\000\000\000\000\300\31 2\337\333\376\177", '\000' <repeats 26 times>, "\356=\023\305^\177\000\000\001", '\000' <repeats 23 times>... option_index = 0 On Fri, Mar 3, 2023 at 12:50 AM Henning Westerholt &lt;hw(a)gilawa.com&gt; wrote: -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless -- Muhammad Danish Moosa " The core of mans' spirit comes from new experiences. "___ Christopher McCandless