Hi all
I have two different kamailio servers running in a single machine. The first one is a sl proxy and the second one acts as stateful proxy/registrar.
From time to time the stateless one goes out of pkg mem. It has 8MB (also tried with 16 in another setup).
The sl proxy listens the public interface for external connections and the loopback interface for conecting to the stateful proxy. As I have seen since I installed a mem debug kamailio 3.3.2 version there 6 days ago, the internal socket children have grown on mem usage while the external ones remain more or less stable.
As example one uf the external udp receivers uses less than 1MB while one of the internal ones uses almost 6MB. I'm attaching the mem dump of one of those internal udp receiver processes via "ngcp-sercmd lb cfg.set_now_int core mem_dump_pkg 25437" command. I see several tens of thousands of lines like these ones:
N address=0x7fd1207383c0 frag=0x7fd120738390 size=56 used=1 alloc'd from <core>: parser/parse_param.c: parse_params(539) start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
I have no idea about how to continue debugging this. I'm pasting some more info regarding the process list, mem status and kamailio version/flags.
Thanks
sp1:~# ngcp-sercmd lb ps 25416 attendant 25429 udp receiver child=0 sock=77.77.77.77:5060 25430 udp receiver child=1 sock=77.77.77.77:5060 25431 udp receiver child=2 sock=77.77.77.77:5060 25432 udp receiver child=3 sock=77.77.77.77:5060 25433 udp receiver child=4 sock=77.77.77.77:5060 25434 udp receiver child=5 sock=77.77.77.77:5060 25435 udp receiver child=6 sock=77.77.77.77:5060 25436 udp receiver child=7 sock=77.77.77.77:5060 25437 udp receiver child=0 sock=127.0.0.1:5060 25438 udp receiver child=1 sock=127.0.0.1:5060 25439 udp receiver child=2 sock=127.0.0.1:5060 25440 udp receiver child=3 sock=127.0.0.1:5060 25441 udp receiver child=4 sock=127.0.0.1:5060 25442 udp receiver child=5 sock=127.0.0.1:5060 25443 udp receiver child=6 sock=127.0.0.1:5060 25444 udp receiver child=7 sock=127.0.0.1:5060 25445 slow timer 25446 timer 25447 ctl handler 25448 MI FIFO 25449 tcp receiver (generic) child=0 25450 tcp receiver (generic) child=1 25451 tcp receiver (generic) child=2 25452 tcp receiver (generic) child=3 25453 tcp receiver (generic) child=4 25454 tcp receiver (generic) child=5 25455 tcp receiver (generic) child=6 25456 tcp receiver (generic) child=7 25457 tcp receiver (tcp:127.0.0.1:5060) child=8 25458 tcp receiver (tcp:127.0.0.1:5060) child=9 25459 tcp main process
sp1:~# ngcp-sercmd lb pkg.stats { entry: 0 pid: 25416 rank: 0 used: 901136 free: 15417544 real_used: 1359672 } { entry: 1 pid: 25429 rank: 1 used: 909360 free: 15408648 real_used: 1368568 } { entry: 2 pid: 25430 rank: 2 used: 909400 free: 15408608 real_used: 1368608 } { entry: 3 pid: 25431 rank: 3 used: 909352 free: 15408656 real_used: 1368560 } { entry: 4 pid: 25432 rank: 4 used: 909344 free: 15408664 real_used: 1368552 } { entry: 5 pid: 25433 rank: 5 used: 909392 free: 15408712 real_used: 1368504 } { entry: 6 pid: 25434 rank: 6 used: 909344 free: 15408664 real_used: 1368552 } { entry: 7 pid: 25435 rank: 7 used: 909344 free: 15408664 real_used: 1368552 } { entry: 8 pid: 25436 rank: 8 used: 909352 free: 15408656 real_used: 1368560 } { entry: 9 pid: 25437 rank: 9 used: 5884872 free: 2480784 real_used: 14296432 } { entry: 10 pid: 25438 rank: 10 used: 5935272 free: 2359728 real_used: 14417488 } { entry: 11 pid: 25439 rank: 11 used: 5971616 free: 2305912 real_used: 14471304 } { entry: 12 pid: 25440 rank: 12 used: 5821672 free: 2522672 real_used: 14254544 } { entry: 13 pid: 25441 rank: 13 used: 5945584 free: 2310056 real_used: 14467160 } { entry: 14 pid: 25442 rank: 14 used: 5868760 free: 2495264 real_used: 14281952 } { entry: 15 pid: 25443 rank: 15 used: 5903672 free: 2457952 real_used: 14319264 } { entry: 16 pid: 25444 rank: 16 used: 5899208 free: 2453872 real_used: 14323344 } { entry: 17 pid: 25445 rank: -1 used: 983944 free: 15409512 real_used: 2267696 } { entry: 18 pid: 25446 rank: -1 used: 909168 free: 15409512 real_used: 1367704 } { entry: 19 pid: 25447 rank: -2 used: 983664 free: 15407912 real_used: 2267032 } { entry: 20 pid: 0 rank: 0 used: 918424 free: 15400064 real_used: 1377152 } { entry: 21 pid: 25449 rank: 17 used: 989192 free: 15329200 real_used: 1448016 } { entry: 22 pid: 25450 rank: 18 used: 989192 free: 15329200 real_used: 1448016 } { entry: 23 pid: 25451 rank: 19 used: 989192 free: 15329200 real_used: 1448016 } { entry: 24 pid: 25452 rank: 20 used: 989192 free: 15329200 real_used: 1448016 } { entry: 25 pid: 25453 rank: 21 used: 989192 free: 15329200 real_used: 1448016 } { entry: 26 pid: 25454 rank: 22 used: 989192 free: 15329200 real_used: 1448016 } { entry: 27 pid: 25455 rank: 23 used: 989192 free: 15329200 real_used: 1448016 } { entry: 28 pid: 25456 rank: 24 used: 989192 free: 15329200 real_used: 1448016 } { entry: 29 pid: 25457 rank: 25 used: 989192 free: 15329200 real_used: 1448016 } { entry: 30 pid: 25458 rank: 26 used: 989192 free: 15329200 real_used: 1448016 } { entry: 31 pid: 25459 rank: -4 used: 983240 free: 15334096 real_used: 2266992 }
sp1:~# kamailio -V version: kamailio 3.3.2 (x86_64/linux) flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 13:08:06 Oct 31 2012 with gcc 4.4.5