6 Nov
2012
6 Nov
'12
6:09 a.m.
Hi all
I have two different kamailio servers running in a single machine. The first
one is a sl proxy and the second one acts as stateful proxy/registrar.
From time to time the stateless one goes out of pkg mem. It has 8MB (also tried
with 16 in another setup).
The sl proxy listens the public interface for external connections and the
loopback interface for conecting to the stateful proxy. As I have seen since I
installed a mem debug kamailio 3.3.2 version there 6 days ago, the internal
socket children have grown on mem usage while the external ones remain more or
less stable.
As example one uf the external udp receivers uses less than 1MB while one of
the internal ones uses almost 6MB. I'm attaching the mem dump of one of those
internal udp receiver processes via
"ngcp-sercmd lb cfg.set_now_int core mem_dump_pkg 25437" command. I see several
tens of thousands of lines like these ones:
N address=0x7fd1207383c0 frag=0x7fd120738390 size=56 used=1
alloc'd from <core>: parser/parse_param.c: parse_params(539)
start check=f0f0f0f0, end check= c0c0c0c0, abcdefed
I have no idea about how to continue debugging this. I'm pasting some more
info regarding the process list, mem status and kamailio version/flags.
Thanks
sp1:~# ngcp-sercmd lb ps
25416 attendant
25429 udp receiver child=0 sock=77.77.77.77:5060
25430 udp receiver child=1 sock=77.77.77.77:5060
25431 udp receiver child=2 sock=77.77.77.77:5060
25432 udp receiver child=3 sock=77.77.77.77:5060
25433 udp receiver child=4 sock=77.77.77.77:5060
25434 udp receiver child=5 sock=77.77.77.77:5060
25435 udp receiver child=6 sock=77.77.77.77:5060
25436 udp receiver child=7 sock=77.77.77.77:5060
25437 udp receiver child=0 sock=127.0.0.1:5060
25438 udp receiver child=1 sock=127.0.0.1:5060
25439 udp receiver child=2 sock=127.0.0.1:5060
25440 udp receiver child=3 sock=127.0.0.1:5060
25441 udp receiver child=4 sock=127.0.0.1:5060
25442 udp receiver child=5 sock=127.0.0.1:5060
25443 udp receiver child=6 sock=127.0.0.1:5060
25444 udp receiver child=7 sock=127.0.0.1:5060
25445 slow timer
25446 timer
25447 ctl handler
25448 MI FIFO
25449 tcp receiver (generic) child=0
25450 tcp receiver (generic) child=1
25451 tcp receiver (generic) child=2
25452 tcp receiver (generic) child=3
25453 tcp receiver (generic) child=4
25454 tcp receiver (generic) child=5
25455 tcp receiver (generic) child=6
25456 tcp receiver (generic) child=7
25457 tcp receiver (tcp:127.0.0.1:5060) child=8
25458 tcp receiver (tcp:127.0.0.1:5060) child=9
25459 tcp main process
sp1:~# ngcp-sercmd lb pkg.stats
{
entry: 0
pid: 25416
rank: 0
used: 901136
free: 15417544
real_used: 1359672
}
{
entry: 1
pid: 25429
rank: 1
used: 909360
free: 15408648
real_used: 1368568
}
{
entry: 2
pid: 25430
rank: 2
used: 909400
free: 15408608
real_used: 1368608
}
{
entry: 3
pid: 25431
rank: 3
used: 909352
free: 15408656
real_used: 1368560
}
{
entry: 4
pid: 25432
rank: 4
used: 909344
free: 15408664
real_used: 1368552
}
{
entry: 5
pid: 25433
rank: 5
used: 909392
free: 15408712
real_used: 1368504
}
{
entry: 6
pid: 25434
rank: 6
used: 909344
free: 15408664
real_used: 1368552
}
{
entry: 7
pid: 25435
rank: 7
used: 909344
free: 15408664
real_used: 1368552
}
{
entry: 8
pid: 25436
rank: 8
used: 909352
free: 15408656
real_used: 1368560
}
{
entry: 9
pid: 25437
rank: 9
used: 5884872
free: 2480784
real_used: 14296432
}
{
entry: 10
pid: 25438
rank: 10
used: 5935272
free: 2359728
real_used: 14417488
}
{
entry: 11
pid: 25439
rank: 11
used: 5971616
free: 2305912
real_used: 14471304
}
{
entry: 12
pid: 25440
rank: 12
used: 5821672
free: 2522672
real_used: 14254544
}
{
entry: 13
pid: 25441
rank: 13
used: 5945584
free: 2310056
real_used: 14467160
}
{
entry: 14
pid: 25442
rank: 14
used: 5868760
free: 2495264
real_used: 14281952
}
{
entry: 15
pid: 25443
rank: 15
used: 5903672
free: 2457952
real_used: 14319264
}
{
entry: 16
pid: 25444
rank: 16
used: 5899208
free: 2453872
real_used: 14323344
}
{
entry: 17
pid: 25445
rank: -1
used: 983944
free: 15409512
real_used: 2267696
}
{
entry: 18
pid: 25446
rank: -1
used: 909168
free: 15409512
real_used: 1367704
}
{
entry: 19
pid: 25447
rank: -2
used: 983664
free: 15407912
real_used: 2267032
}
{
entry: 20
pid: 0
rank: 0
used: 918424
free: 15400064
real_used: 1377152
}
{
entry: 21
pid: 25449
rank: 17
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 22
pid: 25450
rank: 18
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 23
pid: 25451
rank: 19
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 24
pid: 25452
rank: 20
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 25
pid: 25453
rank: 21
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 26
pid: 25454
rank: 22
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 27
pid: 25455
rank: 23
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 28
pid: 25456
rank: 24
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 29
pid: 25457
rank: 25
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 30
pid: 25458
rank: 26
used: 989192
free: 15329200
real_used: 1448016
}
{
entry: 31
pid: 25459
rank: -4
used: 983240
free: 15334096
real_used: 2266992
}
sp1:~# kamailio -V
version: kamailio 3.3.2 (x86_64/linux)
flags: STATS: Off, USE_IPV6, USE_TCP, USE_TLS, TLS_HOOKS, USE_RAW_SOCKS,
DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC,
DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE,
USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16,
MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 4MB poll method support:
poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled on 13:08:06
Oct 31 2012 with gcc 4.4.5