Hello,
Please also give the Kamailio version that you are using, just to sort out eventual already fixed memory leaks.
About the question regarding the 2GB memory mapping – according to your configuration this should be shared memory (not assuming a memory leak in system memory because of a library etc..).
So, it should be not 2GB per process, but 2GB in total.
And the memory is also not completely used, only 855MB of it:
shmem:real_used_size = 855.574.648
Maximum private per process memory is according to your cfg 12 MB.
Cheers,
Henning
--
Henning Westerholt –
https://skalatan.de/blog/
Kamailio services –
https://gilawa.com
From: sr-dev <sr-dev-bounces@lists.kamailio.org>
On Behalf Of Andrey Deykunov
Sent: Wednesday, April 8, 2020 12:19 PM
To: Kamailio (SER) - Development Mailing List <sr-dev@lists.kamailio.org>
Subject: [sr-dev] High memory consumption during load tests
Hi,
I've noticed more than 14 Gb RSS memory consumption during load test (8000 WSS clients send registrations):
[…]
# ps aux | grep kamailio
hpbx 15948 0.0 0.1 2232328 22920 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15949 1.8 3.8 2232460 789212 ? S 01:40 1:23 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15950 1.8 3.8 2232460 790164 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15951 1.7 3.8 2232460 790656 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15952 1.8 3.8 2232460 790180 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15953 1.7 3.8 2232460 788192 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15954 1.8 3.8 2232460 789484 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15955 1.7 3.8 2232460 789640 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15959 1.8 3.8 2232460 788104 ? S 01:40 1:22 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15960 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15961 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15963 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15964 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15966 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15968 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15969 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15971 0.0 0.0 2232328 8272 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15972 0.0 0.0 2232328 8152 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15973 0.0 0.0 2232328 8152 ? S 01:40 0:02 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15975 0.0 0.0 2232328 8156 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15977 4.8 4.0 2232428 835908 ? S 01:40 3:43 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15979 0.0 2.5 2232452 517192 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15980 0.0 2.5 2232328 515344 ? S 01:40 0:03 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15984 0.0 0.0 2232328 8004 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15986 0.0 0.0 2232328 8276 ? S 01:40 0:00 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15987 2.4 3.9 2232492 817888 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15989 2.4 3.9 2232492 816604 ? S 01:40 1:52 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15991 2.4 3.9 2232492 816952 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15992 2.3 3.9 2232492 817236 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15993 2.4 3.9 2232492 818900 ? S 01:40 1:51 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15994 2.4 3.9 2232492 817692 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15996 2.4 3.9 2232492 816832 ? S 01:40 1:50 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 15998 2.3 3.9 2232492 814336 ? S 01:40 1:49 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
hpbx 16001 2.1 3.1 2232328 655908 ? S 01:40 1:37 /var/lib/ums/sbin/kamailio -m 2048 -M 12 -x qm -P /var/run/kamailio/kamailio.pid -f /var/lib/ums/etc/sipp/sipp.cfg
Shared memory consumption is:
# /var/lib/ums/sbin/kamcmd -s tcp:localhost:2048 stats.get_statistics shmem:
shmem:fragments = 4
shmem:free_size = 1291909000
shmem:max_used_size = 855663448
shmem:real_used_size = 855574648
shmem:total_size = 2147483648
shmem:used_size = 818055392
Also, I don't understand why kamailio maps more than 2 GB per every child process:
# pmap -x $(pidof kamailio) | grep total
total kB 2232332 659732 653680
total kB 2232496 816132 802908
total kB 2232496 818680 805456
total kB 2232496 818544 805320
total kB 2232496 818900 805676
total kB 2232496 817888 804664
total kB 2232496 818436 805212
total kB 2232496 818124 804900
total kB 2232496 818508 805284
total kB 2232332 8344 3648
total kB 2232332 8072 3580
total kB 2232332 517360 512444
total kB 2232456 519752 514580
total kB 2232432 835996 829552
total kB 2232332 8428 3576
total kB 2232332 8832 3852
total kB 2232332 10380 4604
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232332 8480 3564
total kB 2232464 789152 777596
total kB 2232464 790128 778484
total kB 2232464 790904 779004
total kB 2232464 788796 777240
total kB 2232464 790284 778384
total kB 2232464 790732 779176
total kB 2232464 790360 778716
total kB 2232464 790512 778612
total kB 2232332 22920 8992
Any ideas what is happening here?
Andrey