Hi folk! Have a production server - used as webrtc to sip proxy - it is built on kamailio v 4.4.2 - Server handles connections from ~4000 incoming SIP trunks (transport=TLS). Kamailio provides auth for this trunks. - Kamailio receives also connections from Webrtc app. Calls from Webrtc routed to one of the 4000 sip trunks.
Server is running on Debian GNU/Linux 7
The number of calls on the system is small. Most of the load is generated by registration/re-registrations of SIP trunks
The problem is following: - output of kamcmd core.shmmem and kamcmd pkg.stats shows constant using of memory - output of htop shows constantly increasing memory usage, e.g. after one week working: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command ..... 6013 kamailio 20 0 744M 134M 130M S 0.0 3.6 2:28.80 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6002 kamailio 20 0 744M 134M 130M S 0.0 3.6 2:30.93 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6012 kamailio 20 0 744M 134M 130M S 0.0 3.5 2:35.00 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 5991 kamailio 20 0 744M 133M 129M S 0.0 3.5 2:05.88 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6018 kamailio 20 0 744M 133M 129M S 0.0 3.5 2:31.90 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio .....
Despite running with: PKG_MEMORY=32, Amount of used memory per process is higher. Finishes all with monit killing kamailio, because of memory usage more than allowed
studied this tutorial: https://www.kamailio.org/wiki/tutorials/troubleshooting/memory but did not find a way how to dig up the problem's root.
Tried also to reproduce the issue in lab environment (having clone of production server). Generated load with incoming registration using PJSIP, but with no success - memory usage is constant.
kamailio -v version: kamailio 4.4.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.7.2
Any suggestion/idea how to troubleshoot deeper would be highly apprecieted
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/memory-is-not-freed-in-system-while-... Sent from the Users mailing list archive at Nabble.com.
Hello,
double checking to see if you looked at:
- https://www.kamailio.org/wiki/tutorials/troubleshooting/memory#os_memory_rep...
Are you using http queries (curl library)?
Cheers, Daniel
On 18/10/16 16:39, Vasiliy Ganchev wrote:
Hi folk! Have a production server
- used as webrtc to sip proxy
- it is built on kamailio v 4.4.2
- Server handles connections from ~4000 incoming SIP trunks (transport=TLS).
Kamailio provides auth for this trunks.
- Kamailio receives also connections from Webrtc app. Calls from Webrtc
routed to one of the 4000 sip trunks.
Server is running on Debian GNU/Linux 7
The number of calls on the system is small. Most of the load is generated by registration/re-registrations of SIP trunks
The problem is following:
- output of kamcmd core.shmmem and kamcmd pkg.stats shows constant using of
memory
- output of htop shows constantly increasing memory usage, e.g. after one
week working: PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command ..... 6013 kamailio 20 0 744M 134M 130M S 0.0 3.6 2:28.80 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6002 kamailio 20 0 744M 134M 130M S 0.0 3.6 2:30.93 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6012 kamailio 20 0 744M 134M 130M S 0.0 3.5 2:35.00 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 5991 kamailio 20 0 744M 133M 129M S 0.0 3.5 2:05.88 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio 6018 kamailio 20 0 744M 133M 129M S 0.0 3.5 2:31.90 /usr/sbin/kamailio -f /etc/kamailio/kamailio.cfg -P /var/run/kamailio/kamailio.pid -m 512 -M 32 -u kamailio -g kamailio .....
Despite running with: PKG_MEMORY=32, Amount of used memory per process is higher. Finishes all with monit killing kamailio, because of memory usage more than allowed
studied this tutorial: https://www.kamailio.org/wiki/tutorials/troubleshooting/memory but did not find a way how to dig up the problem's root.
Tried also to reproduce the issue in lab environment (having clone of production server). Generated load with incoming registration using PJSIP, but with no success - memory usage is constant.
kamailio -v version: kamailio 4.4.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown compiled with gcc 4.7.2
Any suggestion/idea how to troubleshoot deeper would be highly apprecieted
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/memory-is-not-freed-in-system-while-... Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel!
Thanks for the pointing out!
Currently re-checking system memory usage and monit behavior.
Will report results later (seems that there is nothing to be done on the kamailio's point of view)
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/memory-is-not-freed-in-system-while-... Sent from the Users mailing list archive at Nabble.com.
Hello,
from kamailio point of view, the only major component relying on system memory is the MI/RPC framework/commands. Otherwise, likely to be an external library.
Cheers, Daniel
On 19/10/16 11:46, Vasiliy Ganchev wrote:
Hi Daniel!
Thanks for the pointing out!
Currently re-checking system memory usage and monit behavior.
Will report results later (seems that there is nothing to be done on the kamailio's point of view)
Cheers!
-- View this message in context: http://sip-router.1086192.n5.nabble.com/memory-is-not-freed-in-system-while-... Sent from the Users mailing list archive at Nabble.com.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Vasiliy Ganchev