12 Oct
2018
12 Oct
'18
8:47 a.m.
### Description
Kamailio pipelimit module causes high CPU load (20 - 30 %) on servers with > 48 GB RAM
even if Kamailio is idle (no active calls).
strace shows that every second `open /pro/stat` is called even if timer_interval is set to
10 seconds
`modparam("pipelimit", "timer_interval", 10)`
`1539343630.677932 open("/proc/stat", O_RDONLY) = 6`
`1539343630.678278 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) =
6`
`1539343630.678418 open("/proc/net/udp", O_RDONLY) = 6`
`1539343630.679448 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.722522 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.753095 open("/proc/net/udp", O_RDONLY) = 6`
`1539343630.753838 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343630.783710 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.677966 open("/proc/stat", O_RDONLY) = 6`
`1539343631.678378 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) =
6`
`1539343631.678534 open("/proc/net/udp", O_RDONLY) = 6`
`1539343631.679784 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.717329 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.760763 open("/proc/net/udp", O_RDONLY) = 6`
`1539343631.761786 open("/proc/net/tcp", O_RDONLY) = 6`
`1539343631.805426 open("/proc/net/tcp", O_RDONLY) = 6`
strace shows that with 8 GB RAM the read syscall takes 100us (average) - with 48GB it
takes 586us (average)!
see 5 seconds interval strace for 48GB:
`2018-09-21 07:55:38,201`
`strace: Process 6907 attached`
`strace: Process 6907 detached`
`% time seconds usecs/call calls errors syscall`
`------ ----------- ----------- --------- --------- ----------------`
`99.79 1.259694 586 2148 read`
` 0.06 0.000761 19 41 open`
` 0.04 0.000469 8 61 61 pause`
` 0.04 0.000462 13 36 munmap`
` 0.03 0.000336 8 41 close`
` 0.02 0.000257 7 36 mmap`
` 0.02 0.000234 3 80 61 rt_sigreturn`
` 0.01 0.000113 3 36 fstat`
` 0.00 0.000017 3 5 kill`
`------ ----------- ----------- --------- --------- ----------------`
`100.00 1.262343 2484 122 total`
`2018-09-21 07:55:43,225`
In the documentation (https://kamailio.org/docs/modules/5.1.x/modules/pipelimit.html) I
found the following:
As reading the CPU load average is relatively
expensive (opening /proc/stat, parsing it, etc), this only happens once every
timer_interval seconds and consequently the FEEDBACK value is only at these intervals
recomputed. This in turn makes it difficult for the drop rate to adjust quickly. Worst
case scenarios are request rates going up/down instantly by thousands - it takes up to 20
seconds for the controller to adapt to the new request rate.
IMHO the doc is misleading because the periodic timer used for this is hardcoded with
1000ms.
see
`( 349 pipelimit.c ) timer_add(pl_timer, MS_TO_TICKS(1000)); /* Start it after 1000ms */`
Conclusion:
Pipelimit polls periodically every second /proc/stats and /proc/net/udp|tcp. The modparam
timer_interval does not affect the timer period.
On servers with >48 GB this causes high CPU load.
#### Reproduction
always reproducible.
### Possible Solutions
Use modparam timer_interval to initialize the `pl_timer` interval
### Additional Information
version: kamailio 5.1.2 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST,
DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC,
DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER,
USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024,
BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
* **Operating System**: Centos 7.4
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1675