### Description
Kamailio pipelimit module causes high CPU load (20 - 30 %) on servers with > 48 GB RAM even if Kamailio is idle (no active calls).
strace shows that every second `open /pro/stat` is called even if timer_interval is set to 10 seconds `modparam("pipelimit", "timer_interval", 10)`
`1539343630.677932 open("/proc/stat", O_RDONLY) = 6` `1539343630.678278 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 6` `1539343630.678418 open("/proc/net/udp", O_RDONLY) = 6` `1539343630.679448 open("/proc/net/tcp", O_RDONLY) = 6` `1539343630.722522 open("/proc/net/tcp", O_RDONLY) = 6` `1539343630.753095 open("/proc/net/udp", O_RDONLY) = 6` `1539343630.753838 open("/proc/net/tcp", O_RDONLY) = 6` `1539343630.783710 open("/proc/net/tcp", O_RDONLY) = 6` `1539343631.677966 open("/proc/stat", O_RDONLY) = 6` `1539343631.678378 open("/sys/devices/system/cpu/online", O_RDONLY|O_CLOEXEC) = 6` `1539343631.678534 open("/proc/net/udp", O_RDONLY) = 6` `1539343631.679784 open("/proc/net/tcp", O_RDONLY) = 6` `1539343631.717329 open("/proc/net/tcp", O_RDONLY) = 6` `1539343631.760763 open("/proc/net/udp", O_RDONLY) = 6` `1539343631.761786 open("/proc/net/tcp", O_RDONLY) = 6` `1539343631.805426 open("/proc/net/tcp", O_RDONLY) = 6`
strace shows that with 8 GB RAM the read syscall takes 100us (average) - with 48GB it takes 586us (average)!
see 5 seconds interval strace for 48GB:
`2018-09-21 07:55:38,201` `strace: Process 6907 attached` `strace: Process 6907 detached` `% time seconds usecs/call calls errors syscall` `------ ----------- ----------- --------- --------- ----------------` `99.79 1.259694 586 2148 read` ` 0.06 0.000761 19 41 open` ` 0.04 0.000469 8 61 61 pause` ` 0.04 0.000462 13 36 munmap` ` 0.03 0.000336 8 41 close` ` 0.02 0.000257 7 36 mmap` ` 0.02 0.000234 3 80 61 rt_sigreturn` ` 0.01 0.000113 3 36 fstat` ` 0.00 0.000017 3 5 kill` `------ ----------- ----------- --------- --------- ----------------` `100.00 1.262343 2484 122 total` `2018-09-21 07:55:43,225`
In the documentation (https://kamailio.org/docs/modules/5.1.x/modules/pipelimit.html) I found the following:
As reading the CPU load average is relatively expensive (opening /proc/stat, parsing it, etc), this only happens once every timer_interval seconds and consequently the FEEDBACK value is only at these intervals recomputed. This in turn makes it difficult for the drop rate to adjust quickly. Worst case scenarios are request rates going up/down instantly by thousands - it takes up to 20 seconds for the controller to adapt to the new request rate.
IMHO the doc is misleading because the periodic timer used for this is hardcoded with 1000ms. see `( 349 pipelimit.c ) timer_add(pl_timer, MS_TO_TICKS(1000)); /* Start it after 1000ms */`
Conclusion: Pipelimit polls periodically every second /proc/stats and /proc/net/udp|tcp. The modparam timer_interval does not affect the timer period. On servers with >48 GB this causes high CPU load.
#### Reproduction
always reproducible.
### Possible Solutions
Use modparam timer_interval to initialize the `pl_timer` interval
### Additional Information
version: kamailio 5.1.2 (x86_64/linux) flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: unknown
* **Operating System**: Centos 7.4
@tpeham - after quick check in the code, it seems also that the counter is reset every second, so that means it should work only for timer_interval=1. I am traveling and cannot do any testing in an easy way, if no one takes over, then I will look more into it during next week.
Can you try with latest master (or with the commit fb408fb) and see if now is all ok?
@miconda - First of all, thank you very much for your support! I took a look at the changes.
- What is the idea behind the pl_timer_mode? I did not get that ... - One thing I faced yesterday: if we are using RED or TAILDROP algorithm the /proc polling is not used - right? So maybe you could introduce additionally a modparam to enable / disable /proc/polling - what do you think?
@tpeham - `timer_mode` is just to control which timer process is used internally, so far it was the fast timer, but that is kind of busy resources, so this parameter allows to shift the execution of pipelimit timer function to another process, which can help reducing cpu usage/load of fast timer in some busy systems.
Your second point suggestion is good to avoid doing useless operations when not needed.
Closed #1675.
Closing this one, if any issue found after the latest additions on this topic referenced above, better open a new issue, given there has been changes in the code compared with the time when this issue was reported.