100% CPU usage after NTP date update

List overview All Threads
Download

newer

older

issues with RTP on different hosts...

Multi-Array with AVPs

Stefan Kohlhauser

22 Oct 2015 22 Oct '15

11:45 a.m.

Hello everyone! Setup: I am currently using a Kamailio 4.2.3 on an embedded device (low CPU and RAM). The device is not able to store the current date, so after a reboot its 1. Jan 1970 until NTP updates the date. Problem: If the date is updated after the Kamailio has started (a leap of 45 years) the Kamailio uses up all the CPU and renders the device useless for a very long time. Notes: There are no incomming requests at the time. (But the Kamailio is not responsive during that time anyway.) Adjusting the internal Kamailio-time seems to take longer when the date update leap is bigger. My questions: 1) Can I prevent the Kamailio from using up the entire CPU after a date update? Or maybe do the internal ticks-update more gracefully? 2) For my better understanding: What happens within the Kamailio after adjust_ticks has updated the internal date? Thanks in advance for your help! Best regards, Stefan

Show replies by date

Daniel-Constantin Mierla

22 Oct 22 Oct

11:50 a.m.

Hello, can you do 'top' and see what is the pid of the process using a lot of cpu, then get the backtrace with gdb? gdb /path/to/kamailio PID bt full It will help to see what kamailio is trying to do. Cheers, Daniel On 22/10/15 10:45, Stefan Kohlhauser wrote:

...

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com

Stefan Kohlhauser

28 Oct 28 Oct

6:59 p.m.

Stefan Kohlhauser

29 Oct 29 Oct

11:40 a.m.

Same mail as last time but now in plain text. Sorry :( Gesendet: Mittwoch, 28. Oktober 2015 um 16:59 Uhr Von: "Stefan Kohlhauser" <stefan.lists(a)gmx.net> An: miconda(a)gmail.com, sr-users(a)lists.sip-router.org Betreff: Re: [SR-Users] 100% CPU usage after NTP date update Hello Daniel! We have managed to track the problem down. What causes our troubles lies in the file timer.c in the function timer_handler. Or to be more exact in the for-loop: for (prev_ticks=prev_ticks+1; prev_ticks!=saved_ticks; prev_ticks++) timer_run(prev_ticks); In our case, after the NTP time change, this tries to bring the prev_ticks (representing the year 1970) up to the saved_ticks (our current time provided by NTP) tick by tick. This is quite some calculating for 45 years. Especially on an embedded device. As a workaround we have added an additional if-clause in the adjust_ticks function that simply ignores if the 'delta' between system time and ticks gets too high. We have run a first test where this workaround seems to be ok but we'll run some further tests. Do you think this workaround might cause some problems? I.e. might there be a problem if the system time jumps ahead too far but we do not adjust the ticks? (We hope not because a time-leap backwards is ignored too :) ) Apart from our workaround: Do you think there might be a way to handle this problem less CPU consuming (or to bring the ticks up to date in a bigger bulk)? Best regards, Stefan Gesendet: Donnerstag, 22. Oktober 2015 um 09:50 Uhr Von: "Daniel-Constantin Mierla" <miconda(a)gmail.com> An: "Kamailio (SER) - Users Mailing List" <sr-users(a)lists.sip-router.org> Betreff: Re: [SR-Users] 100% CPU usage after NTP date update Hello, can you do 'top' and see what is the pid of the process using a lot of cpu, then get the backtrace with gdb? gdb /path/to/kamailio PID bt full It will help to see what kamailio is trying to do. Cheers, Daniel On 22/10/15 10:45, Stefan Kohlhauser wrote:

...

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda[http://twitter.com/#!/miconda] - http://www.linkedin.com/in/miconda[http://www.linkedin.com/in/miconda] Book: SIP Routing With Kamailio - http://www.asipto.com[http://www.asipto.com] _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users[http://lists.… SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users[http://lists.…

Daniel-Constantin Mierla

2 Nov 2 Nov

11:09 a.m.

Hello, thanks for troubleshooting this issue. Can you make a pull request on github.com kamailio project? Then we can review and analyze better the impact. Ultimately, we can make this a config options, as i think it is something good to have. The option could be the value after which the delta is ignored (because is too big) and timers jumps to execute current ticks. Cheers, Daniel On 28/10/15 16:59, Stefan Kohlhauser wrote:

...

Hello Daniel! We have managed to track the problem down. What causes our troubles lies in the file timer.c in the function timer_handler. Or to be more exact in the for-loop: for (prev_ticks=prev_ticks+1; prev_ticks!=saved_ticks; prev_ticks++) timer_run(prev_ticks); In our case, after the NTP time change, this tries to bring the prev_ticks (representing the year 1970) up to the saved_ticks (our current time provided by NTP) tick by tick. This is quite some calculating for 45 years. Especially on an embedded device. As a workaround we have added an additional if-clause in the adjust_ticks function that simply ignores if the 'delta' between system time and ticks gets too high. We have run a first test where this workaround seems to be ok but we'll run some further tests. Do you think this workaround might cause some problems? I.e. might there be a problem if the system time jumps ahead too far but we do not adjust the ticks? (We hope not because a time-leap backwards is ignored too :) ) Apart from our workaround: Do you think there might be a way to handle this problem less CPU consuming (or to bring the ticks up to date in a bigger bulk)? Best regards, Stefan *Gesendet:* Donnerstag, 22. Oktober 2015 um 09:50 Uhr *Von:* "Daniel-Constantin Mierla" <miconda(a)gmail.com> *An:* "Kamailio (SER) - Users Mailing List" <sr-users(a)lists.sip-router.org> *Betreff:* Re: [SR-Users] 100% CPU usage after NTP date update Hello, can you do 'top' and see what is the pid of the process using a lot of cpu, then get the backtrace with gdb? gdb /path/to/kamailio PID bt full It will help to see what kamailio is trying to do. Cheers, Daniel On 22/10/15 10:45, Stefan Kohlhauser wrote:

Hello everyone! Setup: I am currently using a Kamailio 4.2.3 on an embedded device (low CPU

and RAM). The device is not able to store the current date, so after a reboot its 1. Jan 1970 until NTP updates the date.

Problem: If the date is updated after the Kamailio has started (a leap of 45

years) the Kamailio uses up all the CPU and renders the device useless for a very long time.

Notes: There are no incomming requests at the time. (But the Kamailio is

not responsive during that time anyway.)

Adjusting the internal Kamailio-time seems to take longer when the

date update leap is bigger.

My questions: 1) Can I prevent the Kamailio from using up the entire CPU after a

date update? Or maybe do the internal ticks-update more gracefully?

2) For my better understanding: What happens within the Kamailio

after adjust_ticks has updated the internal date?

Thanks in advance for your help! Best regards, Stefan _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com _______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users(a)lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users

-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat

3308

days inactive

3319

days old

sr-users@lists.kamailio.org

Manage subscription

4 comments

2 participants

tags (0)

participants (2)

Daniel-Constantin Mierla
Stefan Kohlhauser