Re: [Users] Re: [Devel] "detached" timer

wrong again :)

as I mentioned in my previous email, the "detached timer" was more an maker that something else was going wrong - there was no amplification.

and as TR clearly said, the problem was with DB connectivity and had nothing to do with TM timers.

regards, bogdan Jiri Kuthan wrote: >Actually more likely it has been both. The root problem lies in the timer subsystem >and may be amplified by other troubles (or amplify those). > >-jiri > >At 01:35 30/03/2007, T.R. Missner wrote: > >>FYI All >> >>This turned out to be a database write ( acc ) that was blocking due to a raid card problem. >> >> >> >>T.R. Missner wrote: >> >>>Is it possible the locked state I am seeing with openser leads to the "detached" timer? >>>Since the "detached" timer is a race, it would make sense to see the race condition after openser locks up and messages buffer up in the stack. >>>When a bunch of messages are processed all at once by multiple threads the race condition would occur. >>>Does this make sense? >>> >>>Maybe I have been focusing on the wrong place. >>> >>>Ignoring the "detached" timer what could cause openser to hang for a couple seconds then clear every 5 - 10 minutes? >>> >>>Ideas? >>> >>>We are seeing this on 3 different productions servers. >>> >>>Thanks >>> >>>TR >>> >>>using openser1.1.1 >>> >>> >>> >>>T.R. Missner wrote: >>> >>>>Bogdan, >>>> >>>>I have been chasing this for days and done lots of debugging. >>>>using 1.1.1 >>>>While looking at the network trace at the time of these messages ( I usually see at least 5 in a row with differing hex values ) I see many incoming packets coming into the box and no response from the proxy for somewhere between 5 - 10 seconds, then a flood a responses from the proxy. >>>>I can email you a sample pcap file if you like. >>>>As part of my debugging I forced a 100 reply at the very top of my cfg file. >>>>The forced 100 was not sent during the locked up time leading me to believe openser was not processing incoming packets. >>>>I have now seen this on multiple servers in different locations. Likely a particular customer call flow is causing this but I have not been able to pin it down to the exact customer. These proxies run pretty fast during the day so finding a pattern leading up the this issue is difficult. What could I add to the Log output to identify the offending sip-callid? Is sip-callid or branch tag or anything similar easily accessible in any of the data structs in timer.c? >>>> >>>>TR >>>> >>>>Bogdan-Andrei Iancu wrote: >>>> >>>>>Hi TR, >>>>> >>>>>it is race between expire even (from timer) and inserting again on a timer list. >>>>> 1 is the final response timer list (fr_timer) >>>>> 3 id the wait timer list (wt_timer) >>>>> >>>>>I would say there is no way this could leas to a any kind of lock. >>>>> >>>>>what version are you using? what makes you say it locks? >>>>> >>>>>regards, >>>>>bogdan >>>>> >>>>>T.R. Missner wrote: >>>>> >>>>>>Does anyone know what causes this? >>>>>> >>>>>>*/set_timer for 1 list called on a "detached" timer -- ignoring /* >>>>>> >>>>>>I also see >>>>>> >>>>>>*/set_timer for 3 list called on a "detached" timer -- ignoring /* >>>>>> >>>>>> >>>>>> >>>>>>When this happens Openser seems to lock up for 10 seconds or so. >>>>>> >>>>>>>From searching it appears this is caused by a race but I am not sure what the race is or why this results in an unresponsive openser instance for multiple seconds. >>>>>> >>>>>>Transaction expiration racing reply? >>>>>> >>>>>> >>>>>>Desperately need to understand how this could be triggered so I can get customer to adjust system. >>>>>> >>>>>>Any way to adjust? >>>>>> >>>>>>tried tweaking fr_inv_timer but no joy. >>>>>> >>>>>> >>>>>> >>>>>>TR >>>>>> >>>>> >>>>> >>>>> >>>>>-- >>>>>Jiri Kuthan http://iptel.org/~jiri/