Hello, On 12/22/11 6:19 PM, Andreas Granig wrote: Got the cause of the issue. What happens is that there's an AOR which registers ever 120 seconds. For some reason, the location entry is in usrloc cache, but not in db. What happens now is that usrloc tries an "update" query in the db, because it still assumes that the entry is there, which obviously fails. If you remove the entry from usrloc (kamctl ul rm <aor>), then on the next re-registration it's both inserted into the cache and into the db. Wondering how it could happen to get out of sync, and how we could improve this. Maybe using a "replace into" instead of "update", at least for mysql? Suggestions? is the timer interval parameter of usrloc higher than 120sec? http://kamailio.org/docs/modules/3.2.x/modules_k/usrloc.html#id2494575 IIRC, there should be anyhow a flag to mark if the record is in db or not, and based on that do insert or update, maybe something is lost there. If you do 'kamctl ul show __aor__', what are the values for flags fields? Another option, perhaps more portable, but with two db hits is: update and if fails then insert -- considering that these should be corner cases, maybe the performance is not affected much. A blended version is even better, if the db driver supports replace, do replace instead of update (I don't know if replace is faster/slower than update). Cheers, Daniel Andreas On 12/22/2011 05:12 PM, Andreas Granig wrote: Hi, Could you please tell me which of the three timer processes ("timer", "slow timer" or "timer nh") is responsible for cleaning up the internal usrloc cache? Looks like every now and then the cleanup of the internal location cache is starting to fail. Funny thing is that expired locations are removed from the mysql backend, but not from the internal cache. We're running kamailio 3.1.5, are there any known issues fixed since that version? In the meanwhile we're trying to pin the issue down, but maybe someone has a clue... Andreas

Right, that has to be done, but there are some cases when db can become inconsistent, due to database unavailability, and then some trick have to be done at db layer, example: - db is unavailable, phone unregisters, contact deleted from memory but not from database - phone register again, usrloc will try insert and will fail - in this case it should be update if insert fails (or replace)

The other way around could happen when mistakenly deleting/changing records in db, which should not happen, but Murphy says opposite.

It was just an example, I haven't gone to all corner cases that can happen from human or (self or different) application errors. There were couple of similar reports in the past, related to conflicts of insert/update, update/insert, delete/update a.s.o. cases, so I proposed to go for a portable solution, not for one which valid to a db driver only -- configurable or not, is different thing than the specific topic.

shouldn't the db layer and driver be smart enough to do insert ... on duplicate key update at least where it's supported?

Hi, On 12/23/2011 12:28 AM, Daniel-Constantin Mierla wrote: Of course you can never rule out admin errors or application errors, but anyways I'll highly favor a more resilient approach.

I've tried to reproduce the most obvious scenario, which is registering a subscriber, delete it from the underlying db table, then re-register again. The expiry value in the cache is refreshed, but it's never written back to the db table. There are most likely other, more subtle scenarios, and our customers approved that they didn't mess with the db manually. The state was always CS_SYNC and Flags was 0. I don't know the details of the srdb layer, but probably it's possible to find a way to return the "rows affected" after an update in order to know whether to try an insert afterwards. Would be possible with mysql, not sure about pgsql, oracle, dbtext etc. We'll take a look how we could tackle that.

I don't think that a log message would help very much, because kamailio won't know about the missing entry in the db (unless you evaluate the result of the update), at least in this particular case.