21 Oct
2024
21 Oct
'24
6:04 p.m.
So far I've been thinking of this topos SHM htable stuff as a replacement
for existing storages (redis/mysql).
Now, I am thinking of this topos SHM htable as a cache between kama procs
and existing storages. So this should happen:
1. on store, always keep in shm htable and write in storage db
2. on load, check shm htable first, and if not found, load from storage db
in shm htable
Should be an improvement to existing active-active setups, when redis db
becomes unavailable.
Of course, DMQ sync of this topos SHM htable can be done, in best effort
way. Can happen that db storage is unavailable and DMQ sync is too slow at
the same time => topos will still fail.
Thanks,
Stefan
On Mon, Oct 21, 2024 at 4:28 PM Stefan Mititelu <
stefan.mititelu(a)net2phone.com> wrote:
Hi,
Thanks for the feedback for this.
My motivation is related to an active-active scenario and supposing that
redis db becomes unavailable => topology info will then become unavailable.
Now that I am double thinking about DMQ sync, I think you are right. If DMQ
does not sync fast enough, there will be no topology info available too...
Looking in existing code, and doing some tests with redis, I see topos
does 2 types of redis store:
1. for dialog "d:..."
2. for each transaction "b:..."
I. Suppose a BYE comes to a kamailio => topos needs to have the nr 1.
already synced or will send back "404 Not here" and not forward the BYE
II. Suppose a 200OK for that BYE comes to a diiferent kamailio => topos
needs to have the nr 2. already synced or will have bad Via header and log
error
Thank you,
Stefan
On Mon, Oct 21, 2024 at 10:02 AM Henning Westerholt <hw(a)gilawa.com> wrote:
Hi Stefan,
certainly, a new storage mode “shared memory” could be added, to avoid
any additional dependencies for topos. This sounds useful.
I personally don’t see a large case for adding also DMQ synchronisation
capabilities for it. The DMQ synchronisation is not 100% reliable in all
corner cases, there are no locks or other synchronization mechanism
implemented. As a loss of topology related information would cause
immediate issues for message processing, it might be not the best choice
especially as there is already redis as a distributed in memory storage
available.
But maybe I just misunderstood the motivation for the DMQ synchronization
for topos information. If its e.g. more about an active/passive fail-over
setup, data-consistency issues are of course not a big issue.
Cheers,
Henning
*From:* Stefan Mititelu via sr-dev <sr-dev(a)lists.kamailio.org>
*Sent:* Friday, October 18, 2024 10:45 AM
*To:* sr-dev(a)lists.kamailio.org
*Cc:* Stefan Mititelu <stefan.mititelu(a)net2phone.com>
*Subject:* [sr-dev] topos module with storage "shm"
Hi,
I am thinking of this idea to add shm storage for topos module. Main
motivation for this is that if db backend is not available, there will be
no store/load happening.
Did anyone else thought of this or is doing something similar already?
I see 2 milestones here:
1. Add code in topos to keep a shm hash table with all the information
needed, guarded via locks. The api functions should be very similar to what
tps_storage.c has, just it will do ops directly in memory not on db.
2. Find a way to synchronize this topos hash table among multiple
kamailios. This should be similar to how htable module syncronize via DMQ.
Sync may happen either for each new cell, for batches or for entire topos
hash table.
What do you think of this? Any opinions, comments, appreciated.
Thank you,
Stefan Mititelu