Hi,
Is there a simple way to lookup contacts directly from a DB table instead of memory? A quick look into the register module doesn't show anything like that.
The idea behind that is to build a redundant SER cluster. All nodes write their registrations directly into a MySQL master database, which replicates the contacts to the slave databases. The SERs then lookup contacts from their local slave databases. The master database will be secured by a MySQL cluster.
This is because if I replicate registers on SIP layer (for example with t_replicate()), I have to synchronize the location databases manually from time to time, because register replications get lost on heavy SIP load, and there is actually no way to automatically load contacts which are stored on other nodes while one SER was down. Their is also no need to send replicated registers over the net which reduces SIP messages.
I don't really know how much impact this approach will have regarding performance (going around the internal location memory), but I think it will not hurt as much as having out-of-synch location tables on the SER nodes.
Any comments on this approach? Anybody who is also interested in such approach?
Andy
Have a look at: http://www.iptel.org/ser/doc/modules/html/usrloc.html#AEN153
However, this probably does not do all you need, as you still need lookup to synchronize with the db. This could maybe be done with the timer_interval parameter (lower it). I haven't had a look at the code implementing the flush/synchronization, but that could be a start.
I haven't really validated the idea of using database replication/cluster, so there are probably other things to consider as well. I wouldn't worry about heavy SIP load and lost registers because you should a long time before that have taken measures to keep your systems at a lower load... However, synchronization of servers that went down is an issue. Haven't really seen any discussions on the list on replication in itself as a redundancy issue. Anyone?
Locations are not really that critical, are they? If you use registration intervals of ex. 10 minutes, how do you even manage to synchronize the databases manually?
Any more thoughts on your usage scenario and why you would put the effort into moving t_replicate to SQL replication? g-)
Andreas Granig wrote:
Hi,
Is there a simple way to lookup contacts directly from a DB table instead of memory? A quick look into the register module doesn't show anything like that.
The idea behind that is to build a redundant SER cluster. All nodes write their registrations directly into a MySQL master database, which replicates the contacts to the slave databases. The SERs then lookup contacts from their local slave databases. The master database will be secured by a MySQL cluster.
This is because if I replicate registers on SIP layer (for example with t_replicate()), I have to synchronize the location databases manually from time to time, because register replications get lost on heavy SIP load, and there is actually no way to automatically load contacts which are stored on other nodes while one SER was down. Their is also no need to send replicated registers over the net which reduces SIP messages.
I don't really know how much impact this approach will have regarding performance (going around the internal location memory), but I think it will not hurt as much as having out-of-synch location tables on the SER nodes.
Any comments on this approach? Anybody who is also interested in such approach?
Andy
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
See comments inline...
Greger V. Teigre wrote:
Have a look at: http://www.iptel.org/ser/doc/modules/html/usrloc.html#AEN153
Synchronization *to* a DB works of course out of the box (needed for saving locations).
But for clustering you will need to synchronize your memory *from* the DB or from the memory of the other nodes on the fly to get locations from other nodes so you can lookup them again.
Locations are not really that critical, are they? If you use registration intervals of ex. 10 minutes, how do you even manage to synchronize the databases manually?
We've about 20.000 users, which would result in ~2.900.000 successful registrations per day or ~35 successful registerations per second, and that means ~70 register messages per second (because of the www_challenge).
Replicating on SIP layer means that every successful REGISTER has to be sent to every server, so IMHO this just will not scale.
Cheers, Andy
:-) You never know where people are when they ask questions. Better start with the basics... See inline comments.
Have a look at: http://www.iptel.org/ser/doc/modules/html/usrloc.html#AEN153
Synchronization *to* a DB works of course out of the box (needed for saving locations).
But for clustering you will need to synchronize your memory *from* the DB or from the memory of the other nodes on the fly to get locations from other nodes so you can lookup them again.
Yes, I did a quick look at the usrloc module code and can see that resync from DB should be possible, but will probably take some time and I don't think that is a good path.
Actually, Java Rocxx (Paul) and I tried to get a carrier-grade platform discussion going in December, I think, but the reponse was not really there. We don't use sql clusters, but have distributed data centers with IPsec VPN connecting the centers. Each center is as self-contained as possible. I think Paul mentioned that his approach was to use the FIFO queue with a TCP/IP wrapper. An ul_add FIFO command to each server should probably do the trick. I assume that the sql synchronization will not take less bandwidth?! Unfortunately, Paul's company probably would like to keep their solution secret, it's part of their commercial product. However, I would be interested in a joint effort that could be contributed to the project. A SOAP wrapper comes to mind as a solution to the network exchange, but maybe even the SOAP exhange will add too much overhead? Well, probably not, but just thinking out aloud here. What would be the best replication implementation in ser? A new module? The code would trigger a network exchange and would have to be forked. Honestly, I haven't looked at the inner workings of ser, so I don't know. Any pointers, anybody?
Locations are not really that critical, are they? If you use registration intervals of ex. 10 minutes, how do you even manage to synchronize the databases manually?
We've about 20.000 users, which would result in ~2.900.000 successful registrations per day or ~35 successful registerations per second, and that means ~70 register messages per second (because of the www_challenge). Replicating on SIP layer means that every successful REGISTER has to be sent to every server, so IMHO this just will not scale.
Excellent! Don't make it a problem until it is :-) Do you do load balancing with DNS SRV or Vovida load balancer or similar? Is the MySQL cluster networked or in the same physical location? (I don't know much about mySQL clusters...) Paul claimed voicemail to be their single, biggest problem, citing sems to be the most promising. We don't even offer voicemail, but have looked at voicemail solutions from the PSTN domain :-( How have you solved voicemail? And of course RTP proxy :-) I like rtpproxy because its simple, you have good control and it's written in C. However, it has no load balancing and you don't really have control over sessions (stats). It looks like most large installation have chosen python-based mediaproxy due to these two things (as well as video-stream support). Any thoughts?
g-)
Greger V. Teigre wrote:
Yes, I did a quick look at the usrloc module code and can see that resync from DB should be possible, but will probably take some time and I don't think that is a good path.
Yes, either you'll lose the fast memory access if always looking up the DB or you synch back from the DB periodically which would mean that it could be the case that a location isn't up to date or has expired on a node.
possible. I think Paul mentioned that his approach was to use the FIFO queue with a TCP/IP wrapper. An ul_add FIFO command to each server should probably do the trick.
Also a possible approach, maybe we should think about something like that.
I assume that the sql synchronization will not take less bandwidth?!
A rough estimate shows that it's at least less then sending a SIP message. In fact it's more or less the sql statement that has to be transmitted.
But basically in my case I don't see the network as bottleneck since all our equipment is located in a 100MBit LAN.
A SOAP wrapper comes to mind as a solution to the network exchange, but maybe even the SOAP exhange will add too much overhead?
We currently handle call forwards (conditional and unconditional) with location databases too, and here a cronjob runs every minute for inserting the CFC and CFUC locations by calling serctl (so it's related to Paul's approach of location handling, we just don't use the FIFO directly).
But it's planned to use SOAP for realtime provisioning of this stuff, so thanks for the hint, maybe this could be done with the location database too. For this a kind of soap module would be great to prevent exec_msg calls for each registration. Of course also just thinking loud ;o)
Do you do load balancing with DNS SRV or Vovida load balancer or similar?
It's currently DNS SRV, but I'd like to set up some kind of load balancer (maybe based on http://www.linuxvirtualserver.org/software/ipvs.html), just because so much UACs don't get the SRV handling right, and it's always a fight to get potential new UAC vendors to fix it.
Is the MySQL cluster networked or in the same physical location? (I don't know much about mySQL clusters...)
It will be in the same VLAN. MySQL clustering is still beta, so a first step would a simple replication (let the SERs write their locations into the master database and lookup the location in the local slave databases), because replication works well with MySQL. The master database could be secured by a heartbeat cluster.
The advantage of the MySQL cluster would be that the SERs could write also into their local database. But maybe it would generally be a good idea to move the databases to own dedicated machines. We'll see.
Paul claimed voicemail to be their single, biggest problem, citing sems to be the most promising. We don't even offer voicemail, but have looked at voicemail solutions from the PSTN domain :-( How have you solved voicemail?
We use Asterisk for voicemail. The asterisk instances run on the same machine as the SERs, and a SER always redirects voicemail calls to the local Asterisk, with a failure route to a second one. Not that redundant and scaleable, but I don't see voicemail as operation critical as normal call routing. The voicemail storage is shared via NFS by the way.
And of course RTP proxy :-) I like rtpproxy because its simple, you have good control and it's written in C. However, it has no load balancing and you don't really have control over sessions (stats). It looks like most large installation have chosen python-based mediaproxy due to these two things (as well as video-stream support). Any thoughts?
We use mediaproxy here, because it's scalable via DNS SRV and it supports T.38 (I sent a patch to the serdev list some months ago to handle reinvites, and the guys at AG Projects thought about implementing something like that, but I haven't heard anything about that since then).
Cheers, Andy
-
Andreas Granig -
Andreas Granig -
Greger V. Teigre