Dear Andreas,
As I indicated in an earlier email, I would be interested in taking part in
a joint effort to further develop ser's high-availability and scalability
(HAS). I would probably have to do some development anyway, and I would
prefer to see such support in the public domain. In Nov/Dec I called for
responses on a SOAP-based provisioning interface, but heard nothing, so here
is an overlap of interests.
Please see inline comments.
Andreas Granig wrote:
Hi guys,
I'd like to propose another possibility for a highly-available and
scalable system design based on SER:
The problems I've encountered for scalable systems are:
- Distribution of the user location and alias location among the nodes
(user location is based on registrations, alias location comes from
web interfaces and is used for call forwarding).
- Reloading up to date location tables after breakdown and recovery
of a node
If we are to define the elements required as part of a HAS-effort, I think I
would also include load balancing across servers in a single data center,
not only distribution of usrloc/alias replication across centers. You
mentioned yourself the problem of clients not supporting DNS SRV. I think
some kind of reference design using Linux HA or virtual server would be
useful for the community. I have seen several people asking questions, but I
haven't seen documentation of a successful implementation. The reference
design must of course handle NAT traversal and the possibility of RTP
proxying.
Anybody with experiences or thoughts?
So I'm just thinking loud about the following
provisioning system:
- Write a client which fulfills the this demands:
- Receive one or more locations from SER via a SER module or from a
web application and distribute them to other
known clients. Take care of retransmissions if a client isn't
reachable or reports a temporary failure.
- Receive one or more locations from other clients and write
them into the SER FIFO. If writing into the FIFO fails, try to
write directly into the database (location-table, alias-table
etc.). Report a temporary failure if this also fails.
yes, I see you here define a more general network wrapper to the FIFO. Why
not extend the scope right now to support any FIFO implemented command, also
future?
If FIFO writing is not possible, I believe an alert should be made and
the request left in a transaction queue. There is probably a reason for the
FIFO error and also writing directly to the database will mess up the synch
between DB and server.
Maybe a centralized server should be used which
receives the locations
from the clients and distributes them to other clients, so that the
nodes just know about the server and nothing about other nodes. This
would make integration of new nodes easy.
On the other hand, it's another single point of failure, so a
decentralized solution should be considered. But that would mean that
you've to inform every node about the existence of a new node.
I agree that that each node should be independent. I once implemented a
token ring-like system where each node would report its existance.
Something similar would be appropriate. We could even use SIP REGISTER from
each client using a specifically crafted username and use NOTIFYs to update
everybody about changes :-) Each client could use its local ser server to
lookup contacts. Well, this could create a dependency loop, but it was sort
of a cool idea ...
Bringing up new clients will of course need some special thoughts in
terms of their status and at what point in replication they should enter.
Probably each transaction should use a unique transaction id...
The protocol used between the nodes should be simple
and fast. So I
think SOAP drops out here. Maybe XMLRPC or ICE
(
http://www.zeroc.com/ice.html) could be used.
Why do you think SOAP should be dropped?
If this is to be a general module for both provisioning and replication
across servers, we must handle many different usage scenarios, ex.
provisioning over the Internet, and we should use a framework where we can
get as much as possible for "free", ex. SSL/TLS, authentication etc.
In my experience, SOAP can be both simple and fast and there are far
more auto-generation tools/frameworks etc for SOAP. A simple RPC-type SOAP
message is not really far from XML-RPC, basically an envelope extra. The
ability to auto-generate a client based on a WSDL file is pretty nice, as
well. With regards to ICE, we should probably avoid using toolkits where
commercial licenses must be bought for commercial usage.
I have used gSOAP (
http://www.cs.fsu.edu/~engelen/soap.html), which is a
powerful, but very simple to use C/C++.
I'd be willing to release these parts as GPL for
creating an open
framework for carrier-grade SER integration, so any feedback,
improvements or flames are highly welcome.
Ditto.
I have here contributed to increasing the scope... But I think the correct
approach would be to make something simple in the beginning, maybe only the
provisioning and the queue and a text-based config file for clients to
replicate to. We could then add the ser module for forwarding usrloc to the
local client.
Andreas: What's your time scope?
Core developers of ser: I think we need your opinion(s) before we start
anything?!
Regards,
Greger