Hi gentlemen,
I'm trying to find a suitable topology to deploy on our SIP network.
We started testing UCARP, but it seems it's not going to do for us. Our topology involves 2 servers with physical IPs on them and 2 floating IPs (to balance load 50% on each). The balancing is manual, we are going to point half our servers (measured in load) to one floating IP and so on. When a server fails, the other will handle the load assuming the other floating IP.
We found an issue managing IPs, if we have to make Kamailio listen to a floating IP we have to set: listen=0.0.0.0 and then record_route will just pass that IP on the record route headers, making destination servers not to respond.
If we try to use record_route_preset() there, when server A fails, all responses come with the "wrong" IP on record_route when server B assumes A's IP.
I'd like to know if some of you found a tidier solution and would like to share some topology and implementation tips. DNS SRV is also on our list, but maybe there is something out there that can help us with this.
Thanks in advance, Uriel
On Mon, Mar 2, 2009 at 6:06 PM, Uriel Rozenbaum uriel.rozenbaum@gmail.com wrote:
Hi gentlemen,
I'm trying to find a suitable topology to deploy on our SIP network.
We started testing UCARP, but it seems it's not going to do for us. Our topology involves 2 servers with physical IPs on them and 2 floating IPs (to balance load 50% on each). The balancing is manual, we are going to point half our servers (measured in load) to one floating IP and so on. When a server fails, the other will handle the load assuming the other floating IP.
We found an issue managing IPs, if we have to make Kamailio listen to a floating IP we have to set: listen=0.0.0.0 and then record_route will just pass that IP on the record route headers, making destination servers not to respond.
If we try to use record_route_preset() there, when server A fails, all responses come with the "wrong" IP on record_route when server B assumes A's IP.
I'd like to know if some of you found a tidier solution and would like to share some topology and implementation tips. DNS SRV is also on our list, but maybe there is something out there that can help us with this.
I did not try this yet but linux-ha + DRBD seems to be a nice solution.
http://www.linux-ha.org/ http://www.drbd.org/ http://www.linux-ha.org/DRBD
Regards Luciano
Luciano Afranllie wrote:
I did not try this yet but linux-ha + DRBD seems to be a nice solution.
http://www.linux-ha.org/ http://www.drbd.org/ http://www.linux-ha.org/DRBD
Yep. That's what we do.
Alex Balashov writes:
I did not try this yet but linux-ha + DRBD seems to be a nice solution.
http://www.linux-ha.org/ http://www.drbd.org/ http://www.linux-ha.org/DRBD
drbd is ok, but i'm not convinced about linux-ha. based on what i have read, you cannot use linux ha to verify that sip proxy on your host is running, but only that the host itself is reachable. if so, it won't help you, for example, in your proxy has crashed.
-- juha
2009/3/3 Juha Heinanen jh@tutpro.com:
Alex Balashov writes:
> > I did not try this yet but linux-ha + DRBD seems to be a nice solution. > > > > http://www.linux-ha.org/ > > http://www.drbd.org/ > > http://www.linux-ha.org/DRBD
drbd is ok, but i'm not convinced about linux-ha. based on what i have read, you cannot use linux ha to verify that sip proxy on your host is running, but only that the host itself is reachable. if so, it won't help you, for example, in your proxy has crashed.
Hi Juha, under my experience with HeartBeat you are really wrong ;)
HeartBeat v2 allows creating custom and powerful OCF[1] scripts which can manage a service and monitorize it in. In case a service crashes HeartBeat tries to start it during some time, and in case of no success then the service is started in other node.
If you want to monitorize Kamailio you just need to create a OCF resource script [2], and when it is called with argument "monitor" it must returns 0 if the resource is running, 7 if it is stopped and anything else if it is failed.
Please don't confuse HeartBeat v-1 with v-2 (v-2 allows CRM syntax which implements monitorization and other cool features).
I can sure you that HeartBeat-v2 allows reliable and funny failover systems :)
[1] OCF: http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?re... [2] http://www.linux-ha.org/OCFResourceAgent
Iñaki Baz Castillo writes:
HeartBeat v2 allows creating custom and powerful OCF[1] scripts which can manage a service and monitorize it in. In case a service crashes HeartBeat tries to start it during some time, and in case of no success then the service is started in other node.
ok, looks like whaht i have read was based on v1.
what if network connection to the active node has failed? can the hearbeat monitor be located on a third monitoring host that really has vision on how the service looks like from outside? if not, i'm not still convinced.
-- juha
Juha Heinanen wrote:
Iñaki Baz Castillo writes:
HeartBeat v2 allows creating custom and powerful OCF[1] scripts which can manage a service and monitorize it in. In case a service crashes HeartBeat tries to start it during some time, and in case of no success then the service is started in other node.
ok, looks like whaht i have read was based on v1.
what if network connection to the active node has failed? can the hearbeat monitor be located on a third monitoring host that really has vision on how the service looks like from outside? if not, i'm not still convinced.
That is not inherently part of Heartbeat, but it would be trivial to script using the OCF functionality Inaki mentioned.
Alex Balashov writes:
That is not inherently part of Heartbeat, but it would be trivial to script using the OCF functionality Inaki mentioned.
looks like that is exactly what i have done, i.e., instead using heartbeat (where two hosts monitor each other), i use a script that monitors the sip proxy by sip means from the network and switches the service to another host in case of failure.
drbd can be used to keep the db files up to date between the hosts instead of mysql cluster, which i don't like at all. so as i wrote earlier: dbbd is ok, but linux ha even in v2 is still not good for this job.
-- juha
Juha Heinanen wrote:
Alex Balashov writes:
That is not inherently part of Heartbeat, but it would be trivial to script using the OCF functionality Inaki mentioned.
looks like that is exactly what i have done, i.e., instead using heartbeat (where two hosts monitor each other), i use a script that monitors the sip proxy by sip means from the network and switches the service to another host in case of failure.
drbd can be used to keep the db files up to date between the hosts instead of mysql cluster, which i don't like at all. so as i wrote earlier: dbbd is ok, but linux ha even in v2 is still not good for this job.
MySQL replication can potentially be a lighter-weight solution to the NBD ("cluster") layer, but that too isn't particularly seamless.
I would have to disagree with your assessment about Linux-HA, but it really depends on the precise nature of the expectation and requirement. Nevertheless, we have deployed many OpenSER/Kamailio systems in high-volume production environments and it has proven adequate for such purposes.
2009/3/3 Juha Heinanen jh@tutpro.com:
drbd can be used to keep the db files up to date between the hosts instead of mysql cluster, which i don't like at all.
DBRD for MyDWL data replication? that's really dangerous. Even if you do a simple "scp" of a MyISAM database between two similar hosts, the copy will become "not optimized".
so as i wrote earlier: dbbd is ok, but linux ha even in v2 is still not good for this job.
I don't agree. How do you change the DBR status from slave to active and from active to slave? with a command from a remote computer? I think this is really better if it's done in localhost with powerfull and tested HeartBeat commands (IMHO).
On Tuesday 03 March 2009, Iñaki Baz Castillo wrote:
2009/3/3 Juha Heinanen jh@tutpro.com:
drbd can be used to keep the db files up to date between the hosts instead of mysql cluster, which i don't like at all.
DBRD for MyDWL data replication? that's really dangerous. Even if you do a simple "scp" of a MyISAM database between two similar hosts, the copy will become "not optimized".
Hi Iñaki,
what do you mean by not optimized? If the slave is not active (not running) and one use DRDB just for synchronizing the data, how this could be dangerous?
Cheers,
Henning
2009/3/3 Henning Westerholt henning.westerholt@1und1.de:
DBRD for MyDWL data replication? that's really dangerous. Even if you do a simple "scp" of a MyISAM database between two similar hosts, the copy will become "not optimized".
Hi Iñaki,
what do you mean by not optimized? If the slave is not active (not running) and one use DRDB just for synchronizing the data, how this could be dangerous?
Simpler than that. Forget DB replication stuff. I just mean that if you have a MySQL 1 with a large MyISAM table and you copy it to other server using "scp/rsync", you will get a non optimized or corrupted table (even if both servers have same CPU, architecture and file system).
I've experimented it by copying with "rsync" a 2 GB long MyISAM table. After restarting the MySQL-2 and run a SQL command to check tables [1], you probably will get a "NOT OPTIMIZED TABLE" and you must to repair it. This is the best case.
PD: This is based on my experience with MySQL. However I'have no too much knowledge on it.
[1] http://dev.mysql.com/doc/refman/5.1/en/check-table.html
On Tuesday 03 March 2009, Iñaki Baz Castillo wrote:
what do you mean by not optimized? If the slave is not active (not running) and one use DRDB just for synchronizing the data, how this could be dangerous?
Simpler than that. Forget DB replication stuff. I just mean that if you have a MySQL 1 with a large MyISAM table and you copy it to other server using "scp/rsync", you will get a non optimized or corrupted table (even if both servers have same CPU, architecture and file system).
I've experimented it by copying with "rsync" a 2 GB long MyISAM table. After restarting the MySQL-2 and run a SQL command to check tables [1], you probably will get a "NOT OPTIMIZED TABLE" and you must to repair it. This is the best case.
PD: This is based on my experience with MySQL. However I'have no too much knowledge on it.
Hi Iñaki,
ok, thanks for the clarification. This sounds strange. I mean, how is this different to a normal backup/ restore cycle when some machine crash? I'd expect that it simply continue to use the restored data.
Cheers,
Henning
2009/3/3 Henning Westerholt henning.westerholt@1und1.de:
Simpler than that. Forget DB replication stuff. I just mean that if you have a MySQL 1 with a large MyISAM table and you copy it to other server using "scp/rsync", you will get a non optimized or corrupted table (even if both servers have same CPU, architecture and file system).
I've experimented it by copying with "rsync" a 2 GB long MyISAM table. After restarting the MySQL-2 and run a SQL command to check tables [1], you probably will get a "NOT OPTIMIZED TABLE" and you must to repair it. This is the best case.
I would like to add that the "scp" copy was performed after stoping MySQL 1, if not, be sure that you can get a *corrupted* table (more dangerous when using InnoDB). But anyway, being MysQL 1 stoped I got a "non optimized" table after copying it to other server using "scp".
PD: This is based on my experience with MySQL. However I'have no too much knowledge on it.
Hi Iñaki,
ok, thanks for the clarification. This sounds strange. I mean, how is this different to a normal backup/ restore cycle when some machine crash? I'd expect that it simply continue to use the restored data.
I don't know. Perhaps MyISAM format relies on file system nodes, so doing a "cp" or "scp" is not secure (or maybe it's secure but you must repair the table after it).
Iñaki Baz Castillo writes:
what do you mean by not optimized? If the slave is not active (not running) and one use DRDB just for synchronizing the data, how this could be dangerous?
Simpler than that. Forget DB replication stuff. I just mean that if you have a MySQL 1 with a large MyISAM table and you copy it to other server using "scp/rsync", you will get a non optimized or corrupted table (even if both servers have same CPU, architecture and file system).
I've experimented it by copying with "rsync" a 2 GB long MyISAM table. After restarting the MySQL-2 and run a SQL command to check tables [1], you probably will get a "NOT OPTIMIZED TABLE" and you must to repair it. This is the best case.
inaki,
drdb keeps exact copy of the raw disk image of master on the slave. it has nothing to do with copying of files.
-- juha
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
Iñaki Baz Castillo writes:
what do you mean by not optimized? If the slave is not active (not running) and one use DRDB just for synchronizing the data, how this could be dangerous?
Simpler than that. Forget DB replication stuff. I just mean that if you have a MySQL 1 with a large MyISAM table and you copy it to other server using "scp/rsync", you will get a non optimized or corrupted table (even if both servers have same CPU, architecture and file system).
I've experimented it by copying with "rsync" a 2 GB long MyISAM table. After restarting the MySQL-2 and run a SQL command to check tables [1], you probably will get a "NOT OPTIMIZED TABLE" and you must to repair it. This is the best case.
inaki,
drdb keeps exact copy of the raw disk image of master on the slave. it has nothing to do with copying of files.
Yes, for sure. But in case of data corruption wouldn't it be very dangerous?
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
Iñaki Baz Castillo writes:
I mean in the DRB replication process.
there are provisions in the protocol to detect data corruption when data is copied to disk of the secondary.
Yes, I know it, I've used DBRD :) But I didn't know how reliable that protocol is.
Thanks.
Guys, let's go a step backwards.
I'm now trying HeartBeat and succeded making one node starting when the other is down, that is fine.
Now we have the following scenario:
2 Floating IPs: 192.168.200.80 and .82
Kam1: Fixed IP 192.168.200.81, master for .80 and backup for .82 Kam2: Fixed IP 192.168.200.83, master for .82 and backup for .80
Then I'll have both servers up, load-sharing 50% of the load each.
When any of them has an issue I need the other to take the load but without restarting Kamailio (as i don't want to refresh cache). If I use 'listen=0.0.0.0' I'll have record_route to write wrong things on the SIP Header:
*Record-Route: sip:0.0.0.0;ftag=as24719265;lr=on* *Via: SIP/2.0/UDP 0.0.0.0;branch=z9hG4bKf5c5.04e5a224.0* Via: SIP/2.0/UDP 192.168.200.11:5060;branch=z9hG4bK41474ecd;rport=5060 From: "541160911100" <sip:541160911100@192.168.200.81sip%3A541160911100@192.168.200.81
;tag=as24719265
To: <sip:1054111556446967@192.168.200.81sip%3A1054111556446967@192.168.200.81
Contact: <sip:541160911100@192.168.200.11sip%3A541160911100@192.168.200.11
Call-ID: 68dd3a3971ec68ab08b3086c59ac276d@192.168.200.81 CSeq: 102 INVITE User-Agent: Metrotel Max-Forwards: 69 Remote-Party-ID: "541160911100" <sip:541160911100@192.168.200.81sip%3A541160911100@192.168.200.81
;privacy=off;screen=no
Date: Mon, 02 Mar 2009 18:23:25 GMT Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY Content-Type: application/sdp Content-Length: 320
If I use 'listen=eth0' then I'll have to restart Kamailio when an event occurs.
Is there any workaround you may know to solve this? Any module or trick to check the received address in order to use it with listen=0.0.0.0?
Thanks! Uriel
On Tue, Mar 3, 2009 at 7:56 PM, Iñaki Baz Castillo ibc@aliax.net wrote:
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
Iñaki Baz Castillo writes:
I mean in the DRB replication process.
there are provisions in the protocol to detect data corruption when data is copied to disk of the secondary.
Yes, I know it, I've used DBRD :) But I didn't know how reliable that protocol is.
Thanks.
-- Iñaki Baz Castillo
Kamailio (OpenSER) - Users mailing list Users@lists.kamailio.org http://lists.kamailio.org/cgi-bin/mailman/listinfo/users http://lists.openser-project.org/cgi-bin/mailman/listinfo/users
El Miércoles, 4 de Marzo de 2009, Uriel Rozenbaum escribió:
When any of them has an issue I need the other to take the load but without restarting Kamailio (as i don't want to refresh cache). If I use 'listen=0.0.0.0' I'll have record_route to write wrong things on the SIP Header:
HeartBeat should manage the virtual IP (the service IP) and also the services (as Kamailio). This is, when HearBeat detects that node-1 doesn't work, it must up the virtual IP in node-2, and *after it*, start Kamailio in node-2. A service managed by HeartBeat CANNOT be running in both nodes, it's "anti-HeartBeat".
2009/3/3 Juha Heinanen jh@tutpro.com:
what if network connection to the active node has failed?
What do you mean exactly? do you mean the connection between nodes? Connection between nodes (for pinging and status sending) can be achieved via various ETH interfaces or serial link. If node 2 detects that node 1 doesnt' reply then node 2 takes the control of the virtual IP and services.
Note that HeartBeat-v2 allows more complex escenarios with various nodes and groups of services running in different nodes at the same time.
can the hearbeat monitor be located on a third monitoring host that really has vision on how the service looks like from outside? if not, i'm not still convinced.
HA runs on several nodes, and all the nodes send status information to others. It's not a centralized monitorization system but a distributed one.
Iñaki Baz Castillo writes:
What do you mean exactly? do you mean the connection between nodes? Connection between nodes (for pinging and status sending) can be achieved via various ETH interfaces or serial link. If node 2 detects that node 1 doesnt' reply then node 2 takes the control of the virtual IP and services.
what i tried to say that if your two sip proxy host, which, for performance reasons, should be physically close to each other, i.e., in the same lan, see each other up and running, it does not mean that the service is reachable from the outside. you may have, for example, some network equipment broken that affects reachability to one host and not to the other.
HA runs on several nodes, and all the nodes send status information to others. It's not a centralized monitorization system but a distributed one.
what kind of algorithm it has to choose which host becomes the active one if both hosts come up at about same time after powers loss?
-- juha
Juha Heinanen wrote:
what i tried to say that if your two sip proxy host, which, for performance reasons, should be physically close to each other, i.e., in the same lan, see each other up and running, it does not mean that the service is reachable from the outside. you may have, for example, some network equipment broken that affects reachability to one host and not to the other.
So, include a test for reachability to the outside as part of your OCF monitoring operations in the Heartbeat duty cycle.
Alex Balashov writes:
So, include a test for reachability to the outside as part of your OCF monitoring operations in the Heartbeat duty cycle.
where do you send the packet of this test? what if that node is not reachable, but other nodes on the outside are?
in my opinion, it is better to test from outside in, because if your test node is broken, it does not cause any harm to the service node (assuming that the test node gets fixed in due time).
-- juha
Juha Heinanen wrote:
Alex Balashov writes:
So, include a test for reachability to the outside as part of your OCF monitoring operations in the Heartbeat duty cycle.
where do you send the packet of this test? what if that node is not reachable, but other nodes on the outside are?
Same situation can apply to a fixed outside node from which you are testing into the system.
Alex Balashov wrote:
Juha Heinanen wrote:
Alex Balashov writes:
So, include a test for reachability to the outside as part of your OCF monitoring operations in the Heartbeat duty cycle.
where do you send the packet of this test? what if that node is not reachable, but other nodes on the outside are?
Same situation can apply to a fixed outside node from which you are testing into the system.
Send to many different outside nodes as part of the test if this is really a concern.
Alex Balashov writes:
Same situation can apply to a fixed outside node from which you are testing into the system.
yes, that node may very well go down, but as i wrote, it would not affect the operation of the service. but if a service node thinks that there is a problem even when there isn't any, it will affect the service.
-- juha
Juha Heinanen wrote:
Alex Balashov writes:
Same situation can apply to a fixed outside node from which you are testing into the system.
yes, that node may very well go down, but as i wrote, it would not affect the operation of the service. but if a service node thinks that there is a problem even when there isn't any, it will affect the service.
Then build a better test.
The point is that these methodological flaws are valid, but do not inhere in Linux-HA. Like many things, it's just a tool - a very polymorphic tool capable of easily doing many things. How to best apply it and achieve your goals is much more a function of the implementor's judgment.
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
HA runs on several nodes, and all the nodes send status information to
> others. It's not a centralized monitorization system but a distributed > one.
what kind of algorithm it has to choose which host becomes the active one if both hosts come up at about same time after powers loss?
When using HeartBeat, the services are not started by the "init" process, instead HeartBeat starts them, and if only does it after talking with HeartBeat's in other nodes. So there is no risk.
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
what i tried to say that if your two sip proxy host, which, for performance reasons, should be physically close to each other, i.e., in the same lan, see each other up and running, it does not mean that the service is reachable from the outside. you may have, for example, some network equipment broken that affects reachability to one host and not to the other.
You can create some OCF scripts that, in some way, test the connection from outside. Not easy, but feasible. For example, such a script could log into an external machine via SSH and do a PING or UDP test against the hosts.
Iñaki Baz Castillo wrote:
El Martes, 3 de Marzo de 2009, Juha Heinanen escribió:
what i tried to say that if your two sip proxy host, which, for performance reasons, should be physically close to each other, i.e., in the same lan, see each other up and running, it does not mean that the service is reachable from the outside. you may have, for example, some network equipment broken that affects reachability to one host and not to the other.
You can create some OCF scripts that, in some way, test the connection from outside. Not easy, but feasible. For example, such a script could log into an external machine via SSH and do a PING or UDP test against the hosts.
There are many ways to skin that cat. I think Juha's point may be that Linux-HA doesn't natively provide an easy way to do this based on the assumption that it is a common use case / item of desiderata.
Personally, I would argue that it's just not that important. If you lose outside reachability you've got other problems. The types of failures Linux-HA looks to mitigate are related to physical node failure.
If the (LAN) network between the nodes gives out, Linux-HA has provisions called STONITH (Shoot The Other Node In The Head) to mitigate the cluster "split brain" problem/prevent the nodes believing that they are each the master and the other has failed. It doesn't work perfectly (which, evidently, is something to which we are accustomed in the Kamailio community :-), but it works pretty well.
-
Alex Balashov -
Henning Westerholt -
Iñaki Baz Castillo -
jh＠tutpro.com -
Luciano Afranllie -
Uriel Rozenbaum