Hi all,
I've read different approaches towards load balancing, expecially in these two threads: http://lists.iptel.org/pipermail/serusers/2005-May/019107.html http://lists.iptel.org/pipermail/serusers/2005-April/018058.html
Unfortunately it seems that at the moment there's not a "state of the art" solution, so here is my "two cent" idea.
Project goal: to develop a near-linearly scalable platform with NAT traversal support and call control.
Proposed solution: multiple instances of SER will be reacheable via an unique virtual IP handled by LVS (with udp persistence), all SERs will use the same MySQL backend, an array of Asterisk servers will act as SBCs (Session Border Controller).
Every SER will have a "third level" SIP domain associated to it, for instance UA_A will be registered as UA_A@ser1.domain.com and UA_B will be registered as UAC_B@ser2.domain.com. UAC_A will be able to call UA_B as "B" or as "B@domain.com"; SRV records for "domain.com" will be LVS virtual IP, SRV records for third level domain will be internal, not visible to Uas and associated to IP addresses:ports of every SER instance.
To achieve this, we will need to store a domain for any UA in the "location" table, any instance of SER will do this, UAs don't need to know anything about "internal" domains. (Is this to be considered an "hack"?)
## Without call control, there would be only one call leg: When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
## Instead, we want to be able to control the call process and to handle RTP nat too, so let's use Asterisk: -First Leg (left): UA_A calls UA_B, INVITE reaches SER_1 (LVS persistence), SER_1 finds out that UA_B is @ domain ser2.domain.com but its dialplan tells him to send the INVITE to one of the Asterisk servers (one of them). SER modifies the INVITE sent to * with Contact: (ip:port) of SER_2. -Second Leg (right): Asterisk (after obtaining authorization and call duration from AAA server) will look for UA_B at SER_2 and send the INVITE to SER_2 with SDP modified for the RTP, SER_2 will then forward INVITE (solving NAT problems) to UA_B ip:port (parameters taken from memory or DB).
This kind of platform would permit call forking too: if UA_B were registered through two different proxies SER_1 would fork the call to two different "call instances" of Asterisk(s).
Note that we want Asterisk to mangle SDP only when both UA_A and UA_B have private IP addresses
Have you got any suggestion/comment? (expecially regarding: - more SERs using the same MySQL backend - binding a third level domain at the proxy
Thanks and kind regards! Pietro Ravasio
-- Ing. Pietro Ravasio - pietro.ravasio@abbeynet.it Enterprise Market Manager Abbeynet S.p.A. - http://www.abbeynet.it TEL: +390702109044 GSM: +393282172055 FAX: +390702109055
Pietro Ravasio writes:
When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
what happens if ser_1 or ser_2 is down when this call takes place?
Have you got any suggestion/comment?
i don't at all like an idea of mixing a third component (asterisk) to this. the whole thing should be doable with sers and load balancers alone.
-- juha
would lvs do this, I mean doesnt it have a problem with udp/SIP judging by the threads u mentioned and those on lvs list.
Iqbal
On 5/12/2005, "Juha Heinanen" jh@tutpro.com wrote:
Pietro Ravasio writes:
When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
what happens if ser_1 or ser_2 is down when this call takes place?
Have you got any suggestion/comment?
i don't at all like an idea of mixing a third component (asterisk) to this. the whole thing should be doable with sers and load balancers alone.
-- juha
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
LVS can do UDP load distribution based on UA's IP address. i think that is what Pietro suggests. Then the receiving SER must know where the call should go. Such a solution will anyway be reliant on a single server, as Juha pointed out. g-)
Iqbal wrote:
would lvs do this, I mean doesnt it have a problem with udp/SIP judging by the threads u mentioned and those on lvs list.
Iqbal
On 5/12/2005, "Juha Heinanen" jh@tutpro.com wrote:
Pietro Ravasio writes:
When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
what happens if ser_1 or ser_2 is down when this call takes place?
Have you got any suggestion/comment?
i don't at all like an idea of mixing a third component (asterisk) to this. the whole thing should be doable with sers and load balancers alone.
-- juha
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
so just to get a summary out of this, is there a way to do load balancing, and f so is this without LVS and dispatcher, I havent tried either, just figured I'd wait until the dust had settled.
iqbal
Greger V. Teigre wrote:
LVS can do UDP load distribution based on UA's IP address. i think that is what Pietro suggests. Then the receiving SER must know where the call should go. Such a solution will anyway be reliant on a single server, as Juha pointed out. g-)
Iqbal wrote:
would lvs do this, I mean doesnt it have a problem with udp/SIP judging by the threads u mentioned and those on lvs list.
Iqbal
On 5/12/2005, "Juha Heinanen" jh@tutpro.com wrote:
Pietro Ravasio writes:
When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
what happens if ser_1 or ser_2 is down when this call takes place?
Have you got any suggestion/comment?
i don't at all like an idea of mixing a third component (asterisk) to this. the whole thing should be doable with sers and load balancers alone.
-- juha
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
.
I agree with Juha: Mixing Asterisk into this is not something I will want to do. The whole point of load balancing SER is to increase redundancy and scalability. Adding Asterisk to the mix will increase the costs too much. doing authentication in asterisk is also something I would not want to do, it should be handled by ser. I think that the dispatcher module of ser can be used cheaper instead of using LVS, which will not give you call-id persistance anyway. AFAIU, your scenario is quite similar to what I described in this thread: http://lists.iptel.org/pipermail/serusers/2005-May/019118.html Just a different implementation...
Have you got any suggestion/comment? (expecially regarding:
- more SERs using the same MySQL backend
I'm not sure I understand how you would store the locations. By manipulating the registered contact? Especially:
To achieve this, we will need to store a domain for any UA in the "location" table, any instance of SER will do this, UAs don't need to know anything about "internal" domains. (Is this to be considered an "hack"?)
What does this imply in terms of ser.cfg config or possibly ser code change?
- binding a third level domain at the proxy
If it's work, it's a workaround. No better or worse than other workarounds... g-)
Pietro Ravasio wrote:
Hi all,
I've read different approaches towards load balancing, expecially in these two threads: http://lists.iptel.org/pipermail/serusers/2005-May/019107.html http://lists.iptel.org/pipermail/serusers/2005-April/018058.html
Unfortunately it seems that at the moment there's not a "state of the art" solution, so here is my "two cent" idea.
Project goal: to develop a near-linearly scalable platform with NAT traversal support and call control.
Proposed solution: multiple instances of SER will be reacheable via an unique virtual IP handled by LVS (with udp persistence), all SERs will use the same MySQL backend, an array of Asterisk servers will act as SBCs (Session Border Controller).
Every SER will have a "third level" SIP domain associated to it, for instance UA_A will be registered as UA_A@ser1.domain.com and UA_B will be registered as UAC_B@ser2.domain.com. UAC_A will be able to call UA_B as "B" or as "B@domain.com"; SRV records for "domain.com" will be LVS virtual IP, SRV records for third level domain will be internal, not visible to Uas and associated to IP addresses:ports of every SER instance.
To achieve this, we will need to store a domain for any UA in the "location" table, any instance of SER will do this, UAs don't need to know anything about "internal" domains. (Is this to be considered an "hack"?)
## Without call control, there would be only one call leg: When UA_A calls UA_B, its INVITE will reach the same SER it has previously registered to (SER_1, thanks to LVS persistence), SER_1 will then look for UA_B and will notice that it is registered as UA_B@ser2.domain.com, DNS will be queried and SER_2 will be contacted in order to handle INVITE and send it to UA_B through already opened UDP channel (resolving NAT).
## Instead, we want to be able to control the call process and to handle RTP nat too, so let's use Asterisk: -First Leg (left): UA_A calls UA_B, INVITE reaches SER_1 (LVS persistence), SER_1 finds out that UA_B is @ domain ser2.domain.com but its dialplan tells him to send the INVITE to one of the Asterisk servers (one of them). SER modifies the INVITE sent to * with Contact: (ip:port) of SER_2. -Second Leg (right): Asterisk (after obtaining authorization and call duration from AAA server) will look for UA_B at SER_2 and send the INVITE to SER_2 with SDP modified for the RTP, SER_2 will then forward INVITE (solving NAT problems) to UA_B ip:port (parameters taken from memory or DB).
This kind of platform would permit call forking too: if UA_B were registered through two different proxies SER_1 would fork the call to two different "call instances" of Asterisk(s).
Note that we want Asterisk to mangle SDP only when both UA_A and UA_B have private IP addresses
Have you got any suggestion/comment? (expecially regarding:
- more SERs using the same MySQL backend
- binding a third level domain at the proxy
Thanks and kind regards! Pietro Ravasio
on Friday, May 13, 2005 11:48 AM Greger V. Teigre wrote:
I agree with Juha: Mixing Asterisk into this is not something I will want to do. The whole point of load balancing SER is to increase redundancy and scalability. Adding Asterisk to the mix will increase the costs too much.
The only reason I've thought to use Asterisk is that I want a "credit reservation" method to handle call status that lets me stop the call when the credit is finished. Now this is something I wouldn't have liked to do in SER since I didn't want to use it statefully.
doing authentication in asterisk is also something I would not want to do, it should be handled by ser.
Authentication at the moment is planned to be handled by Ser, authorization and accounting by Asterisk (that is the only element that knows both call legs)
Anyway, at the moment "mixing Asterisk with SER" is not my major concern, I'm pretty worried about trying to use a particular outbound proxy for calls to a NATed UA:
I think that the dispatcher module of ser can be used cheaper instead of using LVS, which will not give you call-id persistence anyway. AFAIU, your scenario is quite similar to what I described in this thread: http://lists.iptel.org/pipermail/serusers/2005-May/019118.html Just a different implementation...
Yes, I now agree with you in using this approach rather than "multiple internal domains" one. The only things I'd like to modify are: - storing location on every message, not only on REGISTER, in case a ser dies and lvs udp persistence sends the next UA message to another SER (for instance my Sipura phone sends a bunch of NOTIFY messages in order to keep NAT open, if a ser dies and if phone is already authenticated, I want to update "its SER") - using only one table for every SER storing in a different column the proxy (taken from dst_ip) who has handled the message. This way I don't have to preprovision all SER instances in all ser.cfg routing logics. At this point:
.) if I use a B2B user agent in order to handle NAT (for instance asterisk) - I'd send INVITE to my B2B UA (t_relay_to_udp("b2bua_lvs_virtual_ip")) with rewritten Contact: field with the IP number of the (outbound) SER UAB has registered/sent its latest message to (taken from DB)
..) otherwise, in a "pure SER" environment, - I'd send INVITE directly to the SER proxy which IP:port is taken from DB with the same logic as above
Using only one location table could be useful in order to create a flexible routing logic: if the server taken from DB is not online anymore, send message to another one (randomly chosen), hoping that UAS is not behind nat. This logic could be inserted in B2BUA dialing plans or in ser.cfg.
If it's work, it's a workaround. No better or worse than other workarounds... g-)
At the moment I start thinking that any technique that tries to bind an outbound SER for all calls to a set of user agents, is going to be a workaround... Am I wrong? :-/
Pietro Ravasio writes:
The only reason I've thought to use Asterisk is that I want a "credit reservation" method to handle call status that lets me stop the call when the credit is finished. Now this is something I wouldn't have liked to do in SER since I didn't want to use it statefully.
things get far too complicated unless you solve one problem at a time. resilient load balancing is one problem and pre-paid is another one.
-- juha
Pietro Ravasio wrote:
Anyway, at the moment "mixing Asterisk with SER" is not my major concern, I'm pretty worried about trying to use a particular outbound proxy for calls to a NATed UA:
I think that the dispatcher module of ser can be used cheaper instead of using LVS, which will not give you call-id persistence anyway. AFAIU, your scenario is quite similar to what I described in this thread: http://lists.iptel.org/pipermail/serusers/2005-May/019118.html Just a different implementation...
Yes, I now agree with you in using this approach rather than "multiple internal domains" one. The only things I'd like to modify are:
- storing location on every message, not only on REGISTER, in case a
ser dies and lvs udp persistence sends the next UA message to another SER (for instance my Sipura phone sends a bunch of NOTIFY messages in order to keep NAT open, if a ser dies and if phone is already authenticated, I want to update "its SER")
I'm not sure if I like the idea of doing an sql write for every single message. That's will be a massive performance hit. But the idea is good. What about doing LVS NAT?
- using only one table for every SER storing in a different column
the proxy (taken from dst_ip) who has handled the message. This way I don't have to preprovision all SER instances in all ser.cfg routing logics. At this point:
But then you have to modify SER, which is back to where we started: 1. If you have to modify SER to do load balancing, what would be the best way to do it? 2. The answer is most likely: Do it in a module like dispatcher, but add call-id persistence 3. Doing level 7 load balancing can only be effective in the kernel
So why not focus your attention on creating call-id persistence in LVS? ;-)
.) if I use a B2B user agent in order to handle NAT (for instance asterisk) - I'd send INVITE to my B2B UA (t_relay_to_udp("b2bua_lvs_virtual_ip")) with rewritten Contact: field with the IP number of the (outbound) SER UAB has registered/sent its latest message to (taken from DB)
..) otherwise, in a "pure SER" environment,
- I'd send INVITE directly to the SER proxy which IP:port is taken
from DB with the same logic as above
Yes, but how do you know the server just went down and you still haven't updated the IP:port of the UA? I assume you will use t_replicate and save_memory to make sure that all servers have the updated registration info in memory? (as DB writethrough works fine, but other ser's will not load the new location upon receiving an INVITE) You probably have to do t_replicate for EVERY message to update the IP:port of the UA and you're into massive overload of SIP messages (used by t_replicate).
Using only one location table could be useful in order to create a flexible routing logic: if the server taken from DB is not online anymore, send message to another one (randomly chosen), hoping that UAS is not behind nat. This logic could be inserted in B2BUA dialing plans or in ser.cfg.
I don't like the word "hoping"...
If it's work, it's a workaround. No better or worse than other workarounds... g-)
At the moment I start thinking that any technique that tries to bind an outbound SER for all calls to a set of user agents, is going to be a workaround... Am I wrong? :-/
No, I don't think so. That is trying to do something at the application level that people have long figured out is better done at lower levels... In fact, I believe the simplest solution would be to make sure that LVS can select ANY SER for load balancing and use LVS NAT (which is quite scalable and is the way Cisco does it). The simplest way to modify ser for this would be to make sure that all lookup("location") calls actually make an sql read. If each ser instance flushes each save to db and a shared sql cluster is used as backend, you are good to go.
The other way is to implement call-id persistence in IPVS (LVS' load balancer). I like this better, because you do it in kernel mode and don't add a whole lot of new DB reads...
g-)
----- Original Message ----- From: "Greger V. Teigre" greger@teigre.com To: "Pietro Ravasio" pietro.ravasio@abbeynet.it; serusers@lists.iptel.org Sent: Wednesday, May 18, 2005 7:42 AM Subject: Re: [Serusers] More on load balancing: LVS + SRVs, SER + Asterisk
Pietro Ravasio wrote:
Anyway, at the moment "mixing Asterisk with SER" is not my major concern, I'm pretty worried about trying to use a particular outbound proxy for calls to a NATed UA:
I think that the dispatcher module of ser can be used cheaper instead of using LVS, which will not give you call-id persistence anyway. AFAIU, your scenario is quite similar to what I described in this thread: http://lists.iptel.org/pipermail/serusers/2005-May/019118.html Just a different implementation...
Yes, I now agree with you in using this approach rather than "multiple internal domains" one. The only things I'd like to modify are:
- storing location on every message, not only on REGISTER, in case a
ser dies and lvs udp persistence sends the next UA message to another SER (for instance my Sipura phone sends a bunch of NOTIFY messages in order to keep NAT open, if a ser dies and if phone is already authenticated, I want to update "its SER")
I'm not sure if I like the idea of doing an sql write for every single message. That's will be a massive performance hit. But the idea is good. What about doing LVS NAT?
Not for every single message, but just for REGISTER or for any other message from UA not in server registration info memory. In this case the SER (receiveing the message from "unknown UA") will update info's in proxyIP table and load, from location table, registration infos. Now this will be the proxy referring to UA.
Hi,
is there anyone who is not using SER in statefull mode (i.e. using SER Accounting which you usually dont want to do) who did run into serious performance problems using a modern multiprocessor system?
IPTel states that an compaq ipaq would be able to switch the bay area during peak time, therefore i cant think of any scenario (except some MAJOR carriers maybe) where most SIP providers would run into critical load-balancing issues except with rtp-proxying. I am quite positive that one can handle massive sip-request load by using high-end machines with maybe several network cards.
Regarding the prepaid scenario, do u guys thinks its possible to use an tcl script on a cisco to extract the timeout from a custom sip header-line (ex. "X-Timeout: 320") inserted by a custom SER module?
best regards,
Arnd
Arnd Vehling wrote:
Hi,
is there anyone who is not using SER in statefull mode (i.e. using SER Accounting which you usually dont want to do) who did run into serious performance problems using a modern multiprocessor system?
IPTel states that an compaq ipaq would be able to switch the bay area during peak time, therefore i cant think of any scenario (except some MAJOR carriers maybe) where most SIP providers would run into critical load-balancing issues except with rtp-proxying. I am quite positive that one can handle massive sip-request load by using high-end machines with maybe several network cards.
Regarding the prepaid scenario, do u guys thinks its possible to use an tcl script on a cisco to extract the timeout from a custom sip header-line (ex. "X-Timeout: 320") inserted by a custom SER module?
I do not know any method to access the SIP message in a TCL script. But you can use Radius authentication on the Cisco box. The radius server responds with a certain h323-attribute which will tell the Cisco box the max call duration.
regards, klaus
best regards,
Arnd
-
Arnd Vehling -
Greger V. Teigre -
Iqbal -
Juha Heinanen -
Klaus Darilion -
Marco Figus -
Pietro Ravasio