---- Original Message ----
From: Tina
To: Java Rockx
Cc: Greger V. Teigre ; serusers@lists.iptel.org
Sent: Monday, April 11, 2005 06:37 PM
Subject: Re: [Serusers] still no help - usrloc synchronization

> Pretty good news, Paul. I've only tested IP tunneling. It worked for
> me. The problem I described was born in my thoughts. I've also read
> some complaints from SIP-LVS users. The only configuration where
> Call-Id stickness does not necessary is LVS-NAT. Unfortunately, NAT
> becomes bottleneck very fast.
>
>
> Java Rockx <javarockx@gmail.com> wrote:
> Tina,
>
> I wish I had more information on that, but that LVS stuff is a black
> art to me. One of our engineers whipped up an LVS configuration that
> seems quite happy with the Call-ID "stickness" you described. All I
> can say right now is that it is apparently possible to do this with
> 100% open source. (i'm keeping my fingers crossed) :-)
>
> Regards,
> Paul
>
>
> On Apr 10, 2005 11:03 PM, Tina <kramarv@yahoo.com> wrote:
> Paul,
> thanks a lot, it helps a lot. The thing I do not understand - how you
> are going to solve call-id stickness (see my another post). Since
> we're also use LVS I had to find something proprietary, I am afraid
> this is not a best solution, but the only I have today.
> KRs,
> Tina
>
> Java Rockx <javarockx@gmail.com> wrote:
> Tina,
>
> I really don't know how the LVS server is configured because our
> network guy that we acquired from RedHat set all that stuff up. I do
> know that we basically have this set up:
>
> I hope the formatting goes well :-)
>
> +-----------+     +----------+     +-------+    +---------+
> +--------------------+
>> internet |------| Cisco |------| LVS |-----| Cisco |----|
>> Application     | cloud    |      | 3600   |      |         |     |
>> 3600 |    | & DB Servers |
> +-----------+     +----------+    +--------+    +---------+
> +--------------------+
>
> So! the "Application & DB Servers" box represents multiple servers
> (all are dedicated to their service). These include SER, MySQL,
> Apache, configuration management, monitoring, RTP proxies, etc.
>
> MySQL is active-active so we have two-way replication. We only have 2
> MySQL servers. And this is all we plan on ever having as MySQL is
> more than capable of handling anything ser can throw at it (given the
> right hardware).
>
> Each ser server is 100% oblivious to other SER servers except for
> when handling REGISTER messages. Here is basically the ser.cfg for
> each SER server:
>
> listen=10.3.0.221 # this will be 10.3.0.222 on the peer sip proxy
> modparam("usrloc", "db_mode", 1) # write-through
>
> route {
>        if (method=="REGISTER") {
>             if (src_ip==10.3.0.221) { # ip of peer ser proxy
>                   save_memory("location");
>                   break;
>             } else {
>                   save("location");
>                   t_replicate("10.3.0.222");
>                   break;
>             };
>        };
> }
>
> We have identifed a deficiency in the usrloc module urecord.c file
> whereby when using write-though mode the save_memory() function still
> attempt to update MySQL. This is a very bad thing because it causes a
> primary key violation on the location table.
>
> Yesterday I posted a patch to serdev to fix this, but today we
> identified another case where save_memory() tried to insert in to the
> location table. I'll post a revised patch to serdev as soon as I get
> a chance.
>
> So here is what happens. Assume I have two sip routers and two MySQL
> servers as:
>
> sip-01 is 10.3.0.221
> sip-02 is 10.3.0.222
> db-01 is 10.2.0.21
> db-02 is 10.2.0.22
>
> Now let's just assume that sip-01 is connected to db-01 and sip-02 is
> connected to db-02.
>
> When a SIP UA connects to a sip router (via LVS), assume sip-01 in
> this example and REGISTERs, sip-01 will call save("location") which
> updates/inserts to the location table. sip-01 then calls t_replicate
> to sip-02.
>
> sip-02 recieves the REGISTER message and calls
> save_memory("location") to update its in-memory cache only. It should
> never hit db-02.
>
> Now db-01 eventually (within a second or two) replicates to db-02 and
> now sip-02 is good to go, even in the event of a restart.
>
> Our difficulties mostly surrounded the fact t! hat ser was inserting
> the physical IP of eth0 on the server as the top VIA rather than the
> VIP as required. We solved this by using record_route_preset() and
> passing the VIP as the IP. We also had to bring up a dummy interface
> in order to get ser to honor the VIP.
>
> Hope this helps.
>
> Regards,
> Paul
>
>
>
> On Apr 8, 2005 7:19 PM, Tina <kramarv@yahoo.com> wrote:
> Paul,
> we are using LVS as well. It creates template connection according to
> source Ip (e.g. UAC) and forwards to one of the tunneled IP. I tested
> this a couple of weeks ago, ser was happy and inserted VIP into via
> header, the only issue is to route the calls into right server.
> I agree, there is no such thing as SIP-level clustering. However, how
> would you solve location problem by DB replication - 1) usrloc does
> not update from DB....2) even though, I am afraid it can become
> bottleneck and lock down users ...
>
> KRs,
> Tina
>
>
> Java Rockx <javarockx@gmail.com> wrote:
> See my inline comments.
>
> Regards,
> Paul
>
>
> On Apr 8, 2005 1:08 PM, Greger V. Teigre <greger@teigre.com> wrote:
> Hi Tina,
>
>> I enjoy reading your posts, thanks a lot for the great work you are
>> doing for ser users!
>
> Thanks :-)
>
>> I am going to consider that kind of scenario as well (load balancer
>> returning Virtual IP address). I think for such configuration server
>> side has to keep session information and always forward a call to the
>> right server.
>
> I believe the load balancer can be in front and transparantly
> translate addresses in the SIP messages:
> User agent always sees a.b.c.d
>                          |
> Load balancer with public IP a.b.c.d
>        |                               |
> ser1 (10.0.0.2)          ser2 (10.0.0.3)
>
> Paul: You have a similar load-balancing solution, right? Do you use a
> commerical load balancer that you can reveal the name of? :-)
>
> Our platform is 100% open source. Our load balancer is LVS. I have no
> idea how it actually works. One if our engineers from RedHat set all
> that networking stuff up and it's way beyond me.
>
>
>
> The load balancer need to understand SIP (sort of) and use something
> (ex. IP address of user agent) to select SER server to send to and
> rewrite incoming to 10.0.0.2 and 10.0.0.3 respectively and outgoing
> to a.b.c.d. ser1 and ser2 will happily believe they are alone and
> communicate directly with user agent using their own private
> addresses. Both should know about all users (usrloc). Paul just
> recently posted a small patch for a setup where they use two-way
> replication between two
>
> That small patch will need a slight revision. We discovered through
> further testing that that patch only fixed attempts to INSERT into
> the location table. We didn't cover the UPDATE route, unfortunately.
> I believe Andrew has that patch done and is testing it. So I'll post
> it to serdev as soon as we know it works - possibly later today.
>
>
>
> mysql servers and where each SER instance will save all REGISTERs it
> receives from user agents, but only save to memory those replicated
> from the other SER peer. Klaus pointed out that the same can be done
> by not doing mysql replication, but have one DB for each SER and just
> use t_replicate() and save_noreply() (for two servers, for more than
> two you need to use forward_tcp()).
>
> I have heard complaints that this approach will generate too much
> network traffic. Also, t_replicate() does not guarantee delivery of
> the replication (forward_tcp is better, but not fool proof). So, the
> "best" solution would probably be to hope that Paul's
>
> I fully agree. Replication at the sip proxy level, IMHO, is not a
> good idea. From a purist point of view, replication is not a function
> of a SIP router - nor should it be. It is a function of the database
> and we believe that the MySQL replication is a better approach from a
> speed and reliabilty point of view - not to mention that it
> simplifies the tasks of the sip router.
>
>
>
> recent TCP/UDP patch for network access to FIFO commands get included
> in the CVS. Then, a new replication module should use XMLRPC or
> whatever protocol the patch is using to forward (in a guaranteed
> mode) the usrloc information, which then would be injected directly
> into SER's memory, just like save() would, and of course in whatever
> db_mode you would like. The replicaton module would have to keep a
> queue (of course, flushed to a DB or something) in case of network
> problems, so that the replication would be done (sooner or later)
> regardless of problems.
>     Of course, this is a major undertaking, so for most replication
> scenarioes, I would think the forward_tcp() + save_noreply() would
> work just fine.
>
> g-)
>
>> "Greger V. Teigre" <greger@teigre.com> wrote:
>> I was thinking about a load balancing scenario where the load
>> balancer will replace the IP addresses.
>> g-)
>>
>> ---- Original Message ----
>> From: Tina
>> To: Greger V. Teigre
>> Sent: Wednesday, April 06, 2005 09:45 PM
>> Subject: Re: [Serusers] still no help - usrloc synchronization
>>
>>> Thank you for givingme the scenario with "restricted IP" NAT, I am
>>> starting to find some acceptable solution.
>>> Unfortunately, "one-public-IP" approach is not free from problems
>>> also. If your SIP router inserts this "one-public-IP" into the VIA
>>> header, the reply routing goes via wrong SIP server...
>>> If your SIP server inserts its real-IP-address - the scenario
>>> mentioned above is ! still not resolved.
>>> Any comments?
>>> Tina
>>>
>>> "Greger V. Teigre" <greger@teigre.com> wrote:
>>> See inline.
>>>
>>>> If you use DNS server for load balancing... the client receives one
>>>> of your domain IP addresses according to SRV. I don't see the
>>>> problem
>>>> with a call here, cause UAC asks t! he address only once (before
>>>> sending INVITE). UAC already has the IP for BYE/reINVITEs. So why
>>>> would you replicate INVITEs?
>>>
>>> I would never replicate INVITEs, I would just make sure that they
>>> are proxied through the correct SER server (i.e. IP).
>>>
>>> The problems depends on your setup. If you have SERs with different
>>> IPs, ex UA1 has registered with server A and UA2 h! as registered
>>> with server B: If UA2 wants to call UA1 and UA is behind an IP
>>> restricted NAT, server A is stored in the NAT table of the NAT in
>>> front of UA1. If server B sends an INVITE to UA1, the INVITE will
>>> be refused by UA1's NAT.
>>>     This is why a "one public IP" in front of a load balancing
>>> cluster probably is a good way to go.
>>>
>>>> If you use IPVS/LVS... I believe you can force SER to insert it's
>>>> public IP into VIA,! so there is no problem with replies. With
>>>> regard
>>>> to another requests, I believe load balancer keeps connection
>>>> template, then when another request comes it would be forwarded to
>>>> the same ser.
>>>
>>> Yes. There are different "keys" to use to load balance SIP messages.
>>> One good way from a NAT point of view is to use originating IP
>> ! ;> address. What you must remember is that the problem is not on
>> the
>>> server side, but on the client side. The NAT will in many
>>> situations stop incoming UDP packets if the originating ip:port is
>>> not already stored in the NAT table. The Via header does not
>>> matter for the NAT. g-)
>>>
>>>> Any comments?
>>>>
>>>> "Greger V. Teigre" <greger@teigre.com> wrote:
>>>> Yes, I believe that is so. But still you get a problem if the NAT
>>>> is restricted, port-restricted or symmetric... The best would be to
>>>> load>> balance and always make sure that a given client is handled
>>>> through a given
>>>> SER (REGISTER and INVITEs). That includes forwarding INVITEs from
>>>> one
>>>> SER to
>>>> another... OR you must load balance in front of your servers with
>>>> one
>>>> common
>>>> public IP.
>>>> g-)
>>>>
>>>> Matt Schulte wrote:
>>>>> Ack, I didn't even think about NAT. Would these be added before it
>>>>> gets sent off to the second proxy? ie:
>>>>>
>>>>> if (!src_ip==blah.netlogic.net) {
>>>>> add_rcv_param();
>>>>> t_replicate("blah.netlogic.net", "999");
>>>>> };
>>>>>
>>>>> -----Original Message-----
>>>>> From: Greger V. Teigre [mailto:greger@teigre.com]
>>>>> Sent: Tuesday, April 05, 2005 7:49 AM
>>>>> To: Matt Schulte; kramarv@yahoo.com
>>>>> ! ; Cc: serusers@lists.iptel.org
>>>>> Subject: ! Re: [Serusers] still no help - usrloc synchronization
>>>>>
>>>>>
>>>>> Well, you still have the NAT issues unless you do load balancing
>>>>> and your
>>>>> SER servers have the same public IP.
>>>>> Have you looked at 0.9.0 nathelper function add_rcv_param() ? It
>>>>> will add received info to the contact header for the other SER to
>>>>> process. Haven't really tried yet...
>>>>> g-)
>>>>>
>>>>> Matt Schulte wrote:
>>>>>> I'm starting to lean this direction, using t_replicate and all. I
>>>>>> could never get usrloc (db mode) to function properly..
>>>>>> t_replicate is
>>>>>
>>>>>> a dirty but very effective workaround.
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Greger V. Teigre [mailto:greger@teigre.com]
>>>>>> Sent: Saturday, April 02,! 2005 1:33 AM
>>>>>> To: kramarv@yahoo.com
>>>>>> Cc: serusers@lists.iptel.org
>>>>>> Subject: Re: [Serusers] still ! no help - usrloc synchronization
>>>>>>
>>>>>>
>>>>>> Have a look at this thread:
>>>>>> http://lists.iptel.org/pipermail/serusers/2005-January/014669.html
>>>>>> g-)
>>>>>>
>>>>>> Java Rockx wrote:
>>>>>>> Tina,
>>>>>>>
>>>>>>> I thought I saw you post the other day that you did not want to
>>>>>>> use t_replicate(), however, this is probably your best bet to
>>>>>>> getting this
>>>>>>
>>>>>>> to work, IMHO.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Paul
>>>>>>>
>>>>>>> On Apr 1, 2005 4:08 PM, Tina wrote:
>>>>>> ! >>
>>>>>>> ! ;> Hi, please help me, I'm stuck with it!!!!!
>>>>>>>> I am trying to set up several sers with a shared MySQL database
>>>>>>>> for location service.
>>>>>>>>
>>>>>>>> I set in each ser.cfg:
>>>>>>>>
>>>>>>>> modparam("usrloc", "db_mode", 2)
>>>>>>>> modparam("usrloc",
>>>>>>>> "db_url","sql://ser:heslo@192.168.25.163/ser")
>>>>>>>>
>>>>>>>> and the servers are not synchronized.
>>>>>>>> The I set
>>>>>>>> modparam("usrloc", "db_mode", 2)
>>>>>>>>
>>>>>>>>
>>>>>>>> made UAC (Xlite) register to! one of the servers.
>>>>>>>> I see it via usrloc, but there is no record in "location" mySQL
>>>>>>>> table....So others do not see the client and I'm unable to make
>>>>>>>> calls....
>>>>>>>>
>>>>>>>>
>>>>>>>> Please help how to work with usrloc and mySQL...
>>>>>>>>
>>>>>>>> Tina,
>>>>>>>> software engineer
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>> Do you Yahoo!?
>>>>>>>> Better first dates. More second dates. Yahoo! Personals
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Serusers mailing list
>>>>>>>> serusers@lists.iptel.org
>>>>>>>> http://lists.iptel.org/mailman/listinfo/serusers
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Serusers mailing list
>>>>>>> serusers@lists.iptel.org
>>>>>>> http://lists.iptel.org/mailman/listinfo/serusers
>>>>>>
>>>>>> _______________________________________________
>>>>>> Serusers mailing list
>>>>>> serusers@lists.iptel.org
>>>>>> http://lists.iptel.org/mailman/listinfo/serusers
>>>>
>>>>
>>>>
>>>>
>>>> Yahoo! Messenger
>>>> Show us what our next emoticon should look like. Join the fun.
>>>
>>>
>>> Do you Yahoo!?
>>> Better first dates. More second dates. Yahoo! Personals
>>
>>
>> Yahoo! Messenger
>> Show us what our next emoticon should look like. Join the fun.
>
> _______________________________________________
> Serusers mailing list
> serusers@lists.iptel.org
> http://lists.iptel.org/mailman/listinfo/serusers
>
>
>
>
>
>
>
> Do you Yahoo!?
> Make Yahoo! your home page
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com