Hello,
I just realized that I had the dispatcher configured using a hash of Call-ID. That means, after recvfrom there must be an extra processing finding the Call-ID header in message, to calculate a hash and then forward() message. The more the processing, the more cases when 200 could arrive before 180. I just changed it to round robin, and the amount decreased a lot, but it's still there. If I send a burst of 1000 messages, about 5 of them leave out of order every time.
Best regards,
Luis
On 4/9/20 1:48 PM, Luis Rojas G. wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and I don't see it so unlikely at all that a process loses cpu just after recvfrom(). It's just as probable as to lose it just before, or when writing on a cache or just before of after sendto(). If there are many messages going through, some of them will fall in this scenario. if I try sending a burst of 100 messages, I see two or three presenting the scenario.
Just forward() with a single process does not give the capacity. I'm getting almost 1000caps. More than that and start getting errores, retransmissions, etc. And this is just one way. I need to receive the call to go back to the network (our application is a B2BUA), so I will be down to 500caps, with a simple scenario, with no reliable responses, reinvites, updates, etc. I will end up having as many standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK. Very small, like 10ms, should be enough. Simple and it should work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
then the overtaking is in between reading from the socket and getting to parsing the call-id value -- the cpu is lost by first reader after recvfrom() and the second process get enough cpu time to go ahead further. I haven't encountered this case, but as I said previously, it is very unlikely, but still possible. I added the route_locks_size because in the past I had cases when processing of some messages took longer executing config (e.g., due to authentication, accounting, ..) and I needed to be sure they are processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending out (using forward()) gives the capacity, if you don't do any other complex processing. Or if you do more complex processing, use a dispatcher process with forwarding to local host or in a similar manner try to use mqueue+rtimer for dispatching using shared memory queues.
Of course, it is open source and there is also the C coding way, to add a synchronizing mechanism to protect against parallel execution of the code from recvfrom() till call-id lock is acquired.
Cheers, Daniel
Hello,
dispatcher has nothing to do with handling sip replies. It is intended only for routing sip requests. If you use dispatcher for replies, you do it wrong, just let kamailio route them based on Via headers.
So maybe I was looking at the wrong message flow processing, I was speaking mainly about the case when the caller sends quickly the reINVITE after the ACK to the initial INVITE 200ok and the reINVITE gets to callee before the ACK. That was more of a branching in discussion on Alex' remarks and the situation that I enocountered in the past and created troubles. Never had to deal with troubles caused by change of order between 180 and 200. In IP world, if the time between 180 and 200 is very short, it doesn't matter at all, because the 180 is for start play a ring tone, which a human may not even hear it when 200 comes 50ms after it.
If you face the re-ordering for replies, then Kamailio doesn't do much internally if you don't have reply_route{} (as well as no onsend_route) in config file, provided that you do not use tm module for sending out (and by that no onreply_route or failure_route).
For a sip reply, kamilio is parsing the headers to find the 2nd Via header and use that address to send out the reply. The request route is not executed for sip replies.
What you can try is to set number of kamailio processes not to exceed the number of CPU cores, so there is "no real competition" to get CPU cycles. It could improve a bit, but still not a 100% accuracy (ie., there are other processes running on the system).
Cheers, Daniel
On 09.04.20 21:29, Luis Rojas G. wrote:
Hello,
I just realized that I had the dispatcher configured using a hash of Call-ID. That means, after recvfrom there must be an extra processing finding the Call-ID header in message, to calculate a hash and then forward() message. The more the processing, the more cases when 200 could arrive before 180. I just changed it to round robin, and the amount decreased a lot, but it's still there. If I send a burst of 1000 messages, about 5 of them leave out of order every time.
Best regards,
Luis
On 4/9/20 1:48 PM, Luis Rojas G. wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and I don't see it so unlikely at all that a process loses cpu just after recvfrom(). It's just as probable as to lose it just before, or when writing on a cache or just before of after sendto(). If there are many messages going through, some of them will fall in this scenario. if I try sending a burst of 100 messages, I see two or three presenting the scenario.
Just forward() with a single process does not give the capacity. I'm getting almost 1000caps. More than that and start getting errores, retransmissions, etc. And this is just one way. I need to receive the call to go back to the network (our application is a B2BUA), so I will be down to 500caps, with a simple scenario, with no reliable responses, reinvites, updates, etc. I will end up having as many standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK. Very small, like 10ms, should be enough. Simple and it should work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
then the overtaking is in between reading from the socket and getting to parsing the call-id value -- the cpu is lost by first reader after recvfrom() and the second process get enough cpu time to go ahead further. I haven't encountered this case, but as I said previously, it is very unlikely, but still possible. I added the route_locks_size because in the past I had cases when processing of some messages took longer executing config (e.g., due to authentication, accounting, ..) and I needed to be sure they are processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending out (using forward()) gives the capacity, if you don't do any other complex processing. Or if you do more complex processing, use a dispatcher process with forwarding to local host or in a similar manner try to use mqueue+rtimer for dispatching using shared memory queues.
Of course, it is open source and there is also the C coding way, to add a synchronizing mechanism to protect against parallel execution of the code from recvfrom() till call-id lock is acquired.
Cheers, Daniel
-- Luis Rojas Software Architect Sixbell Los Leones 1200 Providencia Santiago, Chile Phone: (+56-2) 22001288 mailto:luis.rojas@sixbell.com http://www.sixbell.com
Hello, Daniel,
Yes, yes, yes, you are right. I got confused for a moment. Yes, the criteria for dispatcher is only for the request. And so, it will have no effect on the replies. On the replies only the via headers are considered. Wel, moving to round robin increased the throughput for a single instance. Now it can process over 1000 CAPS.
For the scenario ack-reinvite, the solution adding a small delay for re-invite, using something like async_ms_sleep() will solve it, so I am not worried ( I mentioned on a previous post that I have seen also that scenario happening with this operator. Re-invite immediately after ACK, and it caused us problems)
My problem is still 180-200. it will not matter the number of processes or cores. In the end, it's a classical multi-process/multi-threaded race condition. Considering the architecture of Kamailio, with multiple processes, the problem will appear. And the more the traffic, the more close in time 180 and 200 are, the more it will happen. With my currents test, with 180 and 200 very close, I am getting around 0.5% of cases suffering from that condition.
I know, if you think in "only SIP", yes, it's not so important the 180. it's important in my case, because my customer is very complicated, and they will not like to see messages coming to our platform in one order and going out in other. And the second : it's not only SIP. they usually have interworking, and 180 then will carry an ISUP ACM body that is important. As I mentioned in a previous post, for instance, the Backward Call Indicators, with very important subfields like the Charge Indicator.
I understand. It's UDP. Messages can be lost on the network. OK. Messages can arrive out of order. OK. But i't s pity that if messages were not lost and arrived in order, they leave kamailio out of order.
So far the only solution I see is to try to insert a small delay before forwarding the 200.
Best rgards,
Luis
On 4/9/20 3:58 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
dispatcher has nothing to do with handling sip replies. It is intended only for routing sip requests. If you use dispatcher for replies, you do it wrong, just let kamailio route them based on Via headers.
So maybe I was looking at the wrong message flow processing, I was speaking mainly about the case when the caller sends quickly the reINVITE after the ACK to the initial INVITE 200ok and the reINVITE gets to callee before the ACK. That was more of a branching in discussion on Alex' remarks and the situation that I enocountered in the past and created troubles. Never had to deal with troubles caused by change of order between 180 and 200. In IP world, if the time between 180 and 200 is very short, it doesn't matter at all, because the 180 is for start play a ring tone, which a human may not even hear it when 200 comes 50ms after it.
If you face the re-ordering for replies, then Kamailio doesn't do much internally if you don't have reply_route{} (as well as no onsend_route) in config file, provided that you do not use tm module for sending out (and by that no onreply_route or failure_route).
For a sip reply, kamilio is parsing the headers to find the 2nd Via header and use that address to send out the reply. The request route is not executed for sip replies.
What you can try is to set number of kamailio processes not to exceed the number of CPU cores, so there is "no real competition" to get CPU cycles. It could improve a bit, but still not a 100% accuracy (ie., there are other processes running on the system).
Cheers, Daniel
On 09.04.20 21:29, Luis Rojas G. wrote:
Hello,
I just realized that I had the dispatcher configured using a hash of Call-ID. That means, after recvfrom there must be an extra processing finding the Call-ID header in message, to calculate a hash and then forward() message. The more the processing, the more cases when 200 could arrive before 180. I just changed it to round robin, and the amount decreased a lot, but it's still there. If I send a burst of 1000 messages, about 5 of them leave out of order every time.
Best regards,
Luis
On 4/9/20 1:48 PM, Luis Rojas G. wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and I don't see it so unlikely at all that a process loses cpu just after recvfrom(). It's just as probable as to lose it just before, or when writing on a cache or just before of after sendto(). If there are many messages going through, some of them will fall in this scenario. if I try sending a burst of 100 messages, I see two or three presenting the scenario.
Just forward() with a single process does not give the capacity. I'm getting almost 1000caps. More than that and start getting errores, retransmissions, etc. And this is just one way. I need to receive the call to go back to the network (our application is a B2BUA), so I will be down to 500caps, with a simple scenario, with no reliable responses, reinvites, updates, etc. I will end up having as many standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK. Very small, like 10ms, should be enough. Simple and it should work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
then the overtaking is in between reading from the socket and getting to parsing the call-id value -- the cpu is lost by first reader after recvfrom() and the second process get enough cpu time to go ahead further. I haven't encountered this case, but as I said previously, it is very unlikely, but still possible. I added the route_locks_size because in the past I had cases when processing of some messages took longer executing config (e.g., due to authentication, accounting, ..) and I needed to be sure they are processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending out (using forward()) gives the capacity, if you don't do any other complex processing. Or if you do more complex processing, use a dispatcher process with forwarding to local host or in a similar manner try to use mqueue+rtimer for dispatching using shared memory queues.
Of course, it is open source and there is also the C coding way, to add a synchronizing mechanism to protect against parallel execution of the code from recvfrom() till call-id lock is acquired.
Cheers, Daniel
-- Luis Rojas Software Architect Sixbell Los Leones 1200 Providencia Santiago, Chile Phone: (+56-2) 22001288 mailto:luis.rojas@sixbell.com http://www.sixbell.com
-- Daniel-Constantin Mierla --www.asipto.com www.twitter.com/miconda --www.linkedin.com/in/miconda
An ACM in a 180?
On Thu, 9 Apr 2020 at 21:37, Luis Rojas G. luis.rojas@sixbell.com wrote:
Hello, Daniel,
Yes, yes, yes, you are right. I got confused for a moment. Yes, the criteria for dispatcher is only for the request. And so, it will have no effect on the replies. On the replies only the via headers are considered. Wel, moving to round robin increased the throughput for a single instance. Now it can process over 1000 CAPS.
For the scenario ack-reinvite, the solution adding a small delay for re-invite, using something like async_ms_sleep() will solve it, so I am not worried ( I mentioned on a previous post that I have seen also that scenario happening with this operator. Re-invite immediately after ACK, and it caused us problems)
My problem is still 180-200. it will not matter the number of processes or cores. In the end, it's a classical multi-process/multi-threaded race condition. Considering the architecture of Kamailio, with multiple processes, the problem will appear. And the more the traffic, the more close in time 180 and 200 are, the more it will happen. With my currents test, with 180 and 200 very close, I am getting around 0.5% of cases suffering from that condition.
I know, if you think in "only SIP", yes, it's not so important the 180. it's important in my case, because my customer is very complicated, and they will not like to see messages coming to our platform in one order and going out in other. And the second : it's not only SIP. they usually have interworking, and 180 then will carry an ISUP ACM body that is important. As I mentioned in a previous post, for instance, the Backward Call Indicators, with very important subfields like the Charge Indicator.
I understand. It's UDP. Messages can be lost on the network. OK. Messages can arrive out of order. OK. But i't s pity that if messages were not lost and arrived in order, they leave kamailio out of order.
So far the only solution I see is to try to insert a small delay before forwarding the 200.
Best rgards,
Luis
On 4/9/20 3:58 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
dispatcher has nothing to do with handling sip replies. It is intended only for routing sip requests. If you use dispatcher for replies, you do it wrong, just let kamailio route them based on Via headers.
So maybe I was looking at the wrong message flow processing, I was speaking mainly about the case when the caller sends quickly the reINVITE after the ACK to the initial INVITE 200ok and the reINVITE gets to callee before the ACK. That was more of a branching in discussion on Alex' remarks and the situation that I enocountered in the past and created troubles. Never had to deal with troubles caused by change of order between 180 and 200. In IP world, if the time between 180 and 200 is very short, it doesn't matter at all, because the 180 is for start play a ring tone, which a human may not even hear it when 200 comes 50ms after it.
If you face the re-ordering for replies, then Kamailio doesn't do much internally if you don't have reply_route{} (as well as no onsend_route) in config file, provided that you do not use tm module for sending out (and by that no onreply_route or failure_route).
For a sip reply, kamilio is parsing the headers to find the 2nd Via header and use that address to send out the reply. The request route is not executed for sip replies.
What you can try is to set number of kamailio processes not to exceed the number of CPU cores, so there is "no real competition" to get CPU cycles. It could improve a bit, but still not a 100% accuracy (ie., there are other processes running on the system).
Cheers, Daniel On 09.04.20 21:29, Luis Rojas G. wrote:
Hello,
I just realized that I had the dispatcher configured using a hash of Call-ID. That means, after recvfrom there must be an extra processing finding the Call-ID header in message, to calculate a hash and then forward() message. The more the processing, the more cases when 200 could arrive before 180. I just changed it to round robin, and the amount decreased a lot, but it's still there. If I send a burst of 1000 messages, about 5 of them leave out of order every time.
Best regards,
Luis
On 4/9/20 1:48 PM, Luis Rojas G. wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and I don't see it so unlikely at all that a process loses cpu just after recvfrom(). It's just as probable as to lose it just before, or when writing on a cache or just before of after sendto(). If there are many messages going through, some of them will fall in this scenario. if I try sending a burst of 100 messages, I see two or three presenting the scenario.
Just forward() with a single process does not give the capacity. I'm getting almost 1000caps. More than that and start getting errores, retransmissions, etc. And this is just one way. I need to receive the call to go back to the network (our application is a B2BUA), so I will be down to 500caps, with a simple scenario, with no reliable responses, reinvites, updates, etc. I will end up having as many standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK. Very small, like 10ms, should be enough. Simple and it should work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
then the overtaking is in between reading from the socket and getting to parsing the call-id value -- the cpu is lost by first reader after recvfrom() and the second process get enough cpu time to go ahead further. I haven't encountered this case, but as I said previously, it is very unlikely, but still possible. I added the route_locks_size because in the past I had cases when processing of some messages took longer executing config (e.g., due to authentication, accounting, ..) and I needed to be sure they are processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending out (using forward()) gives the capacity, if you don't do any other complex processing. Or if you do more complex processing, use a dispatcher process with forwarding to local host or in a similar manner try to use mqueue+rtimer for dispatching using shared memory queues.
Of course, it is open source and there is also the C coding way, to add a synchronizing mechanism to protect against parallel execution of the code from recvfrom() till call-id lock is acquired.
Cheers, Daniel
-- Luis Rojas Software Architect SixbellLos Leones 1200 Providencia Santiago, Chile https://www.google.com/maps/search/Los+Leones+1200%0AProvidencia%0ASantiago,+Chile?entry=gmail&source=g Phone: (+56-2) 22001288mailto:luis.rojas@sixbell.com luis.rojas@sixbell.comhttp://www.sixbell.com https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981766356&sdata=D2B0niyYm9fstMXOG9b%2BMzlbc3pZmt72mkL3PPPy7kY%3D&reserved=0
-- Daniel-Constantin Mierla -- www.asipto.com https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com%2F&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981766356&sdata=DO4yFB3gccT5ZXuv37oOLIri%2BF5jTnA4kHQaIjV0YbQ%3D&reserved=0www.twitter.com/miconda https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981776349&sdata=x%2BMcm9Dv%2FGY4fg54k6EbQIwHyeAMe4Y6hZrNCfOG4Q4%3D&reserved=0 -- www.linkedin.com/in/miconda https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981786347&sdata=b86Wa3h84oy0BcVl6pLOqLwgi4Fssya2re7Y1GOiEd4%3D&reserved=0
-- Luis Rojas Software Architect SixbellLos Leones 1200 Providencia Santiago, Chile https://www.google.com/maps/search/Los+Leones+1200%0AProvidencia%0ASantiago,+Chile?entry=gmail&source=g Phone: (+56-2) 22001288mailto:luis.rojas@sixbell.com luis.rojas@sixbell.comhttp://www.sixbell.com
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Sorry, I read an ANM... Nevermind
On Fri, 10 Apr 2020 at 02:19, David Villasmil < david.villasmil.work@gmail.com> wrote:
An ACM in a 180?
On Thu, 9 Apr 2020 at 21:37, Luis Rojas G. luis.rojas@sixbell.com wrote:
Hello, Daniel,
Yes, yes, yes, you are right. I got confused for a moment. Yes, the criteria for dispatcher is only for the request. And so, it will have no effect on the replies. On the replies only the via headers are considered. Wel, moving to round robin increased the throughput for a single instance. Now it can process over 1000 CAPS.
For the scenario ack-reinvite, the solution adding a small delay for re-invite, using something like async_ms_sleep() will solve it, so I am not worried ( I mentioned on a previous post that I have seen also that scenario happening with this operator. Re-invite immediately after ACK, and it caused us problems)
My problem is still 180-200. it will not matter the number of processes or cores. In the end, it's a classical multi-process/multi-threaded race condition. Considering the architecture of Kamailio, with multiple processes, the problem will appear. And the more the traffic, the more close in time 180 and 200 are, the more it will happen. With my currents test, with 180 and 200 very close, I am getting around 0.5% of cases suffering from that condition.
I know, if you think in "only SIP", yes, it's not so important the 180. it's important in my case, because my customer is very complicated, and they will not like to see messages coming to our platform in one order and going out in other. And the second : it's not only SIP. they usually have interworking, and 180 then will carry an ISUP ACM body that is important. As I mentioned in a previous post, for instance, the Backward Call Indicators, with very important subfields like the Charge Indicator.
I understand. It's UDP. Messages can be lost on the network. OK. Messages can arrive out of order. OK. But i't s pity that if messages were not lost and arrived in order, they leave kamailio out of order.
So far the only solution I see is to try to insert a small delay before forwarding the 200.
Best rgards,
Luis
On 4/9/20 3:58 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
dispatcher has nothing to do with handling sip replies. It is intended only for routing sip requests. If you use dispatcher for replies, you do it wrong, just let kamailio route them based on Via headers.
So maybe I was looking at the wrong message flow processing, I was speaking mainly about the case when the caller sends quickly the reINVITE after the ACK to the initial INVITE 200ok and the reINVITE gets to callee before the ACK. That was more of a branching in discussion on Alex' remarks and the situation that I enocountered in the past and created troubles. Never had to deal with troubles caused by change of order between 180 and 200. In IP world, if the time between 180 and 200 is very short, it doesn't matter at all, because the 180 is for start play a ring tone, which a human may not even hear it when 200 comes 50ms after it.
If you face the re-ordering for replies, then Kamailio doesn't do much internally if you don't have reply_route{} (as well as no onsend_route) in config file, provided that you do not use tm module for sending out (and by that no onreply_route or failure_route).
For a sip reply, kamilio is parsing the headers to find the 2nd Via header and use that address to send out the reply. The request route is not executed for sip replies.
What you can try is to set number of kamailio processes not to exceed the number of CPU cores, so there is "no real competition" to get CPU cycles. It could improve a bit, but still not a 100% accuracy (ie., there are other processes running on the system).
Cheers, Daniel On 09.04.20 21:29, Luis Rojas G. wrote:
Hello,
I just realized that I had the dispatcher configured using a hash of Call-ID. That means, after recvfrom there must be an extra processing finding the Call-ID header in message, to calculate a hash and then forward() message. The more the processing, the more cases when 200 could arrive before 180. I just changed it to round robin, and the amount decreased a lot, but it's still there. If I send a burst of 1000 messages, about 5 of them leave out of order every time.
Best regards,
Luis
On 4/9/20 1:48 PM, Luis Rojas G. wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and I don't see it so unlikely at all that a process loses cpu just after recvfrom(). It's just as probable as to lose it just before, or when writing on a cache or just before of after sendto(). If there are many messages going through, some of them will fall in this scenario. if I try sending a burst of 100 messages, I see two or three presenting the scenario.
Just forward() with a single process does not give the capacity. I'm getting almost 1000caps. More than that and start getting errores, retransmissions, etc. And this is just one way. I need to receive the call to go back to the network (our application is a B2BUA), so I will be down to 500caps, with a simple scenario, with no reliable responses, reinvites, updates, etc. I will end up having as many standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200 OK. Very small, like 10ms, should be enough. Simple and it should work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
MICONDA@GMAIL.COM appears similar to someone who previously sent you email, but may not be that person. Learn why this could be a risk http://aka.ms/LearnAboutSenderIdentification Feedback http://aka.ms/SafetyTipsFeedback
Hello,
then the overtaking is in between reading from the socket and getting to parsing the call-id value -- the cpu is lost by first reader after recvfrom() and the second process get enough cpu time to go ahead further. I haven't encountered this case, but as I said previously, it is very unlikely, but still possible. I added the route_locks_size because in the past I had cases when processing of some messages took longer executing config (e.g., due to authentication, accounting, ..) and I needed to be sure they are processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending out (using forward()) gives the capacity, if you don't do any other complex processing. Or if you do more complex processing, use a dispatcher process with forwarding to local host or in a similar manner try to use mqueue+rtimer for dispatching using shared memory queues.
Of course, it is open source and there is also the C coding way, to add a synchronizing mechanism to protect against parallel execution of the code from recvfrom() till call-id lock is acquired.
Cheers, Daniel
-- Luis Rojas Software Architect SixbellLos Leones 1200 Providencia Santiago, Chile https://www.google.com/maps/search/Los+Leones+1200%0AProvidencia%0ASantiago,+Chile?entry=gmail&source=g Phone: (+56-2) 22001288mailto:luis.rojas@sixbell.com luis.rojas@sixbell.comhttp://www.sixbell.com https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981766356&sdata=D2B0niyYm9fstMXOG9b%2BMzlbc3pZmt72mkL3PPPy7kY%3D&reserved=0
-- Daniel-Constantin Mierla -- www.asipto.com https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.asipto.com%2F&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981766356&sdata=DO4yFB3gccT5ZXuv37oOLIri%2BF5jTnA4kHQaIjV0YbQ%3D&reserved=0www.twitter.com/miconda https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.twitter.com%2Fmiconda&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981776349&sdata=x%2BMcm9Dv%2FGY4fg54k6EbQIwHyeAMe4Y6hZrNCfOG4Q4%3D&reserved=0 -- www.linkedin.com/in/miconda https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fmiconda&data=02%7C01%7C%7Cdc7f541ef3c74e85e61008d7dcc0562d%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637220590981786347&sdata=b86Wa3h84oy0BcVl6pLOqLwgi4Fssya2re7Y1GOiEd4%3D&reserved=0
-- Luis Rojas Software Architect SixbellLos Leones 1200 Providencia Santiago, Chile https://www.google.com/maps/search/Los+Leones+1200%0AProvidencia%0ASantiago,+Chile?entry=gmail&source=g Phone: (+56-2) 22001288mailto:luis.rojas@sixbell.com luis.rojas@sixbell.comhttp://www.sixbell.com
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
-
Daniel-Constantin Mierla -
David Villasmil -
Luis Rojas G.