We are using kamailio 5.0 as an inbound dispatcher. Sometimes in testing we see it take over half a second to relay a 200 OK. The logs show the new dialog being creating for the 200 OK within .0001 seconds of receiving the inbound 200 but it takes another .5 seconds for the relayed 200 to actually hit the wire. Here is a screenshot showing the timing of the packets. The box is an EC2 m3.medium under almost zero load (1-2 calls).
Is this normal? What should I be looking at?
Screenshot below: 10.0.23.34 is the kamailio box. Carrier IP obfuscated. You can see the carrier resends the 200 OK because of the delay.
We have encountered similar in AWS. Unfortunately, the answer has turned out to be "because AWS". Try using instances more expressly marked for network throughput performance. But sometimes things like this happen anyway because AWS.
On May 2, 2018 3:49:28 PM EDT, Daniel Greenwald dgreenwald@gmail.com wrote:
We are using kamailio 5.0 as an inbound dispatcher. Sometimes in testing we see it take over half a second to relay a 200 OK. The logs show the new dialog being creating for the 200 OK within .0001 seconds of receiving the inbound 200 but it takes another .5 seconds for the relayed 200 to actually hit the wire. Here is a screenshot showing the timing of the packets. The box is an EC2 m3.medium under almost zero load (1-2 calls).
Is this normal? What should I be looking at?
Screenshot below: 10.0.23.34 is the kamailio box. Carrier IP obfuscated. You can see the carrier resends the 200 OK because of the delay.
-- Alex
-- Sent via mobile, please forgive typos and brevity.
So what's the solution? Don't run Kamailio in AWS? Surely we are not the only ones doing that...
On Wed, May 2, 2018 at 5:37 PM, Alex Balashov abalashov@evaristesys.com wrote:
We have encountered similar in AWS. Unfortunately, the answer has turned out to be "because AWS". Try using instances more expressly marked for network throughput performance. But sometimes things like this happen anyway because AWS.
On May 2, 2018 3:49:28 PM EDT, Daniel Greenwald dgreenwald@gmail.com wrote:
We are using kamailio 5.0 as an inbound dispatcher. Sometimes in testing we see it take over half a second to relay a 200 OK. The logs show the new dialog being creating for the 200 OK within .0001 seconds of receiving the inbound 200 but it takes another .5 seconds for the relayed 200 to actually hit the wire. Here is a screenshot showing the timing of the packets. The box is an EC2 m3.medium under almost zero load (1-2 calls).
Is this normal? What should I be looking at?
Screenshot below: 10.0.23.34 is the kamailio box. Carrier IP obfuscated. You can see the carrier resends the 200 OK because of the delay.
-- Alex
-- Sent via mobile, please forgive typos and brevity.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
No, just have to learn to pick the right kinds of instances. I can't tell you what the right ones are off the top of my head, unfortunately.
On May 3, 2018 10:02:10 AM EDT, Daniel Greenwald dgreenwald@gmail.com wrote:
So what's the solution? Don't run Kamailio in AWS? Surely we are not the only ones doing that...
On Wed, May 2, 2018 at 5:37 PM, Alex Balashov abalashov@evaristesys.com wrote:
We have encountered similar in AWS. Unfortunately, the answer has
turned
out to be "because AWS". Try using instances more expressly marked
for
network throughput performance. But sometimes things like this happen anyway because AWS.
On May 2, 2018 3:49:28 PM EDT, Daniel Greenwald
wrote:
We are using kamailio 5.0 as an inbound dispatcher. Sometimes in testing we see it take over half a second to relay a 200 OK. The logs show the
new
dialog being creating for the 200 OK within .0001 seconds of
receiving
the inbound 200 but it takes another .5 seconds for the relayed 200 to actually hit the wire. Here is a screenshot showing the timing of the
packets.
The box is an EC2 m3.medium under almost zero load (1-2 calls).
Is this normal? What should I be looking at?
Screenshot below: 10.0.23.34 is the kamailio box. Carrier IP obfuscated. You can see the carrier resends the 200 OK because of the delay.
-- Alex
-- Sent via mobile, please forgive typos and brevity.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Alex
-- Sent via mobile, please forgive typos and brevity.
PS. A good rule of thumb is that AWS has undisclosed and semi-disclosed (and rather low) network iop and PPS limits EVERYWHERE, including on instances whose specs are otherwise sufficient for the job. If in doubt, try a bigger instance.
-- Alex
-- Sent via mobile, please forgive typos and brevity.
Interesting points. In this case though. here is why I don't think its an AWS network limit: 1)This instance was in staging and had almost no traffic at that time. Why would a PPS limit have been hit?
2) The capture is taken directly on the kamailio box. If a network limit were reached the packet would be delayed across "the wire" which would only be visible in a capture on the far side. Here we see the delay locally on the box.
On Thu, May 3, 2018 at 10:19 AM, Alex Balashov abalashov@evaristesys.com wrote:
PS. A good rule of thumb is that AWS has undisclosed and semi-disclosed (and rather low) network iop and PPS limits EVERYWHERE, including on instances whose specs are otherwise sufficient for the job. If in doubt, try a bigger instance.
-- Alex
-- Sent via mobile, please forgive typos and brevity.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
enable latency logging and then you should discover if some action inside kamailio takes longer or if it is something with underneath layers. See related parameters in the core:
- https://www.kamailio.org/wiki/cookbooks/devel/core#latency_cfg_log
Cheers, Daniel
On 03.05.18 20:04, Daniel Greenwald wrote:
Interesting points. In this case though. here is why I don't think its an AWS network limit: 1)This instance was in staging and had almost no traffic at that time. Why would a PPS limit have been hit?
- The capture is taken directly on the kamailio box. If a network
limit were reached the packet would be delayed across "the wire" which would only be visible in a capture on the far side. Here we see the delay locally on the box.
On Thu, May 3, 2018 at 10:19 AM, Alex Balashov <abalashov@evaristesys.com mailto:abalashov@evaristesys.com> wrote:
PS. A good rule of thumb is that AWS has undisclosed and semi-disclosed (and rather low) network iop and PPS limits EVERYWHERE, including on instances whose specs are otherwise sufficient for the job. If in doubt, try a bigger instance. -- Alex -- Sent via mobile, please forgive typos and brevity. _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Alex Balashov -
Daniel Greenwald -
Daniel-Constantin Mierla