Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from internet and forwards it to FS servers in local network.When i execute "ss" command i see very high value in RECV-Q column.I THINK IT IS NOT NORMAL.PLEASE CORRECT ME IF I AM WRONG. ==================================================== [root@fep-1 proc]# ss -u -a -n -e | grep 5060 UNCONN 0 0 10.50.8.1:5060 *:* ino:6831630 sk:fd <-> UNCONN 0 0 10.50.7.18:5060 *:* ino:6831629 sk:fe <-> UNCONN *1183104* 0 10.50.7.254:5060 *:* ino:6831627 sk:ff <-> UNCONN *84864* 0 2607:f900:1:3::254:5060 :::* ino:6831628 sk:100 v6only:1 <-> ======================================================
Initially i thought that there is something in script which must be causing kamailio to process UDP request slower but i faced same issue with a very simple script where i simply reply with stateless 200 OK for each sip request : request_route {
sl_send_reply("200","OK");exit; $avp(uuid) = $rm + "-" + $ci;
........ ......... ......... .......
Server configuration : OS : CENTOS 7 Kernel : 4.16 CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz RAM : 32 GB
Please help me debugging this issue.Thanks in advance
No, this is not normal. But you almost certainly have some external I/O operation which is blocking the worker thread. There isn't much explanation otherwise.
This article (disclaimer: I wrote it) may help:
http://www.evaristesys.com/blog/tuning-kamailio-for-high-throughput-and-perf...
Hello,
On 28.08.18 12:42, sagar malam wrote:
Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from internet and forwards it to FS servers in local network.When i execute "ss" command i see very high value in RECV-Q column.I THINK IT IS NOT NORMAL.PLEASE CORRECT ME IF I AM WRONG. ==================================================== [root@fep-1 proc]# ss -u -a -n -e | grep 5060 UNCONN 0 0 10.50.8.1:5060 http://10.50.8.1:5060 *:* ino:6831630 sk:fd <-> UNCONN 0 0 10.50.7.18:5060 http://10.50.7.18:5060 *:* ino:6831629 sk:fe <-> UNCONN *1183104* 0 10.50.7.254:5060 http://10.50.7.254:5060 *:* ino:6831627 sk:ff <-> UNCONN *84864* 0 2607:f900:1:3::254:5060 :::* ino:6831628 sk:100 v6only:1 <-> ======================================================
Initially i thought that there is something in script which must be causing kamailio to process UDP request slower but i faced same issue with a very simple script where i simply reply with stateless 200 OK for each sip request : request_route {
sl_send_reply("200","OK");exit; $avp(uuid) = $rm + "-" + $ci;
........ ......... ......... .......
Server configuration : OS : CENTOS 7 Kernel : 4.16 CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz RAM : 32 GB
Please help me debugging this issue.Thanks in advance
if you have performance issues just with a very simple config sending a stateless sip reply, then check your system/firewall configuration/limits. Specially on centos, I have seen a lot of restrictive traffic rates limits set by selinux. Also, if you run in a virtual machine, there can be limits enforced by the vm platform.
If you still cannot sort out, I would just run similar tests on a vanilla debian.
Cheers, Daniel
Thanks for prompt response.
Some more information :
We are running it on virtual machine(Xenserver). Selinux and IPtables are disabled.I will check if VM platform is putting some limits or not. Further i have same Kamailio script in another server and i am not facing this issue there. OS max UDP buffer limit parameters is configured as below : net.core.rmem_max=26214400 net.core.rmem_default=26214400
Any other clue is most welcome.
On Tue, Aug 28, 2018 at 5:01 PM Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 28.08.18 12:42, sagar malam wrote:
Hello,
I am using Kamailio as a SIP proxy.So it receives SIP packet from internet and forwards it to FS servers in local network.When i execute "ss" command i see very high value in RECV-Q column.I THINK IT IS NOT NORMAL.PLEASE CORRECT ME IF I AM WRONG. ==================================================== [root@fep-1 proc]# ss -u -a -n -e | grep 5060 UNCONN 0 0 10.50.8.1:5060 *:* ino:6831630 sk:fd <-> UNCONN 0 0 10.50.7.18:5060 *:* ino:6831629 sk:fe <-> UNCONN *1183104* 0 10.50.7.254:5060 *:* ino:6831627 sk:ff <-> UNCONN *84864* 0 2607:f900:1:3::254:5060 :::* ino:6831628 sk:100 v6only:1 <-> ======================================================
Initially i thought that there is something in script which must be causing kamailio to process UDP request slower but i faced same issue with a very simple script where i simply reply with stateless 200 OK for each sip request : request_route {
sl_send_reply("200","OK");exit; $avp(uuid) = $rm + "-" + $ci;........ ......... ......... .......
Server configuration : OS : CENTOS 7 Kernel : 4.16 CPU : 5 X Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz RAM : 32 GB
Please help me debugging this issue.Thanks in advance
if you have performance issues just with a very simple config sending a stateless sip reply, then check your system/firewall configuration/limits. Specially on centos, I have seen a lot of restrictive traffic rates limits set by selinux. Also, if you run in a virtual machine, there can be limits enforced by the vm platform.
If you still cannot sort out, I would just run similar tests on a vanilla debian.
Cheers, Daniel
-- Daniel-Constantin Mierla -- www.asipto.comwww.twitter.com/miconda -- www.linkedin.com/in/miconda Kamailio World Conference -- www.kamailioworld.com
On Tue, Aug 28, 2018 at 05:12:01PM +0530, sagar malam wrote:
Further i have same Kamailio script in another server and i am not facing this issue there.
That pretty well points to the problem being caused by the runtime environment rather than the Kamailio config or anything it does.
Alex, I agree but i want to find root cause of this problem.Can you suggest me what next i need to look at OS level ? Thanks
On Tue, Aug 28, 2018 at 5:15 PM Alex Balashov abalashov@evaristesys.com wrote:
On Tue, Aug 28, 2018 at 05:12:01PM +0530, sagar malam wrote:
Further i have same Kamailio script in another server and i am not facing this issue there.
That pretty well points to the problem being caused by the runtime environment rather than the Kamailio config or anything it does.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 <(706)%20510-6800> / +1-800-250-5920 <(800)%20250-5920> (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Am Dienstag, 28. August 2018, 13:52:09 CEST schrieb sagar malam:
I agree but i want to find root cause of this problem.Can you suggest me what next i need to look at OS level ?
Hello Sagar,
as already suggested - I would try to sort out the other variables in this setup:
- compare the server that you have that is working correctly to the one that is not working correctly
- try with this particular setup on a "bare metal" instance or move for test to a VM host without any other I/O heavy virtual instances
- try with another Linux (Debian)
- try to start Kamailio with debug logging enabled and have a closer look to the logging timestamps, if there is maybe a suspicious delay during some operations
Best regards,
Henning
Hello Henning, Thanks for your inputs.
Issue got fixed after upgrading kernel version to 4.18.5( latest one). So there was something wrong in Kernel 4.16.7.
On Wed, Aug 29, 2018 at 1:09 AM Henning Westerholt hw@kamailio.org wrote:
Am Dienstag, 28. August 2018, 13:52:09 CEST schrieb sagar malam:
I agree but i want to find root cause of this problem.Can you suggest me what next i need to look at OS level ?
Hello Sagar,
as already suggested - I would try to sort out the other variables in this setup:
- compare the server that you have that is working correctly to the one
that is not working correctly
- try with this particular setup on a "bare metal" instance or move for
test to a VM host without any other I/O heavy virtual instances
try with another Linux (Debian)
try to start Kamailio with debug logging enabled and have a closer look
to the logging timestamps, if there is maybe a suspicious delay during some operations
Best regards,
Henning
-- Henning Westerholt https://skalatan.de/blog/
On Tue, Aug 28, 2018 at 01:31:04PM +0200, Daniel-Constantin Mierla wrote:
if you have performance issues just with a very simple config sending a stateless sip reply, then check your system/firewall configuration/limits. Specially on centos, I have seen a lot of restrictive traffic rates limits set by selinux. Also, if you run in a virtual machine, there can be limits enforced by the vm platform.
Really? When did SELinux get into the PPS enforcement business?
Well, I can't really say what limits selinux puts in place or other CentOS specific apps, but I encountered various strange situations where no centos sysadmin I interacted with could give a solution, even with selinux disabled, so could be something else...
Coming in my mind: limit on number of new TCP connections per second or number of forked processes per second - they are the reason for these "delay" core parameters for mod/child init.
Using Debian stable was always smooth an without such strange limits...
Cheers, Daniel
On Tue, 28 Aug 2018, 13:45 Alex Balashov, abalashov@evaristesys.com wrote:
On Tue, Aug 28, 2018 at 01:31:04PM +0200, Daniel-Constantin Mierla wrote:
if you have performance issues just with a very simple config sending a stateless sip reply, then check your system/firewall configuration/limits. Specially on centos, I have seen a lot of restrictive traffic rates limits set by selinux. Also, if you run in a virtual machine, there can be limits enforced by the vm platform.
Really? When did SELinux get into the PPS enforcement business?
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-
Alex Balashov -
Daniel-Constantin Mierla -
Henning Westerholt -
sagar malam