12 Jan
2025
12 Jan
'25
6:20 p.m.
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
12 Jan
12 Jan
7:52 p.m.
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same
script logic with a native cfg and observing similar numbers? If it’s a simple script, you
can maybe just repeat the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test
with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
Sent: Sonntag, 12. Januar 2025 08:21
To: sr-users(a)lists.kamailio.org
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
13 Jan
13 Jan
2:32 a.m.
I haven't converted this test to native configuration yet. That is my next step to
rule out any general issues. I've never experienced any performance issues like this
before and since this was my first test using KEMI I assumed it was related in some
fashion. I've read through performance optimization related posts and documents that
I've been able to find.
-dan
________________________________
From: Henning Westerholt <hw(a)gilawa.com>
Sent: Sunday, January 12, 2025 03:52
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same
script logic with a native cfg and observing similar numbers? If it’s a simple script, you
can maybe just repeat the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test
with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
Sent: Sonntag, 12. Januar 2025 08:21
To: sr-users(a)lists.kamailio.org
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
1:56 p.m.
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I
had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to
a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue
identifying configuration issues.
-dan
________________________________
From: Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
Sent: Sunday, January 12, 2025 10:32
To: Henning Westerholt <hw(a)gilawa.com>om>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to
rule out any general issues. I've never experienced any performance issues like this
before and since this was my first test using KEMI I assumed it was related in some
fashion. I've read through performance optimization related posts and documents that
I've been able to find.
-dan
________________________________
From: Henning Westerholt <hw(a)gilawa.com>
Sent: Sunday, January 12, 2025 03:52
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same
script logic with a native cfg and observing similar numbers? If it’s a simple script, you
can maybe just repeat the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test
with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
Sent: Sonntag, 12. Januar 2025 08:21
To: sr-users(a)lists.kamailio.org
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
6:31 p.m.
Hello,
the performance of KEMI was compared when using Kamailio exported
functions, in the way that a function implemented inside Kamailio has
pretty much same performance when using it inside a kemi script or
inside native config file. For example, if one uses
KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...)
inside kamailio.cfg, then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script
and it has poor performances, then of course kamailio is slowed down, it
is nothing that can be done by kamailio code to improve that.
If you need to access database, sqlops functions are also exported to kemi.
To summarize, the goal was that the kemi engine does not add any (or at
least not significant) performance to execute Kamailio-specific
functions compared to execution by native config interpreter.
Cheers,
Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote:
Native config resulted in over 10k calls per second
without any drops.
I made some adjustments to the python config, looks like there was an
issue with the way I had been using return. That is handling almost
10k calls now.
However, when adding in other logic such as database calls using
python libraries, back to a few dozen calls before the queue grows and
drops occur.
If performance is expected to be close to native, then I probably need
to continue identifying configuration issues.
-dan
------------------------------------------------------------------------
*From:* Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
*Sent:* Sunday, January 12, 2025 10:32
*To:* Henning Westerholt <hw(a)gilawa.com>om>; Kamailio (SER) - Users
Mailing List <sr-users(a)lists.kamailio.org>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net>
*Subject:* [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe
I haven't converted this test to native configuration yet. That is my
next step to rule out any general issues. I've never experienced any
performance issues like this before and since this was my first test
using KEMI I assumed it was related in some fashion. I've read through
performance optimization related posts and documents that I've been
able to find.
-dan
------------------------------------------------------------------------
*From:* Henning Westerholt <hw(a)gilawa.com>
*Sent:* Sunday, January 12, 2025 03:52
*To:* Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net>
*Subject:* RE: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve
also tried the same script logic with a native cfg and observing
similar numbers? If it’s a simple script, you can maybe just repeat
the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to
observe in your test with KEMI? In this case it would be more of a
generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/ <https://skalatan.de/blog/>
Kamailio services – https://gilawa.com <https://gilawa.com/>
*From:* Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
*Sent:* Sonntag, 12. Januar 2025 08:21
*To:* sr-users(a)lists.kamailio.org
*Cc:* Daniel W. Graham <dan(a)cmsinter.net>
*Subject:* [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If
I exceed 150 calls per second the network receive queue grows and
Kamailio is unable to keep up with requests and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but
there is no impact. I've also attempted some buffer / memory tweaking
at the OS level, again with no impact. Increasing CPU cores and even
running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an
example of the performance they are seeing before I continue down this
path.
Thanks,
- dan
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users(a)lists.kamailio.org
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
14 Jan
14 Jan
8:43 p.m.
Yes, on the same page with all of this. Good to know performance is comparable and I can
focus optimization efforts on non kemi functions.
In this case clickhouse access was needed, so that's where the impact is occurring now
that kemi scripting issues are under control. Performance is continuing to improve.
Thanks Daniel
-dan
________________________________
From: Daniel-Constantin Mierla <miconda(a)gmail.com>
Sent: Monday, January 13, 2025 02:31
To: Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>rg>; Henning
Westerholt <hw(a)gilawa.com>
Cc: Daniel W. Graham <dan(a)cmsinter.net>
Subject: Re: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
Hello,
the performance of KEMI was compared when using Kamailio exported functions, in the way
that a function implemented inside Kamailio has pretty much same performance when using it
inside a kemi script or inside native config file. For example, if one uses
KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...) inside kamailio.cfg,
then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script and it has poor
performances, then of course kamailio is slowed down, it is nothing that can be done by
kamailio code to improve that.
If you need to access database, sqlops functions are also exported to kemi.
To summarize, the goal was that the kemi engine does not add any (or at least not
significant) performance to execute Kamailio-specific functions compared to execution by
native config interpreter.
Cheers,
Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote:
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I
had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to
a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue
identifying configuration issues.
-dan
________________________________
From: Daniel W. Graham via sr-users
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Sent: Sunday, January 12, 2025 10:32
To: Henning Westerholt <hw@gilawa.com><mailto:hw@gilawa.com>; Kamailio (SER) -
Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Cc: Daniel W. Graham <dan@cmsinter.net><mailto:dan@cmsinter.net>
Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to
rule out any general issues. I've never experienced any performance issues like this
before and since this was my first test using KEMI I assumed it was related in some
fashion. I've read through performance optimization related posts and documents that
I've been able to find.
-dan
________________________________
From: Henning Westerholt <hw@gilawa.com><mailto:hw@gilawa.com>
Sent: Sunday, January 12, 2025 03:52
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Cc: Daniel W. Graham <dan@cmsinter.net><mailto:dan@cmsinter.net>
Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same
script logic with a native cfg and observing similar numbers? If it’s a simple script, you
can maybe just repeat the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test
with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel W. Graham via sr-users
<sr-users@lists.kamailio.org><mailto:sr-users@lists.kamailio.org>
Sent: Sonntag, 12. Januar 2025 08:21
To: sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Cc: Daniel W. Graham <dan@cmsinter.net><mailto:dan@cmsinter.net>
Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
To unsubscribe send an email to
sr-users-leave@lists.kamailio.org<mailto:sr-users-leave@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
15 Jan
15 Jan
5:22 p.m.
You can do pull requests with what you think it worth adding to the kemi
tutorial:
-
https://github.com/kamailio/kamailio-docs/tree/master/kamailio-kemi-framewo…
Or the docs of the kemi modules.
Cheers,
Daniel
On 14.01.25 13:49, Alex Balashov via sr-users wrote:
Daniel-Constantin’s point that performance claims and
commitments can
only be applied to native functions probably should be loudly
underscored in the KEMI documentation, as well as in the documentation
for the relevant config interpreter modules.
Many users will not understand Kamailio architecture well enough, and
will assume that because KEMI unlocks the ability to use general
language runtimes and third-party libraries (with unknown concurrency
architecture and performance characteristics), e.g. in Python, it also
allows one to benefit from Kamailio’s high performance with anything
that one might embed in it, in any language.
— Alex
—
Sent from my iPad
Web: https://evaristesys.com
--
Daniel-Constantin Mierla (@ asipto.com)
twitter.com/miconda -- linkedin.com/in/miconda
Kamailio Consultancy, Training and Development Services -- asipto.com
On 14 Jan 2025, at 05:12, Daniel W. Graham via
sr-users
<sr-users(a)lists.kamailio.org> wrote:
Yes, on the same page with all of this. Good to know performance is
comparable and I can focus optimization efforts on non kemi functions.
In this case clickhouse access was needed, so that's where the impact
is occurring now that kemi scripting issues are under control.
Performance is continuing to improve.
Thanks Daniel
-dan
------------------------------------------------------------------------
*From:* Daniel-Constantin Mierla <miconda(a)gmail.com>
*Sent:* Monday, January 13, 2025 02:31
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>rg>; Henning Westerholt <hw(a)gilawa.com>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net>
*Subject:* Re: [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the organization. Do
not click links or open attachments unless you recognize the sender
and know the content is safe
Hello,
the performance of KEMI was compared when using Kamailio exported
functions, in the way that a function implemented inside Kamailio has
pretty much same performance when using it inside a kemi script or
inside native config file. For example, if one uses
KSR.auth.pv_auth_check(...) inside kemi script or pv_auth_check(...)
inside kamailio.cfg, then the performance is very much the same.
Now, if you use a pure python/lua/... library inside your kemi script
and it has poor performances, then of course kamailio is slowed down,
it is nothing that can be done by kamailio code to improve that.
If you need to access database, sqlops functions are also exported to
kemi.
To summarize, the goal was that the kemi engine does not add any (or
at least not significant) performance to execute Kamailio-specific
functions compared to execution by native config interpreter.
Cheers,
Daniel
On 13.01.25 03:56, Daniel W. Graham via sr-users wrote:
Native config resulted in over 10k calls per second without any
drops.
I made some adjustments to the python config, looks like there
was an issue with the way I had been using return. That is
handling almost 10k calls now.
However, when adding in other logic such as database calls using
python libraries, back to a few dozen calls before the queue
grows and drops occur.
If performance is expected to be close to native, then I probably
need to continue identifying configuration issues.
-dan
------------------------------------------------------------------------
*From:* Daniel W. Graham via sr-users
<sr-users(a)lists.kamailio.org> <mailto:sr-users@lists.kamailio.org>
*Sent:* Sunday, January 12, 2025 10:32
*To:* Henning Westerholt <hw(a)gilawa.com> <mailto:hw@gilawa.com>;
Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
<mailto:sr-users@lists.kamailio.org>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net> <mailto:dan@cmsinter.net>
*Subject:* [SR-Users] Re: Performance issues with KEMI
*Caution:* This email originated from outside of the
organization. Do not click links or open attachments unless you
recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That
is my next step to rule out any general issues. I've never
experienced any performance issues like this before and since
this was my first test using KEMI I assumed it was related in
some fashion. I've read through performance optimization related
posts and documents that I've been able to find.
-dan
------------------------------------------------------------------------
*From:* Henning Westerholt <hw(a)gilawa.com> <mailto:hw@gilawa.com>
*Sent:* Sunday, January 12, 2025 03:52
*To:* Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org> <mailto:sr-users@lists.kamailio.org>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net> <mailto:dan@cmsinter.net>
*Subject:* RE: Performance issues with KEMI
*Caution:* This email originated from outside of the
organization. Do not click links or open attachments unless you
recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if
you’ve also tried the same script logic with a native cfg and
observing similar numbers? If it’s a simple script, you can maybe
just repeat the same test. There were benchmarks done for KEMI
some years ago which only showed a small performance difference.
Or do you have generic performance issues which you just happened
to observe in your test with KEMI? In this case it would be more
of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
<https://skalatan.de/blog/>
Kamailio services – https://gilawa.com <https://gilawa.com/>
*From:* Daniel W. Graham via sr-users
<sr-users(a)lists.kamailio.org> <mailto:sr-users@lists.kamailio.org>
*Sent:* Sonntag, 12. Januar 2025 08:21
*To:* sr-users(a)lists.kamailio.org
<mailto:sr-users@lists.kamailio.org>
*Cc:* Daniel W. Graham <dan(a)cmsinter.net> <mailto:dan@cmsinter.net>
*Subject:* [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance
issues. If I exceed 150 calls per second the network receive
queue grows and Kamailio is unable to keep up with requests and
they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations,
but there is no impact. I've also attempted some buffer / memory
tweaking at the OS level, again with no impact. Increasing CPU
cores and even running the test on bare metal results in the same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an
example of the performance they are seeing before I continue down
this path.
Thanks,
- dan
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users(a)lists.kamailio.org <mailto:sr-users@lists.kamailio.org>
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
<mailto:sr-users-leave@lists.kamailio.org> Important: keep the
mailing list in the recipients, do not reply only to the sender!
-- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda --
linkedin.com/in/miconda Kamailio Consultancy, Training and
Development Services -- asipto.com
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users(a)lists.kamailio.org
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only
to the sender!
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions --
sr-users(a)lists.kamailio.org
To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
13 Jan
13 Jan
11:31 p.m.
Hello Daniel,
great that you solved it for the existing KEMI configuration with some adaptions.
Cheers,
Henning
From: Daniel W. Graham <dan(a)cmsinter.net>
Sent: Montag, 13. Januar 2025 03:57
To: Henning Westerholt <hw(a)gilawa.com>om>; Kamailio (SER) - Users Mailing List
<sr-users(a)lists.kamailio.org>
Subject: Re: Performance issues with KEMI
Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an issue with the way I
had been using return. That is handling almost 10k calls now.
However, when adding in other logic such as database calls using python libraries, back to
a few dozen calls before the queue grows and drops occur.
If performance is expected to be close to native, then I probably need to continue
identifying configuration issues.
-dan
________________________________
From: Daniel W. Graham via sr-users
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Sent: Sunday, January 12, 2025 10:32
To: Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>; Kamailio (SER) -
Users Mailing List
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Cc: Daniel W. Graham <dan@cmsinter.net<mailto:dan@cmsinter.net>>
Subject: [SR-Users] Re: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
I haven't converted this test to native configuration yet. That is my next step to
rule out any general issues. I've never experienced any performance issues like this
before and since this was my first test using KEMI I assumed it was related in some
fashion. I've read through performance optimization related posts and documents that
I've been able to find.
-dan
________________________________
From: Henning Westerholt <hw@gilawa.com<mailto:hw@gilawa.com>>
Sent: Sunday, January 12, 2025 03:52
To: Kamailio (SER) - Users Mailing List
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Cc: Daniel W. Graham <dan@cmsinter.net<mailto:dan@cmsinter.net>>
Subject: RE: Performance issues with KEMI
Caution: This email originated from outside of the organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe
Hello Daniel,
I am wondering if your issues are specific to KEMI, e.g. if you’ve also tried the same
script logic with a native cfg and observing similar numbers? If it’s a simple script, you
can maybe just repeat the same test. There were benchmarks done for KEMI some years ago
which only showed a small performance difference.
Or do you have generic performance issues which you just happened to observe in your test
with KEMI? In this case it would be more of a generic performance optimization topic.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: Daniel W. Graham via sr-users
<sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>>
Sent: Sonntag, 12. Januar 2025 08:21
To: sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Cc: Daniel W. Graham <dan@cmsinter.net<mailto:dan@cmsinter.net>>
Subject: [SR-Users] Performance issues with KEMI
Testing out KEMI functionality and running into performance issues. If I exceed 150 calls
per second the network receive queue grows and Kamailio is unable to keep up with requests
and they begin dropping.
KEMI script for testing is just doing a stateless reply to invites.
Using python3s module.
I've played with Kamailio child processes and memory allocations, but there is no
impact. I've also attempted some buffer / memory tweaking at the OS level, again with
no impact. Increasing CPU cores and even running the test on bare metal results in the
same.
Example of receive queue at 150 calls per second -
Netid State Recv-Q Send-Q Local Address:Port
udp UNCONN 337280 0 x.x.x.x:5060
Just wondering if anyone has experienced similar issues or has an example of the
performance they are seeing before I continue down this path.
Thanks,
- dan
21 Jan
21 Jan
1:02 a.m.
I made some adjustments to the python config, looks like there was an
issue with the way I had been using return. That is handling almost 10k
calls now.
Can you share more details on this please? We're migrating to
Python3s KEMI, just to make sure we don't run into some known issue.
Thanks
пн, 13 янв. 2025 г. в 06:35, Daniel W. Graham via sr-users <
sr-users(a)lists.kamailio.org>gt;:
> Native config resulted in over 10k calls per second without any drops.
I made some adjustments to the python config, looks like there was an
issue with the way I had been using return. That is handling almost 10k
calls now.
>
> However, when adding in other logic such as database calls using python
> libraries, back to a few dozen calls before the queue grows and drops occur.
>
> If performance is expected to be close to native, then I probably need to
> continue identifying configuration issues.
>
> -dan
>
> ------------------------------
> *From:* Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
> *Sent:* Sunday, January 12, 2025 10:32
> *To:* Henning Westerholt <hw(a)gilawa.com>om>; Kamailio (SER) - Users Mailing
> List <sr-users(a)lists.kamailio.org>
> *Cc:* Daniel W. Graham <dan(a)cmsinter.net>
> *Subject:* [SR-Users] Re: Performance issues with KEMI
>
> *Caution:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe
>
> I haven't converted this test to native configuration yet. That is my next
> step to rule out any general issues. I've never experienced any performance
> issues like this before and since this was my first test using KEMI I
> assumed it was related in some fashion. I've read through performance
> optimization related posts and documents that I've been able to find.
>
> -dan
>
> ------------------------------
> *From:* Henning Westerholt <hw(a)gilawa.com>
> *Sent:* Sunday, January 12, 2025 03:52
> *To:* Kamailio (SER) - Users Mailing List <sr-users(a)lists.kamailio.org>
> *Cc:* Daniel W. Graham <dan(a)cmsinter.net>
> *Subject:* RE: Performance issues with KEMI
>
> *Caution:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe
>
> Hello Daniel,
>
>
>
> I am wondering if your issues are specific to KEMI, e.g. if you’ve also
> tried the same script logic with a native cfg and observing similar
> numbers? If it’s a simple script, you can maybe just repeat the same test.
> There were benchmarks done for KEMI some years ago which only showed a
> small performance difference.
>
>
>
> Or do you have generic performance issues which you just happened to
> observe in your test with KEMI? In this case it would be more of a generic
> performance optimization topic.
>
>
>
> Cheers,
>
>
>
> Henning
>
>
>
> --
>
> Henning Westerholt – https://skalatan.de/blog/
>
> Kamailio services – https://gilawa.com
>
>
>
> *From:* Daniel W. Graham via sr-users <sr-users(a)lists.kamailio.org>
> *Sent:* Sonntag, 12. Januar 2025 08:21
> *To:* sr-users(a)lists.kamailio.org
> *Cc:* Daniel W. Graham <dan(a)cmsinter.net>
> *Subject:* [SR-Users] Performance issues with KEMI
>
>
>
> Testing out KEMI functionality and running into performance issues. If I
> exceed 150 calls per second the network receive queue grows and Kamailio is
> unable to keep up with requests and they begin dropping.
>
>
>
> KEMI script for testing is just doing a stateless reply to invites.
>
>
>
> Using python3s module.
>
>
>
> I've played with Kamailio child processes and memory allocations, but
> there is no impact. I've also attempted some buffer / memory tweaking at
> the OS level, again with no impact. Increasing CPU cores and even running
> the test on bare metal results in the same.
>
>
>
> Example of receive queue at 150 calls per second -
>
> Netid State Recv-Q Send-Q Local Address:Port
>
> udp UNCONN 337280 0 x.x.x.x:5060
>
>
>
> Just wondering if anyone has experienced similar issues or has an example
> of the performance they are seeing before I continue down this path.
>
>
>
> Thanks,
>
>
>
> - dan
> __________________________________________________________
> Kamailio - Users Mailing List - Non Commercial Discussions --
> sr-users(a)lists.kamailio.org
> To unsubscribe send an email to sr-users-leave(a)lists.kamailio.org
> Important: keep the mailing list in the recipients, do not reply only to
> the sender!
>
--
Best regards,
Dmitry Sytchev
2
days inactive
10
days old
9 comments
5 participants
participants (5)
-
Alex Balashov -
Daniel W. Graham -
Daniel-Constantin Mierla -
Dmitry Sytchev -
Henning Westerholt