15 Oct
2015
15 Oct
'15
4:36 a.m.
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio
4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops
again,... ) => starts happening when Kamailio is treating more than 20k
tcp connections, with between 500-1k req/s. During those periods, trying
to connect to kamailio with an openssl s_client -connect will end up in
a timeout or with a long delay before the certificate is received ( 03
CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to
decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is
impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was
deployed with both versions.
15 Oct
15 Oct
7:09 a.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hello,
is the same operating system and server, or are you using some other
machine for the new version? I can't remember right now any big change
in tcp/tls side, besides sni, which should not have any relevant impact
as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of
4.2.6?
Cheers,
Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio
4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops
again,... ) => starts happening when Kamailio is treating more than 20k
tcp connections, with between 500-1k req/s. During those periods, trying
to connect to kamailio with an openssl s_client -connect will end up in
a timeout or with a long delay before the certificate is received ( 03
CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to
decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is
impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was
deployed with both versions.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
8:29 a.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hi Daniel,
It is on the exact same server, same system configuration, same
configuration trimmed down for fitting 3.3, packages installed from the
kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific
traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
Hello,
is the same operating system and server, or are you using some other
machine for the new version? I can't remember right now any big change
in tcp/tls side, besides sni, which should not have any relevant
impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages
of 4.2.6?
Cheers,
Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio
4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops
again,... ) => starts happening when Kamailio is treating more than 20k
tcp connections, with between 500-1k req/s. During those periods, trying
to connect to kamailio with an openssl s_client -connect will end up in
a timeout or with a long delay before the certificate is received ( 03
CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to
decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is
impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was
deployed with both versions.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
19 Oct
19 Oct
5:03 p.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hello,
can you use benchamrk module to see if any part of config got slower? if
executing of config from beginning of request_route to save() is more or
less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound and
doing unregister if the tcp/tls connection is lost. On tls module, there
was some work for SNI. In tcp core, some callbacks were added for
websockets. If I haven't missed something relevant, I can't imagine what
would create this handling issue between the versions.
Cheers,
Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
Hi Daniel,
It is on the exact same server, same system configuration, same
configuration trimmed down for fitting 3.3, packages installed from
the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific
traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Hello,
is the same operating system and server, or are you using some other
machine for the new version? I can't remember right now any big
change in tcp/tls side, besides sni, which should not have any
relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages
of 4.2.6?
Cheers,
Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio
4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops
again,... ) => starts happening when Kamailio is treating more than 20k
tcp connections, with between 500-1k req/s. During those periods, trying
to connect to kamailio with an openssl s_client -connect will end up in
a timeout or with a long delay before the certificate is received ( 03
CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to
decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is
impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was
deployed with both versions.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
20 Oct
20 Oct
9:38 p.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hi Daniel,
I am doing that right now, thank you.
I will keep you posted as told on IRC.
Interesting part to keep everyone in the loop is when I activated the
"latency" logs options, I could see some messages regarding some core
functions taking more than 250ms to complete, or sometimes just a
route(SOMETHING). The other interesting part is that doing some
optimizations on the tls parameters seems to raise the limit where
Kamailio becomes completely unresponsive and starts dropping connections
( where on the 3.3 there was no need ).
I'll be back very soon with more detailed results.
Best,
Tristan.
On 10/19/2015 07:03 AM, Daniel-Constantin Mierla wrote:
Hello,
can you use benchamrk module to see if any part of config got slower?
if executing of config from beginning of request_route to save() is
more or less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound
and doing unregister if the tcp/tls connection is lost. On tls module,
there was some work for SNI. In tcp core, some callbacks were added
for websockets. If I haven't missed something relevant, I can't
imagine what would create this handling issue between the versions.
Cheers,
Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
Hi Daniel,
It is on the exact same server, same system configuration, same
configuration trimmed down for fitting 3.3, packages installed from
the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific
traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com Hello,
is the same operating system and server, or are you using some other
machine for the new version? I can't remember right now any big
change in tcp/tls side, besides sni, which should not have any
relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages
of 4.2.6?
Cheers,
Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio
4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops
again,... ) => starts happening when Kamailio is treating more than 20k
tcp connections, with between 500-1k req/s. During those periods, trying
to connect to kamailio with an openssl s_client -connect will end up in
a timeout or with a long delay before the certificate is received ( 03
CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to
decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is
impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was
deployed with both versions.
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
21 Oct
21 Oct
1:37 a.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hi Daniel,
Same config, same server, only kamailio version is different, starting
the timer in the beginning of the config, and ending it just after the
save("location").
Here is the result of 4.2.6:
INFO: benchmark [benchmark.c:351]: _bm_log_timer(): benchmark (timer
CHECKREG [0]): 2480 [ msgs/total/min/max/avg - LR:
100/2585000/1272/190744/25850.000000 | GB:
1900/79666245/1272/335018/41929.602632]
And 3.3:
INFO: benchmark [benchmark.c:351]: benchmark (timer CHECKREG [0]): 17022
[ msgs/total/min/max/avg - LR: 100/2139618/1239/167551/21396.180000 |
GB: 1900/49367288/1217/192815/25982.783158]
Let me try to pinpoint better now what is the difference here.
PS: I sent you a private mail about this, in case you have time :)
On 10/20/2015 11:38 AM, Tristan Mahé wrote:
Hi Daniel,
I am doing that right now, thank you.
I will keep you posted as told on IRC.
Interesting part to keep everyone in the loop is when I activated the
"latency" logs options, I could see some messages regarding some core
functions taking more than 250ms to complete, or sometimes just a
route(SOMETHING). The other interesting part is that doing some
optimizations on the tls parameters seems to raise the limit where
Kamailio becomes completely unresponsive and starts dropping
connections ( where on the 3.3 there was no need ).
I'll be back very soon with more detailed results.
Best,
Tristan.
On 10/19/2015 07:03 AM, Daniel-Constantin Mierla wrote:
Hello,
can you use benchamrk module to see if any part of config got slower?
if executing of config from beginning of request_route to save() is
more or less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound
and doing unregister if the tcp/tls connection is lost. On tls
module, there was some work for SNI. In tcp core, some callbacks were
added for websockets. If I haven't missed something relevant, I can't
imagine what would create this handling issue between the versions.
Cheers,
Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users Hi Daniel,
It is on the exact same server, same system configuration, same
configuration trimmed down for fitting 3.3, packages installed from
the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some
specific traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com Hello,
is the same operating system and server, or are you using some
other machine for the new version? I can't remember right now any
big change in tcp/tls side, besides sni, which should not have any
relevant impact as described here.
Also, have you installed from git branch 4.2 or the
tarball/packages of 4.2.6?
Cheers,
Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
> Hi List,
>
> I've been reaching a strange thing lately, trying to upgrade to kamailio
> 4.2.6 ( identical config ):
>
> - 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
> easily reproduced ).
> - more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
> at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
> - kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
> accepting new tcp connections, then resume a bit later, then stops
> again,... ) => starts happening when Kamailio is treating more than 20k
> tcp connections, with between 500-1k req/s. During those periods, trying
> to connect to kamailio with an openssl s_client -connect will end up in
> a timeout or with a long delay before the certificate is received ( 03
> CONNECTED ).
> - kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
> connections: after a while, the number of active connections starts to
> decrease on kam4, on kam3 it stays constant.
>
> I was wondering if I was alone in this situation or if someone else is
> impacted, has some advices ?
>
> It does not seem to be a config issue, as it is the same that was
> deployed with both versions.
>
>
>
> _______________________________________________
> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
> sr-users(a)lists.sip-router.org
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
3 Nov
3 Nov
12:50 p.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
Hi Daniel,
Same config, same server, only kamailio version is different, starting
the timer in the beginning of the config, and ending it just after the
save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl
library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf
memory manager, which ensures predictability of duration for memory
operations in all cases. f malloc can be faster for common case, but in
some corner cases (e.g., lots of tls connections dropped at the same
time) can be slower than tlsf.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
9:21 p.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hello Daniel,
Thanks for the new pointers, we will try it !
For the record, we are now able to obtain an almost identical behavior (
still a loss of around 3% left ) by tuning:
- tls parameters.
- tcp parameters.
- optimizations of the config file in many ways.
Those optimisations/tuning were not necessary on 3.3, but it is with 4.2+.
Anyway, we are now satisfied with the performance, but any small new
gain is always good to get :)
Best,
Tristan.
On 11/03/2015 01:50 AM, Daniel-Constantin Mierla wrote:
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
Hi Daniel,
Same config, same server, only kamailio version is different,
starting the timer in the beginning of the config, and ending it just
after the save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl
library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf
memory manager, which ensures predictability of duration for memory
operations in all cases. f malloc can be faster for common case, but
in some corner cases (e.g., lots of tls connections dropped at the
same time) can be slower than tlsf.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
4 Nov
4 Nov
12:01 a.m.
New subject: Huge TLS performance regression migrating from 3.3.7 to 4.2.6
Hello,
On 03/11/15 19:21, Tristan Mahé wrote:
Hello Daniel,
Thanks for the new pointers, we will try it !
For the record, we are now able to obtain an almost identical behavior
( still a loss of around 3% left )
Does this loss refer to performance?
by tuning:
- tls parameters.
- tcp parameters.
- optimizations of the config file in many ways.
Those optimisations/tuning were not necessary on 3.3, but it is with 4.2+.
if you want to list what you had to tune, specially the tcp/tls, then we
can analyze if there were changes to those features or the impact is
from somewhere else.
Cheers,
Daniel
Anyway, we are now satisfied with the performance, but any small new
gain is always good to get :)
Best,
Tristan.
On 11/03/2015 01:50 AM, Daniel-Constantin Mierla wrote:
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
_______________________________________________
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list
sr-users(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users Hi Daniel,
Same config, same server, only kamailio version is different,
starting the timer in the beginning of the config, and ending it
just after the save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl
library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf
memory manager, which ensures predictability of duration for memory
operations in all cases. f malloc can be faster for common case, but
in some corner cases (e.g., lots of tls connections dropped at the
same time) can be slower than tlsf.
Cheers,
Daniel
--
Daniel-Constantin Mierla
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
Book: SIP Routing With Kamailio - http://www.asipto.com
Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
3310
days inactive
3329
days old
8 comments
2 participants
participants (2)
-
Daniel-Constantin Mierla -
Tristan Mahé