Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario, easily reproduced ). - more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7, at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ). - kamailio 4.2.6 intermittent processing of requests ( Kamailio stops accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ). - kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
Hello,
is the same operating system and server, or are you using some other machine for the new version? I can't remember right now any big change in tcp/tls side, besides sni, which should not have any relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of 4.2.6?
Cheers, Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
It is on the exact same server, same system configuration, same configuration trimmed down for fitting 3.3, packages installed from the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
Hello,
is the same operating system and server, or are you using some other machine for the new version? I can't remember right now any big change in tcp/tls side, besides sni, which should not have any relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of 4.2.6?
Cheers, Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
can you use benchamrk module to see if any part of config got slower? if executing of config from beginning of request_route to save() is more or less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound and doing unregister if the tcp/tls connection is lost. On tls module, there was some work for SNI. In tcp core, some callbacks were added for websockets. If I haven't missed something relevant, I can't imagine what would create this handling issue between the versions.
Cheers, Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
Hi Daniel,
It is on the exact same server, same system configuration, same configuration trimmed down for fitting 3.3, packages installed from the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
Hello,
is the same operating system and server, or are you using some other machine for the new version? I can't remember right now any big change in tcp/tls side, besides sni, which should not have any relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of 4.2.6?
Cheers, Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
I am doing that right now, thank you.
I will keep you posted as told on IRC.
Interesting part to keep everyone in the loop is when I activated the "latency" logs options, I could see some messages regarding some core functions taking more than 250ms to complete, or sometimes just a route(SOMETHING). The other interesting part is that doing some optimizations on the tls parameters seems to raise the limit where Kamailio becomes completely unresponsive and starts dropping connections ( where on the 3.3 there was no need ).
I'll be back very soon with more detailed results.
Best,
Tristan.
On 10/19/2015 07:03 AM, Daniel-Constantin Mierla wrote:
Hello,
can you use benchamrk module to see if any part of config got slower? if executing of config from beginning of request_route to save() is more or less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound and doing unregister if the tcp/tls connection is lost. On tls module, there was some work for SNI. In tcp core, some callbacks were added for websockets. If I haven't missed something relevant, I can't imagine what would create this handling issue between the versions.
Cheers, Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
Hi Daniel,
It is on the exact same server, same system configuration, same configuration trimmed down for fitting 3.3, packages installed from the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
Hello,
is the same operating system and server, or are you using some other machine for the new version? I can't remember right now any big change in tcp/tls side, besides sni, which should not have any relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of 4.2.6?
Cheers, Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
Hi Daniel,
Same config, same server, only kamailio version is different, starting the timer in the beginning of the config, and ending it just after the save("location").
Here is the result of 4.2.6: INFO: benchmark [benchmark.c:351]: _bm_log_timer(): benchmark (timer CHECKREG [0]): 2480 [ msgs/total/min/max/avg - LR: 100/2585000/1272/190744/25850.000000 | GB: 1900/79666245/1272/335018/41929.602632]
And 3.3: INFO: benchmark [benchmark.c:351]: benchmark (timer CHECKREG [0]): 17022 [ msgs/total/min/max/avg - LR: 100/2139618/1239/167551/21396.180000 | GB: 1900/49367288/1217/192815/25982.783158]
Let me try to pinpoint better now what is the difference here.
PS: I sent you a private mail about this, in case you have time :)
On 10/20/2015 11:38 AM, Tristan Mahé wrote:
Hi Daniel,
I am doing that right now, thank you.
I will keep you posted as told on IRC.
Interesting part to keep everyone in the loop is when I activated the "latency" logs options, I could see some messages regarding some core functions taking more than 250ms to complete, or sometimes just a route(SOMETHING). The other interesting part is that doing some optimizations on the tls parameters seems to raise the limit where Kamailio becomes completely unresponsive and starts dropping connections ( where on the 3.3 there was no need ).
I'll be back very soon with more detailed results.
Best,
Tristan.
On 10/19/2015 07:03 AM, Daniel-Constantin Mierla wrote:
Hello,
can you use benchamrk module to see if any part of config got slower? if executing of config from beginning of request_route to save() is more or less the same, then setup of connection could be the issue.
Since 3.3., there was some work on usrloc module about gruu/outbound and doing unregister if the tcp/tls connection is lost. On tls module, there was some work for SNI. In tcp core, some callbacks were added for websockets. If I haven't missed something relevant, I can't imagine what would create this handling issue between the versions.
Cheers, Daniel
On 15/10/15 07:29, Tristan Mahé wrote:
Hi Daniel,
It is on the exact same server, same system configuration, same configuration trimmed down for fitting 3.3, packages installed from the kamailio debian wheezy repository.
I can reproduce it just using sipp and a simple REGISTER loop.
Do you want me to run some specific tests ? provide you some specific traces ?
Best,
Tristan.
On 10/14/2015 09:09 PM, Daniel-Constantin Mierla wrote:
Hello,
is the same operating system and server, or are you using some other machine for the new version? I can't remember right now any big change in tcp/tls side, besides sni, which should not have any relevant impact as described here.
Also, have you installed from git branch 4.2 or the tarball/packages of 4.2.6?
Cheers, Daniel
On 15/10/15 03:36, Tristan Mahé wrote:
Hi List,
I've been reaching a strange thing lately, trying to upgrade to kamailio 4.2.6 ( identical config ):
- 35% loss of performance on TLS connections ( Sipp REGISTER scenario,
easily reproduced ).
- more TCP workers needed to ease the TCP queues ( 32 workers on 3.3.7,
at least 64 are needed on 4.2.6 to avoid "queue full" messages in logs ).
- kamailio 4.2.6 intermittent processing of requests ( Kamailio stops
accepting new tcp connections, then resume a bit later, then stops again,... ) => starts happening when Kamailio is treating more than 20k tcp connections, with between 500-1k req/s. During those periods, trying to connect to kamailio with an openssl s_client -connect will end up in a timeout or with a long delay before the certificate is received ( 03 CONNECTED ).
- kamailio 4.2.6 dropping connections, kamailio 3.3.7 keeping those
connections: after a while, the number of active connections starts to decrease on kam4, on kam3 it stays constant.
I was wondering if I was alone in this situation or if someone else is impacted, has some advices ?
It does not seem to be a config issue, as it is the same that was deployed with both versions.
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
Hi Daniel,
Same config, same server, only kamailio version is different, starting the timer in the beginning of the config, and ending it just after the save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf memory manager, which ensures predictability of duration for memory operations in all cases. f malloc can be faster for common case, but in some corner cases (e.g., lots of tls connections dropped at the same time) can be slower than tlsf.
Cheers, Daniel
Hello Daniel,
Thanks for the new pointers, we will try it !
For the record, we are now able to obtain an almost identical behavior ( still a loss of around 3% left ) by tuning: - tls parameters. - tcp parameters. - optimizations of the config file in many ways.
Those optimisations/tuning were not necessary on 3.3, but it is with 4.2+.
Anyway, we are now satisfied with the performance, but any small new gain is always good to get :)
Best,
Tristan.
On 11/03/2015 01:50 AM, Daniel-Constantin Mierla wrote:
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
Hi Daniel,
Same config, same server, only kamailio version is different, starting the timer in the beginning of the config, and ending it just after the save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf memory manager, which ensures predictability of duration for memory operations in all cases. f malloc can be faster for common case, but in some corner cases (e.g., lots of tls connections dropped at the same time) can be slower than tlsf.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
Hello,
On 03/11/15 19:21, Tristan Mahé wrote:
Hello Daniel,
Thanks for the new pointers, we will try it !
For the record, we are now able to obtain an almost identical behavior ( still a loss of around 3% left )
Does this loss refer to performance?
by tuning:
- tls parameters.
- tcp parameters.
- optimizations of the config file in many ways.
Those optimisations/tuning were not necessary on 3.3, but it is with 4.2+.
if you want to list what you had to tune, specially the tcp/tls, then we can analyze if there were changes to those features or the impact is from somewhere else.
Cheers, Daniel
Anyway, we are now satisfied with the performance, but any small new gain is always good to get :)
Best,
Tristan.
On 11/03/2015 01:50 AM, Daniel-Constantin Mierla wrote:
Hello,
On 21/10/15 00:37, Tristan Mahé wrote:
Hi Daniel,
Same config, same server, only kamailio version is different, starting the timer in the beginning of the config, and ending it just after the save("location").
looking more at it, some few hints.
Because the tls does quite a lot of memory operations inside the ssl library, try with mem_join=0.
Also, based on the work done by Camille Oudot, you can try with tlsf memory manager, which ensures predictability of duration for memory operations in all cases. f malloc can be faster for common case, but in some corner cases (e.g., lots of tls connections dropped at the same time) can be slower than tlsf.
Cheers, Daniel -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com Kamailio Advanced Training, Nov 30-Dec 2, Berlin - http://asipto.com/kat
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Tristan Mahé