Description

On systems with a high number of TCP sessions there can be a significant performance regression observed, probably related to the newly added TCP connection tracking feature.

Troubleshooting

Reproduction

No special configuration is necessary, just install the latest 5.7.x release, e.g. 5.7.3 on a production system with a lot of clients connected over TCP or TLS. You need to have a large number of clients connected to be able to observe the regression. For a high number of connections (e.g. more then 20.000 up to 30.000 connections) the Kamailio servers uses about 30% to 50% more CPU as with the old version.

Debugging Data

Two graphs were attached to this issue. The first shows the CPU load before (less load) and after the upgrade (increased load). The second is a flamegraph that shows that over 80% of the CPU time is spent in the newly added function tcp_connection_limit_srcip().

cpu-load-before-after.png (view on web) ![flamegraph](https://github.com/kamailio/kamailio/assets/6481937/5c2e3982-2d5b-4acd-a774-bad8fa64bd0a)

Most of the CPU time is spend in the TCP main process, as expected.

Log Messages

No special log messages could be observed.

SIP Traffic

Possible Solutions

The TCP limit feature should probably be optimized to not cause such a large performance regression. It should be also possible to deactivate it completly and therefore getting a comparable performance as before the feature addition.

Additional Information

Kamailio 5.7.3 and probably also git master version.

Debian 11, Debian 12


Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: <kamailio/kamailio/issues/3759@github.com>