Hi all
Some time ago I asked about a way of limiting the CAPS sent to differnet gateways and in high CPS environment.
I ended up implementing a htable based solution suggested by one member of the list. The solution was more or less like this one:
modparam("htable", "htable", "caps=>size=11;initval=0;autoexpire=5;") modparam("htable", "timer_interval", 10) modparam("htable", "timer_procs", 4)
$var(cps) = $rd + ":" + $timef(%S); $var(cps) = $shtinc(caps=>$var(cps)); if ($var(cps) > $var(capslimit)) { sl_send_reply("503", "CAPS limit reached for this destination $rd"); exit; }
The problem with this approach is that it's not fast enough detecting the limit. If I set 500 CAPS as limit it will sometimes reach 550, 600, or even 700 before it rejects the rest of the requests during that second. I have complains from the destination and they are asking me to be more precise in the ratelimit process.
Would ratelimit/pipelimit improve this? I don't even know if they would accept handling 1 second intervals at this rate.
Any other suggestion?
cheers,
Jon
I don't see how it could fail to be "fast enough", since $shtinc() is atomic. In other words if you were running 600 children (theoretically, please don't actually set children to 600), the operation $var(cps) = $shtinc(caps=>$var(cps)) will only happen one at a time, all other processes will wait until it is free before they can change it.
Regards, Kaufman
________________________________ From: Jon Bonilla (Manwe) via sr-users sr-users@lists.kamailio.org Sent: Monday, December 8, 2025 1:20 PM To: sr-users@lists.kamailio.org sr-users@lists.kamailio.org Cc: Jon Bonilla (Manwe) manwe@sipdoc.net Subject: [SR-Users] High volume ratelimit CPS
CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi all
Some time ago I asked about a way of limiting the CAPS sent to differnet gateways and in high CPS environment.
I ended up implementing a htable based solution suggested by one member of the list. The solution was more or less like this one:
modparam("htable", "htable", "caps=>size=11;initval=0;autoexpire=5;") modparam("htable", "timer_interval", 10) modparam("htable", "timer_procs", 4)
$var(cps) = $rd + ":" + $timef(%S); $var(cps) = $shtinc(caps=>$var(cps)); if ($var(cps) > $var(capslimit)) { sl_send_reply("503", "CAPS limit reached for this destination $rd"); exit; }
The problem with this approach is that it's not fast enough detecting the limit. If I set 500 CAPS as limit it will sometimes reach 550, 600, or even 700 before it rejects the rest of the requests during that second. I have complains from the destination and they are asking me to be more precise in the ratelimit process.
Would ratelimit/pipelimit improve this? I don't even know if they would accept handling 1 second intervals at this rate.
Any other suggestion?
cheers,
Jon
-- PekePBX, the multitenant PBX solution https://urldefense.com/v3/__https://pekepbx.com__;!!KWzduNI!cnP3s-DaOIIkGX7G...
El Mon, 8 Dec 2025 19:49:58 +0000 Ben Kaufman bkaufman@bcmone.com escribió:
I don't see how it could fail to be "fast enough", since $shtinc() is atomic. In other words if you were running 600 children (theoretically, please don't actually set children to 600), the operation $var(cps) = $shtinc(caps=>$var(cps)) will only happen one at a time, all other processes will wait until it is free before they can change it.
Hi Ben
I really don't know. I remember adding a log because the destination was complaining and the cps being sent was higher than the limit before the rejections hit.
I need to set a new box for another destination. I'll try to implement the same solution and reproduce it.
El Mon, 8 Dec 2025 22:38:48 +0100 "Jon Bonilla (Manwe) via sr-users" sr-users@lists.kamailio.org escribió:
El Mon, 8 Dec 2025 19:49:58 +0000 Ben Kaufman bkaufman@bcmone.com escribió:
I don't see how it could fail to be "fast enough", since $shtinc() is atomic. In other words if you were running 600 children (theoretically, please don't actually set children to 600), the operation $var(cps) = $shtinc(caps=>$var(cps)) will only happen one at a time, all other processes will wait until it is free before they can change it.
I really don't know. I remember adding a log because the destination was complaining and the cps being sent was higher than the limit before the rejections hit.
I need to set a new box for another destination. I'll try to implement the same solution and reproduce it.
Hi again
I'm testing this setup using htable and seems promising.
I'm using 60 children (not 600 xD). I have around 15 gateways with a max of 54 CAPS each. I'm setting a counter for each destination and second. If I'm above threshold for that destination I select next destination in dispatcher (limit4).
modparam("htable", "htable","caps=>size=11;initval=0;autoexpire=5;updateexpire=0;") modparam("htable","timer_interval", 5) modparam("htable", "timer_procs", 4)
route[RATELIMIT] { $var(cps_allow) = 0; while($var(cps_allow) == 0) { $var(cps) = $rd + ":" + $timef(%S); $var(cps) = $shtinc(caps=>$var(cps)); if ($var(cps) > CAPSLIMIT) { $var(cps_allow)=0; if (!ds_next_domain()) { send_reply("500","No destinations available"); exit; } } else { $var(cps_allow)=1; } }
And in failover route I also call route ratelimit:
if (t_check_status("408|500|501|502|503")) { xlog("L_NOTICE", "Failover route for status $T_reply_code\n"); if (!ds_next_domain()) { xlog("L_NOTICE", "No more destinations available\n"); send_reply("500", "No destinations available"); exit; } route(RATELIMIT); t_set_fr(180000, OUTBOUNDTIMEOUT); t_on_failure("FAILURE_OUTBOUND_MAIN"); route(RELAY);
I see in the logs everything working fin except that some counters seem to be offset when I'm dealing with failover. For example:
12:53:15.7774 New call - M=INVITE
12:53:15.7776 Destination A cps is 39 below limit 54. ACCEPT
12:53:15.8658 Failover route for status 503
12:53:15.8659 Destination B cps is 43 below limit 54. ACCEPT
12:53:16.0191 Failover route for status 503
12:53:16.0193 Destination C cps is 54 below limit 54. ACCEPT
12:53:16.1500 Failover route for status 503
12:53:16.1501 Destination D cps is 49 below limit 54. ACCEPT
12:53:16.4523 Failover route for status 503
12:53:16.4524 No more destinations available
The problem there is that when failover to destination C and D seems like it's taking the counter of the previous second but the second already changed from 15 to 16. Seems like $timef(%S) is not updating each time I execute route(RATELIMIT) With this scenario I can go above 54 cps.
Any ideas?
cheers,
Jon
Hello Jon,
you probably investigated and decided against it, but there are other, more easy ways of tracking call volume (concurrent calls) over destinations. There is the dialog module with the named profile support, and also the dlgs module offering a more lightweight approach. These two would be my approach to implement something like this.
Cheers,
Henning
-----Original Message----- From: Jon Bonilla (Manwe) via sr-users sr-users@lists.kamailio.org Sent: Montag, 15. Dezember 2025 19:27 To: sr-users@lists.kamailio.org Cc: Jon Bonilla (Manwe) manwe@sipdoc.net Subject: [SR-Users] Re: High volume ratelimit CPS
El Mon, 8 Dec 2025 22:38:48 +0100 "Jon Bonilla (Manwe) via sr-users" sr-users@lists.kamailio.org escribió:
El Mon, 8 Dec 2025 19:49:58 +0000 Ben Kaufman bkaufman@bcmone.com escribió:
I don't see how it could fail to be "fast enough", since $shtinc() is atomic. In other words if you were running 600 children (theoretically, please don't actually set children to 600), the operation $var(cps) = $shtinc(caps=>$var(cps)) will only happen one at a time, all other processes will wait until it is free before they can change it.
I really don't know. I remember adding a log because the destination was complaining and the cps being sent was higher than the limit before the rejections hit.
I need to set a new box for another destination. I'll try to implement the same solution and reproduce it.
Hi again
I'm testing this setup using htable and seems promising.
I'm using 60 children (not 600 xD). I have around 15 gateways with a max of 54 CAPS each. I'm setting a counter for each destination and second. If I'm above threshold for that destination I select next destination in dispatcher (limit4).
modparam("htable", "htable","caps=>size=11;initval=0;autoexpire=5;updateexpire=0;") modparam("htable","timer_interval", 5) modparam("htable", "timer_procs", 4)
route[RATELIMIT] { $var(cps_allow) = 0; while($var(cps_allow) == 0) { $var(cps) = $rd + ":" + $timef(%S); $var(cps) = $shtinc(caps=>$var(cps)); if ($var(cps) > CAPSLIMIT) { $var(cps_allow)=0; if (!ds_next_domain()) { send_reply("500","No destinations available"); exit; } } else { $var(cps_allow)=1; } }
And in failover route I also call route ratelimit:
if (t_check_status("408|500|501|502|503")) { xlog("L_NOTICE", "Failover route for status $T_reply_code\n"); if (!ds_next_domain()) { xlog("L_NOTICE", "No more destinations available\n"); send_reply("500", "No destinations available"); exit; } route(RATELIMIT); t_set_fr(180000, OUTBOUNDTIMEOUT); t_on_failure("FAILURE_OUTBOUND_MAIN"); route(RELAY);
I see in the logs everything working fin except that some counters seem to be offset when I'm dealing with failover. For example:
12:53:15.7774 New call - M=INVITE
12:53:15.7776 Destination A cps is 39 below limit 54. ACCEPT
12:53:15.8658 Failover route for status 503
12:53:15.8659 Destination B cps is 43 below limit 54. ACCEPT
12:53:16.0191 Failover route for status 503
12:53:16.0193 Destination C cps is 54 below limit 54. ACCEPT
12:53:16.1500 Failover route for status 503
12:53:16.1501 Destination D cps is 49 below limit 54. ACCEPT
12:53:16.4523 Failover route for status 503
12:53:16.4524 No more destinations available
The problem there is that when failover to destination C and D seems like it's taking the counter of the previous second but the second already changed from 15 to 16. Seems like $timef(%S) is not updating each time I execute route(RATELIMIT) With this scenario I can go above 54 cps.
Any ideas?
cheers,
Jon
-- PekePBX, the multitenant PBX solution https://pekepbx.com
El Mon, 15 Dec 2025 21:18:51 +0000 Henning Westerholt hw@gilawa.com escribió:
Hello Jon,
you probably investigated and decided against it, but there are other, more easy ways of tracking call volume (concurrent calls) over destinations. There is the dialog module with the named profile support, and also the dlgs module offering a more lightweight approach. These two would be my approach to implement something like this.
Hi Henning
I'm not tracking concurrent calls but call attempts per second. I don't think dlg module and named profiles match this scenario, do they?
cheers,
Jon
Hello Jon,
yes, both modules they track concurrent calls. For an approximation you could use Little's Law [1], but it will be of course not accurate.
If you need to track call attempts accurately for contractual reasons, then you probably need to use another way.
Cheers,
Henning
[1] concurrent calls = calls per second × (ASR/100) × ACD
-----Original Message----- From: Jon Bonilla (Manwe) manwe@sipdoc.net Sent: Montag, 15. Dezember 2025 22:42 To: Henning Westerholt hw@gilawa.com Cc: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Re: High volume ratelimit CPS
El Mon, 15 Dec 2025 21:18:51 +0000 Henning Westerholt hw@gilawa.com escribió:
Hello Jon,
you probably investigated and decided against it, but there are other, more easy ways of tracking call volume (concurrent calls) over destinations. There is the dialog module with the named profile support, and also the dlgs
module
offering a more lightweight approach. These two would be my approach to implement something like this.
Hi Henning
I'm not tracking concurrent calls but call attempts per second. I don't think dlg module and named profiles match this scenario, do they?
cheers,
Jon
El Tue, 16 Dec 2025 07:59:09 +0000 Henning Westerholt hw@gilawa.com escribió:
Hello Jon,
yes, both modules they track concurrent calls. For an approximation you could use Little's Law [1], but it will be of course not accurate.
If you need to track call attempts accurately for contractual reasons, then you probably need to use another way.
Hi Henning
Yes, I need to be almost 100% sure that I don't have pikes. that's why I'm counting every second.
This htable approach seems to be working fine with around 650 caps. My only issue is with failover and $timef(%S) not updating in failure_route (or seems like it doesn't)
Later this week I'll add more gws and increase the cps to 2500 approx. That will be a good performance check for this approach.
cheers,
Jon
El Mon, 15 Dec 2025 19:27:17 +0100 "Jon Bonilla (Manwe) via sr-users" sr-users@lists.kamailio.org escribió:
The problem there is that when failover to destination C and D seems like it's taking the counter of the previous second but the second already changed from 15 to 16. Seems like $timef(%S) is not updating each time I execute
Maybe the solution is to use $TS instead of $timef(%S) ? $TS should reevaluate each time I call it
On Mon, Dec 8, 2025 at 2:42 PM Jon Bonilla (Manwe) via sr-users sr-users@lists.kamailio.org wrote:
Would ratelimit/pipelimit improve this? I don't even know if they would accept handling 1 second intervals at this rate.
pipelimit/ratelimit should do the job just fine. Keep in mind that the clock is not 100% accurate. The true cps might be off limits by some percent under heavy traffic. Run some tests and tune your config accordingly.
-ovidiu
Any idea how to do this for 2 or more instances with LB over DNS? Sorry for hijack thread.
On Tue, 9 Dec 2025, 06:09 Ovidiu Sas via sr-users, < sr-users@lists.kamailio.org> wrote:
On Mon, Dec 8, 2025 at 2:42 PM Jon Bonilla (Manwe) via sr-users sr-users@lists.kamailio.org wrote:
Would ratelimit/pipelimit improve this? I don't even know if they would
accept
handling 1 second intervals at this rate.
pipelimit/ratelimit should do the job just fine. Keep in mind that the clock is not 100% accurate. The true cps might be off limits by some percent under heavy traffic. Run some tests and tune your config accordingly.
-ovidiu __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- sr-users@lists.kamailio.org To unsubscribe send an email to sr-users-leave@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender!
-
Ben Kaufman -
Fuad Trle -
Henning Westerholt -
Jon Bonilla (Manwe) -
Ovidiu Sas