Hi,
we have a script running on our Kamailio servers which dumps all online users with kamctl ul show, parses it and then sends it to database. (For explanation, we started doing it this way because we noticed at least with an older (open)ser release that SER would completely stop working for a couple of seconds while writing registered users to database every 5 minutes. When dumping it, we don't see any disruption.)
With Kamailio 1.5 this works, with Kamailio 3.1 it works in the beginning, but when you pass the 2000 online user mark, it just stops working.
I've started the Kamailio completely empty, then let users register against it. I did a "kamctl ul show" every 5 seconds, this is what the results were:
Domain:: location table=512 records=82 max_slot=2 Domain:: location table=512 records=412 max_slot=4 Domain:: location table=512 records=640 max_slot=5 Domain:: location table=512 records=973 max_slot=7 Domain:: location table=512 records=1209 max_slot=8 Domain:: location table=512 records=1454 max_slot=11 Domain:: location table=512 records=1667 max_slot=12 Domain:: location table=512 records=1907 max_slot=12 500 command 'ul_dump' failed 500 command 'ul_dump' failed 500 command 'ul_dump' failed 500 command 'ul_dump' failed 500 command 'ul_dump' failed 500 command 'ul_dump' failed
Is there a hard limit for dumping users through the fifo? Why? Or should we do it differently?
Best Regards, Sebastian
On Wednesday 17 August 2011, Sebastian Damm wrote:
we have a script running on our Kamailio servers which dumps all online users with kamctl ul show, parses it and then sends it to database. (For explanation, we started doing it this way because we noticed at least with an older (open)ser release that SER would completely stop working for a couple of seconds while writing registered users to database every 5 minutes. When dumping it, we don't see any disruption.)
Hi Sebastian,
what about fixing this bug instead of using this workaround? This sounds really inefficient.
With Kamailio 1.5 this works, with Kamailio 3.1 it works in the beginning, but when you pass the 2000 online user mark, it just stops working.
I've started the Kamailio completely empty, then let users register against it. I did a "kamctl ul show" every 5 seconds, this is what the results were:
Domain:: location table=512 records=82 max_slot=2 [..] 500 command 'ul_dump' failed
Is there a hard limit for dumping users through the fifo? Why? Or should we do it differently?
I don't think there is a hard limit. Do you see any errors (e.g. memory allocation issues) in the server logs? Another idea, if you're somehow stuck to the old/ buggy version of the server, what about using DB_ONLY mode?
Best regards,
Henning
Hi Henning,
On Wed, Aug 17, 2011 at 11:33 AM, Henning Westerholt < henning.westerholt@1und1.de> wrote:
On Wednesday 17 August 2011, Sebastian Damm wrote:
we have a script running on our Kamailio servers which dumps all online users with kamctl ul show, parses it and then sends it to database. (For explanation, we started doing it this way because we noticed at least
with
an older (open)ser release that SER would completely stop working for a couple of seconds while writing registered users to database every 5 minutes. When dumping it, we don't see any disruption.)
what about fixing this bug instead of using this workaround? This sounds really inefficient.
Actually, piping the output through awk and generating a file for LOAD DATA INFILE" is not too bad. Additionally, we need to have that data in more than one DB, which prevents us from using one of the built-in db modes. We use an own DB mode which reads only on startup and then never touches the DB again.
I don't think there is a hard limit. Do you see any errors (e.g. memory allocation issues) in the server logs? Another idea, if you're somehow stuck to the old/ buggy version of the server, what about using DB_ONLY mode?
I don't see any errors in the log. And actually, that's the only bug left preventing our migration from 1.5 to 3.1. And I'd say, DB_ONLY is not an option, should be a lot less performant than if Kamailio keeps the state in memory (although I haven't benchmarked it).
Other ideas?
Best regards, Sebastian
Hello,
On 8/17/11 1:41 PM, Sebastian Damm wrote:
Hi Henning,
On Wed, Aug 17, 2011 at 11:33 AM, Henning Westerholt <henning.westerholt@1und1.de mailto:henning.westerholt@1und1.de> wrote:
On Wednesday 17 August 2011, Sebastian Damm wrote: > we have a script running on our Kamailio servers which dumps all online > users with kamctl ul show, parses it and then sends it to database. (For > explanation, we started doing it this way because we noticed at least with > an older (open)ser release that SER would completely stop working for a > couple of seconds while writing registered users to database every 5 > minutes. When dumping it, we don't see any disruption.) what about fixing this bug instead of using this workaround? This sounds really inefficient.Actually, piping the output through awk and generating a file for LOAD DATA INFILE" is not too bad. Additionally, we need to have that data in more than one DB, which prevents us from using one of the built-in db modes. We use an own DB mode which reads only on startup and then never touches the DB again.
I don't think there is a hard limit. Do you see any errors (e.g. memory allocation issues) in the server logs? Another idea, if you're somehow stuck to the old/ buggy version of the server, what about using DB_ONLY mode?I don't see any errors in the log. And actually, that's the only bug left preventing our migration from 1.5 to 3.1. And I'd say, DB_ONLY is not an option, should be a lot less performant than if Kamailio keeps the state in memory (although I haven't benchmarked it).
Other ideas?
not sure what was the default in latest 1.x, maybe MI was using the system memory manager. Since location records can need quite some memory if there are lot of entries, it may be better to use the system memory directly in MI to build the response to the commands.
In 3.1, the option is to edit lib/kmi/Makefile and turn on the define of MI_SYSTEM_MALLOC
Then recompile/reinstall and try again. Let us know the results.
As alternative, in 3.1, you can do RPC command with sercmd through ctl module in order to dump location records. There you can control the size of the buffers used to build the output:
* http://kamailio.org/docs/modules/stable/modules/ctl.html#binrpc_max_body_siz...
Cheers, Daniel
MI_SYSTEM_MALLOC did the trick, dumping >100k AOR worked that way.
Should this be mentioned somewhere (like documentation) or is a permanent fix needed?
Cheers, Marcus
On Wed, Aug 17, 2011 at 2:08 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 8/17/11 1:41 PM, Sebastian Damm wrote:
Hi Henning,
On Wed, Aug 17, 2011 at 11:33 AM, Henning Westerholt henning.westerholt@1und1.de wrote:
On Wednesday 17 August 2011, Sebastian Damm wrote:
we have a script running on our Kamailio servers which dumps all online users with kamctl ul show, parses it and then sends it to database. (For explanation, we started doing it this way because we noticed at least with an older (open)ser release that SER would completely stop working for a couple of seconds while writing registered users to database every 5 minutes. When dumping it, we don't see any disruption.)
what about fixing this bug instead of using this workaround? This sounds really inefficient.
Actually, piping the output through awk and generating a file for LOAD DATA INFILE" is not too bad. Additionally, we need to have that data in more than one DB, which prevents us from using one of the built-in db modes. We use an own DB mode which reads only on startup and then never touches the DB again.
I don't think there is a hard limit. Do you see any errors (e.g. memory allocation issues) in the server logs? Another idea, if you're somehow stuck to the old/ buggy version of the server, what about using DB_ONLY mode?
I don't see any errors in the log. And actually, that's the only bug left preventing our migration from 1.5 to 3.1. And I'd say, DB_ONLY is not an option, should be a lot less performant than if Kamailio keeps the state in memory (although I haven't benchmarked it).
Other ideas?
not sure what was the default in latest 1.x, maybe MI was using the system memory manager. Since location records can need quite some memory if there are lot of entries, it may be better to use the system memory directly in MI to build the response to the commands.
In 3.1, the option is to edit lib/kmi/Makefile and turn on the define of MI_SYSTEM_MALLOC
Then recompile/reinstall and try again. Let us know the results.
As alternative, in 3.1, you can do RPC command with sercmd through ctl module in order to dump location records. There you can control the size of the buffers used to build the output:
* http://kamailio.org/docs/modules/stable/modules/ctl.html#binrpc_max_body_siz...
Cheers, Daniel
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Oct 10-13, Berlin: http://asipto.com/u/kat http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
Hello,
On 8/25/11 2:21 PM, Marcus Hunger wrote:
MI_SYSTEM_MALLOC did the trick, dumping>100k AOR worked that way.
Should this be mentioned somewhere (like documentation) or is a permanent fix needed?
maybe we could make this default. Documentation about it should be added in several places, like the mi modules and eventually to the MI commands known to dump large about of data, like dumping usrloc, dialplan rules, lcr records ... If you like to add a bit to documentations, you are most welcomed, I can grant you git write access.
Cheers, Daniel
Cheers, Marcus
On Wed, Aug 17, 2011 at 2:08 PM, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 8/17/11 1:41 PM, Sebastian Damm wrote:
Hi Henning,
On Wed, Aug 17, 2011 at 11:33 AM, Henning Westerholt henning.westerholt@1und1.de wrote:
On Wednesday 17 August 2011, Sebastian Damm wrote:
we have a script running on our Kamailio servers which dumps all online users with kamctl ul show, parses it and then sends it to database. (For explanation, we started doing it this way because we noticed at least with an older (open)ser release that SER would completely stop working for a couple of seconds while writing registered users to database every 5 minutes. When dumping it, we don't see any disruption.)
what about fixing this bug instead of using this workaround? This sounds really inefficient.
Actually, piping the output through awk and generating a file for LOAD DATA INFILE" is not too bad. Additionally, we need to have that data in more than one DB, which prevents us from using one of the built-in db modes. We use an own DB mode which reads only on startup and then never touches the DB again.
I don't think there is a hard limit. Do you see any errors (e.g. memory allocation issues) in the server logs? Another idea, if you're somehow stuck to the old/ buggy version of the server, what about using DB_ONLY mode?
I don't see any errors in the log. And actually, that's the only bug left preventing our migration from 1.5 to 3.1. And I'd say, DB_ONLY is not an option, should be a lot less performant than if Kamailio keeps the state in memory (although I haven't benchmarked it).
Other ideas?
not sure what was the default in latest 1.x, maybe MI was using the system memory manager. Since location records can need quite some memory if there are lot of entries, it may be better to use the system memory directly in MI to build the response to the commands.
In 3.1, the option is to edit lib/kmi/Makefile and turn on the define of MI_SYSTEM_MALLOC
Then recompile/reinstall and try again. Let us know the results.
As alternative, in 3.1, you can do RPC command with sercmd through ctl module in order to dump location records. There you can control the size of the buffers used to build the output:
http://kamailio.org/docs/modules/stable/modules/ctl.html#binrpc_max_body_siz...
Cheers, Daniel
-- Daniel-Constantin Mierla -- http://www.asipto.com Kamailio Advanced Training, Oct 10-13, Berlin: http://asipto.com/u/kat http://linkedin.com/in/miconda -- http://twitter.com/miconda
SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
-
Daniel-Constantin Mierla -
Henning Westerholt -
Marcus Hunger -
Sebastian Damm