Hello all,
I’ve been trying to figure out how htable reloads happen. Does it replaces the existing one or creates a new one, populates it from the db and then renames it? The latter would be (I think) ideal, even if it uses more resources.
Thanks!
David
No, the reload actually creates a temporary new hash table (in heap), loads the entries from the DB into it, then iterates through the buckets of that temporary table and replaces the entries (pointers) in the existing table with the entries from the temporary table, then frees the buckets in the temporary table.
Thanks Alex, I'm wondering how this will affect performance if there are many thousands or even millions of records...
Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:03 PM Alex Balashov abalashov@evaristesys.com wrote:
I have several htables with various amounts of data in them ranging from a couple hundred entries to a couple million entries and while the reload itself does take a bit, the vast majority of the delay is the database -> Kamailio loading time. Once the data is in there, the pointer updates haven’t been anything I’ve noticed. I’m sure it takes some measurable amount of time, but in my case it’s so imperceptible that it’s irrelevant.
Brooks Bridges 1200 S. Rogers Circle, Unit 6 Boca Raton, FL 33487 Office: 916-235-2097 Main: 888-444-1111 email: bbridges@call48.commailto:bbridges@call48.com | web: www.call48.comhttp://www.call48.com/ [cid:image001.png@01D84A57.E847A810]
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of David Villasmil Sent: Thursday, April 7, 2022 08:09 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Htable reload
Thanks Alex, I'm wondering how this will affect performance if there are many thousands or even millions of records...
Regards,
David Villasmil email: david.villasmil.work@gmail.commailto:david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:03 PM Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> wrote: No, the reload actually creates a temporary new hash table (in heap), loads the entries from the DB into it, then iterates through the buckets of that temporary table and replaces the entries (pointers) in the existing table with the entries from the temporary table, then frees the buckets in the temporary table.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Confidentiality Notice: This e-mail, and any attachment to it, contains privileged and confidential information intended only for the use of the individual(s) or entity named on the e-mail. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that reading this e-mail is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system.
Thanks Brooks, that's great news!
Regards,
David Villasmil email: david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:20 PM Brooks Bridges bbridges@call48.com wrote:
Another tip for you… You’ll be tempted to crank up the fetch_rows to try to get it to load faster, but I’ve found that increasing that above 5k-ish doesn’t really seem to speed anything up and sometimes causes it to run a bit slower when importing the larger datasets. Obviously there are a lot of factors that go into that, but with a decently specced DB server and Kam server, once I started getting above 5k it did start slowing down again.
Brooks Bridges 1200 S. Rogers Circle, Unit 6 Boca Raton, FL 33487 Office: 916-235-2097 Main: 888-444-1111 email: bbridges@call48.commailto:bbridges@call48.com | web: www.call48.comhttp://www.call48.com/ [cid:image001.png@01D84A64.BBF859F0]
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of David Villasmil Sent: Thursday, April 7, 2022 09:34 To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Htable reload
Thanks Brooks, that's great news!
Regards,
David Villasmil email: david.villasmil.work@gmail.commailto:david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:20 PM Brooks Bridges <bbridges@call48.commailto:bbridges@call48.com> wrote: I have several htables with various amounts of data in them ranging from a couple hundred entries to a couple million entries and while the reload itself does take a bit, the vast majority of the delay is the database -> Kamailio loading time. Once the data is in there, the pointer updates haven’t been anything I’ve noticed. I’m sure it takes some measurable amount of time, but in my case it’s so imperceptible that it’s irrelevant.
Brooks Bridges 1200 S. Rogers Circle, Unit 6 Boca Raton, FL 33487 Office: 916-235-2097 Main: 888-444-1111 email: bbridges@call48.commailto:bbridges@call48.com | web: www.call48.comhttp://www.call48.com/ [cid:image001.png@01D84A64.BBF859F0]
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of David Villasmil Sent: Thursday, April 7, 2022 08:09 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Htable reload
Thanks Alex, I'm wondering how this will affect performance if there are many thousands or even millions of records...
Regards,
David Villasmil email: david.villasmil.work@gmail.commailto:david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:03 PM Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> wrote: No, the reload actually creates a temporary new hash table (in heap), loads the entries from the DB into it, then iterates through the buckets of that temporary table and replaces the entries (pointers) in the existing table with the entries from the temporary table, then frees the buckets in the temporary table.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Confidentiality Notice: This e-mail, and any attachment to it, contains privileged and confidential information intended only for the use of the individual(s) or entity named on the e-mail. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that reading this e-mail is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system. __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Confidentiality Notice: This e-mail, and any attachment to it, contains privileged and confidential information intended only for the use of the individual(s) or entity named on the e-mail. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that reading this e-mail is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system.
Hello,
just a note: if you use that large dataset, especially if the purpose is for routing decisions/prefix matching, you probably will get a better lookup performance with modules that uses another data structure more optimized for this case. Two examples for these modules are mtree and carrierroute. A hash table can degenerate into linear list(s), if the data is formed in a specific way.
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Brooks Bridges Sent: Thursday, April 7, 2022 5:17 PM To: Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Htable reload
I have several htables with various amounts of data in them ranging from a couple hundred entries to a couple million entries and while the reload itself does take a bit, the vast majority of the delay is the database -> Kamailio loading time. Once the data is in there, the pointer updates haven’t been anything I’ve noticed. I’m sure it takes some measurable amount of time, but in my case it’s so imperceptible that it’s irrelevant.
Brooks Bridges 1200 S. Rogers Circle, Unit 6 Boca Raton, FL 33487 Office: 916-235-2097 Main: 888-444-1111 email: bbridges@call48.commailto:bbridges@call48.com | web: www.call48.comhttp://www.call48.com/ [cid:image001.png@01D84ABD.7836EED0]
From: sr-users <sr-users-bounces@lists.kamailio.orgmailto:sr-users-bounces@lists.kamailio.org> On Behalf Of David Villasmil Sent: Thursday, April 7, 2022 08:09 To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org> Subject: Re: [SR-Users] Htable reload
Thanks Alex, I'm wondering how this will affect performance if there are many thousands or even millions of records...
Regards,
David Villasmil email: david.villasmil.work@gmail.commailto:david.villasmil.work@gmail.com phone: +34669448337
On Thu, Apr 7, 2022 at 4:03 PM Alex Balashov <abalashov@evaristesys.commailto:abalashov@evaristesys.com> wrote: No, the reload actually creates a temporary new hash table (in heap), loads the entries from the DB into it, then iterates through the buckets of that temporary table and replaces the entries (pointers) in the existing table with the entries from the temporary table, then frees the buckets in the temporary table.
-- Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
__________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions * sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: * https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users Confidentiality Notice: This e-mail, and any attachment to it, contains privileged and confidential information intended only for the use of the individual(s) or entity named on the e-mail. If the reader of this e-mail is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that reading this e-mail is strictly prohibited. If you have received this e-mail in error, please immediately return it to the sender and delete it from your system.
-
Alex Balashov -
Brooks Bridges -
David Villasmil -
Henning Westerholt