On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
No, you don't need hundreds of children, usualy 16 is maximu what you need. Newer ser versions contain connection pool, so each child will open exactly one database connection and it will be reused across modules.
What about processing that involves slow DNS queries? I thought this would eat up available workers quite quickly and further processing will be blocked? Please correct me if I'm wrong.
You can install local (running on the same host as the proxy) DNS cache that can also cache negative entries (those that do not exist or cannot be looked up), such as dnsmasq. This way the slow query would be performed once and subsequent queries will be answered from the cache. Moreover, kernel maintains a queue of incoming SIP requests and it will continue receiving SIP messages even if all processes are blocked. The kernel starts to drop incoming SIP messages once the queue is full.
Well, I guess it's not a solution. It's a workaround at best. I'd vote for something apache-like. I.e. master process monitoring the number of idle children and forking additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
Also could you (or someone) give me a rough estimation on what is the optimal number of children to serve let's say 10k sip clients as registrar and proxy with moderately complex config file, but not involving DNS queries? INVITEs may involve several database queries to a dedicated database server. I understand that the question is rather vague, I'm just interested in a rough estimation and some real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
On 25-07-2005 13:49, Michael Ulitskiy wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
Well, I guess it's not a solution. It's a workaround at best. I'd vote for something apache-like. I.e. master process monitoring the number of idle children and forking additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
Well, don't forget that SIP mostly runs on top of UDP and if the the user agent does not receive any reply in 500ms then it considers the SIP message lost and retransmits the request. So dropping the request on the server side might be better in such situations than keeping it in the input queue for a long time.
In conclusion, it would be nice to have the possibility to adjust the number of children based on the traffic volume, but we don't have it at the moment and it is not trivial to develop it. The situation is, however, not that bad because SIP runs on top of UDP and user agents will retransmit requests if necessary.
Jan.
I ran in to a similar problem with the postgres driver. I thought I checked in the 'delay open until used' feature, which should substantially get rid of all of those database connections. If I recall correctly, the mysql used to open the database before forking the children. This created two problems for me, one was that postgres needed the db connection created after the fork, and the second was that many of the modules that didn't even touch the database were also opening channels. The 'delay open until used' feature simply did the open when a query was done instead of when the open was done. I was easily running dozens of SER installations, each with 4 children, all connected to the same database.
---greg
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
> No, you don't need hundreds of children, usualy 16 is maximu what you > need. Newer ser versions contain connection pool, so each child will > open exactly one database connection and it will be reused across > modules.
What about processing that involves slow DNS queries? I thought this would eat up available workers quite quickly and further processing will be blocked? Please correct me if I'm wrong.
You can install local (running on the same host as the proxy) DNS cache that can also cache negative entries (those that do not exist or cannot be looked up), such as dnsmasq. This way the slow query would be performed once and subsequent queries will be answered from the cache. Moreover, kernel maintains a queue of incoming SIP requests and it will continue receiving SIP messages even if all processes are blocked. The kernel starts to drop incoming SIP messages once the queue is full.
Well, I guess it's not a solution. It's a workaround at best. I'd vote for something apache-like. I.e. master process monitoring the number of idle children and forking additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
Also could you (or someone) give me a rough estimation on what is the optimal number of children to serve let's say 10k sip clients as registrar and proxy with moderately complex config file, but not involving DNS queries? INVITEs may involve several database queries to a dedicated database server. I understand that the question is rather vague, I'm just interested in a rough estimation and some real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Greg,
I'm not aware of 'delay open until used' feature. Could you please elaborate or point me to the documentation? Thanks,
Michael
On Monday 25 July 2005 02:32 pm, Greg Fausak wrote:
I ran in to a similar problem with the postgres driver. I thought I checked in the 'delay open until used' feature, which should substantially get rid of all of those database connections. If I recall correctly, the mysql used to open the database before forking the children. This created two problems for me, one was that postgres needed the db connection created after the fork, and the second was that many of the modules that didn't even touch the database were also opening channels. The 'delay open until used' feature simply did the open when a query was done instead of when the open was done. I was easily running dozens of SER installations, each with 4 children, all connected to the same database.
---greg
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
> > No, you don't need hundreds of children, usualy 16 is maximu what you > > need. Newer ser versions contain connection pool, so each child will > > open exactly one database connection and it will be reused across > > modules. > > What about processing that involves slow DNS queries? I thought this would > eat up available workers quite quickly and further processing will be blocked? > Please correct me if I'm wrong.
You can install local (running on the same host as the proxy) DNS cache that can also cache negative entries (those that do not exist or cannot be looked up), such as dnsmasq. This way the slow query would be performed once and subsequent queries will be answered from the cache. Moreover, kernel maintains a queue of incoming SIP requests and it will continue receiving SIP messages even if all processes are blocked. The kernel starts to drop incoming SIP messages once the queue is full.
Well, I guess it's not a solution. It's a workaround at best. I'd vote for something apache-like. I.e. master process monitoring the number of idle children and forking additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
Also could you (or someone) give me a rough estimation on what is the optimal number of children to serve let's say 10k sip clients as registrar and proxy with moderately complex config file, but not involving DNS queries? INVITEs may involve several database queries to a dedicated database server. I understand that the question is rather vague, I'm just interested in a rough estimation and some real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
-- Greg Fausak greg@thursday.com
I haven't worked on this in a while.
In dbase.c there is:
static int submit_query(db_con_t* _h, const char* _s) { int rv;
/* ** this bit of nonsense in case our connection get screwed up */ switch(rv = PQstatus(CON_CONNECTION(_h))) { case CONNECTION_OK: break; case CONNECTION_BAD: PLOG("submit_query", "connection reset"); PQreset(CON_CONNECTION(_h)); break; }
I'm pretty sure this is where the 'delay open' happens. I think (this is way back in my memory) the ser children forks happen after the database open. It must work in mysql, but postgres didn't like it and it barfed when process tried to use a descriptor that belonged to another.
-g
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
Greg,
I'm not aware of 'delay open until used' feature. Could you please elaborate or point me to the documentation? Thanks,
Michael
On Monday 25 July 2005 02:32 pm, Greg Fausak wrote:
I ran in to a similar problem with the postgres driver. I thought I checked in the 'delay open until used' feature, which should substantially get rid of all of those database connections. If I recall correctly, the mysql used to open the database before forking the children. This created two problems for me, one was that postgres needed the db connection created after the fork, and the second was that many of the modules that didn't even touch the database were also opening channels. The 'delay open until used' feature simply did the open when a query was done instead of when the open was done. I was easily running dozens of SER installations, each with 4 children, all connected to the same database.
---greg
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
> > > No, you don't need hundreds of children, usualy 16 is maximu what you > > > need. Newer ser versions contain connection pool, so each child will > > > open exactly one database connection and it will be reused across > > > modules. > > > > What about processing that involves slow DNS queries? I thought this would > > eat up available workers quite quickly and further processing will be blocked? > > Please correct me if I'm wrong. > > You can install local (running on the same host as the proxy) DNS > cache that can also cache negative entries (those that do not exist or > cannot be looked up), such as dnsmasq. This way the slow query would > be performed once and subsequent queries will be answered from the > cache. > Moreover, kernel maintains a queue of incoming SIP requests and it > will continue receiving SIP messages even if all processes are > blocked. The kernel starts to drop incoming SIP messages once the > queue is full.
Well, I guess it's not a solution. It's a workaround at best. I'd vote for something apache-like. I.e. master process monitoring the number of idle children and forking additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
Also could you (or someone) give me a rough estimation on what is the optimal number of children to serve let's say 10k sip clients as registrar and proxy with moderately complex config file, but not involving DNS queries? INVITEs may involve several database queries to a dedicated database server. I understand that the question is rather vague, I'm just interested in a rough estimation and some real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
-- Greg Fausak greg@thursday.com
On 25-07-2005 15:34, Greg Fausak wrote:
I haven't worked on this in a while.
In dbase.c there is:
static int submit_query(db_con_t* _h, const char* _s) { int rv;
/* ** this bit of nonsense in case our connection get screwed up */ switch(rv = PQstatus(CON_CONNECTION(_h))) { case CONNECTION_OK: break; case CONNECTION_BAD: PLOG("submit_query", "connection reset"); PQreset(CON_CONNECTION(_h)); break; }I'm pretty sure this is where the 'delay open' happens. I think (this is way back in my memory) the ser children forks happen after the database open. It must work in mysql, but postgres didn't like it and it barfed when process tried to use a descriptor that belonged to another.
Even mysql did not like it. But what you describe was true for old ser versions only. Recent versions (for sure 0.8.14, 0.9.3 and unstable) do not inherit database connections between processes. Every module opens database connections in child_init function (which is called after the fork). That is, by the way, the main reason why ser opened so many database connections and why the connection pool was introduced.
Jan.
-g
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
Greg,
I'm not aware of 'delay open until used' feature. Could you please elaborate or point me to the documentation? Thanks,
Michael
On Monday 25 July 2005 02:32 pm, Greg Fausak wrote:
I ran in to a similar problem with the postgres driver. I thought I checked in the 'delay open until used' feature, which should substantially get rid of all of those database connections. If I recall correctly, the mysql used to open the database before forking the children. This created two problems for me, one was that postgres needed the db connection created after the fork, and the second was that many of the modules that didn't even touch the database were also opening channels. The 'delay open until used' feature simply did the open when a query was done instead of when the open was done. I was easily running dozens of SER installations, each with 4 children, all connected to the same database.
---greg
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
> > > > No, you don't need hundreds of children, usualy 16 is maximu what you > > > > need. Newer ser versions contain connection pool, so each child will > > > > open exactly one database connection and it will be reused across > > > > modules. > > > > > > What about processing that involves slow DNS queries? I thought this would > > > eat up available workers quite quickly and further processing will be blocked? > > > Please correct me if I'm wrong. > > > > You can install local (running on the same host as the proxy) DNS > > cache that can also cache negative entries (those that do not exist or > > cannot be looked up), such as dnsmasq. This way the slow query would > > be performed once and subsequent queries will be answered from the > > cache. > > Moreover, kernel maintains a queue of incoming SIP requests and it > > will continue receiving SIP messages even if all processes are > > blocked. The kernel starts to drop incoming SIP messages once the > > queue is full. > > Well, I guess it's not a solution. It's a workaround at best. I'd vote for something > apache-like. I.e. master process monitoring the number of idle children and forking > additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
> Also could you (or someone) give me a rough estimation on what is the optimal > number of children to serve let's say 10k sip clients as registrar and proxy with > moderately complex config file, but not involving DNS queries? INVITEs may involve > several database queries to a dedicated database server. I understand that the > question is rather vague, I'm just interested in a rough estimation and some > real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
-- Greg Fausak greg@thursday.com
-- Greg Fausak greg@thursday.com
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
Well, then it doesn't help much. I'm not a programmer, but it sounds correct that one process should not access FD that belongs to another process. Anyway only uri_db requires db connection that I'm not using. So I guess it doesn't make much sense to go for it.
Michael
On Monday 25 July 2005 04:34 pm, Greg Fausak wrote:
I haven't worked on this in a while.
In dbase.c there is:
static int submit_query(db_con_t* _h, const char* _s) { int rv;
/* ** this bit of nonsense in case our connection get screwed up */ switch(rv = PQstatus(CON_CONNECTION(_h))) { case CONNECTION_OK: break; case CONNECTION_BAD: PLOG("submit_query", "connection reset"); PQreset(CON_CONNECTION(_h)); break; }I'm pretty sure this is where the 'delay open' happens. I think (this is way back in my memory) the ser children forks happen after the database open. It must work in mysql, but postgres didn't like it and it barfed when process tried to use a descriptor that belonged to another.
-g
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
Greg,
I'm not aware of 'delay open until used' feature. Could you please elaborate or point me to the documentation? Thanks,
Michael
On Monday 25 July 2005 02:32 pm, Greg Fausak wrote:
I ran in to a similar problem with the postgres driver. I thought I checked in the 'delay open until used' feature, which should substantially get rid of all of those database connections. If I recall correctly, the mysql used to open the database before forking the children. This created two problems for me, one was that postgres needed the db connection created after the fork, and the second was that many of the modules that didn't even touch the database were also opening channels. The 'delay open until used' feature simply did the open when a query was done instead of when the open was done. I was easily running dozens of SER installations, each with 4 children, all connected to the same database.
---greg
On 7/25/05, Michael Ulitskiy mdu113@acedsl.com wrote:
On Monday 25 July 2005 01:47 pm, Michael Ulitskiy wrote:
> > > > No, you don't need hundreds of children, usualy 16 is maximu what you > > > > need. Newer ser versions contain connection pool, so each child will > > > > open exactly one database connection and it will be reused across > > > > modules. > > > > > > What about processing that involves slow DNS queries? I thought this would > > > eat up available workers quite quickly and further processing will be blocked? > > > Please correct me if I'm wrong. > > > > You can install local (running on the same host as the proxy) DNS > > cache that can also cache negative entries (those that do not exist or > > cannot be looked up), such as dnsmasq. This way the slow query would > > be performed once and subsequent queries will be answered from the > > cache. > > Moreover, kernel maintains a queue of incoming SIP requests and it > > will continue receiving SIP messages even if all processes are > > blocked. The kernel starts to drop incoming SIP messages once the > > queue is full. > > Well, I guess it's not a solution. It's a workaround at best. I'd vote for something > apache-like. I.e. master process monitoring the number of idle children and forking > additional as needed. Thanks anyway.
Why not start them all from the beginning? Is not like they will eat a lot of resources...
Just because sometimes it's hard to predict how many of them will be needed. And if you grow over time you have a good chance not to notice that the number needs to be increased until you get a whole lot of customers complaints. And with this approach it's hard to deal with temporary activity spikes. And probably more :)
> Also could you (or someone) give me a rough estimation on what is the optimal > number of children to serve let's say 10k sip clients as registrar and proxy with > moderately complex config file, but not involving DNS queries? INVITEs may involve > several database queries to a dedicated database server. I understand that the > question is rather vague, I'm just interested in a rough estimation and some > real-world numbers people use.
Try it and if you can see packets in the socket buffers (netstat) increase the number of children. I would use 20-50 children. The "lots of db connections" problem is solved when using connection pools (or by increasing the maximum connections in mysql's config :-)).
Unfortunately this is not an option for postgres at the moment or I'd definitely go for it. This is actually was the real reason I started this thread. Due to lack of connection pool for postgres I need to determine the optimal number of children. Thanks for the estimation.
You could use also tune the dns timeout (limit the maximum time a dns request can take). For this you would need either unstable (grep NEWS for dns_) or to edit your /etc/resolv.conf and set the timeout and attempts options and remove the search or domain lines.
Andrei
Michael
Serusers mailing list serusers@lists.iptel.org http://lists.iptel.org/mailman/listinfo/serusers
-- Greg Fausak greg@thursday.com
-- Greg Fausak greg@thursday.com
-
Greg Fausak -
Jan Janak -
Michael Ulitskiy