7 Mar
2006
7 Mar
'06
11:32 a.m.
Hi!
I've been running some performance tests on my OpenSER box and while
registering large number of users I faced some very strange problems. At
least they seem strange to me, but I hope someone on this list would be able
to shed some light on this.
So, I use Openser 1.0.1 and I load test it using SIPp. I've done tests
creating about 1000 registrations per second registering 300k-450k different
users. Register rate is constat and for each register different user is
used. What is strange to me that just about when 300k users have registered
the CPU of the PC running the proxy hits 100% usage and messages start
dropping. I made a nice image to illustrate my problem using Excel. I
attached it to this mail. On this test I registered 300k different users at
1000 RPS and after registering those users the registering starts again from
the beginning. As you can see from it the CPU usage rises constantly and
drops immediately after I've registered 300k users and the registering
starts again from the start. So then each user is being registered all over
again, but still the CPU load grows. Does anyone have explanation for this?
Why does the CPU usage grow based on the number of users registered? And why
does it drop for a while when registering starts all over again? I've tried
different usrloc modes, but there seems to be no difference. This one was
done with having usrloc only in the memory.
Regards,
Teemu
--
Teemu Harju
http://www.teemuharju.net
10 Mar
10 Mar
11:18 a.m.
Hi Teemu,
this is a very interesting test :).
the increased CPU load is a result on the increased number of AOR
records in the usrloc hash table - for each register, the usrloc module
will try to see first if the user (AOR) is already registered or
not....this means searching through the hash table...and as loaded the
table is as higher the CPU load will be.
you may experiment by increasing the hash table size in order to reduce
the number of collisions per branch (by default there are only 512 hash
branches, so for 450K of distinctive users, you will have ~ 900 AORs per
branch).
if you want to change it, see the modules/usrloc/dlist.c line 341 (for
unstable) :
if (new_udomain(&(ptr->name), 512, &(ptr->d)) < 0) {
maybe in the future the hash size should be controllable via module
parameter.
NOTE that the size must be a power of 2!!
now, regarding the peeks.....the decreasing load may be a result of
expiring contacts -> the load of the table decrees and so the CPU
load....what is the expire of the registrations?
regards,
bogdan
Teemu Harju wrote:
Hi!
I've been running some performance tests on my OpenSER box and while
registering large number of users I faced some very strange problems.
At least they seem strange to me, but I hope someone on this list
would be able to shed some light on this.
So, I use Openser 1.0.1 and I load test it using SIPp. I've done tests
creating about 1000 registrations per second registering 300k-450k
different users. Register rate is constat and for each register
different user is used. What is strange to me that just about when
300k users have registered the CPU of the PC running the proxy hits
100% usage and messages start dropping. I made a nice image to
illustrate my problem using Excel. I attached it to this mail. On this
test I registered 300k different users at 1000 RPS and after
registering those users the registering starts again from the
beginning. As you can see from it the CPU usage rises constantly and
drops immediately after I've registered 300k users and the registering
starts again from the start. So then each user is being registered all
over again, but still the CPU load grows. Does anyone have explanation
for this? Why does the CPU usage grow based on the number of users
registered? And why does it drop for a while when registering starts
all over again? I've tried different usrloc modes, but there seems to
be no difference. This one was done with having usrloc only in the
memory.
Regards,
Teemu
--
Teemu Harju
http://www.teemuharju.net
------------------------------------------------------------------------
11:51 a.m.
Hi,
Thanks for the reply. I was thinking about something like that since the CPU
load grows. I'll try what you mentioned and report back the result.
The strange thing still is that when the registrations start over again, the
load drops immediately. So, if I follow the number of registered users, then
exactly when the number reaches the 300k (that is with how many users I've
been testing with) the load drops. You can see it also in the graph I had in
the first email. After 5 minutes the load drops and starts growing again
(5*60*1000 = 300k). So then it is re-registering the same users. The expire
times for registrations are the default 3600s, so that should not be the
reason for the drop in CPU load. I've thought that maybe there is something
strange about the test I'm running, but those are just regular REGISTER
messages.
- Teemu
2006/3/10, Bogdan-Andrei Iancu <bogdan(a)voice-system.ro>ro>:
Hi Teemu,
this is a very interesting test :).
the increased CPU load is a result on the increased number of AOR
records in the usrloc hash table - for each register, the usrloc module
will try to see first if the user (AOR) is already registered or
not....this means searching through the hash table...and as loaded the
table is as higher the CPU load will be.
you may experiment by increasing the hash table size in order to reduce
the number of collisions per branch (by default there are only 512 hash
branches, so for 450K of distinctive users, you will have ~ 900 AORs per
branch).
if you want to change it, see the modules/usrloc/dlist.c line 341 (for
unstable) :
if (new_udomain(&(ptr->name), 512, &(ptr->d)) < 0) {
maybe in the future the hash size should be controllable via module
parameter.
NOTE that the size must be a power of 2!!
now, regarding the peeks.....the decreasing load may be a result of
expiring contacts -> the load of the table decrees and so the CPU
load....what is the expire of the registrations?
regards,
bogdan
Teemu Harju wrote:
--
Teemu Harju
http://www.teemuharju.net
Hi!
I've been running some performance tests on my OpenSER box and while
registering large number of users I faced some very strange problems.
At least they seem strange to me, but I hope someone on this list
would be able to shed some light on this.
So, I use Openser 1.0.1 and I load test it using SIPp. I've done tests
creating about 1000 registrations per second registering 300k-450k
different users. Register rate is constat and for each register
different user is used. What is strange to me that just about when
300k users have registered the CPU of the PC running the proxy hits
100% usage and messages start dropping. I made a nice image to
illustrate my problem using Excel. I attached it to this mail. On this
test I registered 300k different users at 1000 RPS and after
registering those users the registering starts again from the
beginning. As you can see from it the CPU usage rises constantly and
drops immediately after I've registered 300k users and the registering
starts again from the start. So then each user is being registered all
over again, but still the CPU load grows. Does anyone have explanation
for this? Why does the CPU usage grow based on the number of users
registered? And why does it drop for a while when registering starts
all over again? I've tried different usrloc modes, but there seems to
be no difference. This one was done with having usrloc only in the
memory.
Regards,
Teemu
--
Teemu Harju
http://www.teemuharju.net
------------------------------------------------------------------------
1:58 p.m.
Problem solved. It seemed that since I was registering the users in
sequential order, it was sort of the worst case scenario for the usrloc.
When registering users in random order the load grows still according to the
number of users, but after all users are registered the number stays
constant.
I didn't yet increase the hash table size, but that might give some more
performance so I might try it.
- Teemu
2006/3/10, Teemu Harju <teemu.harju(a)gmail.com>om>:
Hi,
Thanks for the reply. I was thinking about something like that since the
CPU load grows. I'll try what you mentioned and report back the result.
The strange thing still is that when the registrations start over again,
the load drops immediately. So, if I follow the number of registered users,
then exactly when the number reaches the 300k (that is with how many users
I've been testing with) the load drops. You can see it also in the graph I
had in the first email. After 5 minutes the load drops and starts growing
again (5*60*1000 = 300k). So then it is re-registering the same users. The
expire times for registrations are the default 3600s, so that should not be
the reason for the drop in CPU load. I've thought that maybe there is
something strange about the test I'm running, but those are just regular
REGISTER messages.
- Teemu
2006/3/10, Bogdan-Andrei Iancu <bogdan(a)voice-system.ro >:
--
Teemu Harju
http://www.teemuharju.net
Hi Teemu,
this is a very interesting test :).
the increased CPU load is a result on the increased number of AOR
records in the usrloc hash table - for each register, the usrloc module
will try to see first if the user (AOR) is already registered or
not....this means searching through the hash table...and as loaded the
table is as higher the CPU load will be.
you may experiment by increasing the hash table size in order to reduce
the number of collisions per branch (by default there are only 512 hash
branches, so for 450K of distinctive users, you will have ~ 900 AORs per
branch).
if you want to change it, see the modules/usrloc/dlist.c line 341 (for
unstable) :
if (new_udomain(&(ptr->name), 512, &(ptr->d)) < 0) {
maybe in the future the hash size should be controllable via module
parameter.
NOTE that the size must be a power of 2!!
now, regarding the peeks.....the decreasing load may be a result of
expiring contacts -> the load of the table decrees and so the CPU
load....what is the expire of the registrations?
regards,
bogdan
Teemu Harju wrote:
--
Teemu Harju
http://www.teemuharju.net
Hi!
I've been running some performance tests on my OpenSER box and while
registering large number of users I faced some very strange problems.
At least they seem strange to me, but I hope someone on this list
would be able to shed some light on this.
So, I use Openser 1.0.1 and I load test it using SIPp. I've done tests
creating about 1000 registrations per second registering 300k-450k
different users. Register rate is constat and for each register
different user is used. What is strange to me that just about when
300k users have registered the CPU of the PC running the proxy hits
100% usage and messages start dropping. I made a nice image to
illustrate my problem using Excel. I attached it to this mail. On this
test I registered 300k different users at 1000 RPS and after
registering those users the registering starts again from the
beginning. As you can see from it the CPU usage rises constantly and
drops immediately after I've registered 300k users and the registering
starts again from the start. So then each user is
being registered all
over again, but still the CPU load grows. Does anyone have explanation
for this? Why does the CPU usage grow based on the number of users
registered? And why does it drop for a while when registering starts
all over again? I've tried different usrloc modes, but there seems to
be no difference. This one was done with having usrloc only in the
memory.
Regards,
Teemu
--
Teemu Harju
http://www.teemuharju.net
------------------------------------------------------------------------
2:37 p.m.
Teemu,
Teemu Harju wrote:
Hi,
Thanks for the reply. I was thinking about something like that since
the CPU load grows. I'll try what you mentioned and report back the
result.
that will be great. btw, are you using the devel or stable version of
openser? in devel version you can get more info via the statistics
support: the amount of used memory, the number of AORs and contacts.
The strange thing still is that when the registrations start over
again, the load drops immediately. So, if I follow the number of
registered users, then exactly when the number reaches the 300k (that
is with how many users I've been testing with) the load drops. You can
see it also in the graph I had in the first email. After 5 minutes the
load drops and starts growing again (5*60*1000 = 300k). So then it is
re-registering the same users. The expire times for registrations are
the default 3600s, so that should not be the reason for the drop in
CPU load. I've thought that maybe there is something strange about the
test I'm running, but those are just regular REGISTER messages.
the explanation is that you do not have a constant traffic of REGISTER.
If you have 300K of REGISTER with 1000 REGISTER per second -> all 300K
will be registered in 5 minutes (as you also said). The usrloc uses CPU
power only when processing REGISTERs. once done, there is nothing to
process until the next burst of register comes.
Also you mentioned something about messages being discarded - this may
happen if the burst is high, the number of worker too low and the OS has
a limited queue for UDP packages... solution -> increase the number of
children processes.
regards,
bogdan
- Teemu
2006/3/10, Bogdan-Andrei Iancu <bogdan(a)voice-system.ro
<mailto:bogdan@voice-system.ro>>:
Hi Teemu,
this is a very interesting test :).
the increased CPU load is a result on the increased number of AOR
records in the usrloc hash table - for each register, the usrloc
module
will try to see first if the user (AOR) is already registered or
not....this means searching through the hash table...and as loaded the
table is as higher the CPU load will be.
you may experiment by increasing the hash table size in order to
reduce
the number of collisions per branch (by default there are only 512
hash
branches, so for 450K of distinctive users, you will have ~ 900
AORs per
branch).
if you want to change it, see the modules/usrloc/dlist.c line 341 (for
unstable) :
if (new_udomain(&(ptr->name), 512, &(ptr->d)) < 0) {
maybe in the future the hash size should be controllable via module
parameter.
NOTE that the size must be a power of 2!!
now, regarding the peeks.....the decreasing load may be a result of
expiring contacts -> the load of the table decrees and so the CPU
load....what is the expire of the registrations?
regards,
bogdan
Teemu Harju wrote:
Hi!
I've been running some performance tests on my OpenSER box and while
registering large number of users I faced some very strange
problems.
At least they seem strange to me, but I hope
someone on this list
would be able to shed some light on this.
So, I use Openser 1.0.1 and I load test it using SIPp. I've done
tests
creating about 1000 registrations per second
registering 300k-450k
different users. Register rate is constat and for each register
different user is used. What is strange to me that just about when
300k users have registered the CPU of the PC running the proxy hits
100% usage and messages start dropping. I made a nice image to
illustrate my problem using Excel. I attached it to this mail.
On this
test I registered 300k different users at 1000
RPS and after
registering those users the registering starts again from the
beginning. As you can see from it the CPU usage rises constantly and
drops immediately after I've registered 300k users and the
registering
starts again from the start. So then each user is
being
registered all
over again, but still the CPU load grows. Does
anyone have
explanation
for this? Why does the CPU usage grow based on
the number of users
registered? And why does it drop for a while when registering starts
all over again? I've tried different usrloc modes, but there
seems to
be no difference. This one was done with having
usrloc only in the
memory.
Regards,
Teemu
--
Teemu Harju
http://www.teemuharju.net
------------------------------------------------------------------------
--
Teemu Harju
http://www.teemuharju.net
6836
days inactive
6839
days old
4 comments
2 participants
participants (2)
-
Bogdan-Andrei Iancu -
Teemu Harju