[Users] Question about Processes and their priorities in OpenSER

Hello, I am working on trying to stress OpenSER using a benchmarking utility. I have three questions: 1) I was hoping someone here could tell me how many 'openser' processes I should expect, and what their functions are. For example, I would expect there would be one process for each child, but I find that there are more processes than children as configured in openser.cfg. I have it set to 32 children in openser.cfg, and yet I find that there are 36 processes. The only non-standard module I have configured openSER to use is the mySQL module. I have attached the output of ps auwx | grep openser below. 2) When I observe the benchmark as it is running (see below), I notice one process that seems to have a lower priority, yet take up more CPU time than the rest. Is this process special (this relates to question 1)? 3) Why is only one processor (Cpu2 in the example below) saturated? The server is running on a machine with 2 dual-core processors, for a total of 4 CPUs. I am reasonably sure my benchmark is not network limited, as the server and the client-benchmark machine are connected via 2 Gigabit NICs. I have also discovered that if I renice the process mentioned in question 2 to -20, all 4 processors become saturated, and the throughput of the server more than doubles. The output from top is below. Any insight is greatly appreciated. Thanks, --Ian Fedeli #ps auxw | grep openser root 27038 0.0 0.0 2150456 4536 ? S 10:20 0:00 /usr/local/sbin/openser -E -m 2048 root 27039 1.3 7.4 2150456 609804 ? S 10:20 0:03 /usr/local/sbin/openser -E -m 2048 root 27040 3.7 7.6 2150456 624188 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27041 3.6 7.6 2150456 625852 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27042 3.6 7.5 2150456 614116 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27043 3.6 7.5 2150456 619564 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27044 0.0 0.0 2150460 1020 ? S 10:20 0:00 /usr/local/sbin/openser -E -m 2048 root 27045 3.6 7.5 2150456 615800 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27046 3.8 7.5 2150456 621144 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27047 3.8 7.6 2150456 623408 ? S 10:20 0:10 /usr/local/sbin/openser -E -m 2048 root 27048 3.7 7.6 2150456 627172 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27050 3.8 7.6 2150456 627780 ? R 10:20 0:10 /usr/local/sbin/openser -E -m 2048 root 27051 3.6 7.6 2150456 624980 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27054 3.7 7.6 2150456 622324 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27056 3.8 7.6 2150456 623992 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27058 3.8 7.5 2150456 616816 ? S 10:20 0:10 /usr/local/sbin/openser -E -m 2048 root 27060 3.5 7.5 2150456 620464 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27062 3.6 7.5 2150456 614324 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27064 3.5 7.4 2150456 611200 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27065 3.7 7.5 2150456 616132 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27067 3.5 7.5 2150456 617480 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27070 3.6 7.5 2150456 618024 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27071 3.6 7.5 2150456 619008 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27073 3.7 7.5 2150456 619724 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27076 3.6 7.5 2150456 617448 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27077 3.7 7.6 2150456 624320 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27080 3.6 7.5 2150456 619576 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27081 3.6 7.5 2150456 615472 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27082 3.8 7.5 2150456 615396 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27086 3.7 7.6 2150456 621484 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27088 3.7 7.5 2150456 617260 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27090 3.6 7.5 2150456 616068 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27092 3.6 7.4 2150456 609980 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27093 3.6 7.5 2150456 616160 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27094 3.6 7.4 2150456 610928 ? S 10:20 0:09 /usr/local/sbin/openser -E -m 2048 root 27098 21.8 1.9 2150456 159300 ? R 10:20 0:56 /usr/local/sbin/openser -E -m 2048 #top (before renicing) Tasks: 109 total, 15 running, 94 sleeping, 0 stopped, 0 zombie Cpu0 : 23.0%us, 16.0%sy, 0.0%ni, 34.0%id, 0.0%wa, 3.0%hi, 24.0%si, 0.0%st Cpu1 : 26.7%us, 20.8%sy, 0.0%ni, 50.5%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st Cpu2 : 44.6%us, 53.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st Cpu3 : 25.7%us, 21.8%sy, 0.0%ni, 52.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8175948k total, 1108488k used, 7067460k free, 74800k buffers Swap: 3301316k total, 0k used, 3301316k free, 888692k cached PID USER PPID PR VIRT RES SHR S %CPU P %MEM TIME+ COMMAND 10065 root 10006 25 2100m 12m 11m R 28 2 0.2 0:10.65 openser 10013 root 10006 15 2100m 522m 522m R 13 2 6.5 0:03.03 openser 10010 root 10006 15 2100m 525m 524m R 11 2 6.6 0:03.21 openser 10057 root 10006 15 2100m 432m 431m S 9 0 5.4 0:02.65 openser 10063 root 10006 15 2100m 487m 486m S 9 0 6.1 0:02.83 openser 10011 root 10006 15 2100m 510m 509m R 8 1 6.4 0:03.23 openser 10012 root 10006 15 2100m 511m 510m S 8 3 6.4 0:02.97 openser 10017 root 10006 15 2100m 481m 480m R 8 2 6.0 0:02.80 openser 10021 root 10006 15 2100m 487m 486m S 8 2 6.1 0:03.01 openser 10024 root 10006 15 2100m 484m 484m S 8 0 6.1 0:02.92 openser 10029 root 10006 15 2100m 492m 491m S 8 3 6.2 0:02.65 openser 10033 root 10006 15 2100m 501m 501m S 8 3 6.3 0:03.00 openser 10036 root 10006 15 2100m 476m 475m S 8 3 6.0 0:02.81 openser 10038 root 10006 15 2100m 488m 488m S 8 3 6.1 0:02.88 openser 10048 root 10006 15 2100m 463m 462m R 8 1 5.8 0:02.66 openser #top (after renicing) Tasks: 109 total, 32 running, 77 sleeping, 0 stopped, 0 zombie Cpu0 : 42.9%us, 24.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 6.1%hi, 26.5%si, 0.0%st Cpu1 : 52.0%us, 44.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.0%si, 0.0%st Cpu2 : 51.0%us, 47.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 2.0%si, 0.0%st Cpu3 : 46.0%us, 50.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 4.0%si, 0.0%st Mem: 8175948k total, 1262084k used, 6913864k free, 74808k buffers Swap: 3301316k total, 0k used, 3301316k free, 1032116k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ PPID COMMAND 10317 root 0 -20 2100m 12m 12m S 54 0.2 0:10.84 10258 openser 10279 root 16 0 2100m 490m 489m R 18 6.1 0:02.88 10258 openser 10264 root 16 0 2100m 473m 472m R 14 5.9 0:02.54 10258 openser 10265 root 16 0 2100m 464m 463m R 14 5.8 0:02.73 10258 openser 10266 root 16 0 2100m 481m 480m R 14 6.0 0:02.55 10258 openser 10271 root 16 0 2100m 463m 462m R 14 5.8 0:02.57 10258 openser 10286 root 16 0 2100m 470m 470m R 14 5.9 0:02.41 10258 openser 10304 root 15 0 2100m 434m 433m S 14 5.4 0:02.53 10258 openser 10305 root 16 0 2100m 456m 455m R 14 5.7 0:02.37 10258 openser 10307 root 16 0 2100m 387m 386m R 14 4.8 0:02.22 10258 openser 10262 root 15 0 2100m 476m 475m S 12 6.0 0:02.88 10258 openser 10267 root 15 0 2100m 438m 437m R 12 5.5 0:02.50 10258 openser 10288 root 16 0 2100m 443m 442m R 12 5.6 0:02.33 10258 openser 10289 root 16 0 2100m 458m 457m R 12 5.7 0:02.52 10258 openser