[SR-Users] Database connection handles vs. processes

Hi, By way of illustration, I have one server with children=8, a single UDP listener, and 23 processes total: -- 21117 attendant 21118 udp receiver child=0 sock=xxx.xxx.xxx.xxx:5060 21119 udp receiver child=1 sock=xxx.xxx.xxx.xxx:5060 21120 udp receiver child=2 sock=xxx.xxx.xxx.xxx:5060 21121 udp receiver child=3 sock=xxx.xxx.xxx.xxx:5060 21122 udp receiver child=4 sock=xxx.xxx.xxx.xxx:5060 21123 udp receiver child=5 sock=xxx.xxx.xxx.xxx:5060 21124 udp receiver child=6 sock=xxx.xxx.xxx.xxx:5060 21125 udp receiver child=7 sock=xxx.xxx.xxx.xxx:5060 21126 slow timer 21127 timer 21128 MI FIFO 21129 ctl handler 21130 RTIMER USEC EXEC 21131 RTIMER USEC EXEC 21132 RTIMER USEC EXEC 21133 RTIMER USEC EXEC 21134 RTIMER USEC EXEC 21135 RTIMER USEC EXEC 21136 RTIMER USEC EXEC 21137 Dialog KA Timer 21138 Dialog Clean Timer 21139 tcp main process # kamcmd -s /tmp/kamailio_ctl ps | wc -l 23 -- This appears to result in 23 total database connections to the Postgres server: -- # lsof -n | grep -i '^kam' | grep IPv4 | grep TCP | grep -i :postgres | wc -l 23 -- But I have another Kamailio server that has three listeners, two UDP and one TCP, and children=24, i.e. -- children=24 listen=udp:xxx.xxx.xxx.xxx:5060 listen=udp:yyy.yyy.yyy.yyy:5060 listen=tcp:zzz.zzz.zzz.zzz:5060 -- I am given to understand that the total number of worker processes there will be (children_setting * listeners), i.e. 72 in this case, plus some number of rtimer and supervisory processes, as in my case. And true enough: -- # kamcmd -s /tmp/kamailio_ctl ps | wc -l 86 # kamcmd -s /tmp/kamailio_ctl ps | sed 's/65.254.44.195/yyy.yyy.yyy.yyy/g' | sed 's/65.254.44.194/zzz.zzz.zzz.zzz/g' 22065 main process - attendant 22069 udp receiver child=0 sock=zzz.zzz.zzz.zzz:5060 22070 udp receiver child=1 sock=zzz.zzz.zzz.zzz:5060 22071 udp receiver child=2 sock=zzz.zzz.zzz.zzz:5060 22073 udp receiver child=3 sock=zzz.zzz.zzz.zzz:5060 22074 udp receiver child=4 sock=zzz.zzz.zzz.zzz:5060 22076 udp receiver child=5 sock=zzz.zzz.zzz.zzz:5060 22079 udp receiver child=6 sock=zzz.zzz.zzz.zzz:5060 22080 udp receiver child=7 sock=zzz.zzz.zzz.zzz:5060 22082 udp receiver child=8 sock=zzz.zzz.zzz.zzz:5060 22085 udp receiver child=9 sock=zzz.zzz.zzz.zzz:5060 22086 udp receiver child=10 sock=zzz.zzz.zzz.zzz:5060 22089 udp receiver child=11 sock=zzz.zzz.zzz.zzz:5060 22091 udp receiver child=12 sock=zzz.zzz.zzz.zzz:5060 22093 udp receiver child=13 sock=zzz.zzz.zzz.zzz:5060 22095 udp receiver child=14 sock=zzz.zzz.zzz.zzz:5060 22097 udp receiver child=15 sock=zzz.zzz.zzz.zzz:5060 22099 udp receiver child=16 sock=zzz.zzz.zzz.zzz:5060 22100 udp receiver child=17 sock=zzz.zzz.zzz.zzz:5060 22103 udp receiver child=18 sock=zzz.zzz.zzz.zzz:5060 22105 udp receiver child=19 sock=zzz.zzz.zzz.zzz:5060 22107 udp receiver child=20 sock=zzz.zzz.zzz.zzz:5060 22109 udp receiver child=21 sock=zzz.zzz.zzz.zzz:5060 22111 udp receiver child=22 sock=zzz.zzz.zzz.zzz:5060 22113 udp receiver child=23 sock=zzz.zzz.zzz.zzz:5060 22115 udp receiver child=0 sock=yyy.yyy.yyy.yyy:5060 22117 udp receiver child=1 sock=yyy.yyy.yyy.yyy:5060 22119 udp receiver child=2 sock=yyy.yyy.yyy.yyy:5060 22121 udp receiver child=3 sock=yyy.yyy.yyy.yyy:5060 22123 udp receiver child=4 sock=yyy.yyy.yyy.yyy:5060 22125 udp receiver child=5 sock=yyy.yyy.yyy.yyy:5060 22127 udp receiver child=6 sock=yyy.yyy.yyy.yyy:5060 22129 udp receiver child=7 sock=yyy.yyy.yyy.yyy:5060 22131 udp receiver child=8 sock=yyy.yyy.yyy.yyy:5060 22132 udp receiver child=9 sock=yyy.yyy.yyy.yyy:5060 22135 udp receiver child=10 sock=yyy.yyy.yyy.yyy:5060 22137 udp receiver child=11 sock=yyy.yyy.yyy.yyy:5060 22139 udp receiver child=12 sock=yyy.yyy.yyy.yyy:5060 22141 udp receiver child=13 sock=yyy.yyy.yyy.yyy:5060 22143 udp receiver child=14 sock=yyy.yyy.yyy.yyy:5060 22145 udp receiver child=15 sock=yyy.yyy.yyy.yyy:5060 22147 udp receiver child=16 sock=yyy.yyy.yyy.yyy:5060 22149 udp receiver child=17 sock=yyy.yyy.yyy.yyy:5060 22151 udp receiver child=18 sock=yyy.yyy.yyy.yyy:5060 22153 udp receiver child=19 sock=yyy.yyy.yyy.yyy:5060 22155 udp receiver child=20 sock=yyy.yyy.yyy.yyy:5060 22156 udp receiver child=21 sock=yyy.yyy.yyy.yyy:5060 22159 udp receiver child=22 sock=yyy.yyy.yyy.yyy:5060 22161 udp receiver child=23 sock=yyy.yyy.yyy.yyy:5060 22163 slow timer 22165 timer 22167 secondary timer 22168 ctl handler 22169 RTIMER USEC EXEC 22170 RTIMER USEC EXEC 22172 RTIMER USEC EXEC 22173 RTIMER USEC EXEC 22175 Dialog Clean Timer 22176 TIMER NH 22177 TIMER NH 22178 TIMER NH 22179 tcp receiver (generic) child=0 22180 tcp receiver (generic) child=1 22182 tcp receiver (generic) child=2 22184 tcp receiver (generic) child=3 22186 tcp receiver (generic) child=4 22188 tcp receiver (generic) child=5 22190 tcp receiver (generic) child=6 22192 tcp receiver (generic) child=7 22194 tcp receiver (generic) child=8 22196 tcp receiver (generic) child=9 22198 tcp receiver (generic) child=10 22200 tcp receiver (generic) child=11 22202 tcp receiver (generic) child=12 22204 tcp receiver (generic) child=13 22206 tcp receiver (generic) child=14 22208 tcp receiver (generic) child=15 22210 tcp receiver (generic) child=16 22212 tcp receiver (generic) child=17 22214 tcp receiver (generic) child=18 22216 tcp receiver (generic) child=19 22218 tcp receiver (generic) child=20 22220 tcp receiver (generic) child=21 22223 tcp receiver (generic) child=22 22225 tcp receiver (generic) child=23 22227 tcp main process -- But, for some reason, the number of database connections to Postgres seems to be considerably higher than the number of total processes: -- # lsof -n | grep '^kam' | grep TCP | grep IPv4 | grep ':postgres' | wc -l 114 -- This is not due to orphaned/unterminated Kamailio processes. There's only one process lineage: -- |-kamailio(22065)-+-kamailio(22069) | |-kamailio(22070) | |-kamailio(22071) | |-kamailio(22073) | |-kamailio(22074) | |-kamailio(22076) | |-kamailio(22079) | |-kamailio(22080) | |-kamailio(22082) | |-kamailio(22085) | |-kamailio(22086) | |-kamailio(22089) | |-kamailio(22091) | |-kamailio(22093) | |-kamailio(22095) | |-kamailio(22097) | |-kamailio(22099) | |-kamailio(22100) | |-kamailio(22103) | |-kamailio(22105) | |-kamailio(22107) | |-kamailio(22109) | |-kamailio(22111) | |-kamailio(22113) | |-kamailio(22115) | |-kamailio(22117) | |-kamailio(22119) | |-kamailio(22121) | |-kamailio(22123) | |-kamailio(22125) | |-kamailio(22127) | |-kamailio(22129) | |-kamailio(22131) | |-kamailio(22132) | |-kamailio(22135) | |-kamailio(22137) | |-kamailio(22139) | |-kamailio(22141) | |-kamailio(22143) | |-kamailio(22145) | |-kamailio(22147) | |-kamailio(22149) | |-kamailio(22151) | |-kamailio(22153) | |-kamailio(22155) | |-kamailio(22156) | |-kamailio(22159) | |-kamailio(22161) | |-kamailio(22163) | |-kamailio(22165) | |-kamailio(22167) | |-kamailio(22168) | |-kamailio(22169) | |-kamailio(22170) | |-kamailio(22172) | |-kamailio(22173) | |-kamailio(22175) | |-kamailio(22176) | |-kamailio(22177) | |-kamailio(22178) | |-kamailio(22179) | |-kamailio(22180) | |-kamailio(22182) | |-kamailio(22184) | |-kamailio(22186) | |-kamailio(22188) | |-kamailio(22190) | |-kamailio(22192) | |-kamailio(22194) | |-kamailio(22196) | |-kamailio(22198) | |-kamailio(22200) | |-kamailio(22202) | |-kamailio(22204) | |-kamailio(22206) | |-kamailio(22208) | |-kamailio(22210) | |-kamailio(22212) | |-kamailio(22214) | |-kamailio(22216) | |-kamailio(22218) | |-kamailio(22220) | |-kamailio(22223) | |-kamailio(22225) | `-kamailio(22227) -- Yet, for some reason, some of the processes have two established DB connections from different ports: -- kamailio 22220 root 3u IPv4 189895303 0t0 TCP 127.0.0.1:46550->127.0.0.1:postgres (ESTABLISHED) kamailio 22220 root 86u IPv4 189895067 0t0 TCP 127.0.0.1:46458->127.0.0.1:postgres (ESTABLISHED) kamailio 22223 root 3u IPv4 189895315 0t0 TCP 127.0.0.1:46556->127.0.0.1:postgres (ESTABLISHED) kamailio 22223 root 86u IPv4 189895067 0t0 TCP 127.0.0.1:46458->127.0.0.1:postgres (ESTABLISHED) kamailio 22225 root 3u IPv4 189895321 0t0 TCP 127.0.0.1:46560->127.0.0.1:postgres (ESTABLISHED) kamailio 22225 root 86u IPv4 189895067 0t0 TCP 127.0.0.1:46458->127.0.0.1:postgres (ESTABLISHED) kamailio 22227 root 86u IPv4 189895067 0t0 TCP 127.0.0.1:46458->127.0.0.1:postgres (ESTABLISHED) -- ... while others don't ... -- kamailio 22111 root 3u IPv4 189894820 0t0 TCP 127.0.0.1:46326->127.0.0.1:postgres (ESTABLISHED) kamailio 22113 root 3u IPv4 189894828 0t0 TCP 127.0.0.1:46332->127.0.0.1:postgres (ESTABLISHED) kamailio 22115 root 3u IPv4 189894834 0t0 TCP 127.0.0.1:46334->127.0.0.1:postgres (ESTABLISHED) kamailio 22117 root 3u IPv4 189894842 0t0 TCP 127.0.0.1:46338->127.0.0.1:postgres (ESTABLISHED) kamailio 22119 root 3u IPv4 189894848 0t0 TCP 127.0.0.1:46340->127.0.0.1:postgres (ESTABLISHED) -- So, this gives rise to two questions in my mind: 1. I don't seem to understand the relationship between the number of database connection handles and the total number of child processes. I had always assumed that only SIP workers and other processes specialised into a role with possible "database involvement" (e.g. rtimer, async workers, etc.) would hold database connections. 2. What's going on in the second scenario such that some processes have two connections? Both servers running: -- # kamcmd -s /tmp/kamailio_ctl core.version kamailio 4.4.6 (x86_64/linux) becbde -- Insights much appreciated! -- Alex -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/