31 Jul
2009
31 Jul
'09
4:32 p.m.
Andrei Pelinescu-Onciul wrote:
The first fix solution would then be to make a single receiver process
forked at startup, or a combine acceptor/receiver. My initial reasoning
for forking was that each connection would've got a dedicated process
and as such multiple connections won't be bottlenecked by busy interfaces.
The Diameter connections are then more stable in time than those from
SIP (they are always kept alive between peers). So I don't know if I
understand exactly how your 2nd suggestion would work... I have now the
workers and I could make a static pool of receivers or reuse the
workers. Then I should pass the descriptors from accept() between the
two processes. This does not seem to me as being very standard over
different kernels, so how does ser do it? (could you please just point
me to the lines of code that do the descriptor exchanges?)
This won't be that bad in the end, as anyway in a well provisioned
environment, each peer should have pre-configured all it's possible and
trusted peers. Then I would pre-fork a receiver process on startup for
each, plus one for all the other "unknown" eventual peers.
For the close_extra_socks() issue, now the acceptor/timer, before
forking, opens the named pipe, which had a fd the same as another
unix_sock from another process and thus got closed on fork. I could
maybe open it after the fork, although this won't be just as safe and
I'll need to double check it... But without the assumption that all
forks are only done at start-up, this looked a bit like a bug.
Cheers,
-Dragos
On Jul 29, 2009 at 14:58, Dragos Vingarzan
<dragos.vingarzan(a)gmail.com> wrote:
I don't really want to cause some nasty hack just for this, but if it
works and the performance is better then maybe it's worth.
In short the cdp module works like this:
1. Initialize on module init, but don't fork
2. On module child init, for rank==PROC_MAIN, fork_process() multiple
times for the following:
- 1x acceptor, for all the accepting TCP sockets
- Nx workers, which will actually process the incoming Diameter
messages, after they've been received. Being ser processes, they can do
all that a ser process can.
- 1x timer
3. Later on, when connections are established with other Diameter peers,
fork_process() is called from the acceptor process on incoming
connection or from the timer process on outgoing ones:
- 1xreceiver per each peer, which will receive all incoming messages
and pass them immediately to a task queue for the workers. Also has a
named pipe, where pointers to shm alloced Diameters to be sent out are
signaled. On disconnection, the process is terminated
Generally the receivers are pretty light and don't do too much:
- watching the tcp socket
- receiving diameter messaging, doing the quick binary decode and
putting the message in the task queue for the workers
- in case the message is part of the base protocol, run it through a
simple state machine
- watch a named pipe for signaling messages to be sent out. With this in
place, any ser process can send out Diameter messages.
The trouble that we have now is in these receiver processes, which fork
later from the acceptor or timer and also could terminate on
disconnection, thus being now a bit dynamic during the execution time.
Only marking I don't know if it's a clean and safe procedure...
I see... so it seems quite complicated to add all
the required locks and
to redesign the process_no and my_pid() for not much of a benefit. I did
not see this before.
Well, if this is final and the conclusion is that the restrictions
should be in place there on dynamically forked processes, then I'll
start redesigning my module. It's a not a huge deal, but now the code is
much clear, easier to manage and also potentially faster if each
Diameter TCP connection has it's own process.
We might be able to add some limited dynamic processes support. It
depends a lot on when you fork and what do you want to be able to do
from the dynamically forked processes.
For example we could make drop_my_process() only mark an entry in the
process table as empty and make fork_process() search the table for
empty entries. This would keep process_no valid, but it won't allow tcp
use for processes forked after startup (you probably already have this
problem) and it might have some strange side effects with statistics (a
dyn. forked process might "inherit" the stats of another terminated dyn.
forked process).
But this is
not a must and
one universal acceptor/receiver forked at the beginning could do all the
ops, much like the TCP structure from ser, right? Where there any
performance issues due to some bottlenecks or something like that?
There are 2 possibilities:
- 1 process that handles all the I/O (based on
epoll/kqueue/poll/sigio). This is fast but does not scale well with the
number of cpus.
- 1 process that only accept()s new connections and then sends them to
some workers (similar to ser tcp). This is fast and scales well and
doesn't have the disadvantage of running one process or thread for
each connection.
The main disadvantage is much more complex code.