Andrei Pelinescu-Onciul wrote:
On Jul 29, 2009 at 14:58, Dragos Vingarzan dragos.vingarzan@gmail.com wrote:
I see... so it seems quite complicated to add all the required locks and to redesign the process_no and my_pid() for not much of a benefit. I did not see this before.
Well, if this is final and the conclusion is that the restrictions should be in place there on dynamically forked processes, then I'll start redesigning my module. It's a not a huge deal, but now the code is much clear, easier to manage and also potentially faster if each Diameter TCP connection has it's own process.
We might be able to add some limited dynamic processes support. It depends a lot on when you fork and what do you want to be able to do from the dynamically forked processes. For example we could make drop_my_process() only mark an entry in the process table as empty and make fork_process() search the table for empty entries. This would keep process_no valid, but it won't allow tcp use for processes forked after startup (you probably already have this problem) and it might have some strange side effects with statistics (a dyn. forked process might "inherit" the stats of another terminated dyn. forked process).
I don't really want to cause some nasty hack just for this, but if it works and the performance is better then maybe it's worth.
In short the cdp module works like this: 1. Initialize on module init, but don't fork 2. On module child init, for rank==PROC_MAIN, fork_process() multiple times for the following: - 1x acceptor, for all the accepting TCP sockets - Nx workers, which will actually process the incoming Diameter messages, after they've been received. Being ser processes, they can do all that a ser process can. - 1x timer 3. Later on, when connections are established with other Diameter peers, fork_process() is called from the acceptor process on incoming connection or from the timer process on outgoing ones: - 1xreceiver per each peer, which will receive all incoming messages and pass them immediately to a task queue for the workers. Also has a named pipe, where pointers to shm alloced Diameters to be sent out are signaled. On disconnection, the process is terminated
Generally the receivers are pretty light and don't do too much: - watching the tcp socket - receiving diameter messaging, doing the quick binary decode and putting the message in the task queue for the workers - in case the message is part of the base protocol, run it through a simple state machine - watch a named pipe for signaling messages to be sent out. With this in place, any ser process can send out Diameter messages.
The trouble that we have now is in these receiver processes, which fork later from the acceptor or timer and also could terminate on disconnection, thus being now a bit dynamic during the execution time. Only marking I don't know if it's a clean and safe procedure...
But this is not a must and one universal acceptor/receiver forked at the beginning could do all the ops, much like the TCP structure from ser, right? Where there any performance issues due to some bottlenecks or something like that?
There are 2 possibilities:
- 1 process that handles all the I/O (based on epoll/kqueue/poll/sigio). This is fast but does not scale well with the number of cpus.
- 1 process that only accept()s new connections and then sends them to some workers (similar to ser tcp). This is fast and scales well and doesn't have the disadvantage of running one process or thread for each connection.
The main disadvantage is much more complex code.
The first fix solution would then be to make a single receiver process forked at startup, or a combine acceptor/receiver. My initial reasoning for forking was that each connection would've got a dedicated process and as such multiple connections won't be bottlenecked by busy interfaces.
The Diameter connections are then more stable in time than those from SIP (they are always kept alive between peers). So I don't know if I understand exactly how your 2nd suggestion would work... I have now the workers and I could make a static pool of receivers or reuse the workers. Then I should pass the descriptors from accept() between the two processes. This does not seem to me as being very standard over different kernels, so how does ser do it? (could you please just point me to the lines of code that do the descriptor exchanges?)
This won't be that bad in the end, as anyway in a well provisioned environment, each peer should have pre-configured all it's possible and trusted peers. Then I would pre-fork a receiver process on startup for each, plus one for all the other "unknown" eventual peers.
For the close_extra_socks() issue, now the acceptor/timer, before forking, opens the named pipe, which had a fd the same as another unix_sock from another process and thus got closed on fork. I could maybe open it after the fork, although this won't be just as safe and I'll need to double check it... But without the assumption that all forks are only done at start-up, this looked a bit like a bug.
Cheers, -Dragos