On Wed, Apr 08, 2020 at 04:54:48PM +0100, David Villasmil wrote:
I have a question on this, since 180 arrived before the 200, shouldn’t it go out before? I.e.: FIFO? Or is this not happening because there are multiple processes and one may take the 180 and the other the 200 and they are processes independently?
I guess setting the childs to only 1 should prove this theory.
In theory, it should all be FIFO, of course. In practice, there are stochastic aspects to OS process scheduling and I/O scheduling which make the ingress and egress of messages arriving at Kamailio simultaneously or substantially simultaneously somewhat indeterminate.
And, of course, it depends on how complicated the processing of the messages respectively is, either in computational terms, or external I/O wait, or both.
To Steve Davies' point, that's just the nature of the multiprocess architecture, together with the lack of a distributor thread.
-- Alex
Alex Balashov abalashov@evaristesys.com writes:
On Wed, Apr 08, 2020 at 04:54:48PM +0100, David Villasmil wrote:
I have a question on this, since 180 arrived before the 200, shouldn’t it go out before? I.e.: FIFO? Or is this not happening because there are multiple processes and one may take the 180 and the other the 200 and they are processes independently?
I guess setting the childs to only 1 should prove this theory.
In theory, it should all be FIFO, of course. In practice, there are stochastic aspects to OS process scheduling and I/O scheduling which make the ingress and egress of messages arriving at Kamailio simultaneously or substantially simultaneously somewhat indeterminate.
I don't see this as being about the nature of parallel processing.
Either the protocol specifications have ordering rules (in which case an implementation which does not follow those is defective), or they do not have ordering rules (in which case an implementation that is troubled by lack of ordering is defective).
There is a messy middle ground of course, where significant misordering is inefficient. An example is the transport of IP packets. TCP can deal with misordered packets, but this ends up not so efficient. So there is a notion that a system that causes frequent or gross misordering is not as good as one that doesn't. Compare to ATM, where cells may not be reordered at all -- this is a major burden on implementers.
I am curious which situation we are discussing here. Given that SIP can use UDP, which has no ordering guarantees, it would be surprising if an endpoint can expect ordering of any messages.
On Wed, Apr 08, 2020 at 12:30:45PM -0400, Greg Troxel wrote:
I don't see this as being about the nature of parallel processing.
It absolutely is -- when it comes to explaining why Kamailio behaves the way it does.
It bears remark that it's not a common occurrence; if Kamailio could not be relied upon to forward messages in a FIFO order on a widespread basis, it wouldn't be very useful. This race occurs, in my experience, in highly contended environments, e.g. a VM on a busy hypervisor.
Either the protocol specifications have ordering rules (in which case an implementation which does not follow those is defective), or they do not have ordering rules (in which case an implementation that is troubled by lack of ordering is defective).
And yet, SIP-as-specified is replete with race conditions.
One way to look at this: if you want guaranteed ordering (network-introduced ordering issues not withstanding), use a single child process. If you want "enhanced performance", use multiple child processes, with the understanding that this entails a small amount of risk as a trade-off. :-)
-- Alex
On 08.04.20 18:30, Greg Troxel wrote:
Alex Balashov abalashov@evaristesys.com writes:
On Wed, Apr 08, 2020 at 04:54:48PM +0100, David Villasmil wrote:
I have a question on this, since 180 arrived before the 200, shouldn’t it go out before? I.e.: FIFO? Or is this not happening because there are multiple processes and one may take the 180 and the other the 200 and they are processes independently?
I guess setting the childs to only 1 should prove this theory.
In theory, it should all be FIFO, of course. In practice, there are stochastic aspects to OS process scheduling and I/O scheduling which make the ingress and egress of messages arriving at Kamailio simultaneously or substantially simultaneously somewhat indeterminate.
I don't see this as being about the nature of parallel processing.
Either the protocol specifications have ordering rules (in which case an implementation which does not follow those is defective), or they do not have ordering rules (in which case an implementation that is troubled by lack of ordering is defective).
There is a messy middle ground of course, where significant misordering is inefficient. An example is the transport of IP packets. TCP can deal with misordered packets, but this ends up not so efficient. So there is a notion that a system that causes frequent or gross misordering is not as good as one that doesn't. Compare to ATM, where cells may not be reordered at all -- this is a major burden on implementers.
I am curious which situation we are discussing here. Given that SIP can use UDP, which has no ordering guarantees, it would be surprising if an endpoint can expect ordering of any messages.
SIP is designed to deal (pretty well) with packets received not in the same order as they were sent.
If it is a request with a lower CSeq than a previously received one, it has to ignore it and send a 500 response.
The responses for the active transaction (matching the cseq of the request that created the transactions), the response codes have a priority "rank". 200ok is the highest priority in the way that if a 1xx or [3456]xx is received after it, just discard it.
The ACK-reINVITE seems to be the one that creates the most of the troubles, from my experience. A SIP proxy or a server cannot do much here, because even it can process sequentially, the re-ordering can still happen on the network (before or after it). The clients have the mechanisms in the SIP specs to implement it somehow nice, but I haven't seen it in reality. The receiver of the reINVITE before the ACK can reject it with a Retry-After header. But I bet that the (most of) sender will tear down the call if the reINVITE is not getting a 200ok. As I remember now, I haven't seen it written explicitly in the RFCs (at least those from the early years of SIP, may be in the new ones that update some specs), but it is legit to reject a request and ask to be resent later, without considering it a complete failure.
Cheers, Daniel
On 08.04.20 18:14, Alex Balashov wrote:
On Wed, Apr 08, 2020 at 04:54:48PM +0100, David Villasmil wrote:
I have a question on this, since 180 arrived before the 200, shouldn’t it go out before? I.e.: FIFO? Or is this not happening because there are multiple processes and one may take the 180 and the other the 200 and they are processes independently?
I guess setting the childs to only 1 should prove this theory.
In theory, it should all be FIFO, of course. In practice, there are stochastic aspects to OS process scheduling and I/O scheduling which make the ingress and egress of messages arriving at Kamailio simultaneously or substantially simultaneously somewhat indeterminate.
And, of course, it depends on how complicated the processing of the messages respectively is, either in computational terms, or external I/O wait, or both.
To Steve Davies' point, that's just the nature of the multiprocess architecture, together with the lack of a distributor thread.
A distributor thread (or process) won't help here, if it distributes traffic to other threads (processes) without waiting for them to finish, which ends up to be serial processing. The distributor role for UDP traffic is done by the kernel. For tcp/tls there is a distributor process for connections.
Cheers, Daniel
On Wed, Apr 08, 2020 at 07:22:17PM +0200, Daniel-Constantin Mierla wrote:
A distributor thread (or process) won't help here, if it distributes traffic to other threads (processes) without waiting for them to finish, which ends up to be serial processing. The distributor role for UDP traffic is done by the kernel. For tcp/tls there is a distributor process for connections.
I was imagining a distributor thread that is aware of SIP semantics, but I agree it's a terrible idea. :-)
-- Alex
-
Alex Balashov -
Daniel-Constantin Mierla -
Greg Troxel