9 Jul
2013
9 Jul
'13
7:56 p.m.
Hello,
I've just committed the initial version of the rtpproxy-ng module to a
private branch (rfuchs/rtpproxy-ng) for comments and review. This module
is heavily based on the old rtpproxy module and addresses some, but not
all, of the concerns listed at [1]. Notably asynchronous discovery,
enabling and disabling of proxies is on my to-do list.
This module currently works with mediaproxy-ng [2] and supports full SDP
rewriting. It passes the complete body to the RTP proxy and receives
back a replacement body. As such, some of the features (such as ICE
handling) of the old rtpproxy module are removed from the module and
left to the RTP proxy instead. Other features not currently supported by
mediaproxy-ng, such as media playback or recording, are left
unimplemented (i.e. not rewritten and commented out) in rtpproxy-ng, but
it would be trivial to implement them.
Instead of JSON, it uses the bencode [3] format for the control
protocol, which has a very similar feature set, but is a much simpler
and more efficient encoding. The module includes a bencode "library"
(bencode.[ch]) which is deliberately written without any close ties to
the Kamailio core code, so that it can potentially be used in other
projects with little or no modification. For other languages, such as
Perl or Python, bencode modules already exist.
The control protocol in its current state is described at [4]. Most of
it is made to resemble the flags and other options and features of the
old rtpproxy module. The rtpproxy-ng module is also made to be a drop-in
replacement, all the function names and their syntax is the same. This
should make for easy transitioning. However, as more and more features
are added to the RTP proxies, I feel there's a need for a new set of
module functions with an extended syntax in the future. The control
protocol is made to be flexible and easily extendable.
Comments welcome.
cheerse
[1] http://www.kamailio.org/wiki/devel/rtpproxy-ng
[2] https://github.com/sipwise/mediaproxy-ng
[3] http://en.wikipedia.org/wiki/Bencode
[4] https://github.com/sipwise/mediaproxy-ng#the-ng-control-protocol
10 Jul
10 Jul
5:43 p.m.
Hello,
I understand that the documentation of the protocol is exemplified with
JSON, but the implementation is actually bencode? From the readme it
seems you still keep the key=value format, passing them as a dictionary
in the control protocol. That means it could result in dictionaries
included in other dictionaries and/or lists (e.g., the result of query,
like the last json in the readme from the github) - obviously no longer
that easy to troubleshoot by 'eye'.
Have you got any figures of json vs bencode performances that made you
mind to go for the later?
From my point of view, json would have had the benefits of being easy
to query/display in some languages as well as storage engines. This is
mainly for statistics, when willing to search for various situations,
without a need to pre-process the result from the rtp relay.
Another thing I haven't noticed is about brdiging between two networks
of the same type, are --ip/--ipv6 parameters taking values like rtpproxy
addr1/addr2?
Cheers,
Daniel
On 7/9/13 6:56 PM, Richard Fuchs wrote:
Hello,
I've just committed the initial version of the rtpproxy-ng module to a
private branch (rfuchs/rtpproxy-ng) for comments and review. This module
is heavily based on the old rtpproxy module and addresses some, but not
all, of the concerns listed at [1]. Notably asynchronous discovery,
enabling and disabling of proxies is on my to-do list.
This module currently works with mediaproxy-ng [2] and supports full SDP
rewriting. It passes the complete body to the RTP proxy and receives
back a replacement body. As such, some of the features (such as ICE
handling) of the old rtpproxy module are removed from the module and
left to the RTP proxy instead. Other features not currently supported by
mediaproxy-ng, such as media playback or recording, are left
unimplemented (i.e. not rewritten and commented out) in rtpproxy-ng, but
it would be trivial to implement them.
Instead of JSON, it uses the bencode [3] format for the control
protocol, which has a very similar feature set, but is a much simpler
and more efficient encoding. The module includes a bencode "library"
(bencode.[ch]) which is deliberately written without any close ties to
the Kamailio core code, so that it can potentially be used in other
projects with little or no modification. For other languages, such as
Perl or Python, bencode modules already exist.
The control protocol in its current state is described at [4]. Most of
it is made to resemble the flags and other options and features of the
old rtpproxy module. The rtpproxy-ng module is also made to be a drop-in
replacement, all the function names and their syntax is the same. This
should make for easy transitioning. However, as more and more features
are added to the RTP proxies, I feel there's a need for a new set of
module functions with an extended syntax in the future. The control
protocol is made to be flexible and easily extendable.
Comments welcome.
cheerse
[1] http://www.kamailio.org/wiki/devel/rtpproxy-ng
[2] https://github.com/sipwise/mediaproxy-ng
[3] http://en.wikipedia.org/wiki/Bencode
[4] https://github.com/sipwise/mediaproxy-ng#the-ng-control-protocol
_______________________________________________
sr-dev mailing list
sr-dev(a)lists.sip-router.org
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
6:14 p.m.
On 07/10/13 10:43, Daniel-Constantin Mierla wrote:
Hello,
I understand that the documentation of the protocol is exemplified with
JSON, but the implementation is actually bencode? From the readme it
seems you still keep the key=value format, passing them as a dictionary
in the control protocol. That means it could result in dictionaries
included in other dictionaries and/or lists (e.g., the result of query,
like the last json in the readme from the github) - obviously no longer
that easy to troubleshoot by 'eye'.
Correct, that's about the only disadvantage bencode has over JSON.
However, large JSON structures can also be quite cumbersome to read if
condensed enough, unless you run them through a pretty-printer (which,
btw, I had to do with that query example from the readme). The same
process applies to bencode structures.
Have you got any figures of json vs bencode
performances that made you
mind to go for the later?
I don't. But since the primary purpose of the control protocol is to
pass back and forth SDP bodies, the fact that those don't have to be
copied and escaped/unescaped every time a request is sent or a reply is
received should give a clear advantage to bencode.
From my point of view, json would have had the
benefits of being easy
to query/display in some languages as well as storage engines. This is
mainly for statistics, when willing to search for various situations,
without a need to pre-process the result from the rtp relay.
Assuming you're talking about searching in, or running stats over, the
raw encoded data blobs that occur on the wire, bencode actually makes
this easier, since the encoding is deterministic, i.e. there's only one
way to encode any particular structure or data. With JSON on the other
hand, the encoded version could look like anything and you'd have to get
into complicated regular expressions to get what you're looking for,
while still leaving a chance of missing it due to unexpected escaping.
Another thing I haven't noticed is about brdiging
between two networks
of the same type, are --ip/--ipv6 parameters taking values like rtpproxy
addr1/addr2?
Mediaproxy-ng currently only supports one address each for IPv4/6. But
this is not a limitation of the control protocol or the module, as it
still supports the same external/internal flags like the old one.
cheers
6:32 p.m.
On 7/10/13 5:14 PM, Richard Fuchs wrote:
That's is indeed an inconvenience, although most of sdps don't have
quotes, a walk through has to be done to be safe. The impact might not
be that significant, though.
I was referring to post-processing -- i.e., do the queries periodically
or for each call at the end, store the results in a storage system and
then generate reports. With json, in some systems you can simply refer
to it as a structure, like:
result.totals.input.rtp.bytes>1000
This kind of operations makes very easy generating reports.
The question was ofr mediaproxy-ng app, because I was referring to
--ip/--ipv6 parameters -- wanted to see if it supports bridging networks
of same address family.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
On 07/10/13 10:43, Daniel-Constantin Mierla wrote:
Are there pretty-printing tools for bencode? Or just 'gateways' to
json/xml pretty-printing.
Hello,
I understand that the documentation of the protocol is exemplified with
JSON, but the implementation is actually bencode? From the readme it
seems you still keep the key=value format, passing them as a dictionary
in the control protocol. That means it could result in dictionaries
included in other dictionaries and/or lists (e.g., the result of query,
like the last json in the readme from the github) - obviously no longer
that easy to troubleshoot by 'eye'.
Correct, that's about the only disadvantage bencode has over JSON.
However, large JSON structures can also be quite cumbersome to read if
condensed enough, unless you run them through a pretty-printer (which,
btw, I had to do with that query example from the readme). The same
process applies to bencode structures. Have you got any figures of json vs bencode
performances that made you
mind to go for the later?
I don't. But since the primary purpose of the control protocol is to
pass back and forth SDP bodies, the fact that those don't have to be
copied and escaped/unescaped every time a request is sent or a reply
is received should give a clear advantage to bencode. From my point of view, json would have had the
benefits of being easy
to query/display in some languages as well as storage engines. This is
mainly for statistics, when willing to search for various situations,
without a need to pre-process the result from the rtp relay.
Assuming you're talking about searching in, or running stats over, the
raw encoded data blobs that occur on the wire, bencode actually makes
this easier, since the encoding is deterministic, i.e. there's only
one way to encode any particular structure or data. With JSON on the
other hand, the encoded version could look like anything and you'd
have to get into complicated regular expressions to get what you're
looking for, while still leaving a chance of missing it due to
unexpected escaping. Another thing I haven't noticed is about
brdiging between two networks
of the same type, are --ip/--ipv6 parameters taking values like rtpproxy
addr1/addr2?
Mediaproxy-ng currently only supports one address each for IPv4/6. But
this is not a limitation of the control protocol or the module, as it
still supports the same external/internal flags like the old one.
7:07 p.m.
On 07/10/13 11:32, Daniel-Constantin Mierla wrote:
Are there pretty-printing tools for bencode? Or just
'gateways' to
json/xml pretty-printing.
There's an online one here: http://pauw.student.utwente.nl/bencode.html
Or you can easily make one yourself. For example in Perl, using one of
the existing bencode modules, it's about a 2-liner script. Same in PHP,
or Python, or ...
I was referring to post-processing -- i.e., do the
queries periodically
or for each call at the end, store the results in a storage system and
then generate reports. With json, in some systems you can simply refer
to it as a structure, like:
result.totals.input.rtp.bytes>1000
This kind of operations makes very easy generating reports.
I'm not sure I understand the problem. In order to access elements of a
JSON object, the whole JSON string has to be parsed and processed. The
same is true for bencode. After decoding, elements can be accessed in
whatever way the language and library used supports. There's no reason
why bencode data can't be accessed the same way JSON data can be
accessed. The structure of the data is the same.
Unless you're talking about an immutable system which only supports but
can't support JSON. In that case you can either directly translate
bencode to JSON, or teach either the RTP proxy or the module to also
produce JSON output when needed. But I'm not familiar with any such
systems, so can't really provide any useful input.
cheers
9:28 p.m.
On 7/10/13 6:07 PM, Richard Fuchs wrote:
It's again about existing tools/storage systems -- they have built-in to
access json as an object/variable, no need to parse yourself and build a
custom/own query language. Just for example, iirc, at some point I saw a
patch/presentation that MariaDB can do optimized searching withing json
object by matching with dedicated functions (as well as capability of
setting just a value inside json docs).
On 07/10/13 11:32, Daniel-Constantin Mierla wrote:
It was not about 'do it yourself', but commodity for
average users to
deal with it ...
Are there pretty-printing tools for bencode? Or
just 'gateways' to
json/xml pretty-printing.
There's an online one here: http://pauw.student.utwente.nl/bencode.html
Or you can easily make one yourself. For example in Perl, using one of
the existing bencode modules, it's about a 2-liner script. Same in
PHP, or Python, or ... I was referring to post-processing -- i.e., do
the queries periodically
or for each call at the end, store the results in a storage system and
then generate reports. With json, in some systems you can simply refer
to it as a structure, like:
result.totals.input.rtp.bytes>1000
This kind of operations makes very easy generating reports.
I'm not sure I understand the problem. In order to access elements of
a JSON object, the whole JSON string has to be parsed and processed. The same is true for bencode. After decoding, elements
can be accessed
in whatever way the language and library used supports. There's no
reason why bencode data can't be accessed the same way JSON data can
be accessed. The structure of the data is the same.
That will required development in all these systems, both can be seen as
serialization mechanisms, but their structure is different, so existing
json functions in one system cannot be reused.
Unless you're talking about an immutable system which only supports
but can't support JSON. In that case you can either directly translate
bencode to JSON, or teach either the RTP proxy or the module to also
produce JSON output when needed. But I'm not familiar with any such
systems, so can't really provide any useful input.
You implemented and it was
your decision, so I don't have anything
against using bencode in this case -- in the light of requests for
comments you asked for, I tried to figure out if parsing faster bencoded
strings pays off when balancing with the need to interact with/from
different components in a complete operational platform.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
10:50 p.m.
On 07/10/13 14:28, Daniel-Constantin Mierla wrote:
It was not about 'do it yourself', but
commodity for average users to
deal with it ...
It's again about existing tools/storage systems --
they have built-in to
access json as an object/variable, no need to parse yourself and build a
custom/own query language. Just for example, iirc, at some point I saw a
patch/presentation that MariaDB can do optimized searching withing json
object by matching with dedicated functions (as well as capability of
setting just a value inside json docs).
That will required development in all these systems,
both can be seen as
serialization mechanisms, but their structure is different, so existing
json functions in one system cannot be reused.
You implemented and it was your decision, so I
don't have anything
against using bencode in this case -- in the light of requests for
comments you asked for, I tried to figure out if parsing faster bencoded
strings pays off when balancing with the need to interact with/from
different components in a complete operational platform.
I think all those cases that you mention are outside of the scope of a
control protocol between a SIP proxy and an RTP proxy. I don't see an
average user having much interest in the details of the protocol and
they probably wouldn't find the contents any more interesting if they
were in JSON format. Apart from that, there's no reason why an RTP proxy
shouldn't be able to export stats or what not via a different protocol
and/or in another format. But I don't think this specific protocol is
the place to think about interaction with other existing systems, since
those almost certainly wouldn't be able to speak the protocol anyway,
even if it uses JSON encoding, no matter how you design it. Unless you
go above and beyond and make it HTTP-based or some such...
Since you asked for specific numbers comparing the performance, here are
some. I took the "offer" command example from the mediaproxy-ng docs,
which isn't all that complex, with an average sized SDP body, and ran it
in a tight decode-encode-free loop. I increased the number of loop
iterations until I got a measurable run time. I did this for both the
bencode and the equivalent JSON representation, where the encode step
each encoded back to the original format. Using the json-c library, I
got a runtime of 18.9 seconds, while with bencode it runs in 2.3
seconds. This is further reduced to 1.6 seconds when the "print as
string" encoding step is skipped and the bencode representation instead
is put into an iovec array, which the rtpproxy-ng module does. In the
grand scheme of things this may or may not make a difference, but why go
slower if you can go faster...
Anyway, just my 2 bits worth.
cheers
11 Jul
11 Jul
1:45 p.m.
On 7/10/13 9:50 PM, Richard Fuchs wrote:
On 07/10/13 14:28, Daniel-Constantin Mierla wrote:
My focus in the discussion was related to the query -- getting
statistics of the sessions, which can be very useful to humans. But also
I meant post-processing -- polling those statistics, save them to a
backend and then generate reports when needed.
Similar is the way we have the control protocol in kamailio, e.g.,
'kamctl stats' just displays what is sent back by kamailio.
It was not about 'do it yourself', but
commodity for average users to
deal with it ...
It's again about existing tools/storage
systems -- they have built-in to
access json as an object/variable, no need to parse yourself and build a
custom/own query language. Just for example, iirc, at some point I saw a
patch/presentation that MariaDB can do optimized searching withing json
object by matching with dedicated functions (as well as capability of
setting just a value inside json docs).
That will required development in all these
systems, both can be seen as
serialization mechanisms, but their structure is different, so existing
json functions in one system cannot be reused.
You implemented and it was your decision, so I
don't have anything
against using bencode in this case -- in the light of requests for
comments you asked for, I tried to figure out if parsing faster bencoded
strings pays off when balancing with the need to interact with/from
different components in a complete operational platform.
I think all those cases that you mention are outside of the scope of a
control protocol between a SIP proxy and an RTP proxy. I don't see an
average user having much interest in the details of the protocol and
they probably wouldn't find the contents any more interesting if they
were in JSON format. Apart from that, there's no reason why an RTP
proxy shouldn't be able
to export stats or what not via a different protocol and/or in another
format. But I don't think this specific protocol is the place to think
about interaction with other existing systems, since those almost
certainly wouldn't be able to speak the protocol anyway, even if it
uses JSON encoding, no matter how you design it. Unless you go above
and beyond and make it HTTP-based or some such...
Since you asked for specific numbers comparing the performance, here
are some. I took the "offer" command example from the mediaproxy-ng
docs, which isn't all that complex, with an average sized SDP body,
and ran it in a tight decode-encode-free loop. I increased the number
of loop iterations until I got a measurable run time. I did this for
both the bencode and the equivalent JSON representation, where the
encode step each encoded back to the original format. Using the json-c
library, I got a runtime of 18.9 seconds, while with bencode it runs
in 2.3 seconds. This is further reduced to 1.6 seconds when the "print
as string" encoding step is skipped and the bencode representation
instead is put into an iovec array, which the rtpproxy-ng module does.
In the grand scheme of things this may or may not make a difference,
but why go slower if you can go faster...
Going faster is good as long as the
overall impact is not actually
making the overall integration/system more complex.
Anyhow, I wanted to see if there were any strong technical arguments for
using bencode or just more like a preference.
Cheers,
Daniel
--
Daniel-Constantin Mierla - http://www.asipto.com
http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda
2:58 p.m.
On 07/11/13 06:45, Daniel-Constantin Mierla wrote:
My focus in the discussion was related to the query --
getting
statistics of the sessions, which can be very useful to humans. But also
I meant post-processing -- polling those statistics, save them to a
backend and then generate reports when needed.
Similar is the way we have the control protocol in kamailio, e.g.,
'kamctl stats' just displays what is sent back by kamailio.
Maybe the rtpproxy-ng module should include a similar CLI tool then. One
that talks the control protocol to the RTP proxy and outputs the
returned data in different formats, plain text, JSON, HTML tables,
whatever. I have such scripts ready and can include them in the source.
It would also make sense to include other commands in the protocol then,
e.g. one that produces a list of all running calls. The "query" command
was primarily meant to support the $rtpstat PV, which incidentally I've
just remembered that I have left unimplemented, mostly because I don't
know what people use it for.
Going faster is good as long as the overall impact is
not actually
making the overall integration/system more complex.
Anyhow, I wanted to see if there were any strong technical arguments for
using bencode or just more like a preference.
No reason other than the performance benefits and no obvious (to me)
disadvantages.
cheers
5:50 p.m.
Richard Fuchs writes:
Maybe the rtpproxy-ng module should include a similar
CLI tool then. One
that talks the control protocol to the RTP proxy and outputs the
returned data in different formats, plain text, JSON, HTML tables,
whatever. I have such scripts ready and can include them in the source.
It would also make sense to include other commands in the protocol then,
e.g. one that produces a list of all running calls.
yes, and one of the commands could be 'debug <rtpproxy_socket>
<debug_level>'
-- juha
4153
days inactive
4155
days old
9 comments
3 participants
participants (3)
-
Daniel-Constantin Mierla -
Juha Heinanen -
Richard Fuchs