Hi,
I have a ~21 kB HTTP response that I need to handle with http_query()
from the utils module.
There are two separate obstacles to this. The first is insufficient size
of the default PV text buffer, which is solvable by increasing it using
the 'pv_buffer_size' core parameter.
The second is the CURL wrapper in modules/utils/functions.c itself. It
uses the write_function() callback to pump the data into a buffer:
char *stream;
...
stream = NULL;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_function);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &stream);
As far as I can tell from how write_function() is constructed, it
expects to be called once:
size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream)
{
/* Allocate memory and copy */
char* data;
data = (char*)pkg_malloc((size* nmemb) + 1);
if (data == NULL) {
LM_ERR("cannot allocate memory for stream\n");
return CURLE_WRITE_ERROR;
}
memcpy(data, (char*)ptr, size* nmemb);
data[nmemb] = '\0';
*((char**) stream) = data;
return size* nmemb;
}
That's not the case for a 21 kB reply:
size_t write_function( void *ptr, size_t size, size_t nmemb, void *stream)
{
/* Allocate memory and copy */
char* data;
LM_INFO("Write function called with additional buffer: %u\n",
(unsigned int) size * nmemb);
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 1193
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 1448
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 10136
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 1448
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 1448
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 4344
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 1448
0(9696) INFO: utils [functions.c:56]: write_function(): Write function
called with additional buffer: 327
So, I'm not sure what the easiest way to go here is.
1) Does this function need to be rewritten to realloc() and concatenate?
2) Is there an option in CURL that can use internal buffering and only
invoke the callback once all the response chunks are received?
3) Would increasing the curl.h constant CURL_MAX_WRITE_SIZE help?
Input is appreciated!
-- Alex
--
Alex Balashov - Principal
Evariste Systems LLC
235 E Ponce de Leon Ave
Suite 106
Decatur, GA 30030
United States
Tel: +1-678-954-0670
Web:
http://www.evaristesys.com/,
http://www.alexbalashov.com/