Awkwardness of C API for making tuples

[Dave Opstad]

One of the functions in a C extension I'm writing needs to return a
tuple of integers, where the length of the tuple is only known at
runtime. I'm currently doing a loop calling PyInt_FromLong to make the
integers, then PyTuple_New, and finally a loop calling PyTuple_SET_ITE M
to set the tuple's items. Whew.

Does anyone know of a simpler way? I can't use Py_BuildValue because I
don't know at compile-time how many values there are going to be. And
there doesn't seem to be a PyTuple_FromArr ay() function.

If I'm overlooking something obvious, please clue me in!

Thanks,
Dave Opstad

[Diez B. Roggisch]

Re: Awkwardness of C API for making tuples

> One of the functions in a C extension I'm writing needs to return a[color=blue]
> tuple of integers, where the length of the tuple is only known at
> runtime. I'm currently doing a loop calling PyInt_FromLong to make the
> integers, then PyTuple_New, and finally a loop calling PyTuple_SET_ITE M
> to set the tuple's items. Whew.
>
> Does anyone know of a simpler way? I can't use Py_BuildValue because I
> don't know at compile-time how many values there are going to be. And
> there doesn't seem to be a PyTuple_FromArr ay() function.
>
> If I'm overlooking something obvious, please clue me in![/color]

If you already know _how_ to create a function like PyTuple_FromArr ay() -
why don't you define it yourself and use it?

--
Regards,

Diez B. Roggisch

[John Machin]

Re: Awkwardness of C API for making tuples

Dave Opstad wrote:[color=blue]
> One of the functions in a C extension I'm writing needs to return a
> tuple of integers, where the length of the tuple is only known at
> runtime. I'm currently doing a loop calling PyInt_FromLong to make[/color]
the[color=blue]
> integers,[/color]

What is the purpose of this first loop?

In what variable-length storage are you storing these (Python) integers
during this first loop? Something you created with (a) PyMem_Malloc (b)
malloc (c) alloca (d) your_own_malloc ?
[color=blue]
> then PyTuple_New, and finally a loop calling PyTuple_SET_ITE M
> to set the tuple's items. Whew.[/color]

Whew indeed.
[color=blue]
> Does anyone know of a simpler way? I can't use Py_BuildValue because[/color]
I[color=blue]
> don't know at compile-time how many values there are going to be. And[/color]
[color=blue]
> there doesn't seem to be a PyTuple_FromArr ay() function.
>
> If I'm overlooking something obvious, please clue me in![/color]

1. Determine the length of the required tuple; this may need a loop,
but only to _count_ the number of C longs that you have.
2. Use PyTuple_New.
3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem .

Much later, after thoroughly testing your code, gingerly change
PyTuple_SetItem to PyTuple_SET_ITE M. Benchmark the difference. Is it
anywhere near what you saved by cutting out the store_in_temp_a rray
thing in the first loop?

[Dave Opstad]

Re: Awkwardness of C API for making tuples

In article <1107289261.185 768.116790@f14g 2000cwb.googleg roups.com>,
"John Machin" <sjmachin@lexic on.net> wrote:
[color=blue]
> What is the purpose of this first loop?[/color]

Error handling. If I can't successfully create all the PyInts then I can
dispose the ones I've made and not bother making the tuple at all.[color=blue]
>
> In what variable-length storage are you storing these (Python) integers
> during this first loop? Something you created with (a) PyMem_Malloc (b)
> malloc (c) alloca (d) your_own_malloc ?[/color]

(b) malloc. The sequence here is: 1) malloc; 2) check for malloc
success; 3) loop to create PyInts (if failure, Py_DECREF those made so
far and free the malloc'ed buffer); 4) create new tuple (error checks
again); and 5) PyTuple_SET_ITE M (no error checks needed)
[color=blue]
> 1. Determine the length of the required tuple; this may need a loop,
> but only to _count_ the number of C longs that you have.
> 2. Use PyTuple_New.
> 3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem .[/color]

This would certainly be simpler, although I'm not sure I'm as clear as
to what happens if, say, in the middle of this loop a PyInt_FromLong
fails. I know that PyTuple_SetItem steals the reference; does that mean
I could just Py_DECREF the tuple and all the pieces will be
automagically freed? If so, I'll take your recommendation and rework the
logic this way.

Thanks!
Dave

[John Machin]

Re: Awkwardness of C API for making tuples


Dave Opstad wrote:[color=blue]
> In article <1107289261.185 768.116790@f14g 2000cwb.googleg roups.com>,
> "John Machin" <sjmachin@lexic on.net> wrote:
>[color=green]
> > What is the purpose of this first loop?[/color]
>
> Error handling. If I can't successfully create all the PyInts then I[/color]
can[color=blue]
> dispose the ones I've made and not bother making the tuple at all.[color=green]
> >
> > In what variable-length storage are you storing these (Python)[/color][/color]
integers[color=blue][color=green]
> > during this first loop? Something you created with (a) PyMem_Malloc[/color][/color]
(b)[color=blue][color=green]
> > malloc (c) alloca (d) your_own_malloc ?[/color]
>
> (b) malloc. The sequence here is: 1) malloc; 2) check for malloc
> success; 3) loop to create PyInts (if failure, Py_DECREF those made[/color]
so[color=blue]
> far and free the malloc'ed buffer); 4) create new tuple (error checks[/color]
[color=blue]
> again); and 5) PyTuple_SET_ITE M (no error checks needed)[/color]

Don't. If you _must_ allocate your own storage, use PyMem_Malloc.
[color=blue]
>[color=green]
> > 1. Determine the length of the required tuple; this may need a[/color][/color]
loop,[color=blue][color=green]
> > but only to _count_ the number of C longs that you have.
> > 2. Use PyTuple_New.
> > 3. Loop to fill the tuple, using PyInt_FromLong and[/color][/color]
PyTuple_SetItem .[color=blue]
>
> This would certainly be simpler, although I'm not sure I'm as clear[/color]
as[color=blue]
> to what happens if, say, in the middle of this loop a PyInt_FromLong
> fails. I know that PyTuple_SetItem steals the reference; does that[/color]
mean[color=blue]
> I could just Py_DECREF the tuple and all the pieces will be
> automagically freed? If so, I'll take your recommendation and rework[/color]
the[color=blue]
> logic this way.[/color]

This is what I believe happens. However even if you did need to do more
cleaning up, you shouldn't penalise the normal case i.e. when
PyInt_FromLong works. The only failure cause AFAIK is running out of
memory.
This should be rare unless it's triggered by your calling malloc :-)

[Fredrik Lundh]

Re: Awkwardness of C API for making tuples

Dave Opstad wrote:
[color=blue]
> This would certainly be simpler, although I'm not sure I'm as clear as
> to what happens if, say, in the middle of this loop a PyInt_FromLong
> fails. I know that PyTuple_SetItem steals the reference; does that mean
> I could just Py_DECREF the tuple and all the pieces will be
> automagically freed?[/color]

yes. tuples release their members when you release the tuple (this is pretty
obvious, if you think about it). they also handle uninitialized items property
(this is also pretty obvious).

here's the code, btw (from tupledealloc in Objects/tupleobject.c):

while (--i >= 0)
Py_XDECREF(op->ob_item[i]);

</F>

[Dave Cole]

Re: Awkwardness of C API for making tuples

John Machin wrote:[color=blue]
> Dave Opstad wrote:
>[color=green]
>>One of the functions in a C extension I'm writing needs to return a
>>tuple of integers, where the length of the tuple is only known at
>>runtime. I'm currently doing a loop calling PyInt_FromLong to make[/color]
>
> the
>[color=green]
>>integers,[/color]
>
>
> What is the purpose of this first loop?
>
> In what variable-length storage are you storing these (Python) integers
> during this first loop? Something you created with (a) PyMem_Malloc (b)
> malloc (c) alloca (d) your_own_malloc ?
>
>[color=green]
>>then PyTuple_New, and finally a loop calling PyTuple_SET_ITE M
>>to set the tuple's items. Whew.[/color]
>
>
> Whew indeed.
>
>[color=green]
>>Does anyone know of a simpler way? I can't use Py_BuildValue because[/color]
>
> I
>[color=green]
>>don't know at compile-time how many values there are going to be. And[/color]
>
>[color=green]
>>there doesn't seem to be a PyTuple_FromArr ay() function.
>>
>>If I'm overlooking something obvious, please clue me in![/color]
>
>
> 1. Determine the length of the required tuple; this may need a loop,
> but only to _count_ the number of C longs that you have.
> 2. Use PyTuple_New.
> 3. Loop to fill the tuple, using PyInt_FromLong and PyTuple_SetItem .
>
> Much later, after thoroughly testing your code, gingerly change
> PyTuple_SetItem to PyTuple_SET_ITE M. Benchmark the difference. Is it
> anywhere near what you saved by cutting out the store_in_temp_a rray
> thing in the first loop?[/color]

Shouldn't something like this (uncompiled) do what you want?

- Dave

PyObject *PyTuple_FromAr ray(int *values, int num_values)
{
PyObject *tuple;
int i;

tuple = PyTuple_New(num _values);
if (tuple == NULL)
return NULL;

for (i = 0; i < num_values; i++) {
PyObject *obj;

obj = PyInt_FromLong( value[i]);
if (obj == NULL
|| PyTuple_SetItem (tuple, i, obj) != 0) {
Py_DECREF(tuple );
return NULL;
}
}

return tuple;
}


--
http://www.object-craft.com.au

[Fredrik Lundh]

Re: Awkwardness of C API for making tuples

Dave Cole wrote:
[color=blue]
> for (i = 0; i < num_values; i++) {
> PyObject *obj;
>
> obj = PyInt_FromLong( value[i]);
> if (obj == NULL
> || PyTuple_SetItem (tuple, i, obj) != 0) {
> Py_DECREF(tuple );
> return NULL;
> }
> }[/color]

in theory, if PyInt_FromLong succeeds, and PyTuple_SetItem fails, you'll leak
an object.

</F>