Tuple slices

[George Sakkis]

Why does slicing a tuple returns a new tuple instead of a view of the existing one, given that
tuples are immutable ? I ended up writing a custom ImmutableSequen ce class that does this, but I
wonder why it is not implemented for tuples.

George

[Fredrik Lundh]

Re: Tuple slices

George Sakkis wrote:
[color=blue]
> Why does slicing a tuple returns a new tuple instead of a view of the existing one, given that
> tuples are immutable ?[/color]

really?
[color=blue][color=green][color=darkred]
>>> a = 1, 2, 3
>>> b = a[:]
>>> a is b[/color][/color][/color]
True

</F>

[Steven Bethard]

Re: Tuple slices

Fredrik Lundh wrote:[color=blue]
> George Sakkis wrote:
>[color=green]
>>Why does slicing a tuple returns a new tuple instead of a view of the existing one, given that
>>tuples are immutable ?[/color]
>
> really?
>
>[color=green][color=darkred]
>>>>a = 1, 2, 3
>>>>b = a[:]
>>>>a is b[/color][/color]
> True[/color]

My impression was that full tuple copies didn't actually copy, but that
slicing a subset of a tuple might. Not exactly sure how to test this, but:

py> a = 1, 2, 3
py> a[:2] is a[:2]
False

So _something_ at least is different between the two slices...

Steve

[Pedro Werneck]

Re: Tuple slices

On Mon, 24 Jan 2005 18:45:46 +0100
"Fredrik Lundh" <fredrik@python ware.com> wrote:
[color=blue]
> George Sakkis wrote:
>[color=green]
> > Why does slicing a tuple returns a new tuple instead of a view of
> > the existing one, given that tuples are immutable ?[/color]
>
> really?[/color]

Well... seems like this case is optimized to return the original tuple
just incrementing its reference count and returning

tupleobject.c, 330-335

if (ilow == 0 && ihigh == a->ob_size && PyTuple_CheckEx act(a)) {
Py_INCREF(a);
return (PyObject *)a;
}

[color=blue]
>[color=green][color=darkred]
> >>> a = 1, 2, 3
> >>> b = a[:]
> >>> a is b[/color][/color]
> True
>
> </F>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list[/color]

[Pedro Werneck]

Re: Tuple slices

On Mon, 24 Jan 2005 18:45:46 +0100
"Fredrik Lundh" <fredrik@python ware.com> wrote:
[color=blue]
> George Sakkis wrote:
>[color=green]
> > Why does slicing a tuple returns a new tuple instead of a view of
> > the existing one, given that tuples are immutable ?[/color]
>
> really?[/color]


Well... seems like this case (slicing the whole tuple) is optimized to
return the original tuple just incrementing its reference count and returning


tupleobject.c, 330-335

if (ilow == 0 && ihigh == a->ob_size && PyTuple_CheckEx act(a)) {
Py_INCREF(a);
return (PyObject *)a;
}

[color=blue]
>[color=green][color=darkred]
> >>> a = 1, 2, 3
> >>> b = a[:]
> >>> a is b[/color][/color]
> True
>
> </F>
>
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list[/color]

[Fredrik Lundh]

Re: Tuple slices

Steven Bethard wrote:
[color=blue][color=green][color=darkred]
>>>>>a = 1, 2, 3
>>>>>b = a[:]
>>>>>a is b[/color]
>> True[/color]
>
> My impression was that full tuple copies didn't actually copy, but that slicing a subset of a
> tuple might. Not exactly sure how to test this, but:
>
> py> a = 1, 2, 3
> py> a[:2] is a[:2]
> False[/color]

yup. and to figure out why things are done this way, consider this case:
[color=blue][color=green][color=darkred]
>>> a = give_me_a_huge_ tuple()
>>> len(a)[/color][/color][/color]
(a rather large number)[color=blue][color=green][color=darkred]
>>> b = a[:2]
>>> del a[/color][/color][/color]

(IIRC, I proposed to add "substrings " when I implemented the Unicode string
type, but that idea was rejected, for the very same "and how do you get rid of
the original object" reason)

</F>

[Steven Bethard]

Re: Tuple slices

Fredrik Lundh wrote:[color=blue]
> Steven Bethard wrote:[color=green]
>>
>>My impression was that full tuple copies didn't actually copy, but that slicing a subset of a
>>tuple might. Not exactly sure how to test this, but:
>>
>>py> a = 1, 2, 3
>>py> a[:2] is a[:2]
>>False[/color]
>
> yup. and to figure out why things are done this way, consider this case:
>[color=green][color=darkred]
> >>> a = give_me_a_huge_ tuple()
> >>> len(a)[/color][/color]
> (a rather large number)[color=green][color=darkred]
> >>> b = a[:2]
> >>> del a[/color][/color]
>
> (IIRC, I proposed to add "substrings " when I implemented the Unicode string
> type, but that idea was rejected, for the very same "and how do you get rid of
> the original object" reason)[/color]

Ahh. Yeah, that seems sensible. I don't think I've ever written code
like that, but if I did, I'd almost certainly want it to work as it does
now...

Steve

[Jeff Epler]

Re: Tuple slices

The cpython implementation stores tuples in memory like this:
[common fields for all Python objects]
[common fields for all variable-size python objects, including tuple size]
[fields specific to tuple objects, if any]
[array of PyObject*, one for each item in the tuple]
This way of storing variable-size Python objects was chosen in part
because it reuqires only one allocation for an object, not two.
However, there is no way for one tuple to point to a slice of another
tuple.

there's no reason that some other python implementation couldn't make a
different choice.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.1 (GNU/Linux)

iD8DBQFB9V1/Jd01MZaTXX0RAnD MAJ0f2v26tba9j4 6KsYV3SkylB51Kl QCfeTtK
YYuTzz1nulvDc8A d9p78AGo=
=+SO0
-----END PGP SIGNATURE-----

[George Sakkis]

Re: Tuple slices


"Fredrik Lundh" <fredrik@python ware.com> wrote in message
news:mailman.12 14.1106591959.2 2381.python-list@python.org ...[color=blue]
> Steven Bethard wrote:
>[color=green][color=darkred]
> >>>>>a = 1, 2, 3
> >>>>>b = a[:]
> >>>>>a is b
> >> True[/color]
> >
> > My impression was that full tuple copies didn't actually copy, but that slicing a subset of a
> > tuple might. Not exactly sure how to test this, but:
> >
> > py> a = 1, 2, 3
> > py> a[:2] is a[:2]
> > False[/color]
>
> yup. and to figure out why things are done this way, consider this case:
>[color=green][color=darkred]
> >>> a = give_me_a_huge_ tuple()
> >>> len(a)[/color][/color]
> (a rather large number)[color=green][color=darkred]
> >>> b = a[:2]
> >>> del a[/color][/color]
>
> (IIRC, I proposed to add "substrings " when I implemented the Unicode string
> type, but that idea was rejected, for the very same "and how do you get rid of
> the original object" reason)
>
> </F>[/color]

Fair enough. So perhaps the question is whether such cases are more regular than something like:
a = give_me_a_huge_ tuple()
slices = [a[i:j] for i in xrange(len(a)) for j in xrange(i+1, len(a)+1)]

George

[Peter Hansen]

Re: Tuple slices

George Sakkis wrote:[color=blue]
> Fair enough. So perhaps the question is whether such cases are more regular than something like:
> a = give_me_a_huge_ tuple()
> slices = [a[i:j] for i in xrange(len(a)) for j in xrange(i+1, len(a)+1)][/color]

I believe the general usage of tuples tends to mean that
"give_me_a_huge _tuple()" just doesn't happen except with
those who insist on using tuples as though they were nothing
other than read-only lists.

If you use a tuple the way it was apparently intended, you
are extraordinarily unlikely to find yourself with a
huge one requiring slicing in such a way that you care
whether it is a "view" or a new object.

-Peter

[Terry Reedy]

Re: Tuple slices


"George Sakkis" <gsakkis@rutger s.edu> wrote in message
news:35kn4mF4o4 4ufU1@individua l.net...[color=blue]
> Why does slicing a tuple returns a new tuple instead of a
> view of the existing one, given that
> tuples are immutable ? I ended up writing a custom
> ImmutableSequen ce class that does this, but I
> wonder why it is not implemented for tuples.[/color]

Numpy and Numarray both do this -- generate views into arrays -- because
they are specifically aimed at large datasets for which the memory savings
overrides the complications.

Aside from the problem of not being able to delete the underlying object,
the view object for a tuple would have to be a new type of object with a
new set of methods. So there would be one more thing to learn. And so
'type(o) is tuple' would generally have to be replaced by
'isinstance(typ e(o), (tuple,tuplevie w)).

For slices that are used within expressions and then discarded,
a= (1,2,3)
b = a(1:) + a(:1)
an internal tuple view might be an interesting optimization.

Terry J. Reedy

[George Sakkis]

Re: Tuple slices

"Peter Hansen" <peter@engcorp. com> wrote in message news:A9OdnRotCd qJ-2jcRVn-jA@powergate.ca ...[color=blue]
> George Sakkis wrote:[color=green]
> > Fair enough. So perhaps the question is whether such cases are more regular than something like:
> > a = give_me_a_huge_ tuple()
> > slices = [a[i:j] for i in xrange(len(a)) for j in xrange(i+1, len(a)+1)][/color]
>
> I believe the general usage of tuples tends to mean that
> "give_me_a_huge _tuple()" just doesn't happen except with
> those who insist on using tuples as though they were nothing
> other than read-only lists.
>
> If you use a tuple the way it was apparently intended, you
> are extraordinarily unlikely to find yourself with a
> huge one requiring slicing in such a way that you care
> whether it is a "view" or a new object.
>
> -Peter[/color]

Actually my initial motivation was not a huge tuple I had to slice many times. It was something much
less extraordinarily unlikely, a recursive function with a sequence parameter:

def foo(sequence):
# base_case
# do_stuff()
combine(foo(seq uence[:n]),
foo(sequence[n:]))

Having each slice be a view of the original sequence instead of a fresh copy would be a Good Thing
(tm).

George

[George Sakkis]

Re: Tuple slices

"Terry Reedy" <tjreedy@udel.e du> wrote in message
news:mailman.12 26.1106605134.2 2381.python-list@python.org ...[color=blue]
>
> Aside from the problem of not being able to delete the underlying object,
> the view object for a tuple would have to be a new type of object with a
> new set of methods.[/color]

It *could*, but it doesn't have to. One can represent a view as essentially an object with a pointer
to a memory buffer and a (start,stop,ste p) triple. Then a "real tuple" is just a "view" with the
triple being (0, len(sequence), 1).

George

[Jeff Shannon]

Re: Tuple slices

[My newsreader crapped out on sending this; apologies if it appears
twice.]

George Sakkis wrote:
[color=blue]
> "Terry Reedy" <tjreedy@udel.e du> wrote in message
> news:mailman.12 26.1106605134.2 2381.python-list@python.org ...
>[color=green]
>>Aside from the problem of not being able to delete the underlying object,
>>the view object for a tuple would have to be a new type of object with a
>>new set of methods.[/color]
>
> It *could*, but it doesn't have to. One can represent a view as essentially an object with a pointer
> to a memory buffer and a (start,stop,ste p) triple. Then a "real tuple" is just a "view" with the
> triple being (0, len(sequence), 1).[/color]

Except that that's not how Python tuples *are* constructed, and it'd
be a pretty big deal to redo the architecture of all tuples to support
this relatively special case.

Even if this re-architecting were done, you're still constructing a
new object -- the difference is that you're creating this
(start,stop,ste p) triple instead of duplicating a set of PyObject*
pointers, and then doing math based on those values instead of
straightforward pointer access. I'm not at all convinced that this
really saves you a significant amount for tuple slices (really, you're
still constructing a new tuple anyhow, aren't you?), and it's going to
cost a bit in both execution time and complexity in the common case
(accessing tuples without slicing). If there's a big cost in object
construction, it's probably going to be the memory allocation, and for
a reasonable tuple the size of the memory required is not going to
significantly affect the allocation time.

Jeff Shannon
Technician/Programmer
Credit International

[George Sakkis]

Re: Tuple slices


"Jeff Shannon" <jeff@ccvcorp.c om> wrote in message news:10vb3enf8q vld4@corp.super news.com...[color=blue]
> [My newsreader crapped out on sending this; apologies if it appears
> twice.]
>
> George Sakkis wrote:
>[color=green]
> > "Terry Reedy" <tjreedy@udel.e du> wrote in message
> > news:mailman.12 26.1106605134.2 2381.python-list@python.org ...
> >[color=darkred]
> >>Aside from the problem of not being able to delete the underlying object,
> >>the view object for a tuple would have to be a new type of object with a
> >>new set of methods.[/color]
> >
> > It *could*, but it doesn't have to. One can represent a view as essentially an object with a[/color][/color]
pointer[color=blue][color=green]
> > to a memory buffer and a (start,stop,ste p) triple. Then a "real tuple" is just a "view" with the
> > triple being (0, len(sequence), 1).[/color]
>
> Except that that's not how Python tuples *are* constructed, and it'd
> be a pretty big deal to redo the architecture of all tuples to support
> this relatively special case.
>
> Even if this re-architecting were done, you're still constructing a
> new object -- the difference is that you're creating this
> (start,stop,ste p) triple instead of duplicating a set of PyObject*
> pointers, and then doing math based on those values instead of
> straightforward pointer access. I'm not at all convinced that this
> really saves you a significant amount for tuple slices (really, you're
> still constructing a new tuple anyhow, aren't you?), and it's going to
> cost a bit in both execution time and complexity in the common case
> (accessing tuples without slicing). If there's a big cost in object
> construction, it's probably going to be the memory allocation, and for
> a reasonable tuple the size of the memory required is not going to
> significantly affect the allocation time.
>
> Jeff Shannon
> Technician/Programmer
> Credit International
>[/color]

You're probably right about the allocation time, but my main concern is the memory required for each
slice, which can be O(n) wrt the length of the whole tuple. I would like this to be constant (at
least if there was a way around to the problem of deleting the underlying sequence).

George