Question about references/copies

[Henning Kage]

I'm using Python only for some months now and I'm wondering, whether such
assignments as above are creating bitwise copies of an object or just
recieve a reference. That means I wanted to know, wheter Python in general
differs between references and copies:

class someclass:
def __init__( self, otherobject):
self.someattrib ute = otherobject

And my second question is, whether I should use a cast in such cases or
not (I know, a cast isn't mandatory here...)

Thanks a lot!

[Anthony Baxter]

Re: Question about references/copies

On Sat, 28 Aug 2004 09:52:36 +0200, Henning Kage <c0dec@gmx.de > wrote:[color=blue]
> I'm using Python only for some months now and I'm wondering, whether such
> assignments as above are creating bitwise copies of an object or just
> recieve a reference. That means I wanted to know, wheter Python in general
> differs between references and copies:[/color]

It's all references. If you want a copy, use the copy module.
[color=blue]
> class someclass:
> def __init__( self, otherobject):
> self.someattrib ute = otherobject
>
> And my second question is, whether I should use a cast in such cases or
> not (I know, a cast isn't mandatory here...)[/color]

A cast? Python has no static typing, so I'm unsure how or why you'd do a cast.

[Henning Kage]

Re: Question about references/copies

Am Sat, 28 Aug 2004 18:11:46 +1000 schrieb Anthony Baxter:
[color=blue]
> It's all references. If you want a copy, use the copy module.[/color]

Ok, thanks.
[color=blue]
> A cast? Python has no static typing, so I'm unsure how or why you'd do a cast.[/color]

As I said, I'm new to Python and as other languages do support such casts
(as I know, in C# e.g. you have to do such casts if you are iterating
over array lists and have to access the stored objects), I thought of it
in Python too.

[Alex Martelli]

Re: Question about references/copies

Henning Kage <c0dec@gmx.de > wrote:
[color=blue]
> I'm using Python only for some months now and I'm wondering, whether such
> assignments as above[/color]

"As above" _where_?
[color=blue]
> are creating bitwise copies of an object or just
> recieve a reference. That means I wanted to know, wheter Python in general
> differs between references and copies:
>
> class someclass:
> def __init__( self, otherobject):
> self.someattrib ute = otherobject[/color]

Python makes no copy unless you explicitly ask for a copy -- what you
get is always a reference. There are several ways to ask for copies --
the most general and powerful ones you'll find in standard library
modules copy (both copy.copy and copy.deepcopy -- all other ways of
asking for a copy, except copy.deepcopy, get shallow copies).

The only area where one might have some doubt is _slicing_. When you
code x=y[z:t] you're getting a (shallow) copy, if y belongs to any of
the builtin types that implement slicing, BUT there are important
third-party extensions whose types behave differently -- specifically,
Numeric; if the type of y is Numeric.array, then x shares (some of) the
data of y.

So we can say that Python _allows_ third-party and user-coded types with
slicing that works by data sharing, even though the preferred semantics
of slicing is by (shallow) copy.

[color=blue]
> And my second question is, whether I should use a cast in such cases or
> not (I know, a cast isn't mandatory here...)[/color]

Python does not really have the concept of a cast. Something that looks
like a cast to you is probably one of the ways to ask for a copy: for
example, if x is a list, list(x) is a (shallow) copy of x -- if y is a
dict, dict(y) is a (shallow) copy of y, and so on.

If you take an argument X that is of any iterable type, and you need to
perform local operations on the items of X (which will not risk altering
the original value of X), it is quite common to start with something
like:
X = list(X)
if X was already a list this makes a copy (so the original does not get
altered); if X was, say, a tuple, an open file, a dict, a string, ...,
in other words any bound iterable, this in any case ensures X is now a
list with that iterable's items (be careful: some iterables change state
when iterated upon -- if X was a file, that file object is now at
end-of-file, and the caller may need, if feasible, to call its method
seek to get that file back to the previous state, for example -- so the
"will not risk altering" tidbit above does need qualification).

So anyway you might now call such mutator methods on X as sort, reverse,
extend, pop -- all the nice methods that list offers and other iterable
types don't -- and in the end presumably return or store some result
based on the suitably-mutated X.

Another similar operation that you may do reasonably frequently as your
familiarity with Python grows is to ensure you have an _iterator_ for a
generically _iterable_ argument, and sometimes the best way is something
like:
X = iter(X)
you don't get many methods in X this way (basically, only X.next...) but
you do ensure that X "keeps state" that tracks its current iteration.
Note a big difference here: if X is already an iterator, iter(X) will
NOT make a copy -- it wll return a reference to X unchanged. An example
generator using this technique:

def interleave(X, Y):
X = iter(X)
Y = iter(Y)
while True:
yield X.next()
yield Y.next()

not necessarily the absolute best approach, but pretty clear -- and the
calls to iter are indispensable here, in case X and/or Y as originally
passed were iterABLES that are not iterATORS such as lists etc.


Alex

[Terry Reedy]

Re: Question about references/copies


"Henning Kage" <c0dec@gmx.de > wrote in message
news:pan.2004.0 8.28.07.52.35.4 22774@gmx.de...[color=blue]
> I'm using Python only for some months now and I'm wondering, whether such
> assignments as above are creating bitwise copies of an object or just
> recieve a reference. That means I wanted to know, wheter Python in
> general
> differs between references and copies:[/color]

Better to think of Python this way: it has objects and bindings of objects
to one or more targets (names and slots of composite objects). Binding
statements (target = expression) always and only assign targets to objects.

Terry J. Reedy

[Alex Martelli]

Re: Question about references/copies

Terry Reedy <tjreedy@udel.e du> wrote:
[color=blue]
> "Henning Kage" <c0dec@gmx.de > wrote in message
> news:pan.2004.0 8.28.07.52.35.4 22774@gmx.de...[color=green]
> > I'm using Python only for some months now and I'm wondering, whether such
> > assignments as above are creating bitwise copies of an object or just
> > recieve a reference. That means I wanted to know, wheter Python in
> > general
> > differs between references and copies:[/color]
>
> Better to think of Python this way: it has objects and bindings of objects
> to one or more targets (names and slots of composite objects). Binding
> statements (target = expression) always and only assign targets to objects.[/color]

Yes, the binding per se never does anything more, if it is to a bare
name (if it's to an indexing, a slicing, or an attribute reference aka a
dotted name, things can become more interesting sometimes). And
normally the expression itself makes no copies unless copies are
specifically requested in it. Slicing though is the interesting part.

Slicing always creates a new object (except that whole-object slicing of
an immutable sliceable need not). But the interesting issue is with the
TARGETS, the SLOTS, inside that new object. Are they the SAME slots as
in the object being sliced? If said object is a list or instance of
array.array -- no, the slots are new and separate ones.

E.g., consider:
[color=blue][color=green][color=darkred]
>>> z = []
>>> a = [ z, z, 23 ]
>>> sys.getrefcount (z)[/color][/color][/color]
4[color=blue][color=green][color=darkred]
>>> b = a[:]
>>> sys.getrefcount (z)[/color][/color][/color]
6

Here, slicing does create three new slots, two of which are additional
references to the same list object z refers to. However:
[color=blue][color=green][color=darkred]
>>> z = []
>>> a = Numeric.zeros(( 3,), 'O')
>>> a[0]=a[1]=z
>>> sys.getrefcount (z)[/color][/color][/color]
4[color=blue][color=green][color=darkred]
>>> b = a[:]
>>> sys.getrefcount (z)[/color][/color][/color]
4

Here, slicing "shares" the SAME slots that 'a' already had, so there is
no change in the reference count of the list object z refers to.


Alex

[Arthur]

Re: Question about references/copies

On Sat, 28 Aug 2004 11:32:39 +0200, aleaxit@yahoo.c om (Alex Martelli)
wrote:
[color=blue]
>Henning Kage <c0dec@gmx.de > wrote:
>[color=green]
>> I'm using Python only for some months now and I'm wondering, whether such
>> assignments as above[/color]
>
>"As above" _where_?
>[color=green]
>> are creating bitwise copies of an object or just
>> recieve a reference. That means I wanted to know, wheter Python in general
>> differs between references and copies:
>>
>> class someclass:
>> def __init__( self, otherobject):
>> self.someattrib ute = otherobject[/color]
>
>Python makes no copy unless you explicitly ask for a copy -- what you
>get is always a reference. There are several ways to ask for copies --
>the most general and powerful ones you'll find in standard library
>modules copy (both copy.copy and copy.deepcopy -- all other ways of
>asking for a copy, except copy.deepcopy, get shallow copies).[/color]

I've felt strongly that this key piece of Python learning would be
and much, much more inevitable if there was a (preferred) consistent
way to ask for a copy, across objects - and/or that the copy module
was something other than one of XXX importable moudles.

That dicts and lists (for example) have totally different syntax for
copy, helps - I promise - to misdirect and confuse. I happen to
consider this entire area a significant wart. I think I am entitled to
consider it so.

I had tried to bring this up to in various forums, most particularly
edu-sig - where I thought this knid of discussion might be of
particular significance - and was rather rudely asked by Guido to take
it elsewhere. Where? not clear.

Ray Hettinger asked Gudio for permission to at least put a few words
about the copy module in the tutorial section mentioning a few of the
key importable modules - to at least, I think,. give new users a clue
that there was something of conceptual significance in all this. And
was refused.

My only participation on python-dev was to ask Gudio to reconsider

And was - rudely - dismissed.

I promise Brett C. and Anthony B. that there is more of a 2-way
street than they might think. For some of us trying to make our own
kind of contribution in our own kind of way.

Art

[Alex Martelli]

Re: Question about references/copies

Arthur <ajsiegel@opton line.com> wrote:
...[color=blue]
> I've felt strongly that this key piece of Python learning would be
> and much, much more inevitable if there was a (preferred) consistent
> way to ask for a copy, across objects -[/color]

There is: type(obj)(obj) is (almost) always the best way to get a
(shallow) copy of copyable objects (won't work with noncopyables such as
files and other iterators, of course) of built-in types (whether
user-coded types support that popular convention is of course another
issue -- it depend on the author of each of these types).

If you know what type obj is, say a list; or, if don't really care
whether obj is (say) a list or a tuple or ... 'cause what you want is a
list anyway, then the normal way to spell this is of course list(obj).
[color=blue]
> and/or that the copy module
> was something other than one of XXX importable moudles.[/color]

Well, it's Python-coded, so it seems quite natural to me that it be a
perfectly normal importable module.

[color=blue]
> That dicts and lists (for example) have totally different syntax for
> copy, helps - I promise - to misdirect and confuse. I happen to[/color]

dict(mydict) and list(mylist) appear quite similar to me. Sure,
mydict.copy() has stuck around from the dark ages and won't be even
considered for removal, due to backwards compatibility, until Python 3.0
some years from now; and mylist[:] also builds a shallow copy; but
despite their undeserved popularity these are not "the syntax for copy"
in my book, any more than, say, dict(mydict.ite ritems()) for a dict or
[x for x in mylist] for a list, even though THOSE happen to make shallow
copies too (and I HAVE seen them used in code intended as "serious").
[color=blue]
> consider this entire area a significant wart. I think I am entitled to
> consider it so.[/color]

Given the apparent dearth of information in the matter, you may be
right. I'll do my best to keep clarifying my viewpoint in my books...
[color=blue]
> I had tried to bring this up to in various forums, most particularly
> edu-sig - where I thought this knid of discussion might be of
> particular significance - and was rather rudely asked by Guido to take
> it elsewhere. Where? not clear.[/color]

Not sure why edu-sig would be more appropriate than any generic Python
forum, actually. But not particularly inappropriate either, I think.


Alex

[Arthur]

Re: Question about references/copies

On Sat, 28 Aug 2004 16:02:59 +0200, aleaxit@yahoo.c om (Alex Martelli)
wrote:
[color=blue]
>Arthur <ajsiegel@opton line.com> wrote:
> ...[color=green]
>> I've felt strongly that this key piece of Python learning would be
>> and much, much more inevitable if there was a (preferred) consistent
>> way to ask for a copy, across objects -[/color]
>
>There is: type(obj)(obj) is (almost) always the best way to get a
>(shallow) copy of copyable objects (won't work with noncopyables such as
>files and other iterators, of course) of built-in types (whether
>user-coded types support that popular convention is of course another
>issue -- it depend on the author of each of these types).[/color]
[color=blue]
>If you know what type obj is, say a list; or, if don't really care
>whether obj is (say) a list or a tuple or ... 'cause what you want is a
>list anyway, then the normal way to spell this is of course list(obj).[/color]

Well there is a lot of divergence in practice, as I think you know.
And beyond that no real consensus on what is preferable, as far as I
have been able to determnine Though hearing Alex declare it as
preferable is the beginning of the formation of some consensus, one
hopes.
[color=blue]
>[color=green]
>> and/or that the copy module
>> was something other than one of XXX importable moudles.[/color]
>
>Well, it's Python-coded, so it seems quite natural to me that it be a
>perfectly normal importable module.[/color]

I understand, now, the problems with treating the copy module
otherwise than it is. But it is a strange duck - conceptually in the
middle of things, but with limited functional importance. I thought
only that Ray's idea of setting it off a bit, by mention in the
tutorial, had considerable merit.[color=blue]
>
>[color=green]
>> That dicts and lists (for example) have totally different syntax for
>> copy, helps - I promise - to misdirect and confuse. I happen to[/color]
>
>dict(mydict) and list(mylist) appear quite similar to me. Sure,
>mydict.copy( ) has stuck around from the dark ages and won't be even
>considered for removal, due to backwards compatibility, until Python 3.0
>some years from now; and mylist[:] also builds a shallow copy; but
>despite their undeserved popularity these are not "the syntax for copy"
>in my book, any more than, say, dict(mydict.ite ritems()) for a dict or
>[x for x in mylist] for a list, even though THOSE happen to make shallow
>copies too (and I HAVE seen them used in code intended as "serious").[/color]

I agree that dict(mydict) and list(mylist) - it that is what one saw
with consitentcy in practice - would go a long way. I'll try to
practice it, having gotten into the list[:] habit.

And ...

It was my understanding - perhaps wrong - that dict.copy() was an
add-on method to dict of non-ancient origin. How do I check?
[color=blue]
>[color=green]
>> consider this entire area a significant wart. I think I am entitled to
>> consider it so.[/color]
>
>Given the apparent dearth of information in the matter, you may be
>right. I'll do my best to keep clarifying my viewpoint in my books...[/color]

That would be excellant!

ARt

[Alex Martelli]

Re: Question about references/copies

Arthur <ajsiegel@opton line.com> wrote:
...[color=blue][color=green]
> >If you know what type obj is, say a list; or, if don't really care
> >whether obj is (say) a list or a tuple or ... 'cause what you want is a
> >list anyway, then the normal way to spell this is of course list(obj).[/color]
>
> Well there is a lot of divergence in practice, as I think you know.[/color]

Sure, I did say I've seen [x for x in thelist] used.
[color=blue]
> And beyond that no real consensus on what is preferable, as far as I[/color]

Hmmm -- I've never seen a debate trying to determine that consensus,
actually.
[color=blue]
> have been able to determnine Though hearing Alex declare it as
> preferable is the beginning of the formation of some consensus, one
> hopes.[/color]

You're very kind, but I've hardly ever been able to convince GvR of
anything whatsoever;-)

[color=blue][color=green][color=darkred]
> >> and/or that the copy module
> >> was something other than one of XXX importable moudles.[/color]
> >
> >Well, it's Python-coded, so it seems quite natural to me that it be a
> >perfectly normal importable module.[/color]
>
> I understand, now, the problems with treating the copy module
> otherwise than it is. But it is a strange duck - conceptually in the
> middle of things, but with limited functional importance. I thought[/color]

Yes, I see your point. Anna and I just co-wrote a Cookbook recipe about
copying, so we ended up debating several such points and she also
brought this one up, btw
[color=blue]
> only that Ray's idea of setting it off a bit, by mention in the
> tutorial, had considerable merit.[/color]

Yep, I see that it might.
[color=blue]
> I agree that dict(mydict) and list(mylist) - it that is what one saw
> with consitentcy in practice - would go a long way. I'll try to
> practice it, having gotten into the list[:] habit.[/color]

myseq[:] has the advantage of being polymorphic over list, tuple, str,
and array.array -- list(myseq) would produce a new list no matter what
the type of myseq (it's more polymorphic too, which is generally fine
but may perhaps hide problems if myseq is, say, a dict, unexpectedly
turning it into a sequence of its keys in arbitrary hash order). I find
that when people are copying a dubious-type sequence they're often happy
with getting a list anyway (so they KNOW they can use list's wide
variety of methods), the only trouble case being indeed list(somedict).
[color=blue]
>
> And ...
>
> It was my understanding - perhaps wrong - that dict.copy() was an
> add-on method to dict of non-ancient origin. How do I check?[/color]

http://www.python.org/doc/versions.html

pointers to all versions of docs since 1.4 which came out in 1996.

to see that method copy was already in 1.5 (1998):

http://www.python.org/doc/1.5/lib/node13.html#SECTION0031600000000000000

00

I can't see it in 1.4 so I assume it was introduced between '96 and '98.

dict appeared in 2.2 (late 2001), see

http://www.python.org/doc/2.2/lib/built-in-funcs.html

[color=blue][color=green][color=darkred]
> >> consider this entire area a significant wart. I think I am entitled to
> >> consider it so.[/color]
> >
> >Given the apparent dearth of information in the matter, you may be
> >right. I'll do my best to keep clarifying my viewpoint in my books...[/color]
>
> That would be excellant![/color]

Anna (my wife and coeditor), hearing from me a summary of this
discussion, decided (and easily convinced me) that copying needs to be
the first recipe in the book, and we have already written and committed
it, so, unless we get a veto from David Ascher, the other coeditor, or
from O'Reilly, that might help. The recipe as it currently stands shows
copy.copy as the preferred way, with copy.deepcopy and calling the
object's type as secondary ways for particular needs, and L[:] as well
as [x for x in L] being specifically shown as wrong-headed approaches.
(We'll see how that survives tech review etc;-).


Alex