Weakrefs to classes that derive from str

**Steven Bethard** · Jul 18 '05, 11:14 PM

Re: Weakrefs to classes that derive from str

Ron Garret wrote:[color=blue]
> Why doesn't this work?
>[color=green][color=darkred]
>>>>from weakref import ref
>>>>class C(str): pass[/color][/color]
> ...[color=green][color=darkred]
>>>>ref(C())[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> TypeError: cannot create weak reference to 'C' object[/color]

Note that you don't need the class redirection:

py> ref('')
Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
TypeError: cannot create weak reference to 'str' object

But I don't know why strings aren't valid arguments to ref...

STeVe

**Ron Garret** · Jul 18 '05, 11:14 PM

Re: Weakrefs to classes that derive from str

In article <bvednVgyNqMbpt ffRVn-ig@comcast.com> ,
Steven Bethard <steven.bethard @gmail.com> wrote:
[color=blue]
> Ron Garret wrote:[color=green]
> > Why doesn't this work?
> >[color=darkred]
> >>>>from weakref import ref
> >>>>class C(str): pass[/color]
> > ...[color=darkred]
> >>>>ref(C())[/color]
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > TypeError: cannot create weak reference to 'C' object[/color]
>
> Note that you don't need the class redirection:
>
> py> ref('')
> Traceback (most recent call last):
> File "<interacti ve input>", line 1, in ?
> TypeError: cannot create weak reference to 'str' object
>
> But I don't know why strings aren't valid arguments to ref...[/color]

None of the native types (int, float, list, tuple, etc.) can have weak
references, but wrapping them in a class is supposed to get around that.
And it does -- for all classes except str.

rg

**Steven Bethard** · Jul 18 '05, 11:14 PM

Re: Weakrefs to classes that derive from str

Ron Garret wrote:[color=blue][color=green]
>>Note that you don't need the class redirection:
>>
>>py> ref('')
>>Traceback (most recent call last):
>> File "<interacti ve input>", line 1, in ?
>>TypeError: cannot create weak reference to 'str' object
>>
>>But I don't know why strings aren't valid arguments to ref...[/color]
>
> None of the native types (int, float, list, tuple, etc.) can have weak
> references, but wrapping them in a class is supposed to get around that.
> And it does -- for all classes except str.[/color]

Interesting. Is the wrapping thing documented somewhere? I didn't see
it in the documentation for weakref.ref (though I have been known to be
blind occasionally) ;)

STeVe

**Peter Hansen** · Jul 18 '05, 11:15 PM

Re: Weakrefs to classes that derive from str

Steven Bethard wrote:[color=blue]
> Ron Garret wrote:[color=green]
>> None of the native types (int, float, list, tuple, etc.) can have weak
>> references, but wrapping them in a class is supposed to get around
>> that. And it does -- for all classes except str.[/color]
>
> Interesting. Is the wrapping thing documented somewhere? I didn't see
> it in the documentation for weakref.ref (though I have been known to be
> blind occasionally) ;)[/color]

I believe it's here: http://docs.python.org/lib/module-weakref.html
if you search for the string "Not all" and read the next two
paragraphs.

On the other hand, it says (there) only that "several builtin
types such as list and dict ... can add support through
subclassing", and does not say anything about int, str, etc...

-Peter

**Steven Bethard** · Jul 18 '05, 11:15 PM

Re: Weakrefs to classes that derive from str

Peter Hansen wrote:[color=blue]
> I believe it's here: http://docs.python.org/lib/module-weakref.html
> if you search for the string "Not all" and read the next two
> paragraphs.
>
> On the other hand, it says (there) only that "several builtin
> types such as list and dict ... can add support through
> subclassing", and does not say anything about int, str, etc...[/color]

Ahh, thanks for the help. I guess there are at least two solutions to
the OP's problem then:

(1) Document that str and subclasses of str can't be weakreffed (easy)
(2) Change str so that it (or at least its subclasses) can be weakreffed
(hard)

Probably either way a feature request should be filed.

STeVe

**Ron Garret** · Jul 18 '05, 11:15 PM

Re: Weakrefs to classes that derive from str

In article <tu-dnbtOEZLOOtffRV n-hA@powergate.ca >,
Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
> Steven Bethard wrote:[color=green]
> > Ron Garret wrote:[color=darkred]
> >> None of the native types (int, float, list, tuple, etc.) can have weak
> >> references, but wrapping them in a class is supposed to get around
> >> that. And it does -- for all classes except str.[/color]
> >
> > Interesting. Is the wrapping thing documented somewhere? I didn't see
> > it in the documentation for weakref.ref (though I have been known to be
> > blind occasionally) ;)[/color]
>
> I believe it's here: http://docs.python.org/lib/module-weakref.html
> if you search for the string "Not all" and read the next two
> paragraphs.
>
> On the other hand, it says (there) only that "several builtin
> types such as list and dict ... can add support through
> subclassing", and does not say anything about int, str, etc...[/color]

Empirically:
[color=blue][color=green][color=darkred]
>>> from weakref import ref
>>> def foo(c):[/color][/color][/color]
.... class C(c): pass
.... ref(C())
....[color=blue][color=green][color=darkred]
>>> foo(int)
>>> foo(float)
>>> foo(dict)
>>> foo(list)
>>> foo(str)[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object[color=blue][color=green][color=darkred]
>>> foo(tuple)[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object[color=blue][color=green][color=darkred]
>>> foo(long)[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 3, in foo
TypeError: cannot create weak reference to 'C' object[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Ah, it appears that non-immediate immutable types don't support
weakrefs. Hm...

rg

**Peter Hansen** · Jul 18 '05, 11:15 PM

Re: Weakrefs to classes that derive from str

Ron Garret wrote:[color=blue][color=green][color=darkred]
>>>>foo(int)
>>>>foo(float )
>>>>foo(dict)
>>>>foo(list)
>>>>foo(str)[/color][/color]
> TypeError: cannot create weak reference to 'C' object
>[color=green][color=darkred]
>>>>foo(tuple )[/color][/color]
> TypeError: cannot create weak reference to 'C' object
>[color=green][color=darkred]
>>>>foo(long)[/color][/color]
> TypeError: cannot create weak reference to 'C' object
>
> Ah, it appears that non-immediate immutable types don't support
> weakrefs. Hm...[/color]

I see the same results you do, and yet I don't understand
the comment. Can you please explain what "immediate"
means in this context?

-Peter

**Raymond Hettinger** · Jul 18 '05, 11:15 PM

Re: Weakrefs to classes that derive from str

[Ron Garret][color=blue]
> Why doesn't this work?
>[color=green][color=darkred]
> >>> from weakref import ref
> >>> class C(str): pass[/color][/color]
> ...[color=green][color=darkred]
> >>> ref(C())[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> TypeError: cannot create weak reference to 'C' object[/color]
. . .[color=blue]
> Everything but strs.[/color]

Also subclasses of tuple are not weak referencable.

The issue is in the design of the C structure as a variable-sized immutable
object. Both strings and tuples allocate as a single unit of memory that holds
both the header information and the content information (the characters in a
string or the array of object pointers for a tuple). Since the size varies from
one string or tuple to the next, there is no straight-forward way for a subclass
to add an additional header field pointing to a list of weak references.

For lists and dicts, this is not a problem because the object is allocated in
two sections, a fixed size header component and a pointer to another area of
memory to hold the contents of the collection. This makes it possible for a
subclass to graft-on a weak reference pointer at a known, fixed offset from the
beginning of the header.

There are two ways to fix this. One is to add a weak reference pointer to every
string object -- that way you wouldn't even have to subclass it. Another way is
to break the object into two pieces as is done for mutable containers like dicts
and lists.

Both approaches consume time and space. In general, that is not a big deal, but
fast, memory efficient strings and tuples are at the center of all things
Python. The need to weak reference this objects is somewhat rare in comparison
to the frequency of their other uses. It did not make sense to always pay a
time/space penalty just to create the possibility of weak referencing.

While the design decision is unlikely to change, the docs could certainly be
improved. A doc patch would be welcome.

FWIW, the work-arounds are to weak-reference instances of UserString or to
create a custom class with a has-a relationship instead of an is-a relationship.

Raymond Hettinger

**Ron Garret** · Jul 18 '05, 11:16 PM

Re: Weakrefs to classes that derive from str

In article <-4KdnSsFwtp0Rtff RVn-oA@powergate.ca >,
Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
> Ron Garret wrote:[color=green][color=darkred]
> >>>>foo(int)
> >>>>foo(float )
> >>>>foo(dict)
> >>>>foo(list)
> >>>>foo(str)[/color]
> > TypeError: cannot create weak reference to 'C' object
> >[color=darkred]
> >>>>foo(tuple )[/color]
> > TypeError: cannot create weak reference to 'C' object
> >[color=darkred]
> >>>>foo(long)[/color]
> > TypeError: cannot create weak reference to 'C' object
> >
> > Ah, it appears that non-immediate immutable types don't support
> > weakrefs. Hm...[/color]
>
> I see the same results you do, and yet I don't understand
> the comment. Can you please explain what "immediate"
> means in this context?[/color]

"Immediate" means a data type that will fit entirely in a machine
register, and therefore doesn't need to actually be stored in memory.
Ints and floats are the only two immediate types in Python. They are
immutable, but you can still create weakrefs to classes that derive from
them.

rg

**Ron Garret** · Jul 18 '05, 11:16 PM

Re: Weakrefs to classes that derive from str

In article <RbB2e.52616$u7 6.11019@trndny0 8>,
"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote:
[color=blue]
> [Ron Garret][color=green]
> > Why doesn't this work?
> >[color=darkred]
> > >>> from weakref import ref
> > >>> class C(str): pass[/color]
> > ...[color=darkred]
> > >>> ref(C())[/color]
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > TypeError: cannot create weak reference to 'C' object[/color]
> . . .[color=green]
> > Everything but strs.[/color]
>
> Also subclasses of tuple are not weak referencable.
>
> The issue is in the design of the C structure as a variable-sized immutable
> object. Both strings and tuples allocate as a single unit of memory that
> holds
> both the header information and the content information (the characters in a
> string or the array of object pointers for a tuple). Since the size varies
> from
> one string or tuple to the next, there is no straight-forward way for a
> subclass
> to add an additional header field pointing to a list of weak references.
>
> For lists and dicts, this is not a problem because the object is allocated in
> two sections, a fixed size header component and a pointer to another area of
> memory to hold the contents of the collection. This makes it possible for a
> subclass to graft-on a weak reference pointer at a known, fixed offset from
> the
> beginning of the header.
>
> There are two ways to fix this. One is to add a weak reference pointer to
> every
> string object -- that way you wouldn't even have to subclass it. Another way
> is
> to break the object into two pieces as is done for mutable containers like
> dicts
> and lists.
>
> Both approaches consume time and space. In general, that is not a big deal,
> but
> fast, memory efficient strings and tuples are at the center of all things
> Python. The need to weak reference this objects is somewhat rare in
> comparison
> to the frequency of their other uses. It did not make sense to always pay a
> time/space penalty just to create the possibility of weak referencing.
>
> While the design decision is unlikely to change, the docs could certainly be
> improved. A doc patch would be welcome.
>
> FWIW, the work-arounds are to weak-reference instances of UserString or to
> create a custom class with a has-a relationship instead of an is-a
> relationship.[/color]

Thanks for the detailed explanation. I understand now why you can't
create weakrefs to these types. What I don't understand still is why
you can't create weakrefs to user-defined classes that inherit from
these types. I would think that instances of user-defined classes have
the same header structure regardless of what they inherit from. This
would seem to be supported by the fact that you can create weakrefs to
instances of user-defined classes that inherit from int and float.

rg

**Raymond Hettinger** · Jul 18 '05, 11:16 PM

Re: Weakrefs to classes that derive from str

[Ron Garret][color=blue]
> Thanks for the detailed explanation. I understand now why you can't
> create weakrefs to these types. What I don't understand still is why
> you can't create weakrefs to user-defined classes that inherit from
> these types. I would think that instances of user-defined classes have
> the same header structure regardless of what they inherit from. This
> would seem to be supported by the fact that you can create weakrefs to
> instances of user-defined classes that inherit from int and float.[/color]

It is an over-statement to say that it can't be done. In fact, Michael Hudson
is already working on a patch.

It is more accurate to say that the current mechanism doesn't allow it.
Michael's solution is to build a new mechanism.

The existing mechanism has a subclass extend the superclass's structure:

[--someobj--][--subclassdata--]
^
|
|---- offset to wr table ---

The offset is fixed for the type and must be the same across instances.

This is a problem for tuples and ints because someobj is of varying length:

[--tuple header, elem0, elem1, elem2--]
[--tuple header, elem0 ]

In contrast, ints and floats floats have no problem because they are always the
same size:

[--int header, int value--]

Raymond Hettinger

**Ron Garret** · Jul 18 '05, 11:16 PM

Re: Weakrefs to classes that derive from str

In article <RgJ2e.26268$I1 6.3183@trndny03 >,
"Raymond Hettinger" <vze4rx4y@veriz on.net> wrote:
[color=blue]
> [Ron Garret][color=green]
> > Thanks for the detailed explanation. I understand now why you can't
> > create weakrefs to these types. What I don't understand still is why
> > you can't create weakrefs to user-defined classes that inherit from
> > these types. I would think that instances of user-defined classes have
> > the same header structure regardless of what they inherit from. This
> > would seem to be supported by the fact that you can create weakrefs to
> > instances of user-defined classes that inherit from int and float.[/color]
>
> It is an over-statement to say that it can't be done. In fact, Michael
> Hudson
> is already working on a patch.
>
> It is more accurate to say that the current mechanism doesn't allow it.
> Michael's solution is to build a new mechanism.
>
> The existing mechanism has a subclass extend the superclass's structure:
>
> [--someobj--][--subclassdata--]
> ^
> |
> |---- offset to wr table ---
>
> The offset is fixed for the type and must be the same across instances.
>
> This is a problem for tuples and ints because someobj is of varying length:
>
> [--tuple header, elem0, elem1, elem2--]
> [--tuple header, elem0 ]
>
> In contrast, ints and floats floats have no problem because they are always
> the
> same size:
>
> [--int header, int value--]
>
>
> Raymond Hettinger[/color]

I see. Thanks!

rg

Weakrefs to classes that derive from str

Weakrefs to classes that derive from str

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment