Why less emphasis on private data?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Bruno Desthuilliers

    #31
    Re: Why less emphasis on private data?

    John Nagle a écrit :
    sturlamolden wrote:
    >
    >time.swift@gmai l.com wrote:
    >>
    >>Coming from a C++ / C# background, the lack of emphasis on private data
    >>seems weird to me. I've often found wrapping private data useful to
    >>prevent bugs and enforce error checking..
    >>>
    >>It appears to me (perhaps wrongly) that Python prefers to leave class
    >>data public. What is the logic behind that choice?
    >>
    >>
    >>
    >The designers of Java, C++, C#, Ada95, Delphi, etc. seem to think that
    >if an object's 'internal' variables or states cannot be kept private,
    >programmers get an irresistible temptation to mess with them in
    >malicious ways.
    >
    >
    If you're not clear on encapsulation issues,
    encapsulation != data hiding
    you probably haven't
    done extensive maintenance programming on code written by others.
    I did.
    Finding out who can mess with a variable when debugging the code of
    others is not fun.
    # before
    class Toto(object):
    def __init__(self, x):
    self._x = x

    # after
    class Toto(object):
    def __init__(self, x):
    self._x = x

    @apply
    def _x():
    def fget(self):
    return self._real_x
    def fset(self, value):
    import pdb; pdb.set_trace()
    self._real_x = value
    return property(**loca ls)

    This is of course a braindead implementation - a better one would use
    either the inspect module of the sys._getframe() hack to retrieve useful
    debug infos (left as an excercice to the reader...)

    Because Python doesn't have explicit declarations, scope of
    variables is
    a touchy issue.
    ???
    If you write "x = 1" within a function, that will
    create a local "x" if "x" doesn't exist, or alter a global "x" if "x" was
    previously created in the global context.
    Err... May I suggest you to read these two pages:


    But at least global variables
    are local to the namespace; we don't have clashes across files. So
    it's not too bad. JavaScript has the convention that newly created
    variables are global by default.
    Unless preceded by the 'var' keyword...
    Big mistake.
    Mmm... which one ?
    The underscore thing makes sense. Single underscore
    variables are "protected" in the C++ sense, and double underscore
    variables are "private", not visible from inherited classes.
    It's hard to misuse such variables by accident. I'd be tempted
    to prohibit access to underscore variables other than via "self._x"
    forms, so they'd be inaccessable outside the object.
    # foo.py
    class Foo(object):
    def __init__(self, x):
    self._x = x
    def __repr__(self):
    return "<Foo %s>" % self._x

    # bar.py
    def bar(self):
    self.y = self._x

    # baaz.py
    from foo import Foo
    from bar import bar
    Foo.bar = bar

    f = Foo([42])
    f.bar()
    f.y.append('got cha')
    print f

    It's undesirable
    from a maintenance standpoint to have an unenforced convention
    If it's a convention, it doesn't have to be inforced. If it's inforced,
    it's not a convention anymore.

    While we're at it, I've found it very valuable to be able to mess with
    implementation when doing maintenance on somewhat large (and somewhat
    messy) Python systems...
    like
    a lead underscore. The maintenance programmer can't trust its meaning.
    >
    As Python grows up, and larger systems are written in it, these
    issues become more important.
    If you go that way, then you'll also want to introduce declarative
    static typing and remove all possibility to dynamically modify classes
    or add/replace attributes and/or methods on a per-instance basis. If you
    want Java, you know where to find it.

    Comment

    • Steven D'Aprano

      #32
      Re: Why less emphasis on private data?

      On Sun, 07 Jan 2007 04:09:13 -0800, Paul Rubin wrote:
      "Felipe Almeida Lessa" <felipe.lessa@g mail.comwrites:
      >What is the chance of having to inherit from two classes from
      >different modules but with exactly the same name *and* the same
      >instance variable name?
      >>
      >Of course you're being very pessimistic or extremely unlucky.
      >
      If you want to write bug-free code, pessimism is the name of the game.
      I wonder whether Paul uses snow chains all year round, even in the blazing
      summer? After all, "if you want to drive safely, pessimism is the name of
      the game".

      In the last couple of weeks comp.lang.pytho n has had (at least) two
      practical examples of the pros and cons of private attributes.

      The pro: there was discussion about replacing the optparse module's
      implementation with argparse, leaving the interface the same. This was
      complicated by the fact that optparse exposes its internal variables,
      making the job of duplicating the interface significantly harder. However
      this was surely a design choice, not an accident. Having private
      attributes won't save you if you choose not to make your attributes
      private.

      The con: there was a fellow who (for some reason) actually needed to
      access a class' private attributes. To the best of my knowledge, he was
      over 18 and, while new to Python, an experienced programmer, so I believe
      him when he said he had eliminated all other alternatives. (And if he
      were wrong, if he was incompetent -- Not My Problem. It isn't for me to
      take a hammer off him so he doesn't hit his thumb with it.) In his case,
      Python's name mangling of private attributes was an inconvenience, not a
      help.

      Compared to all the "what-ifs" and "maybes" and hypotheticals in this
      thread, there were two practical cases that revolved around private
      variables. In one, we see that they aren't a panacea: data hiding doesn't
      help when the data isn't hidden. In the other, we see that one developer's
      private attribute is just what another developer needs to solve a problem.


      --
      Steven.

      Comment

      • Paul Rubin

        #33
        Re: Why less emphasis on private data?

        "Paul Boddie" <paul@boddie.or g.ukwrites:
        Consider that the above three class definitions might be in separate
        files and you see how clumsy this gets.
        >
        What are you trying to show with the above? The principal benefit of
        using private attributes set on either the class or the instance is to
        preserve access, via self, to those attributes defined in association
        with (or within) a particular class in the inheritance hierarchy, as
        opposed to providing access to the "most overriding" definition of an
        attribute. This is demonstrated more effectively with a method on class A:
        Right, the problem is if those methods start changing the "private"
        variable. I should have been more explicit about that.

        class A:
        def __init__(self):
        self.__x = 3
        def foo(self):
        return self.__x

        class B(A): pass

        class A(B):
        def bar(self):
        self.__x = 5 # clobbers private variable of earlier class named A

        Comment

        • Paul Rubin

          #34
          Re: Why less emphasis on private data?

          Steven D'Aprano <steve@REMOVE.T HIS.cybersource .com.auwrites:
          If you want to write bug-free code, pessimism is the name of the game.
          >
          I wonder whether Paul uses snow chains all year round, even in the blazing
          summer? After all, "if you want to drive safely, pessimism is the name of
          the game".
          No. I'm willing to accept a 10**-5 chance of hitting a freak
          snowstorm in summer, since I drive in summer at most a few hundred
          times a year, so it will take me 100's of years before I'm likely to
          encounter such a storm. There are millions of drivers, so if they all
          take a similar chance, then a few times a year we'll see in the paper
          that someone got caught in a storm, which is ok. Usually there's no
          real consequence beyond some inconvenience of waiting for a tow truck.

          Tow truck or ambulance operators, on the other hand, should keep
          chains available all year around, since they have to service the needs
          of millions of users, have to be ready for freak summer storms.

          As a software developer wanting to deploy code on a wide scale, I'm
          more like a tow truck operator than an individual car driver.
          Alternatively, as a coder I "drive" a lot more often. If some Python
          misfeature introduces a bug with probability 10**-5 per line of code,
          then a 100 KLoc program is likely to have such a bug somewhere. It
          doesn't take 100's of years.

          Comment

          • Paul Rubin

            #35
            Re: Why less emphasis on private data?

            Dennis Lee Bieber <wlfraed@ix.net com.comwrites:
            I'd be quite concerned about the design environment rather than the
            immediate code... Probably need something ugly like...
            >
            from mod1 import B as B1
            from mod2 import B as B2
            class A(B1, B2):
            ....
            Interesting. I just tried that. mod1.py contains:

            class B:
            def foo(self): self.__x = 'mod1'

            mod2.py contains:

            class B:
            def bar(self): self.__x = 'mod2'

            And the test is:

            from mod1 import B as B1
            from mod2 import B as B2

            class A(B1, B2): pass

            a = A()
            a.foo()
            print a._B__x
            a.bar()
            print a._B__x

            Sure enough, mod2 messes up mod1's private variable.

            Comment

            • Steven D'Aprano

              #36
              Re: Why less emphasis on private data?

              On Sun, 07 Jan 2007 19:30:05 -0800, Paul Rubin wrote:
              Steven D'Aprano <steve@REMOVE.T HIS.cybersource .com.auwrites:
              If you want to write bug-free code, pessimism is the name of the game.
              >>
              >I wonder whether Paul uses snow chains all year round, even in the blazing
              >summer? After all, "if you want to drive safely, pessimism is the name of
              >the game".
              >
              No. I'm willing to accept a 10**-5 chance of hitting a freak
              snowstorm in summer, since I drive in summer at most a few hundred
              times a year, so it will take me 100's of years before I'm likely to
              encounter such a storm. There are millions of drivers, so if they all
              take a similar chance, then a few times a year we'll see in the paper
              that someone got caught in a storm, which is ok. Usually there's no
              real consequence beyond some inconvenience of waiting for a tow truck.
              >
              Tow truck or ambulance operators, on the other hand, should keep
              chains available all year around, since they have to service the needs
              of millions of users, have to be ready for freak summer storms.
              >
              As a software developer wanting to deploy code on a wide scale, I'm
              more like a tow truck operator than an individual car driver.
              Alternatively, as a coder I "drive" a lot more often. If some Python
              misfeature introduces a bug with probability 10**-5 per line of code,
              then a 100 KLoc program is likely to have such a bug somewhere. It
              doesn't take 100's of years.
              That's an irrelevant argument. We're not talking about random bugs in
              random places of code, we're talking about one specific type of bug which
              can only occur in a handful of very restricted set of circumstances, e.g.
              you have to inherit from two classes which not only have exactly the same
              name but they also have the same private attribute.

              Your argument is that Python's strategy for dealing with private
              attributes is insufficiently pessimistic, because it doesn't deal with
              those circumstances. Fine. I agree. Python isn't pessimistic. Does it need
              to be? Just how often do you inherit from two identically-named classes
              both of which use identically-named private attributes?

              You suggested that coders (and by extension, Python) should behave with
              equal pessimism whether they are subclassing two identically-named
              classes or not. That's equivalent to the argument that one should use snow
              chains whether it is snowing or not -- it only considers the benefit of
              the extra protection, without considering the costs.

              Python's private attribute handling balances convenience and protection,
              giving more weight to convenience, trading off some protection. And
              convenience gives increased productivity, easier debugging, few bugs
              overall, and other Good Things. It would probably change the character of
              Python unacceptably much to push that balance the other way.

              Don't get me wrong, it is a good thing for you to alert people to the
              circumstances that Python's strategy breaks down, so that they can "use
              snow chains" in those circumstances. And, hey, if somebody reads this
              thread and is motivated to find a better strategy that doesn't change the
              nature of the language by too much, great. (This happened once before:
              multiple inheritance was broken in classic classes, and new classes where
              added partly to fix that.)

              But chances are, the majority of Pythonistas will think that having to
              use snow chains once in a very great while is an acceptable trade-off to
              the smooth ride Python gives the rest of the time.


              --
              Steven D'Aprano

              Comment

              • Paul Rubin

                #37
                Re: Why less emphasis on private data?

                Steven D'Aprano <steve@REMOVEME .cybersource.co m.auwrites:
                Just how often do you inherit from two identically-named classes
                both of which use identically-named private attributes?
                I have no idea how often if ever. I inherit from library classes all
                the time, without trying to examine what superclasses they use. If my
                subclass happens to have the same name as a superclass of some library
                class (say Tkinter) this could happen. Whether it ever DOES happen, I
                don't know, I could only find out by examining the implementation
                details of every library class I ever use, and I could only prevent it
                by remembering those details. That is an abstraction leak and is
                dangerous and unnecessary. The name mangling scheme is a crock. How
                often does anyone ever have a good reason for using it, except maybe
                in something like a debugger that can just as easily reach inside the
                actual class descriptors and get all the variables out?

                Comment

                • Paul Boddie

                  #38
                  Re: Why less emphasis on private data?

                  Paul Rubin wrote:
                  >
                  Right, the problem is if those methods start changing the "private"
                  variable. I should have been more explicit about that.
                  >
                  class A:
                  def __init__(self):
                  self.__x = 3
                  def foo(self):
                  return self.__x
                  >
                  class B(A): pass
                  >
                  class A(B):
                  def bar(self):
                  self.__x = 5 # clobbers private variable of earlier class named A
                  Has this ever been reported as a bug in Python? I could imagine more
                  sophisticated "name mangling": something to do with the identity of the
                  class might be sufficient, although that would make the tolerated
                  "subversive " access to private attributes rather difficult.

                  Paul

                  Comment

                  • Paul Rubin

                    #39
                    Re: Why less emphasis on private data?

                    "Paul Boddie" <paul@boddie.or g.ukwrites:
                    Has this ever been reported as a bug in Python? I could imagine more
                    sophisticated "name mangling": something to do with the identity of the
                    class might be sufficient, although that would make the tolerated
                    "subversive " access to private attributes rather difficult.
                    If you mean the object id, I don't think you can use it for name
                    mangling, since the mangled names have to survive code marshalling
                    and you may end up with different object id's.

                    I've just never encountered any legitimate use for the "subversive "
                    access and if it's really necessary, it's better to do it through some
                    kind of well-designed reflection interface in the class, rather than
                    with a crock like name mangling.

                    Comment

                    • Duncan Booth

                      #40
                      Re: Why less emphasis on private data?

                      "Paul Boddie" <paul@boddie.or g.ukwrote:
                      Paul Rubin wrote:
                      >>
                      >Right, the problem is if those methods start changing the "private"
                      >variable. I should have been more explicit about that.
                      >>
                      >class A:
                      > def __init__(self):
                      > self.__x = 3
                      > def foo(self):
                      > return self.__x
                      >>
                      >class B(A): pass
                      >>
                      >class A(B):
                      > def bar(self):
                      > self.__x = 5 # clobbers private variable of earlier class named A
                      >
                      Has this ever been reported as a bug in Python? I could imagine more
                      sophisticated "name mangling": something to do with the identity of the
                      class might be sufficient, although that would make the tolerated
                      "subversive " access to private attributes rather difficult.
                      >
                      Paul
                      >
                      If it worries you then you can always check for it and disallow any
                      hierarchies where it could be a problem. For that matter PyChecker ought to
                      be able to catch this situation (maybe it already does, I haven't looked).
                      >>class SafetyNet(type) :
                      def __new__(cls, name, bases, dct):
                      print "new",name
                      c = type.__new__(cl s, name, bases, dct)
                      assert not name in [b.__name__ for b in c.__mro__[1:]]
                      return c

                      >>__metaclass __ = SafetyNet
                      >>class A: pass
                      new A
                      >>class B(A): pass
                      new B
                      >>class A(B): pass
                      new A

                      Traceback (most recent call last):
                      File "<pyshell#2 4>", line 1, in <module>
                      class A(B): pass
                      File "<pyshell#1 7>", line 5, in __new__
                      assert not name in [b.__name__ for b in c.__mro__[1:]]
                      AssertionError

                      Comment

                      • Steven D'Aprano

                        #41
                        Re: Why less emphasis on private data?

                        On Sun, 07 Jan 2007 23:49:21 -0800, Paul Rubin wrote:
                        Steven D'Aprano <steve@REMOVEME .cybersource.co m.auwrites:
                        >Just how often do you inherit from two identically-named classes
                        >both of which use identically-named private attributes?
                        >
                        I have no idea how often if ever.
                        You've established that there's a name conflict when you do so, which
                        leads to bugs. So how often do you get bitten by that particular type of
                        bug?

                        I inherit from library classes all
                        the time, without trying to examine what superclasses they use. If my
                        subclass happens to have the same name as a superclass of some library
                        class (say Tkinter) this could happen. Whether it ever DOES happen, I
                        don't know, I could only find out by examining the implementation
                        details of every library class I ever use, and I could only prevent it
                        by remembering those details.
                        class MySubClass(Some Superclass):
                        try:
                        __my_private_at tribute
                        except AttributeError:
                        __my_private_at tribute = some_value
                        else:
                        raise ValueError("Nam e conflict with private attribute!")

                        Problem solved.

                        *wink*

                        That is an abstraction leak and is
                        dangerous and unnecessary. The name mangling scheme is a crock. How
                        often does anyone ever have a good reason for using it,
                        Exactly. I never use it.

                        The truth of the matter is, MyClass.__priva te is not private at all. It is
                        still a public attribute with a slightly unexpected name. In other words,
                        if you want to code defensively, you should simply assume that Python has
                        no private attributes, and code accordingly.

                        Problem solved.



                        --
                        Steven.

                        Comment

                        • Paul Boddie

                          #42
                          Re: Why less emphasis on private data?

                          Steven D'Aprano wrote:
                          >
                          The truth of the matter is, MyClass.__priva te is not private at all. It is
                          still a public attribute with a slightly unexpected name. In other words,
                          if you want to code defensively, you should simply assume that Python has
                          no private attributes, and code accordingly.
                          >
                          Problem solved.
                          Well, it isn't really solved - it's more avoided than anything else.
                          ;-)

                          Still, if one deconstructs the use of private data in various
                          programming languages, one can identify the following roles (amongst
                          others):

                          1. The prevention of access to data from program sections
                          not belonging to a particular component.
                          (The classic "keep out" mechanism.)
                          2. The enforcement of distinct namespaces within components.
                          (Making sure that subclass attributes and superclass attributes
                          can co-exist.)
                          3. To support stable storage layouts and binary compatibility.

                          Most Python adherents don't care too much about #1, and Python isn't
                          driven by the need for #3, mostly due to the way structures (modules,
                          classes, objects) are accessed by the virtual machine. However, one
                          thing which does worry some people is #2, and in a way it's the
                          forgotten but more significant benefit of private data.

                          Before I became completely aware of the significance of #2, I remember
                          using various standard library classes which are meant to be subclassed
                          and built upon, thinking that if I accidentally re-used an attribute
                          name then the operation of such classes would be likely to fail in
                          fairly bizarre ways. Of course, a quick browse of the source code for
                          sgmllib.SGMLPar ser informed me of the pitfalls, and I'm sure that
                          various tools could also be informative without the need to load
                          sgmllib.py into a text editor, but if I had been fully aware of the
                          benefits of private attributes and could have been sure that such
                          attributes had been used (again, a tool might have given such
                          assurances) then I wouldn't have needed to worry.

                          So I suppose that to "code accordingly" in the context of your advice
                          involves a manual inspection of the source code of superclasses or the
                          usage of additional tools. Yet I suppose that this isn't necessarily
                          unusual behaviour when working with large systems.

                          Paul

                          Comment

                          • Paul Rubin

                            #43
                            Re: Why less emphasis on private data?

                            Steven D'Aprano <steve@REMOVE.T HIS.cybersource .com.auwrites:
                            I have no idea how often if ever.
                            >
                            You've established that there's a name conflict when you do so, which
                            leads to bugs. So how often do you get bitten by that particular type of bug?
                            I don't know. Likely zero, possibly not. I'm sure I've written many
                            bugs that have never been detected by me or anyone else. I've
                            probably written bugs that crashed an application for some user but
                            they just cursed me out and never bothered to tell me about the crash.
                            Maybe I've even written bugs that leaked a user's private data without
                            the user noticing, but discovered by some attacker intercepting the
                            data who is cackling about the bug while keeping it secret. There's
                            no way for me to think I'll ever find out.

                            I'd much prefer to be able to say of any type of bug, "the number is
                            exactly zero as a known fact, because it's inherent in Python's design
                            that it's impossible to have that type of bug". Language designs
                            should aim to let programmers say things like that as often as possible.
                            class MySubClass(Some Superclass): ...
                            raise ValueError("Nam e conflict with private attribute!")
                            Problem solved.
                            No good, Python allows creating classes and attributes on the fly.
                            The superclass could create its private variable after the subclass is created.
                            The truth of the matter is, MyClass.__priva te is not private at all. It is
                            still a public attribute with a slightly unexpected name. In other words,
                            if you want to code defensively, you should simply assume that Python has
                            no private attributes, and code accordingly.
                            >
                            Problem solved.
                            Well, "problem moved", not "problem solved". Now you have the problem
                            of having to know the names of every attribute any related class might
                            use when you write your own class. That is why other languages have
                            private variables and Python has name mangling--to solve a real problem.
                            Except Python's solution is a leaky kludge.

                            Comment

                            • Hendrik van Rooyen

                              #44
                              Re: Why less emphasis on private data?

                              "Paul Rubin" <http://phr.cx@NOSPAM.i nvalidwrote:
                              If you want to write bug-free code, pessimism is the name of the game.
                              A healthy touch of paranoia does not come amiss either...

                              And even then things foul up in strange ways because your head
                              is never quite literal enough.

                              When you hear a programmer use the word "probabilit y" -
                              then its time to fire him, as in programming even the lowest
                              probability is a certainty when you are doing millions of
                              things a second.

                              But this is off topic, really - I don't think that hiding things make
                              much difference, especially as the python hiding is not absolute.

                              - Hendrik


                              Comment

                              • hg

                                #45
                                Re: Why less emphasis on private data?

                                sturlamolden wrote:
                                The designers of Java, C++, C#, Ada95, Delphi, etc. seem to think that
                                if an object's 'internal' variables or states cannot be kept private,
                                programmers get an irresistible temptation to mess with them in
                                malicious ways. But if you are that stupid, should you be programming
                                in any language? The most widely used language is still C, and there is
                                no concept of private data in C either, nor is it needed.

                                void test(void)
                                {
                                static int i;
                                }


                                Do you agree that i is "private" to test ?

                                hg

                                Comment

                                Working...