Why does python not have a mechanism for data hiding?

[Marc 'BlackJack' Rintsch]

Re: Why does python not have a mechanism for data hiding?

On Wed, 04 Jun 2008 13:50:42 +1000, Ben Finney wrote:

alex23 <wuwei23@gmail. comwrites:
>

>So the basic answers I'm seeing that "do just fine" are:
>>
>1. Don't test private functions.
>2. Add functionality _to_ the private functions for testing.
>3. Change the interface for the purpose of testing.
>>
>All of which seem exceptionally inefficient and run counter to the
>whole purpose of unit testing.

>
It seems you have a different idea of what unit testing is for from
me.

For me it's about finding bugs where documentation and implementation
disagree. And if you document private functions it makes sense to me to
also test if they work as documented. Because the official API relies on
the correct implementation of the private parts it uses under the hood.

Isn't the entire point of encapsulation to separate internal
components from the external interface?
>
Why would a unit test, the whole purpose of which is to assert some
aspect of the external behaviour of the unit of code, care about how
that code unit is implemented internally?

One part of writing unit tests is invoking functions with arguments that
you think are "corner cases". For example test if a function that takes a
list doesn't bomb out when you feed the empty list into it. Or if it
handles all errors correctly.

If a function `f()` calls internally `_g()` and that function might even
call other private functions, then you have to know how `f()` works
internally to create input that checks if error handling in `_g()` works
correctly. So it goes against your understanding of unit tests.

What do you do in such a situation? Build something from untested private
parts and just test the assembled piece? I prefer to test the private
functions too. After all the private functions are not private to the
everybody, there *are* functions that rely on them working correctly.

Ciao,
Marc 'BlackJack' Rintsch

[Antoon Pardon]

Re: Why does python not have a mechanism for data hiding?

On 2008-06-03, Lie <Lie.1296@gmail .comwrote:

>
Python has an extremely good design because the BDFL doesn't just
listen to everyone and create a product that tries to please
everybody, no, he listens to those that have good ideas and tells the
stupid ideas to go away and he applies a subjective decision which
more often than not leads to a better python.

I agree that Guido van Rossum has done an excellent job. That doesn't
mean he has to be painted as unfailable in which the ideais he accepts
are good ideas and those he rejects are bad ideas almost by definition.

Guido has been known to change his mind, which is an admirabele quality,
but it does show that at some point he rejected a good idea or accepted
a bad idea.

--
Antoon Pardon

[Antoon Pardon]

Re: Why does python not have a mechanism for data hiding?

On 2008-06-03, sturlamolden <sturlamolden@y ahoo.nowrote:

On Jun 2, 12:40 pm, Antoon Pardon <apar...@forel. vub.ac.bewrote:
>

>I think you completed missed the point.
>>
>This is just a proof of concept thing. In a real example there would
>of course no Set en Get methods but just methods that in the course
>of their execution would access or update the hidden attributes

>
I have to agree with Banks here, you have not provided an example of
data hiding. It does not discriminate between attribute access from
within and from outside the class. You just assume that the attribute
named 'hidden' will be left alone. Also naming it hidden is stupid as
it is visible.

No I don't assume that hidden wil be left alone. hidden is a free
variable in a closure and thus simply can't be accessed except by
local functions that were made accessible (and some mechanism
dependant on the CPython implementation) .

What you need is a mechanism that will thrown an exception whenever an
attribue is accessed from outside the class, but not from inside.

And my example does this. It threw an AttributeError

The mechanism must also be impossible to override with additional
code.

Which as far as I know it is.

--
Antoon Pardon

[Antoon Pardon]

Re: Why does python not have a mechanism for data hiding?

On 2008-06-04, Marc 'BlackJack' Rintsch <bj_666@gmx.net wrote:

>>it makes sense to me to also test if they work as documented.

>>
>If they affect the behaviour of some public component, that's where
>the documentation should be.

>
As I said they are public themselves for someone.

Isn't that contradictory: "Public for someone" I always
thought "public" meant accessible to virtually anyone.
Not to only someone.

--
Antoon Pardon

[NickC]

Re: Why does python not have a mechanism for data hiding?

On May 26, 7:32 am, "Joe P. Cool" <joe.p.c...@goo glemail.comwrot e:

I saw this "don't need it" pattern in discussions about the ternary
"if..else" expression and about "except/finally on the same block
level".
Now Python has both.

if/else was added solely because people kept coming up with ways of
embedding a pseudo conditional inside expressions and writing buggy
code in the process. All it really saves you in practice is a bit of
vertical whitespace, so, no, you still don't need it - but if you
insist on doing it, at least there's now an easy way to do it
correctly.

except/finally on the same block level was trivial to implement once
the reference interpreter switched to an AST based compiler for 2.5.
If you look at the AST, you'll find that it still only has TryExcept
and TryFinally, so again, you still don't need except/finally on the
same block level - all the syntax allows you to do is omit the second
try: line and its associated indentation.

Actually it is very useful to be able to
distinguish
between inside and outside. This is obvious for real world things e.g.
your
TV. Nobody likes to open the rear cover to switch the channel. Similar
arguments apply to software objects. "data hiding" is a harsh name, I
would
call it "telling what matters". The need for this becomes
indispensable in
really big software packages like the Eclipse framework with approx.
100000
classes. If you cannot tell the difference between inside and outside
you
are lost.
>

In Python, the philosophy "we're all consenting adults here" applies.

>
Please don't sell a missing feature as a philosophy. Say you don't
need/want
it. But don't call it philosophy.

Gosh, and here I thought treating programmers as non-idiots was
actually one of the guiding philosophies in the discussion on python-
dev. Good thing we have you here to tell us we're only imagining that.

You shouldn't pretend to know, at the time you write it, all the uses
to which your code will be put.

>
It's *your* *decision* which uses will be available. Your explanation
appears
to me as a fear to decide.

Are you writing application code or library code? For application
code, you have a much greater idea of the uses for your code, so you
can be confident in your decision as to what should and should not be
visible. For library code, however, it's fairly common for a library
to provide something which is almost, but not quite, what the user
needs. Letting users poke around at their own risk is a nice courtesy
that can save them a lot of work in the long run.

So the decision to hide something is still made (by using an
underscore prefix), but an easy mechanism is provided for the library
user to override that decision.

If you want the users of your code to know that an attribute should
not be used as a public API for the code, use the convention of naming
the attribute with a single leading underscore.

>
Littering your class definition with dozens of underscores is exactly
the
line noise we love to criticize in Perl.

Using underscores in names (leading or otherwise) separated by
plaintext keywords is a far cry from multiple different symbols that
mean different things in different contexts and can be chained
together fairly arbitrarily.

Python advertises himself as a full OOP language, but why does it
miss one of the basic principles of OOP?

>

Who taught you that enforced restrictions on attribute access was a
"basic principle" of OO?

>
Nearly every introduction to OOP? Please don't tell me that
encapsulation
does not mean "enforced restriction". If the language has no syntactic
support for encapsulation then it does not have encapsulation.

Module globals aren't visible outside the module without importing it.
Class attributes aren't visible outside the class without derefencing
it.
Instance attributes aren't visible outside an instance without
deferencing one.

*That* is the encapsulation/data hiding which OOP requires, and is the
kind which Python enforces. What you're asking for is encapsulation of
class and instance attributes based on the context in which the
dereferencing occurs (inside the class, inside a subclass of that
class, inside an instance of that class, inside an instance of a
subclass of that class, somewhere else entirely), and that has nothing
to do with the basics of OOP.

On the other hand, if you're so keen on this feature, perhaps you'd
like to make a concrete proposal regarding how you would like the
semantics to work in light of Python dynamic typing model. What will
it do when a method is invoked via the class dict rather than via
attribute retrieval? Can unbound methods access protected or private
attribute? How about descriptor get, set and delete methods? What
happens when a function is added to a class definition after creation
as a new method?

Cheers,
Nick.

[NickC]

Re: Why does python not have a mechanism for data hiding?

On May 26, 2:49 pm, "Russ P." <Russ.Paie...@g mail.comwrote:

I am also bothered a bit by the seeming inconsistency of the rules for
the single underscore. When used at file scope, they make the variable
or function invisible outside the module, but when used at class
scope, the "underscore d" variables or functions are still fully
visible. For those who claim that the client should be left to decide
what to use, why is the client prohibited from using underscored
variables at file scope?

They aren't - the only thing that won't see the underscore prefixed
names is "from x import *". If you do "import x" instead, all the
underscored names will be accessible as attributes of the module.

[Paul Rubin]

Re: Why does python not have a mechanism for data hiding?

NickC <ncoghlan@gmail .comwrites:

if/else was added solely because people kept coming up with ways of
embedding a pseudo conditional inside expressions and writing buggy
code in the process. All it really saves you in practice is a bit of
vertical whitespace, so, no, you still don't need it - but if you
insist on doing it, at least there's now an easy way to do it
correctly.

Come on, it's more than vertical whitespace, it's extraneous variables
and sometimes even extraneous functions and function call overhead.
And Python is supposed to be unbureaucratic. People kept looking for
ways to write conditional expressions instead of spewing the logic
across multiple statements for a reason: the code is often cleaner
that way.

[NickC]

Re: Why does python not have a mechanism for data hiding?

On Jun 4, 4:09 am, "Russ P." <Russ.Paie...@g mail.comwrote:

What is it about leading underscores that bothers me? To me, they are
like a small pebble in your shoe while you are on a hike. Yes, you can
live with it, and it does no harm, but you still want to get rid of it.

With leading underscores, you can see *at the point of dereference*
that the code is accessing private data. With a "this is private"
keyword you have no idea whether you're accessing private or public
data, because the two namespaces get conflated together.

I'll keep my pebble, thanks.

Cheers,
Nick.

[NickC]

Re: Why does python not have a mechanism for data hiding?

On May 25, 8:01 pm, Fuzzyman <fuzzy...@gmail .comwrote:

Python was not really written with 'difficult' customers in mind ;-)

>
True. It's extremely suited to what we do though.Minor difficulties
like this are vastly outweighed by advantages. The difficulties are
real though.

It's interesting to take a look at some of the work Brett Cannon has
done trying to come up with a sandbox for executing Python code that
actually manages to block access to dangerous functions like file() or
urllib.urlopen( ). Powerful introspection capabilities and restricted
access to methods and attributes don't really play well together.

http://svn.python.org/view/python/branches/bcannon-objcap/securing_python.txt?rev=55685&view=markup

(I believe that work is on hiatus while he's been busy with other
projects, such as a more flexible Python-based reimplementatio n of the
import mechanism that would be make it possible to implement the
security restrictions needed to permit limited imports in a sandboxed
interpreter)

One could largely hide private vars with a program that substituted random
names for single _ names, and removed the doc strings for functions,
classes, and methods with such names.

>
We need to *use* those names to display the spreadsheet once the
calculation has finished (and their code has run).
>

Such a program could even put such names in a separate module imported as
'_private_do_no t_use_'.

>
Splitting more of the functionality out is probably part of the best
solution.

Yeah, at this point your only hope is going to be making them go
through such wild contortions to get at the internal data they think
better of it. Actually blocking all access to something written in
Python is fairly tough (you generally need an extension class written
in non-Python code that hides access to certain attributes).

Cheers,
Nick.

[NickC]

Re: Why does python not have a mechanism for data hiding?

On Jun 4, 4:41 pm, Antoon Pardon <apar...@forel. vub.ac.bewrote:

Guido has been known to change his mind, which is an admirabele quality,
but it does show that at some point he rejected a good idea or accepted
a bad idea.

And sometimes the person that talked him into accepting the bad idea
in the first place ends up agreeing with him when he eventually
rejects it ;)

Cheers,
Nick.

P.S. Read the list of references in PEP 343 if you want to know what
I'm talking about *cough*

[Ben Finney]

Re: Why does python not have a mechanism for data hiding?

Marc 'BlackJack' Rintsch <bj_666@gmx.net writes:

On Wed, 04 Jun 2008 15:55:38 +1000, Ben Finney wrote:
>

By definition, "private" functions are not part of the publicly
documented behaviour of the unit. Any behaviour exhibited by some
private component is seen externally as a behaviour of some public
component.

>
But only indirectly

No, that's the point: externally, such behaviour is exhibited by the
public API. Whatever internal moving parts actually lead to the
behaviour doesn't make any difference: it's the external behaviour
that's being discussed in the above.

and it's often harder to predict the corner cases that might trigger
bugs or to test error testing in dependent private functions.

Indeed. The person writing the unit tests should do so with full
knowledge of what the implementation looks like. This allows coverage
of those corner cases you rightly point out exist in many
implementations .

Those unit tests should *not*, though, exercise anything but the
public API, otherwise they're breaking encapsulation. Their assertion
should continue to be just as true after a refactoring of the internal
components as before.

Private functions offer an API that's public to someone, so they
ought to be documented and tested.

No, that's pretty much the point: private functions are intended for
use by nothing except other functions at the same scope. If they're to
be used in other contexts, they're public API, not private.

It's a feature of Python that such bad API design doesn't lead to
hideous workarounds: one can still easily get at the parts of an API
that the programmer mistakenly marked "private". That doesn't make it
any less a mistake to break encapsulation, but it does make it much
more easily fixed.

--
\ "During the Middle Ages, probably one of the biggest mistakes |
`\ was not putting on your armor because you were 'just going down |
_o__) to the corner.'" -- Jack Handey |
Ben Finney

[NickC]

Re: Why does python not have a mechanism for data hiding?

On Jun 4, 9:24 pm, Paul Rubin <http://phr...@NOSPAM.i nvalidwrote:

NickC <ncogh...@gmail .comwrites:

if/else was added solely because people kept coming up with ways of
embedding a pseudo conditional inside expressions and writing buggy
code in the process. All it really saves you in practice is a bit of
vertical whitespace, so, no, you still don't need it - but if you
insist on doing it, at least there's now an easy way to do it
correctly.

>
Come on, it's more than vertical whitespace, it's extraneous variables
and sometimes even extraneous functions and function call overhead.
And Python is supposed to be unbureaucratic. People kept looking for
ways to write conditional expressions instead of spewing the logic
across multiple statements for a reason: the code is often cleaner
that way.

True, but it really was the multitude of buggy workarounds for the
lack of a ternary expression that sealed the deal, rather than the
benefits of ternary expressions in their own right :)

Given that I personally use ternary expressions solely as the right
hand side of an assignment statement, the reduction in vertical
whitespace usage really is the only thing they gain me. I guess if you
embedded them as an argument to a function call or other more
complicated expression then there may be additional savings. I prefer
not to do that though, since such things can get quite difficult to
parse mentally when reading them later.

Cheers,
Nick.

[NickC]

Re: Why does python not have a mechanism for data hiding?

On Jun 4, 9:56 pm, Ben Finney <bignose+hate s-s...@benfinney. id.au>
wrote:

Those unit tests should *not*, though, exercise anything but the
public API, otherwise they're breaking encapsulation. Their assertion
should continue to be just as true after a refactoring of the internal
components as before.

Python must have bad unit tests then - the CPython test suite
explicitly tests private methods all the time.

There's actually an extremely good reason for doing it that way: when
the implementation of an internal method gets broken, the unit tests
flag it explicitly, rather than having to derive the breakage from the
breakage of 'higher level' unit tests (after all, you wouldn't factor
something out into its own method or function if you weren't using it
in at least a couple of different places).

Black box testing (testing only the public API) is certainly
important, but grey box and white box testing that either exploits
knowledge of the implementation when crafting interesting test cases,
or explicitly tests internal APIs can be highly beneficial in
localising faults quickly when something does break (and as any
experienced maintenance programmer will tell you, figuring out what
you actually broke is usually harder than fixing it after you find it).

[Antoon Pardon]

Re: Why does python not have a mechanism for data hiding?

On 2008-06-04, NickC <ncoghlan@gmail .comwrote:

On May 26, 7:32 am, "Joe P. Cool" <joe.p.c...@goo glemail.comwrot e:

>I saw this "don't need it" pattern in discussions about the ternary
>"if..else" expression and about "except/finally on the same block
>level".
>Now Python has both.

>
if/else was added solely because people kept coming up with ways of
embedding a pseudo conditional inside expressions and writing buggy
code in the process. All it really saves you in practice is a bit of
vertical whitespace, so, no, you still don't need it - but if you
insist on doing it, at least there's now an easy way to do it
correctly.

If I remember correctly it was added because one of the python
developers was bitten by a bug in the standard library code
that was caused by the use of the and-or emulation, mentioned
in the FAQ.

And although one indeed doesn't need this. There are a lot
of things in Python one doesn't need. Python could be limited
to single operator expressions. You don't need:

x = a * b + c

You can write it just like this:

x = a * b
x = x + c


And if you want a list comprehension like the following:

ls = [ x * x + 4 for x in xrange(10)]

You can of course write it as follows:

def sqrplus4(a):
rs = a * a
return rs + 4

ls = [sqrplus4(x) for x in xrange(10)]


Now of course noone would defend such a limitation on the grounds
that one doesn't need the general case and that the general case
will only save you some vertical space.

But when it came to the ternary operator that was exactly the
argument used, to defend the lack of it.

In Python, the philosophy "we're all consenting adults here" applies.

>>
>Please don't sell a missing feature as a philosophy. Say you don't
>need/want
>it. But don't call it philosophy.

>
Gosh, and here I thought treating programmers as non-idiots was
actually one of the guiding philosophies in the discussion on python-
dev.

I have heard the argument: "Such a feature will be abused too easily"
and similar too many times to find this credible.

--
Antoon Pardon

[Antoon Pardon]

Re: Why does python not have a mechanism for data hiding?

On 2008-06-04, NickC <ncoghlan@gmail .comwrote:

On Jun 4, 4:09 am, "Russ P." <Russ.Paie...@g mail.comwrote:

>What is it about leading underscores that bothers me? To me, they are
>like a small pebble in your shoe while you are on a hike. Yes, you can
>live with it, and it does no harm, but you still want to get rid of it.

>
With leading underscores, you can see *at the point of dereference*
that the code is accessing private data.

But the leading underscore doesn't tell you whether it is your own
private date, which you can use a you see fit, or those of someone
else, which you have to be very carefull with.

--
Antoon Pardon