How to write Smart Python programs?

[Raj]

Hi,

We just executed a project with Python using TG. The feedback was to
use more python like programming rather than C style code executed in
Python. The feedback is from a Python purist and for some reasons we
cannot solicity his help.

So we'd like to do is to scrub through the codebase and identify places
where the codebase needs improvement, both from styling as well as
design. Is there any website that can provide me with advanced tips
rather than just tutorials coz thats not of much help.

The project involves:
1. A server module that receives messages from a datalogger to populate
a database
2. A DB access module that interfaces with the UI

The technologies we use are TurboGears, SQLAlchemy, CherryPy, Kid
templates.

Any help will be greatly appreciated. If you would like to get involved
in it for $$ (experienced in the tools mentioned above) as well do let
me know. I need help asap as the deadlines are very very short.

Thanks,
Raj.

[Bruno Desthuilliers]

Re: How to write Smart Python programs?

Raj wrote:

Hi,
>
We just executed a project with Python using TG. The feedback was to
use more python like programming rather than C style code executed in
Python. The feedback is from a Python purist and for some reasons we
cannot solicity his help.
>
So we'd like to do is to scrub through the codebase and identify places
where the codebase needs improvement, both from styling as well as
design. Is there any website that can provide me with advanced tips
rather than just tutorials coz thats not of much help.

Googling for "python is not java" may be a good start. Also, here are 2
common C-style smells:

1/ using explicit indexing instead of iteration:

C-smell :
for i in range(len(seq)) :
do_something_wi th(seq[i])

Pythonic:
for item in seq:

do_something_wi th(item)
# or if you need the index too:
for i, item in enumerate(seq):
do_something_wi th(i, item)

2/ functions that returns a status code and modify their arguments. The
pythonic way is to have the function return multiple args and raise an
exception if something went wrong


--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Theerasak Photha]

Re: How to write Smart Python programs?

On 10/11/06, Bruno Desthuilliers <onurb@xiludom. growrote:

2/ functions that returns a status code and modify their arguments.

Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

-- Theerasak

[Fredrik Lundh]

Re: How to write Smart Python programs?

Theerasak Photha wrote:

>2/ functions that returns a status code and modify their arguments.

>
Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

sounds like they (or you) are confusing "arguments" with "free
variables", though. here's an example of the latter:

def outer():
var = [value]
def inner():
var[0] = new value
inner()

here's an example of the former:

def func(var):
var[0] = new value
return something else

myvar = [value]
res = func(myvar)

the second example feels rather contrived; if you need to return
multiple values, just return them.

</F>

[Theerasak Photha]

Re: How to write Smart Python programs?

On 10/11/06, Fredrik Lundh <fredrik@python ware.comwrote:

Theerasak Photha wrote:
>

>2/ functions that returns a status code and modify their arguments.

Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

>
sounds like they (or you) are confusing "arguments" with "free
variables", though. here's an example of the latter:

This is (kind of) what we were talking about:

http://www.c2.com/cgi/wiki?ClosuresThatWorkAroundFinalLimitation

-- Theerasak

[Antoine De Groote]

Re: How to write Smart Python programs?

Googling for "python is not java" may be a good start. Also, here are 2

common C-style smells:

Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this somewhere:

"Getters and setters are evil. Evil, evil, I say! Python objects are not
Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you should
write getters and setters, and then wrap them in 'property'. (1) That
means that until you prove that you need anything more than a simple
attribute access, don't write getters and setters. They are a waste of
CPU time, but more important, they are a waste of programmer time. Not
just for the people writing the code and tests, but for the people who
have to read and understand them as well.

In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters. (2) So in Java, you might as well get the chore out
of the way up front. In Python, this is silly, because you can start
with a normal attribute and change your mind at any time, without
affecting any clients of the class. (3) So, don't write getters and
setters."

For the record, I'm coming from Java, but I have some Python experience
already.

Now here's what I don't understand.

What exactly is meant by (1)? The python library reference includes this
as an example:

class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")

To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

And what does property mean anyway? The explanation in
http://docs.python.org/lib/built-in-funcs.html is not enough for me. Can
anybody point me to a more detailed documentation about this matter? (At
first sight, the Python tutorial doesn't seem to describe this. At least
it is not stated in the TOC, nor in chapter 9 which describes classes...)

And then, by (2) I believe that what is meant is that once some client
code uses public fields one can't make them private anymore because then
the clients are broke. That's clear to me. But I don't understand (3). I
don't know exactly what they mean. How would one want to change their
mind? In what way? I've been thinking about it some time now, but I'm
still kept in the dark. I would appreciate any explanation.

I'd really like to get this, because when I use a language I want to use
it "correctly" , i.e. in the way it is intended, but right now I feel a
bit lost :-(

I'm not done with 'How To Ask Questions The Smart Way' yet, so please
forgive any violations. (http://catb.org/~esr/faqs/smart-questions.html)

Regards,
antoine

[Paul Rubin]

Re: How to write Smart Python programs?

Antoine De Groote <antoine@vo.luw rites:

To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

The library reference states the technical procedures for doing
various things. That is, it explains how the procedures work. It
doesn't say when to use one procedure or another. It's a technical
reference, not a style guide.

And then, by (2) I believe that what is meant is that once some client
code uses public fields one can't make them private anymore because
then the clients are broke. That's clear to me. But I don't understand
(3). I don't know exactly what they mean. How would one want to change
their mind? In what way? I've been thinking about it some time now,
but I'm still kept in the dark. I would appreciate any explanation.

I think it means the example you cited from the library doc is
inadvisable in practice, and should only be used as a template for
more complicated getters and setters. E.g. if you want the client
to be able to set foo.x to an arbitrary value, don't use a setter,
just say

foo.x = 237

and the attribute value gets set directly. If you want to limit the
value of foo.x to the range 0-100, and make setting to outside the
range clip to that range or raise an exception or something, THEN use
a setter. In Python, you can omit the setter at first (so there's no
restriction), then later decide you want to add the restriction, so
you add the setter. In Java, if you omit the setter at first and then
later want to add it, you have to change all the client calls.

[Ben Finney]

Re: How to write Smart Python programs?

Antoine De Groote <antoine@vo.luw rites:

Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this somewhere:
>
"Getters and setters are evil. Evil, evil, I say! Python objects are
not Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you
should write getters and setters, and then wrap them in
'property'. (1) That means that until you prove that you need
anything more than a simple attribute access, don't write getters
and setters. They are a waste of CPU time, but more important, they
are a waste of programmer time. Not just for the people writing the
code and tests, but for the people who have to read and understand
them as well. [...]"
>
What exactly is meant by (1)? The python library reference includes
this as an example:
>
class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
>
To me this seems contradictory. Why would one not want to do
something that is included in the documentation? Or am I just
confused? Does anybody have an idea how to put me in the right
direction?

Note that, to a user of that class, the fact that it's implemented as
a property is opaque. Thus, changing an attribute from a simple value
object, to a property, can be done as and when needed, without
changing any of the code that uses that class.

So, it's best to always code plain public attributes without getters
and setters in the first instance:

>>class Foo(object):

... def __init__(self):
... self.menu = "spam"
...

>>foo = Foo()
>>foo.menu

'spam'

>>foo.menu = "sausage"
>>foo.menu

'sausage'

Because later, when one wants to refactor the implementation of that
attribute without changing the interface, only the implementation
needs to change:

>>class Foo(object):

... def __init__(self):
... self._menu = ["spam", "eggs", "beans"]
... self.current_it em = 0
... def _get_menu(self) :
... return self._menu[self.current_it em]
... def _set_menu(self, value):
... self._menu[self.current_it em] = value
... menu = property(_get_m enu, _set_menu)
...

And the code that uses that attribute can remain the same:

>>foo = Foo()
>>foo.menu

'spam'

>>foo.menu = "sausage"
>>foo.menu

'sausage'

Thus, coding getters and setters without a demonstrated need to do so
in a particular case is a waste of effort and an increase in
complexity, with no benefit.

I'm not done with 'How To Ask Questions The Smart Way' yet, so
please forgive any violations.

Your awareness of that document, and your attempt to follow it
sincerely, is much more important.

--
\ "He that would make his own liberty secure must guard even his |
`\ enemy from oppression." -- Thomas Paine |
_o__) |
Ben Finney

[Bruno Desthuilliers]

Re: How to write Smart Python programs?

Theerasak Photha wrote:

On 10/11/06, Bruno Desthuilliers <onurb@xiludom. growrote:
>

>2/ functions that returns a status code and modify their arguments.

>
Argument modification for lists with one item is *sometimes* used to
emulate full lexical closure. (or at least that's what the folks on
freenode #python told me)

Yes, but that's another point. What I was talking about (perhaps not
clearly) is mutable arguments used as returned values, ie:

C-smell:

def my_func(res):
bar = foo()
if bar is None:
return 1 # error
else:
res['bar'] = bar
res['baaz'] = "blah"
return 0 # no error

pythonic:

def my_func():
bar = foo()
if bar is None:
raise FooBarError("no bar from foo")
return dict(bar=bar, baaz='blah')


--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Bruno Desthuilliers]

Re: How to write Smart Python programs?

Antoine De Groote wrote:

>Googling for "python is not java" may be a good start. Also, here are 2
>common C-style smells:

>
Ok, the first Google result
(http://dirtsimple.org/2004/12/python-is-not-java.html) says this
somewhere:
>
"Getters and setters are evil. Evil, evil, I say! Python objects are not
Java beans. Do not write getters and setters. This is what the
'property' built-in is for. And do not take that to mean that you should
write getters and setters, and then wrap them in 'property'. (1) That
means that until you prove that you need anything more than a simple
attribute access, don't write getters and setters. They are a waste of
CPU time, but more important, they are a waste of programmer time. Not
just for the people writing the code and tests, but for the people who
have to read and understand them as well.
>
In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters. (2) So in Java, you might as well get the chore out
of the way up front. In Python, this is silly, because you can start
with a normal attribute and change your mind at any time, without
affecting any clients of the class. (3) So, don't write getters and
setters."
>
For the record, I'm coming from Java, but I have some Python experience
already.
>
Now here's what I don't understand.
>
What exactly is meant by (1)? The python library reference includes this
as an example:
>
class C(object):
def __init__(self): self.__x = None
def getx(self): return self._x
def setx(self, value): self._x = value
def delx(self): del self._x
x = property(getx, setx, delx, "I'm the 'x' property.")
>
To me this seems contradictory. Why would one not want to do something
that is included in the documentation? Or am I just confused? Does
anybody have an idea how to put me in the right direction?

Have you tried running the above snippet in a python shell ?

>>c = C()
>>c.x = 42
>>c.x

42

And what does property mean anyway?

See above. A property is a computed attribute : you access it like a
'data' attribute, but it really uses getters/setters. The point here is
that client code doesn't know nor need to know if it's a plain attribute
or a computed one. This let you change implementation (turn a plain
attribute into a computed one) without breaking the API. So the real
thing is not "don't use getters and setters", but "don't bother with
getters and setters, you'll be able to add them transparently if and
when needed".

As a side node, the getters/setters of a property are often either
marked as implementation (the _leading_unders core convention) or deleted
from the class's dict once the property is created or 'masked' one way
or another so they don't pollute the class's namespace.

Also and FWIW, properties are just one possible application of the
descriptor protocol. If you want to grasp Python's object model, I'd
recommand that you read about descriptors and metaclasses.

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Antoine De Groote]

Re: How to write Smart Python programs?

Bruno Desthuilliers wrote:

>And what does property mean anyway?

>
See above. A property is a computed attribute : you access it like a
'data' attribute, but it really uses getters/setters. The point here is
that client code doesn't know nor need to know if it's a plain attribute
or a computed one. This let you change implementation (turn a plain
attribute into a computed one) without breaking the API. So the real
thing is not "don't use getters and setters", but "don't bother with
getters and setters, you'll be able to add them transparently if and
when needed".

Ok great, now I understand. It's nice actually :-)

For anyone else trying this for the first time, the following might be
interesting, as I had trouble with this at first:
Don't forget to derive your class from object, otherwise properties
won't work.

As a side node, the getters/setters of a property are often either
marked as implementation (the _leading_unders core convention) or deleted
from the class's dict once the property is created or 'masked' one way
or another so they don't pollute the class's namespace.

You mean like manually removed from the dict? Because (at least on the
system I tried) it doesn't happen automatically. (Windwos XP, Python 2.5)

[Fredrik Lundh]

Re: How to write Smart Python programs?

Antoine De Groote wrote:

Don't forget to derive your class from object, otherwise properties
won't work.

getters work, setters don't (because the old-style objects checks the instance
before it looks for hooks in the class):

>>class Spam:

.... def getx(self):
.... return "x"
.... def setx(self, value):
.... print "set to", value
.... x = property(getx, setx)
....

>>s = Spam()
>>s.x

'x'

>>s.x = "y"
>>s.x

'y'

>As a side node, the getters/setters of a property are often either
>marked as implementation (the _leading_unders core convention) or deleted
>from the class's dict once the property is created or 'masked' one way
>or another so they don't pollute the class's namespace.

>
You mean like manually removed from the dict?

explicitly deleted:

>>class Spam(object):

.... def getx(self):
.... return "x"
.... def setx(self, value):
.... print "set to", value
.... x = property(getx, setx)
.... del getx, setx
....

>>s = Spam()
>>s.x

'x'

>>s.x = "y"

set to y

>>s.getx()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Spam' object has no attribute 'getx'

(note that the class definition is executed in a class-specific namespace; whatever's
left in that namespace when you reach the end of the class statement is added to the
class).

</F>

[Roy Smith]

Re: How to write Smart Python programs?

Antoine De Groote <antoine@vo.luw rote:

In Java, you have to use getters and setters because using public fields
gives you no opportunity to go back and change your mind later to using
getters and setters.

This is a good example of an anti-pattern. The general thought process
here is:

1) "I need the ability to control access to an object's internal state".
This is fine. Decoupling of interface from implementation is, in general,
a good thing, and one of the guiding principles of OOD.

2) "In the language I'm using, the only way to do that is to write getter
and setter functions". Fair enough. You gotta use the tools you have
available to you.

3) "Getters and setters are an essential part of OOD". Here's where things
go wrong. Just like one must learn to decouple interface from
implementation in your programs, so must one learn to decouple them in your
program design. Getters and setters are an language implementation detail.
There's nothing magic about adding the word "get" to the beginning of a
variable name and putting "()" after it. That's just how you spell
"property" in C++ or java.

It used to be said, "You can write Fortran in any language". The
implication there was that a literal line-by-line translation
(transliteratio n?) of a Fortran program into some other language usually
resulted in a mess. I think we can generalize that to "You can write any
language in any other language". Every language has a natural way of doing
things. When you try to bring the old idioms and design processes from one
language over to another, it usually doesn't work.

Once, years ago, I was working in a lisp environment for maybe a year. It
took me a little while to learn the syntax and basic function set, but I
never really "got" it. I was writing C programs in lisp. The same thing
happens when I try to speak Spanish. Even when I used Spanish a fair bit,
I never really thought in Spanish. I thought in English and translated.
That doesn't work very well.

[John Salerno]

Re: How to write Smart Python programs?

Bruno Desthuilliers wrote:

Googling for "python is not java" may be a good start.

I have a question about this section of a blog with that title. I'll ask
the question first so it doesn't get lost at the bottom: does the
following opinion of XML apply to GUIs? (i.e., using an XML resource to
define a GUI layout rather than coding it yourself).

Thanks.

----------------------------------------------

XML is not the answer. It is not even the question. To paraphrase Jamie
Zawinski on regular expressions, "Some people, when confronted with a
problem, think "I know, I'll use XML." Now they have two problems."

This is a different situation than in Java, because compared to Java
code, XML is agile and flexible. Compared to Python code, XML is a boat
anchor, a ball and chain. In Python, XML is something you use for
interoperabilit y, not your core functionality, because you simply don't
need it for that. In Java, XML can be your savior because it lets you
implement domain-specific languages and increase the flexibility of your
application "without coding". In Java, avoiding coding is an advantage
because coding means recompiling. But in Python, more often than not,
code is easier to write than XML. And Python can process code much, much
faster than your code can process XML. (Not only that, but you have to
write the XML processing code, whereas Python itself is already written
for you.)

If you are a Java programmer, do not trust your instincts regarding
whether you should use XML as part of your core application in Python.
If you're not implementing an existing XML standard for interoperabilit y
reasons, creating some kind of import/export format, or creating some
kind of XML editor or processing tool, then Just Don't Do It. At all.
Ever. Not even just this once. Don't even think about it. Drop that
schema and put your hands in the air, now! If your application or
platform will be used by Python developers, they will only thank you for
not adding the burden of using XML to their workload.

(The only exception to this is if your target audience really really
needs XML for some strange reason. Like, they refuse to learn Python and
will only pay you if you use XML, or if you plan to give them a nice GUI
for editing the XML, and the GUI in question is something that somebody
else wrote for editing XML and you get to use it for free. There are
also other, very rare, architectural reasons to need XML. Trust me, they
don't apply to your app. If in doubt, explain your use case for XML to
an experienced Python developer. Or, if you have a thick skin and don't
mind being laughed at, try explaining to a Lisp programmer why your
application needs XML!)

[Bruno Desthuilliers]

Re: How to write Smart Python programs?

Antoine De Groote wrote:

Bruno Desthuilliers wrote:

>>And what does property mean anyway?

>>
>See above. A property is a computed attribute : you access it like a
>'data' attribute, but it really uses getters/setters. The point here is
>that client code doesn't know nor need to know if it's a plain attribute
>or a computed one. This let you change implementation (turn a plain
>attribute into a computed one) without breaking the API. So the real
>thing is not "don't use getters and setters", but "don't bother with
>getters and setters, you'll be able to add them transparently if and
>when needed".

>
Ok great, now I understand. It's nice actually :-)
>
For anyone else trying this for the first time, the following might be
interesting, as I had trouble with this at first:
Don't forget to derive your class from object, otherwise properties
won't work.

FWIW, I don't see the point of still using old-style classes (apart for
compat with old Python versions...)

>As a side node, the getters/setters of a property are often either
>marked as implementation (the _leading_unders core convention) or deleted
>from the class's dict once the property is created or 'masked' one way
>or another so they don't pollute the class's namespace.

>
You mean like manually removed from the dict?

Yes.

Because (at least on the
system I tried) it doesn't happen automatically. (Windwos XP, Python 2.5)

My usual idiom for properties is:

class MyClass(object) :
@apply
def propname():
def fget(self): return self._something * 2
def fset(self, value): self._something = int(value) / 2
return property(**loca ls())


(Don't remember where it comes from...)

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"