exception handling in complex Python programs

[eliben]

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.

Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.

Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.

Eli

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:

http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/

Not really CL specific, and can apply to Python's exceptions.

[Rafe]

Re: exception handling in complex Python programs

On Aug 20, 12:19 am, eliben <eli...@gmail.c omwrote:

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
>
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
>
Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
>
I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
>
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
>
Eli
>
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
>
P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:http://dlweinreb.wordpress.com/2008/...ns-exceptions-...
Not really CL specific, and can apply to Python's exceptions.

Maybe I am oversimplifying (and I am here to learn), but I catch all
exceptions which otherwise would be hard to understand as a user. In
other words, when a better error message is useful.

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)


I suppose you could write a decorator to do this if you want it at the
function level, but that seems a bit to broad. Shouldn't exceptions be
on a case-by-case basis to add protection and return information
exactly where it is needed?

- Rafe

[Chris Mellon]

Re: exception handling in complex Python programs

On Tue, Aug 19, 2008 at 12:19 PM, eliben <eliben@gmail.c omwrote:

Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
>
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
>

Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.

This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.

Catching an exception when you don't know exactly what to do to fix it
is an error. At best, it will make debugging a program harder (because
you're losing context information about the error) and at worst it
adds bugs to your program. The way Javas checked exceptions encourage
empty or otherwise useless exception handlers is a major problem with
them.

There's some fear about presenting exceptions to the end user. That's
a user interface issues, not a software quality or engineering issue,
and it's resolvable with top-level handlers that log tracebacks
somewhere a user can't see them if desired.

Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
>

You should catch anything that you can correct. If you don't have a
specific answer for a specific exception, don't catch it.

I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
>
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
>
Eli
>
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
>

Same rules apply. The only sort-of exception (no pun intended) is that
sometimes you want to re-raise as a different type of exception. Make
sure that you preserve all of the original information (including the
original traceback) if you do this.

P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:

http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-are-really-about/

Not really CL specific, and can apply to Python's exceptions.
--

http://mail.python.org/mailman/listinfo/python-list

>

[Fredrik Lundh]

Re: exception handling in complex Python programs

Rafe wrote:

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...
>
try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)

so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)

I'd be really worried if I found that in a piece of source code I had to
maintain.

</F>

[dbpokorny@gmail.com]

Re: exception handling in complex Python programs

On Aug 19, 10:19 am, eliben <eli...@gmail.c omwrote:

P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?

You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:

def do_something(fi lename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...

where err is a function that generates an error object that your
application understands. I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is

def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
return err(...)
f = open(filename)
...

or, (and this last one I actually used for a web application)

def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
raise MyApplicationsE xceptionType("F ile not found...")
f = open(filename)
...

The last one has the advantage that you can write a request handler
like this

def handle_http_req uest(...):
func = specific_handle r_func(...)
try:
response = func(...)
return response
except MyApplicationsE xceptionType as exc: #3.0 syntax
return error_response( exc,...)

Exceptions you don't expect (i.e. bugs) will get handled by the web
app framework, but you get to handle your own exceptions. Raising your
own exception type can also be employed with the EAFP approach like
this:

def do_something(fi lename):
try:
f = open(filename)
except IOError:
raise MyApplicationsE xceptionType("F ile %s not found" %
filename)
...

If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.

David

[Steven D'Aprano]

Re: exception handling in complex Python programs

On Tue, 19 Aug 2008 11:07:39 -0700, dbpokorny@gmail .com wrote:

def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
return err(...)
f = open(filename)
...

You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This isn't just a theoretical risk. There's a whole class of errors and
security holes based on similar race conditions. I find it amusing that
you consider it "sloppy" to deal with errors raised when actually opening
a file, but then recommend a technique that has a well-known failure mode.

That's not to say that I never use such techniques myself. For quick and
dirty scripts, where I can tolerate the risk of some other process moving
a file behind my back, I've been known to do something similar.



--
Steven

[eliben]

Re: exception handling in complex Python programs

""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>.
"""

This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.

You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.

Any (semi)complete guides on this canonical paradigm online ? I've
only found some references in maillist discussions.

  def do_something(fi lename):
    if not os.access(filen ame,os.R_OK):
      return err(...)
    f = open(filename)
    ...
>

But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?

[Marc 'BlackJack' Rintsch]

Re: exception handling in complex Python programs

On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:

>You want to look up Easier to Ask Forgivness than Permission (EAFP)
>which is touted as the "canonical" error-handling paradigm for Python.

>
Any (semi)complete guides on this canonical paradigm online ? I've only
found some references in maillist discussions.

There's the glossary in the documentation:

http://docs.python.org/tut/node18.html

Look under 'duck-typing', 'EAFP', and 'LBYL'.

Ciao,
Marc 'BlackJack' Rintsch

[Steven D'Aprano]

Re: exception handling in complex Python programs

On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:

""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes head
against keyboard>.
"""

I'm curious about the claim that open() will raise NameError in Python3.
I find it hard to credit that claim, but if it is correct, what's the
justification for that?

This is *exactly* my concern with Python exceptions. You just never know
what can be thrown at you.

It's true that documentation of exceptions is relatively weak in Python.
And some functions can raise a bewildering array of exceptions. See for
example this thread where somebody notes that urllib2.urlopen () can raise
any of six different exceptions:

http://mail.python.org/pipermail/baypiggies/2008-April/003187.html

And I've had it raise socket.error, which makes seven. And the
documentation only mentions one of those exceptions.

However, as Gregory Smith describes, some of those seven exceptions are
subclasses of others, so it is possible to reduce it down to three cases
-- and arguably one of those cases (ValueError) is a bug that needs
fixing, not an exception that needs catching.

That's probably as bad as it gets in Python, at least for the standard
library. Most functions don't raise arbitrary exceptions for sensible
data, and if you pass non-sensible data then you should treat the
exception as a bug in your code and fix it.



--
Steven

[dbpokorny@gmail.com]

Re: exception handling in complex Python programs

On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:

On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail .com wrote:

  def do_something(fi lename):
    if not os.access(filen ame,os.R_OK):
      return err(...)
    f = open(filename)
    ...

>
You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?

This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we
can probably
agree that the following is the simplest, easiest-to-remember way:

def do_something(fi lename):
try:
f = open(filename)
except:
<handle exception>
...

Opening files is a special case where EAFP is the only correct
solution (AFAIK). I still liberally sprinkle LBYL-style "assert
isinstance(...) " and other similar assertions in routines. The point
is that EAFP conflicts with the interest of reporting errors as soon
as possible (on which much has been written see, for instance Ch. 8 -
Defensive Programming in Code Complete), but LBYL conflicts with
correctness when objects can be shared.

Also, look at the man page for access. I have found at least two (one
on my Linux box, another online) that essentially say "never use it."
I completely forgot about this in my last post...

David

[Bruno Desthuilliers]

Re: exception handling in complex Python programs

dbpokorny@gmail .com a écrit :

On Aug 19, 10:19 am, eliben <eli...@gmail.c omwrote:
>

>P.S. There's a common case where a method is passed a filename, to do
>something with a file (say, read data). Should the method catch the
>errors possibly thrown by open(), or leave it to the caller ?

>
You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
This would give rise to the following function:
>
def do_something(fi lename):
try:
f = open(filename)
except IOError:
return err("File %s not found" % filename)
...
>
where err is a function that generates an error object that your
application understands.

Sorry but that's IMHO totally broken.

This "error object" is useless (heck, we *do* have exceptions, don't we
???), *returning* it ruins the whole point of structured exception
handling and take us back to infamous C error code checking (which are
almost never checked), and - icing on the cake - the error message is
very possibly wrong and misleading (IOError dont necessarily mean 'file
not found'). This kind of exception "handling" manages to be worse than
no exception handling at all.

I personally think this is sloppy because you
have to couple the exception type with the function --- between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3

??? I suspect this has nothing to do with any error happening while
opening the file. NameError means the name doesn't exists in the current
namespace nor it's enclosing namespaces. Could it be possible that
open() has been removed from Py3k ?

and an IOError is thrown in the other three cases <bashes
head against keyboard>. The alternative is
>
def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
return err(...)
f = open(filename)
...

This gets even worse. race condition... Things can change between the
call to os.access and the call to open. Well-known antipattern.

or, (and this last one I actually used for a web application)
>
def do_something(fi lename):
if not os.access(filen ame,os.R_OK):
raise MyApplicationsE xceptionType("F ile not found...")

You loose all the useful information you'd have from an IOError raised
by a direct call to open...

f = open(filename)
...

.... IOError that you're still likely to see happen anyway.

The last one has the advantage that you can write a request handler
like this
>
def handle_http_req uest(...):
func = specific_handle r_func(...)
try:
response = func(...)
return response
except MyApplicationsE xceptionType as exc: #3.0 syntax
return error_response( exc,...)

If you want to raise a different exception type - which can indeed be a
sensible thing to do, depending on the context -, you can do it safely
and keep accurate informations:

def do_something(fi lename):
try:
f = open(filename)
except IOError, e
raise MyApplicationsE xceptionType(e. msg)
# could even pass whole traceback etc
# etc...

Exceptions you don't expect (i.e. bugs)

An exception you don't expect is not necessarily a bug. Try unplugging
your lan cable while writing to a socket connected to another computer...

(snip)

If you are writing a library (for instance using a file for persistent
storage), then the answer to your question is "don't catch the
exception." Clients will expect the usual exception to be thrown when
a bad file name is passed.

Indeed.

[Bruno Desthuilliers]

Re: exception handling in complex Python programs

eliben a écrit :

>
This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.

This rarely happen to be a problem in real life. At least not in mine.

Exception that can be expected (ie : IOError when dealing with files)
are usually obvious and more or less documented - or easy to figure out
(like TypeError and ValueError when trying to build an int from an
arbitrary object, KeyError when working with dicts, AttributeError when
inspecting an object, etc) from concrete use.

IOW, it's usually easy to know which exceptions you're able to deal with
at the lower level.

Any other exception is either a programming error - which needs to be
fixed, not hidden - or nothing you can deal with at the lower level - in
which case just let it propagate until some other layer above deal with
it (eventually just logging the error, displaying a user-friendly
message, and crashing if nothing else is possible).

> def do_something(fi lename):
> if not os.access(filen ame,os.R_OK):
> return err(...)
> f = open(filename)
> ...
>>

>
But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?

The above code is a perfect antipattern. It's useless (if you can't
access the file, you'll get an IOError when trying to open it anyway),
it's wrong (things may change between the call to os.access and the call
to open), and it defeats the whole point of exception handling (by
returning some kind of error object instead of using exception handling).

[Rafe]

Re: exception handling in complex Python programs

On Aug 20, 12:47 am, Fredrik Lundh <fred...@python ware.comwrote:

Rafe wrote:

Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...

>

try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)

>
so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)
>
I'd be really worried if I found that in a piece of source code I had to
maintain.
>
</F>

I'm not it was just the first exception that came to mind... It is
pretty rare that I would pass an exception in fact. Maybe as a last-
resort test in some cases.

- Rafe

[Steven D'Aprano]

Re: exception handling in complex Python programs

On Wed, 20 Aug 2008 09:23:22 -0700, dbpokorny@gmail .com wrote:

On Aug 19, 4:12 pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com .auwrote:

>On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail .com wrote:

  def do_something(fi lename):
    if not os.access(filen ame,os.R_OK):
      return err(...)
    f = open(filename)
    ...

>>
>You're running on a multitasking modern machine, right? What happens
>when some other process deletes filename, or changes its permissions,
>in the time after you check for access but before you actually open it?

>
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:
>
def do_something(fi lename):
try:
f = open(filename)
except:
<handle exception>

No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.

If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.

And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.

Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style "assert isinstance(...) "

Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.

BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:

def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.

# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.

y = some_function(x )
# y should now be between -1 and 1.
assert -1 < y < 1
do_something_wi th(y)

and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible

Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:

def do_something(fi lename):
try:
f = open(filename)
except IOError, e:
report_exceptio n(e) # use a GUI, log to a file, whatever...

How could you report the exception any earlier than immediately?




--
Steven

[Marc 'BlackJack' Rintsch]

Re: exception handling in complex Python programs

On Wed, 20 Aug 2008 18:37:02 +0200, Bruno Desthuilliers wrote:

>I personally think this is sloppy because you have to couple the
>exception type with the function --- between file() and open() in
>Python 2 and 3, a NameError is thrown with open() in Python 3

>
??? I suspect this has nothing to do with any error happening while
opening the file. NameError means the name doesn't exists in the current
namespace nor it's enclosing namespaces. Could it be possible that
open() has been removed from Py3k ?

No it's moved/changed but there's still a name for it in the builtin
namespace. `file` on the other hand is gone:

Python 3.0b2 (r30b2:65080, Aug 20 2008, 20:41:17)
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.

>>open

<class 'io.OpenWrapper '>

>>file

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'file' is not defined

Ciao,
Marc 'BlackJack' Rintsch