How to except the unexpected?

[Rene Pijlman]

One of the things I dislike about Java is the need to declare exceptions
as part of an interface or class definition. But perhaps Java got this
right...

I've writen an application that uses urllib2, urlparse, robotparser and
some other modules in the battery pack. One day my app failed with an
urllib2.HTTPErr or. So I catch that. But then I get a urllib2.URLErro r, so
I catch that too. The next day, it encounters a urllib2.HTTPErr or, then a
IOError, a socket.timeout, httplib.Invalid URL,...

How do you program robustly with these modules throwing all those
different (and sometimes undocumented) exceptions at you?

A catchall seems like a bad idea, since it also catches AttributeErrors
and other bugs in the program.

--
René Pijlman

[James Stroud]

Re: How to except the unexpected?

Rene Pijlman wrote:[color=blue]
> One of the things I dislike about Java is the need to declare exceptions
> as part of an interface or class definition. But perhaps Java got this
> right...
>
> I've writen an application that uses urllib2, urlparse, robotparser and
> some other modules in the battery pack. One day my app failed with an
> urllib2.HTTPErr or. So I catch that. But then I get a urllib2.URLErro r, so
> I catch that too. The next day, it encounters a urllib2.HTTPErr or, then a
> IOError, a socket.timeout, httplib.Invalid URL,...
>
> How do you program robustly with these modules throwing all those
> different (and sometimes undocumented) exceptions at you?
>
> A catchall seems like a bad idea, since it also catches AttributeErrors
> and other bugs in the program.
>[/color]

The relevant lines of urllib2, for example, look as such:

class URLError(IOErro r):
class HTTPError(URLEr ror, addinfourl):
class GopherError(URL Error):

This suggests that catching URLError should have caught your HTTPError,
so you might have the chronology backwards above.

E.g.:

py> class BobError(Except ion): pass
....
py> class CarolError(BobE rror): pass
....
py> try:
.... raise CarolError
.... except BobError:
.... print 'got it'
....
got it


Now,

% cat httplib.py | grep -e '^\s*class'

produces the following at one point in its output:

class HTTPException(E xception):
class NotConnected(HT TPException):
class InvalidURL(HTTP Exception):
class UnknownProtocol (HTTPException) :
class UnknownTransfer Encoding(HTTPEx ception):
class UnimplementedFi leMode(HTTPExce ption):
class IncompleteRead( HTTPException):
class ImproperConnect ionState(HTTPEx ception):
class CannotSendReque st(ImproperConn ectionState):
class CannotSendHeade r(ImproperConne ctionState):
class ResponseNotRead y(ImproperConne ctionState):
class BadStatusLine(H TTPException):

Which suggests that "try: except HTTPException:" will be specific enough
as a catchall for this module.

The following, then, should catch everything you mentioned except the
socket timeout:

try:
whatever()
except URLError, HTTPException:
alternative()

But it seems to me that working with the internet as you are doing is
fraught with peril anyway.

James

[Ben Caradoc-Davies]

Re: How to except the unexpected?

James Stroud wrote:[color=blue]
> except URLError, HTTPException:[/color]

Aieee! This catches only URLError and binds the name HTTPException to
the detail of that error. You must write

except (URLError, HTTPException):

to catch both.

--
Ben Caradoc-Davies <ben@wintersun. org>

http://wintersun.org/

"Those who deny freedom to others deserve it not for themselves."
- Abraham Lincoln

[James Stroud]

Re: How to except the unexpected?

Ben Caradoc-Davies wrote:[color=blue]
> James Stroud wrote:
>[color=green]
>> except URLError, HTTPException:[/color]
>
>
> Aieee! This catches only URLError and binds the name HTTPException to
> the detail of that error. You must write
>
> except (URLError, HTTPException):
>
> to catch both.
>[/color]

Oops.

[Roy Smith]

Re: How to except the unexpected?

In article <4408db38$0$218 98$5a62ac22@per-qv1-newsreader-01.iinet.net.au >,
Ben Caradoc-Davies <ben@wintersun. org> wrote:
[color=blue]
> James Stroud wrote:[color=green]
> > except URLError, HTTPException:[/color]
>
> Aieee! This catches only URLError and binds the name HTTPException to
> the detail of that error. You must write
>
> except (URLError, HTTPException):
>
> to catch both.[/color]

This exact issue came up just within the past week or so. I think that
qualifies it as a wart, but I think it's a double wart.

It's certainly a wart that the try statement syntax allows for such
ambiguity. But, I think it's also a wart in how the exceptions were
defined. I like to create a top-level exception class to encompass all the
possible errors in a given module, then subclass that. This way, if you
want to catch anything to goes wrong in a call, you can catch the top-level
exception class without having to enumerate them all.

[Peter Hansen]

Re: How to except the unexpected?

Rene Pijlman wrote:[color=blue]
> One of the things I dislike about Java is the need to declare exceptions
> as part of an interface or class definition. But perhaps Java got this
> right...
>
> I've writen an application that uses urllib2, urlparse, robotparser and
> some other modules in the battery pack. One day my app failed with an
> urllib2.HTTPErr or. So I catch that. But then I get a urllib2.URLErro r, so
> I catch that too. The next day, it encounters a urllib2.HTTPErr or, then a
> IOError, a socket.timeout, httplib.Invalid URL,...
>
> How do you program robustly with these modules throwing all those
> different (and sometimes undocumented) exceptions at you?[/color]

I do it by not micromanaging things. Presumably if you plan to catch an
exception, you have a specific procedure in mind for handling the
problem. Maybe a retry, maybe an alternate way of attempting the same
thing? Look to the code that you are putting in those except:
statements (or that you think you want to put in them) to decide what to
do about this situation. If each type of exception will be handled in a
different manner, then you definitely want to identify each type by
looking at the source or the docs, or doing it empirically.

Most of the time there isn't a whole lot of real "handling" going on in
an exception handler, but merely something like logging and/or reporting
it onscreen in a cleaner fashion than a traceback, then failing anyway.
This is one reason Java does get it wrong: 95% of exceptions don't
need and shouldn't have special handling anyway.

Good code should probably have a very small set of real exception
handling cases, and one or two catchalls at a higher level to avoid
barfing a traceback at the user.
[color=blue]
> A catchall seems like a bad idea, since it also catches AttributeErrors
> and other bugs in the program.[/color]

Generally speaking this won't be a problem if you have your catchalls at
a fairly high level and have proper unit tests for the lower level code
which is getting called. You are doing unit testing, aren't you? ;-)

-Peter

[Steven D'Aprano]

Re: How to except the unexpected?

On Sat, 04 Mar 2006 00:10:17 +0100, Rene Pijlman wrote:
[color=blue]
> I've writen an application that uses urllib2, urlparse, robotparser and
> some other modules in the battery pack. One day my app failed with an
> urllib2.HTTPErr or. So I catch that. But then I get a urllib2.URLErro r, so
> I catch that too. The next day, it encounters a urllib2.HTTPErr or, then a
> IOError, a socket.timeout, httplib.Invalid URL,...
>
> How do you program robustly with these modules throwing all those
> different (and sometimes undocumented) exceptions at you?[/color]

How robust do you want to be? Do you want to take a leaf out of Firefox
and Windows XP by generating an error report and transmitting it back to
the program maintainer?
[color=blue]
> A catchall seems like a bad idea, since it also catches AttributeErrors
> and other bugs in the program.[/color]

ExpectedErrors = (URLError, IOError)
ErrorsThatCantH appen = (LookupError, ArithmeticError , AssertionError)

try:
process_things( )
except ExpectedErrors:
recover_from_er ror_gracefully( )
except ErrorsThatCantH appen:
print "Congratulation s! You have found a program bug!"
print "For a $327.68 reward, please send the following " \
"traceback to Professor Donald Knuth."
raise
except:
print "An unexpected error occurred."
print "This probably means the Internet is broken."
print "If the bug still occurs after fixing the Internet, " \
"it may be a program bug."
log_error()
sys.exit()



--
Steven.

[Paul Rubin]

Re: How to except the unexpected?

Steven D'Aprano <steve@REMOVETH IScyber.com.au> writes:[color=blue]
> try:
> process_things( )
> except ExpectedErrors:
> recover_from_er ror_gracefully( )
> except ErrorsThatCantH appen:
> print "Congratulation s! You have found a program bug!"
> print "For a $327.68 reward, please send the following " \
> "traceback to Professor Donald Knuth."
> raise
> except:
> print "An unexpected error occurred."
> print "This probably means the Internet is broken."[/color]

But this isn't good, it catches asynchronous exceptions like the user
hitting ctrl-C, which you might want to handle elsewhere. What you
want is a way to catch only actual exceptions raised from inside the
try block.

[Steven D'Aprano]

Re: How to except the unexpected?

On Fri, 03 Mar 2006 21:10:22 -0800, Paul Rubin wrote:
[color=blue]
> Steven D'Aprano <steve@REMOVETH IScyber.com.au> writes:[color=green]
>> try:
>> process_things( )
>> except ExpectedErrors:
>> recover_from_er ror_gracefully( )
>> except ErrorsThatCantH appen:
>> print "Congratulation s! You have found a program bug!"
>> print "For a $327.68 reward, please send the following " \
>> "traceback to Professor Donald Knuth."
>> raise
>> except:
>> print "An unexpected error occurred."
>> print "This probably means the Internet is broken."[/color]
>
> But this isn't good, it catches asynchronous exceptions like the user
> hitting ctrl-C, which you might want to handle elsewhere. What you
> want is a way to catch only actual exceptions raised from inside the
> try block.[/color]


It will only catch the KeyboardInterru pt exception if the user actually
hits ctrl-C during the time the code running inside the try block is
executing. It certainly won't catch random ctrl-Cs happening at other
times.

The way to deal with it is to add another except clause to deal with the
KeyboardInterru pt, or to have recover_from_er ror_gracefully( ) deal with
it. The design pattern still works. I don't know if it has a fancy name,
but it is easy to describe:-

catch specific known errors that you can recover from, and recover from
them whatever way you like (including, possibly, re-raising the exception
and letting higher-level code deal with it);

then catch errors that cannot possibly happen unless there is a bug,
and treat them as a bug;

and lastly catch unexpected errors that you don't know how to handle and
die gracefully.

My code wasn't meant as production level code, nor was ExpectedErrors
meant as an exhaustive list. I thought that was too obvious to need
commenting on.

Oh, in case this also wasn't obvious, Donald Knuth won't really pay
$327.68 for bugs in your Python code. He only pays for bugs in his own
code. *wink*



--
Steven.

[Paul Rubin]

Re: How to except the unexpected?

Steven D'Aprano <steve@REMOVETH IScyber.com.au> writes:[color=blue]
> The way to deal with it is to add another except clause to deal with the
> KeyboardInterru pt, or to have recover_from_er ror_gracefully( ) deal with
> it.[/color]

I think adding another except clause for KeyboardInterru pt isn't good
because maybe in Python 2.6 or 2.6 or whatever there will be some
additional exceptions like that and your code will break. For example,
proposals have floated for years of adding ways for threads to raise
exceptions in other threads.

I put up a proposal for adding an AsynchronousExc eption class to
contain all of these types of exceptions, so you can check for that.
[color=blue]
> Oh, in case this also wasn't obvious, Donald Knuth won't really pay
> $327.68 for bugs in your Python code. He only pays for bugs in his own
> code. *wink*[/color]

The solution to that one is obvious. We have to get Knuth using Python.
Anyone want to write a PEP? ;-)

[Rene Pijlman]

Re: How to except the unexpected?

Roy Smith:[color=blue]
>I like to create a top-level exception class to encompass all the
>possible errors in a given module, then subclass that. This way, if you
>want to catch anything to goes wrong in a call, you can catch the top-level
>exception class without having to enumerate them all.[/color]

What do you propose to do with exceptions from modules called by the given
module?

--
René Pijlman

[Rene Pijlman]

Re: How to except the unexpected?

James Stroud:[color=blue]
>Which suggests that "try: except HTTPException:" will be specific enough
>as a catchall for this module.
>
>The following, then, should catch everything you mentioned except the
>socket timeout:[/color]

Your conclusion may be (almost) right in this case. I just don't like this
approach. Basically this is reverse engineering the interface from the
source at the time of writing the app. Even if you get it right, it may
fail next week when someone added an exception to a module.
[color=blue]
>But it seems to me that working with the internet as you are doing is
>fraught with peril anyway.[/color]

Why? It shouldn't be.

--
René Pijlman

[Rene Pijlman]

Re: How to except the unexpected?

Peter Hansen:[color=blue]
>Good code should probably have a very small set of real exception
>handling cases, and one or two catchalls at a higher level to avoid
>barfing a traceback at the user.[/color]

Good point.
[color=blue][color=green]
>> A catchall seems like a bad idea, since it also catches AttributeErrors
>> and other bugs in the program.[/color]
>
>Generally speaking this won't be a problem if you have your catchalls at
>a fairly high level and have proper unit tests for the lower level code
>which is getting called. You are doing unit testing, aren't you? ;-)[/color]

With low coverage, yes. But unit testing isn't the answer for this
particular problem. For example, yesterday my app was surprised by an
httplib.Invalid URL since I hadn't noticed this could be raised by
robotparser (this is undocumented). If that fact goes unnoticed when
writing the exception handling, it will also go unnoticed when designing
test cases. I probably wouldn't have thought of writing a test case with a
first url with some external domain (that triggers robots.txt-fetching)
that's deemed invalid by httplib.

--
René Pijlman

[Rene Pijlman]

Re: How to except the unexpected?

Steven D'Aprano:[color=blue]
>ExpectedErro rs = (URLError, IOError)
>ErrorsThatCant Happen =
>
>try:
> process_things( )
>except ExpectedErrors:
> recover_from_er ror_gracefully( )
>except ErrorsThatCantH appen:
> print "Congratulation s! You have found a program bug!"
> print "For a $327.68 reward, please send the following " \
> "traceback to Professor Donald Knuth."
> raise
>except:
> print "An unexpected error occurred."
> print "This probably means the Internet is broken."
> print "If the bug still occurs after fixing the Internet, " \
> "it may be a program bug."
> log_error()
> sys.exit()[/color]

Yes, I think I'll do something like this. Perhaps combined with Peter's
advice to not micromanage, like so:

Reraise = (LookupError, ArithmeticError , AssertionError) # And then some

try:
process_things( )
except Reraise:
raise
except:
log_error()

--
René Pijlman

[Rene Pijlman]

Re: How to except the unexpected?

Paul Rubin <http://phr.cx@NOSPAM.i nvalid>:[color=blue]
>We have to get Knuth using Python.[/color]

Perhaps a MIX emulator and running TeXDoctest on his books will convince
him..

--
René Pijlman