canonical file access pattern?

[Hans-Joachim Widmaier]

Recently, there was mentioned how someone who had understood Python's
error handling would write the "open and read file with error handling"
idiom. If I remember correctly, it went like this:

try:
f = file(filename, op)
except IOError, e:
# Handle exception
else:
# Use the file
f.close()

I don't think that's sufficient. First of all, opening a file is only
part of the job--reading and writing can go wrong too, and these are
not handled by the above code. Then I've seen OSError as well as IOError
raised occassionaly.

Seems like I have do this, then:

try:
f = file(filename, op)
except (OSError, IOError), e:
# Handle open() error
else:
try:
# read/write to the file
f.close() # flushes write buffers, so can fail, too
except (OSError, IOError), e:
# Handle read/write errors

Is it really so wrong, then, to fold the read/write operations and close()
into the first try suite and handle all those errors in one go?
I'd just like to have a definitive pattern that I can stick to that is
considered bullet-proof.

Hans-Joachim Widmaier

[Rene Pijlman]

Re: canonical file access pattern?

Hans-Joachim Widmaier:[color=blue]
>Then I've seen OSError as well as IOError raised occassionaly.[/color]

The documentation says only IOError can be raised. Is there a bug?

"When a file operation fails for an I/O-related reason, the exception
IOError is raised."

http://www.python.org/doc/current/lib/bltin-file-objects.html

[color=blue]
>Seems like I have do this, then:
>
>try:
> f = file(filename, op)
>except (OSError, IOError), e:
> # Handle open() error
>else:
> try:
> # read/write to the file
> f.close() # flushes write buffers, so can fail, too
> except (OSError, IOError), e:
> # Handle read/write errors[/color]

This is not correct, since the file is not closed when an exception is
thrown by read or write.

If we assume the following:
1. A file that is opened should be closed in all cases
2. Every exception raised by read, write or close should be caught and
handled

Then this would need to be the algorithm:

try:
f = file("spam.txt" , "w")
except IOError, e:
# Handle open() error
pass
else:
try:
try:
# read/write to the file
pass
except IOError, e:
# Handle read/write errors
pass
finally:
try:
f.close()
except IOError, e:
# Handle close error
pass

Not very pretty, but I can't think of a simplification that does not
violate one of the assumptions.

--
René Pijlman

[Hans-Joachim Widmaier]

Re: canonical file access pattern?

Am Tue, 30 Dec 2003 12:00:52 +0100 schrieb Rene Pijlman:
[color=blue]
> The documentation says only IOError can be raised. Is there a bug?
>
> "When a file operation fails for an I/O-related reason, the exception
> IOError is raised."[/color]

Apart from the "usual suspects" like file does not exist or is write
protected, errors aren't that common, so you don't see them all the time.
Maybe my memory is failing, but I have this distinct recollection that I
once used a 'except IOError' and got a OSError, much to my surprise. But
then, this has been quite a while ago. Python was much younger
then, I was still a fledgling Pythonist, and maybe I attributed the
error to the wrong operation ...
[color=blue]
> This is not correct, since the file is not closed when an exception is
> thrown by read or write.[/color]

Oh, sorry. I silently assumed this to be done in the exception handling
suite. Really should have made that explicit.
[color=blue]
> If we assume the following:
> 1. A file that is opened should be closed in all cases 2. Every
> exception raised by read, write or close should be caught and handled
>
> Then this would need to be the algorithm:
>
> try:
> f = file("spam.txt" , "w")
> except IOError, e:
> # Handle open() error
> pass
> else:
> try:
> try:
> # read/write to the file
> pass
> except IOError, e:
> # Handle read/write errors
> pass
> finally:
> try:
> f.close()
> except IOError, e:
> # Handle close error
> pass[/color]

While my first thought was: Why is the finally needed here? The close()
might just as well be the end of the else-suit. But it slowly dawns on me:
If there is another exception than IOError in the read/write suite,
closing of the file wouldn't take place.
[color=blue]
> Not very pretty, but I can't think of a simplification that does not
> violate one of the assumptions.[/color]

Yes, I would second that. Alas, this means that the wonderful pattern

for line in file(filename, "r"):
process_line(li ne)

is unusable for any "production quality" program. ;-(
It certainly is ok for throw-away one-liners and (what I do often!)
interactive file converters.

Thanks for elaborating.

Hans-Joachim Widmaier

[John J. Lee]

Re: canonical file access pattern?

Hans-Joachim Widmaier <hjwidmaier@web .de> writes:
[color=blue]
> Am Tue, 30 Dec 2003 12:00:52 +0100 schrieb Rene Pijlman:[/color]
[...][color=blue][color=green]
> > Then this would need to be the algorithm:
> >
> > try:
> > f = file("spam.txt" , "w")
> > except IOError, e:
> > # Handle open() error
> > pass
> > else:
> > try:
> > try:
> > # read/write to the file
> > pass
> > except IOError, e:
> > # Handle read/write errors
> > pass
> > finally:
> > try:
> > f.close()
> > except IOError, e:
> > # Handle close error
> > pass[/color]
>
> While my first thought was: Why is the finally needed here? The close()
> might just as well be the end of the else-suit. But it slowly dawns on me:
> If there is another exception than IOError in the read/write suite,
> closing of the file wouldn't take place.
>[color=green]
> > Not very pretty, but I can't think of a simplification that does not
> > violate one of the assumptions.[/color]
>
> Yes, I would second that. Alas, this means that the wonderful pattern
>
> for line in file(filename, "r"):
> process_line(li ne)
>
> is unusable for any "production quality" program. ;-([/color]
[...]

If that were true, why did exceptions get invented? If you don't need
to do something different in all those except: clauses, then don't put
them in. As you know, the nice thing about exceptions is that you're
allowed to handle them in sensible places, so typical usage is like
this:

def frob(filename):
f = file(filename, "w")
try:
# read/write to the file
finally:
f.close()

def blah():
try:
frob(filename)
except IOError, e:
# try something else, or
print e.strerror


or, even better:

def blah():
frob(filename)


Shock, horror, where's the except IOError:, you ask?! It's in the
calling function, of course.

Also, if you want to catch both OSError and IOError, note that
EnvironmentErro r is their common base class.


John

[Hans-Joachim Widmaier]

Re: canonical file access pattern?

Am Wed, 31 Dec 2003 14:51:01 +0000 schrieb John J. Lee:

[color=blue]
> If that were true, why did exceptions get invented? If you don't need
> to do something different in all those except: clauses, then don't put
> them in.[/color]

One thing I need is to create an informative and useful error message
[color=blue]
> As you know, the nice thing about exceptions is that you're
> allowed to handle them in sensible places, so typical usage is like
> this:[/color]

[handling exceptions in the caller]
[color=blue]
> Shock, horror, where's the except IOError:, you ask?! It's in the
> calling function, of course.[/color]

Hmm. The farther away from the actual point of error you catch it, the
harder it is to tell exactly what happened and to redo/work around it.
[color=blue]
> Also, if you want to catch both OSError and IOError, note that
> EnvironmentErro r is their common base class.[/color]

Good tip! I didn't know that before.

Hans-Joachim Widmaier

[John J. Lee]

Re: canonical file access pattern?

Hans-Joachim Widmaier <hjwidmaier@web .de> writes:[color=blue]
> Am Wed, 31 Dec 2003 14:51:01 +0000 schrieb John J. Lee:[/color]
[...][color=blue][color=green]
> > If that were true, why did exceptions get invented? If you don't need
> > to do something different in all those except: clauses, then don't put
> > them in.[/color]
>
> One thing I need is to create an informative and useful error message[/color]

..strerror?

[...][color=blue]
> Hmm. The farther away from the actual point of error you catch it, the
> harder it is to tell exactly what happened and to redo/work around it.[/color]
[...]

So in some cases, you have fewer except statments, in others, more.
There's really no "canonical pattern" to it.


John