knowing when file is flushed to disk

[John Pote]

Hello,

I'm using a Python CGI script on a web server to log data from a remote site
every few minutes. I do not want to lose any data for whatever rare reason -
power outage/os crash just at the wrong moment etc. So I would like to know
when the data is actually written to disk and the file closed. At that point
I can signal deleting of the data at the remote site which has very limited
storage.

Is there some way from my Python script to know when the data is actually on
the disk. BTW server OS is Linux. Presumably calling flush() and close() on
the output file will initiate the disk write, but do they wait for the
actual disk write or immediately return leaving the OS to do the write when
it sees fit?

Any thoughts appreciated,

John

[daftspaniel@gmail.com]

Re: knowing when file is flushed to disk

John Pote wrote:

Is there some way from my Python script to know when the data is actually on
the disk. BTW server OS is Linux. Presumably calling flush() and close() on
the output file will initiate the disk write, but do they wait for the
actual disk write or immediately return leaving the OS to do the write when
it sees fit?

All you can do in Python (or similar) is call flush & close and hope
for the best :-)

There are many factors outwith the control of the language e.g.
* Library behaviour
* OS behaviour
* Hardware cache on the disk itself

That said, I've only found it an issue when a computer is under heavy
load.

Hope this helps,
Davy Mitchell

http://www.latedecember.com/sites/personal/davy/

[Neil Hodgson]

Re: knowing when file is flushed to disk

John Pote:

Is there some way from my Python script to know when the data is actually on
the disk. BTW server OS is Linux. Presumably calling flush() and close() on
the output file will initiate the disk write, but do they wait for the
actual disk write or immediately return leaving the OS to do the write when
it sees fit?

No, commonly they will schedule these operations and return quickly.
You can try os.fsync but there are no real guarantees about what that
does either. There's an amusing message from Tim Peters about this:

http://mail.zope.org/pipermail/zodb-dev/2004-July/007689.html

Neil

[John Pote]

Re: knowing when file is flushed to disk

Thanks for the replies. I guessed the situation would be flush() and trust.
The probability of a crash between flush() returning and data actually
written resulting in a trashed disk must be very small. But if you can be
certain without too much effort it's got to be a good idea, so I thought I'd
ask anyway.

How does the banking industry handle this sort of thing? Could be big bucks
if something goes wrong for them!

Thanks again,

John

[Slawomir Nowaczyk]

Re: knowing when file is flushed to disk

On Wed, 09 Aug 2006 16:13:19 +0000 (GMT)
John Pote <johnpote@bluey onder.co.ukwrot e:

#Is there some way from my Python script to know when the data is actually on
#the disk. BTW server OS is Linux. Presumably calling flush() and close() on
#the output file will initiate the disk write, but do they wait for the
#actual disk write or immediately return leaving the OS to do the write when
#it sees fit?

You may want to look into sqlite -- it is a single-file based SQL
database which is known to be extremely robust in face of problems you
describe. One of its design goals was to provide a replacement for
file storage. There is python binding http://pysqlite.org which is,
IIRC, supposed to be in stdlib for Python 2.5

That said, if your disk and/or OS is lying about the fact whether it
has actually wrote the data or not, there is not much you can do.

--
Best wishes,
Slawomir Nowaczyk
( Slawomir.Nowacz yk@cs.lth.se )

If vegetarians love animals so much, why do they eat all their food???