hard disk activity

[VSmirk]

I have a task that involves knowing when a file has changed. But while
for small files this is an easy enough task, checking the modification
dates, or doing a compare on the contents, I need to be able to do this
for very large files.

Is there anything already available in Python that will allow me to
check the hard-disk itself, or that can make my routines aware when a
disk write has occurred?

Thanks for any help,

V

[Rene Pijlman]

Re: hard disk activity

VSmirk:[color=blue]
>I have a task that involves knowing when a file has changed. But while
>for small files this is an easy enough task, checking the modification
>dates,[/color]

Checking the modification time works the same way for large files. Why is
that not good enough?

What's your platform?

--
René Pijlman

[VSmirk]

Re: hard disk activity

I'm working primarily on Windows XP, but my solution needs to be cross
platform.

The problem is that I need more than the fact that a file has been
modified. I need to know what has been modified in that file.

I am needing to synchronize the file on a remote folder, and my current
solution, which simply copies the file if a date comparison or a
content comparison, becomes a bit unmanageable for very large files.
Some of the files I'm working with are hundreds of MB in size, or
larger.

So I need to skip copying a hundred MB file that has had only a few
bytes changed and instead identify which few bytes have changed and
where those changes are. I was thinking having a module that worked
below the file system level, at the device level, might be a place to
look for a solution.

[Paul Rubin]

Re: hard disk activity

"VSmirk" <vania.smirk@gm ail.com> writes:[color=blue]
> I am needing to synchronize the file on a remote folder, and my current
> solution, which simply copies the file if a date comparison or a
> content comparison, becomes a bit unmanageable for very large files.
> Some of the files I'm working with are hundreds of MB in size, or
> larger.[/color]

Why don't you look at the rsync program:

http://samba.anu.edu.au/rsync/

but for that much data, just plopping it all in a huge file is not
a great approach if you can help it. Maybe you can use a database instead.

[VSmirk]

Re: hard disk activity

I agree with you wholeheartedly, but the large files is part of the
business requirements.

Thanks for the link. I'll look into it.

V

[gene tani]

Re: hard disk activity


VSmirk wrote:[color=blue]
> I'm working primarily on Windows XP, but my solution needs to be cross
> platform.
>
> The problem is that I need more than the fact that a file has been
> modified. I need to know what has been modified in that file.
>
> I am needing to synchronize the file on a remote folder, and my current
> solution, which simply copies the file if a date comparison or a
> content comparison, becomes a bit unmanageable for very large files.
> Some of the files I'm working with are hundreds of MB in size, or
> larger.
>
> So I need to skip copying a hundred MB file that has had only a few
> bytes changed and instead identify which few bytes have changed and
> where those changes are. I was thinking having a module that worked
> below the file system level, at the device level, might be a place to
> look for a solution.[/color]

Sounds like the diff'g files part is the crux of it, look at sequence
matching libs like (don't know if they'll handle strings this big:

http://docs.python.org/lib/module-difflib.html

for watching files' last-mod flags:

http://www.amk.ca/python/simple/dirwatch.html

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/215418

http://python-fam.sourceforge.net/

http://pyinotify.sourceforge.net/

(there's a few recipes in the online cookbook, in fact)

[Martin P. Hellwig]

Re: hard disk activity

Paul Rubin wrote:[color=blue]
> "VSmirk" <vania.smirk@gm ail.com> writes:[color=green]
>> I am needing to synchronize the file on a remote folder, and my current
>> solution, which simply copies the file if a date comparison or a
>> content comparison, becomes a bit unmanageable for very large files.
>> Some of the files I'm working with are hundreds of MB in size, or
>> larger.[/color]
>
> Why don't you look at the rsync program:
>
> http://samba.anu.edu.au/rsync/
>
> but for that much data, just plopping it all in a huge file is not
> a great approach if you can help it. Maybe you can use a database instead.[/color]

Perhaps a cvs developer could also give some insight, you could check
subversion's mailinglist (check their website for more info:
http://subversion.tigris.org/)

--
mph

[VSmirk]

Re: hard disk activity

Pretty much, yeah. Except I need diffing a pair of files that exist on
opposite ends of a network, without causing the entire contents of the
file to be transferred over that network.

Now, I have the option of doing this: If I am able to determine that
(for instance) bytes 10468 to 1473 in a 849308 byte file are the only
segment that has changed, I can send that range over the network and
insert it into the right place; and then, with a downtime overnight, I
can do a file-copy synchronization to ensure there were no errors
during the day. (I'm reading this and wondering if it even makes
sense, sorry if it doesn't.)

But the trick in my mind is figuring out which specific bytes have been
written to disk. That's why I was thinking device level. Am I going
to have to work in C++ or Assembler for something like this?

Sorry if this sounds like a newbie question. I've been working with
Python long enough to know that someone out there has already solved
one or another of a really obscure problem. So I thought I'd take a
stab at it.

Thanks everyone for the great links.

V

[Paul Rubin]

Re: hard disk activity

"VSmirk" <vania.smirk@gm ail.com> writes:[color=blue]
> But the trick in my mind is figuring out which specific bytes have been
> written to disk. That's why I was thinking device level. Am I going
> to have to work in C++ or Assembler for something like this?[/color]

No, you can do it in Python. The basic idea is: locally compute a
separate checksum for (say) each 1% chunk of the file. Do the same
thing on the remote side. So for a 1GB file, you compute 100
checksums at each end, each checksum covering 10 MB. Then send the
100 checksums over the network, which is just a few kbytes. Compare
the checksums and you know which 10MB chunks have changed. For the
chunks that have changed, divide them into 100-kbyte sub-chunks and
checksum those, etc. The optimal number of chunks at each level
depends on network speed and various other things. Anyway this is
basically how rsync works.

Doing anything device level will be highly OS dependent.

[VSmirk]

Re: hard disk activity

Aweseme!!! I got as far as segmenting the large file on my own, and I
ran out of ideas. I kind of thought about checksum, but I never put
the two together.

Thanks. You've helped a lot....

V

[Paul Rubin]

Re: hard disk activity

"VSmirk" <vania.smirk@gm ail.com> writes:[color=blue]
> Aweseme!!! I got as far as segmenting the large file on my own, and I
> ran out of ideas. I kind of thought about checksum, but I never put
> the two together.
>
> Thanks. You've helped a lot....[/color]

The checksum method I described works ok if bytes change in the middle
of the file but don't get inserted (piecs of the file don't move
around). If you insert on byte in the middle of a 1GB file (so it
becomes 1GB+1 byte) then all the checksums after the middle block
change, which is no good for your purpose.

Rsync is a very clever program. Rather than re-implement its
algorithm maybe you should just install it and use it, either directly
(instead of writing a Python program) or under control of a Python
program, using os.system or the subprocess module.

[VSmirk]

Re: hard disk activity

Thanks for the head's up. I was so giddy with the simplicity of the
solution, I stopped trying to poke holes in it.

I agree with your philosophy of not "reinventin g the wheel", but I did
notice two things: First, the link you provided claims in the features
section that rsync if for *nix systems, so I am assuming I'll need a
port of it for windows systems; however looking at a Python rsync
module I found, it looks like it's just doing file-copy (which I have
already solved).

So I'm wondering if you know off-hand which windows port does this
checksum validation you outlined.

[ironkan]

Re: hard disk activity

Maybe an example will help

file A

abef | 1938 | 4bac | 0def | 8675

file B

adef | 0083 | abfd | 3356 | 2465

File A is different from file B and you want to have File A look like
File B. So do the segmentation (I have chosen ' | ' as the divide
between segments).

After that do checksums on each segment. For each segment's checksum
that differ there's a discrepancy between the two segments. So make
the changes to have one segment look like the other segment.

In this example the first segment's checksum would be the same whereas
the checksum for segments 2, 3, 4, and 5 will be different. So modify
the bits and bytes accordingly.

You may want to pursue this subject further by looking into various
error correction algorithms.

VSmirk wrote:[color=blue]
> Aweseme!!! I got as far as segmenting the large file on my own, and I
> ran out of ideas. I kind of thought about checksum, but I never put
> the two together.
>
> Thanks. You've helped a lot....
>
> V[/color]

[Paul Rubin]

Re: hard disk activity

"VSmirk" <vania.smirk@gm ail.com> writes:[color=blue]
> So I'm wondering if you know off-hand which windows port does this
> checksum validation you outlined.[/color]

I think rsync has been ported to Windows but I don't know any details.
I don't use Windows.

[Countess Katzenplatzen]

Re: hard disk activity

> So I'm wondering if you know off-hand which windows port does this[color=blue]
> checksum validation you outlined.[/color]

http://www.gaztronics.net/rsync.php is one source.

Just do a Google search for "windows rsync".