hard disk activity

[CatDude]

Re: hard disk activity

> So I'm wondering if you know off-hand which windows port does this[color=blue]
> checksum validation you outlined.[/color]

http://www.gaztronics.net/rsync.php is one source. Just do a Google search
for "windows rsync".

[VSmirk]

Re: hard disk activity

Of course that was the first thing I tried.

But what I meant to say was that at least one port, the python one,
didn't have the checksum validation that Paul was talking about, so I
was wondering if he knew of one that was faithful to the unix port of
it.

Thanks much for the links, though, and all the help.

[Terry Hancock]

Re: hard disk activity

On 13 Feb 2006 13:13:51 -0800
Paul Rubin <"http://phr.cx"@NOSPAM. invalid> wrote:[color=blue]
> "VSmirk" <vania.smirk@gm ail.com> writes:[color=green]
> > Aweseme!!! I got as far as segmenting the large file on
> > my own, and I ran out of ideas. I kind of thought about
> > checksum, but I never put the two together.
> >
> > Thanks. You've helped a lot....[/color]
>
> The checksum method I described works ok if bytes change
> in the middle of the file but don't get inserted (piecs of
> the file don't move around). If you insert on byte in the
> middle of a 1GB file (so it becomes 1GB+1 byte) then all
> the checksums after the middle block change, which is no
> good for your purpose.[/color]

But of course, the OS will (I hope) give you the exact
length of the file, so you *could* assume that the beginning
and end are the same, then work towards the middle.
Somewhere in between, when you hit the insertion point, both
will disagree, and you've found it. Same for deletion.

Of course, if *many* changes have been made to the file,
then this will break down. But then, if that's the case,
you're going to have to do an expensive transfer anyway, so
expensive analysis is justified.

In fact, you could proceed by analyzing the top and bottom
checksum lists at the point of failure -- download that
frame, do a byte by byte compare and see if you can derive
the frameshift. Then compensate, and go back to checksums
until they fail again. Actually, that will work just coming
from the beginning, too.

If instead, the region continues to be unrecognizeable to
the end of the frame, then you need the next frame anyway.

Seems like it could get pretty close to optimal (but we
probably are re-inventing rsync).

Cheers,
Terry

--
Terry Hancock (hancock@Anansi Spaceworks.com)
Anansi Spaceworks http://www.AnansiSpaceworks.com

[VSmirk]

Re: hard disk activity

Terry,

Yeah, I was sketching out a scenario much like that. It does break
things down pretty well, and that gets my file sync scenario up to much
larger files. Even if many changes are made to a file, if you keep
track of the number of bytes and checksum over from 1 to the number of
bytes different by shifting the sequence ( that is [abcd]ef, a[bced]f,
ab[cdef]), until a checksum is a match again, you should be able to
find some point where the checksums match again and you can continue up
(or down) doing only the checksums again without all the overhead.

The question in my mind that I will have to test is how much overhead
this causes.

One of the business rules underlying this task is to work with files
that are being continuously written to, say by logging systems or
database servers. This brings with it some obvious problems of file
access, but even in cases where you don't have file access issues, I am
very concerned about race conditions where one of the already-handled
blocks of data are written to. The synched copy on the remote system
now no longer represents a true image of the local file.

This is one of the reasons I was looking into a device-level solution
that would let me know when a hard disk write had occurred. One
colleagues suggested I was going to have to write assembler to do this,
and I may have to ultimately just use the solutions described here for
files that don't have locking and race-condition issues.

Regardless, it's a fun project, and I have to say this list is one of
the more polite lists I've been involved with. Thanks!

V