Comparing Files - quickly and definitively

[Smithers]

I would appreciate some recommendations for programmaticall y determining if
files differ.

I'm writing a utility that backs up files that customers upload to Web
sites. Rather than mindlessly copying any/all files from each Web site to
the backup server (and wasting space), I'm looking to copy only files that
have been modified since the last backup took place. The files include
anything from PDF to GIF/JPG to XML, text, etc. Max size is currently under
5MB, but that could be increased later depending on customer demand.

I understand that I can look to the LastModified date or other file
properties, but I would prefer something more reliable. By "more reliable" I
mean this: I have noticed that the time can differ by a couple of seconds
after copying a file from one server to another. If the logic were to
compare using those date/times, we would expect "false positives" - files
that appear to be newer (different) based on Date/Time, but are in fact no
different. At least this scenario would happen if the logic looked to the
last backup (on the backup server) and compared against the current file on
a Web server.

So I'm thinking that there may be a more reliable way to determine if the
file content is actually different. While it would be a no-brainer to open
each file and compare the contents, that could be a rather costly
operation - given the large number of files to potentially compare, and
their potential large sizes.

So I'm looking for a reliable means through which to determine which files
have, in fact, been changed - and make that determination with fast
performance.

Suggestions? Ideas?

Thanks!

-S

[Peter Duniho]

Re: Comparing Files - quickly and definitively

Smithers wrote:

[...]
So I'm looking for a reliable means through which to determine which files
have, in fact, been changed - and make that determination with fast
performance.

Depends on your definition of "reliable". Many backup programs use only
the filename, size, and modified date to determine whether the file has
changed. Some even just use the archive bit. When they use these
things, they make sure that they copy not only the file but also the
file attributes they are checking. So if you are relying on the
modified date, for example, you'd have to copy the modified date too (I
know that the Windows Explorer does this when copying the files by hand).

But since these things aren't actually tied the actual file contents,
they aren't actually 100% reliable, though they often are "good enough".

If you really want to know whether the file is different, you have to
compare it somehow. A common method would be to generate and store an
MD5 hash on the file, and then generate the same hash for the file that
is eligible for copying. If the hash is the same, don't copy.

Of course, you would check the file size first, since that's a quick way
to know for sure if the files are different. :)

There is a theoretical possibility of hash collisions even using that
technique, so technically speaking it's not 100% reliable. But it's far
more reliable than looking just at date and file size, and is probably
good enough for almost any real-world application.

Pete

[Smithers]

Re: Comparing Files - quickly and definitively

Thanks Pete - hadn't thought about the hashing alternatives.

"good enough" is criteria I can live with on this. An occasional false
positive won't be the end of the world. It would simply mean that we archive
a file unnecessarily. No big deal. I think I'll go with a comparison of the
date/times after all, do a bunch of testing, and if there are very few false
positives, then we'll be done. We can go with more involved analyses and
possibly hashing if we need to tighten things up later.

-S


"Peter Duniho" <NpOeStPeAdM@Nn OwSlPiAnMk.comw rote in message
news:13fapot1bn hjjc9@corp.supe rnews.com...

Smithers wrote:

>[...]
>So I'm looking for a reliable means through which to determine which
>files have, in fact, been changed - and make that determination with fast
>performance.

>
Depends on your definition of "reliable". Many backup programs use only
the filename, size, and modified date to determine whether the file has
changed. Some even just use the archive bit. When they use these things,
they make sure that they copy not only the file but also the file
attributes they are checking. So if you are relying on the modified date,
for example, you'd have to copy the modified date too (I know that the
Windows Explorer does this when copying the files by hand).
>
But since these things aren't actually tied the actual file contents, they
aren't actually 100% reliable, though they often are "good enough".
>
If you really want to know whether the file is different, you have to
compare it somehow. A common method would be to generate and store an MD5
hash on the file, and then generate the same hash for the file that is
eligible for copying. If the hash is the same, don't copy.
>
Of course, you would check the file size first, since that's a quick way
to know for sure if the files are different. :)
>
There is a theoretical possibility of hash collisions even using that
technique, so technically speaking it's not 100% reliable. But it's far
more reliable than looking just at date and file size, and is probably
good enough for almost any real-world application.
>
Pete