Huge data file

[CSharper]

I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?
Thanks.

[rossum]

Re: Huge data file

On Wed, 23 Jul 2008 13:32:16 -0700 (PDT), CSharper <csharper@gmx.c om>
wrote:

>I have a data file which is 2.1GB (zipped), when the exe runs, it
>copies the content of the zip file to a destination folder. What is
>the best way to achive this? Current implementation is to zip the file
>at the source and at the destination unzip the file and if the file
>dates are different then copy the files over. Is there a better way to
>do this other than zipping and unzipping?
>Thanks.

When you create the big zipped file, also create a very small file at
the same time at the source. The small file contains a date/copy
number/serial number/whatever identifying the current iteration of the
large file.

Copy the small file to the destination first and check its date/etc.
against the date/etc. of the current version of the small file. If
they are different then update the small file and copy over the large
file. This avoids copying over the large file unless you have to -
the small file tells you enough to know whether there is an updated
version of the large file without having to copy the whole thing over
the network.

Zipping is a good way to reduce network load when you do have to get a
new copy of the large file.

A more radical solution would be for the update file to merely contain
the changes from the last time, though this might be too much effort
to set up from scratch. Probably easier to use a database at both
ends and to use the built-in synchronisation routines.

rossum

[=?ISO-8859-1?Q?Arne_Vajh=F8j?=]

Re: Huge data file

CSharper wrote:

I have a data file which is 2.1GB (zipped), when the exe runs, it
copies the content of the zip file to a destination folder. What is
the best way to achive this? Current implementation is to zip the file
at the source and at the destination unzip the file and if the file
dates are different then copy the files over. Is there a better way to
do this other than zipping and unzipping?

You can not avoid zipping and unzipping if you use a zip file.

You can avoid any intermediate files, if your app checks
dates and if necessary extract the file directly to final
destination.

Arne

[Marc Gravell]

Re: Huge data file

Can I ask what is in the data file? Not specifics - but is it
* an archive of lots of other files
* raw binary data
* serialized .NET entities, such as xml

In the first bullet, you could perhaps read the archive with tools like
#ZipLib and only extract / copy the interesting files - but note that
you are on the limit of what this will work with. Alternatively, tools
like "robocopy" will work on the *extracted* files (not the single
archive) to efficiently copy on the changed data; you can use
OS/file-system compression rather a zip to allow this to work happily.

In the second bullet; well - that is tricky; the data is likely to be
fairly dense, so I'd be surprised if it compressed very well anyway...

In the third bullet - perhaps try an alternative data format to reduce
space? I have lots of current ideas in this area if you are interested...

Marc

[Ken Foskey]

Re: Huge data file

On Wed, 23 Jul 2008 13:32:16 -0700, CSharper wrote:

I have a data file which is 2.1GB (zipped), when the exe runs, it copies
the content of the zip file to a destination folder. What is the best
way to achive this? Current implementation is to zip the file at the
source and at the destination unzip the file and if the file dates are
different then copy the files over. Is there a better way to do this
other than zipping and unzipping? Thanks.

You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.

If you are talking a Database (eg Access) then look at replication.

Ta
Ken

[CSharper]

Re: Huge data file

On Jul 24, 6:14 am, Ken Foskey <rmove.fos...@o ptushome.com.au wrote:

On Wed, 23 Jul 2008 13:32:16 -0700, CSharper wrote:

I have a data file which is 2.1GB (zipped), when the exe runs, it copies
the content of the zip file to a destination folder. What is the best
way to achive this? Current implementation is to zip the file at the
source and at the destination unzip the file and if the file dates are
different then copy the files over. Is there a better way to do this
other than zipping and unzipping? Thanks.

>
You could look at the application rsync to allow synchronisation of files
that only change a little bit each time.
>
If you are talking a Database (eg Access) then look at replication.
>
Ta
Ken

Thanks to you all. I really appriciate your comments.