Creating Large zip files in php

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Arkady Renko

    #1

    Creating Large zip files in php

    Gday Guys

    I'm attempting to create zip files on the fly for some highly
    compressible, yet very large files stored on my Web server. At present
    I'm using a class from the Zend library by Eric Mueller which I've
    modified to suit my purposes. The problem occurs when I have files of
    an arbitrarily large size.. say 100mb, whereby I end up with high
    memory usage. At the moment I'm using a workaround whereby the scripts
    may use a very large amount of memory and work on no time limit but
    obviously this is something that must not be done in a production
    environment.

    What I'm hoping is that someone has written a class, or solved this by
    'streaming' a zip file as it's being created to the user to avoid high
    memory usage during the process. My understanding is that using the
    deflate algorithm, you can create sequential 'blocks' of compressed data
    which I imagine would play directly into the hands of what I'd want to
    achieve

    Time is not an issue for my clients, as the 2 minutes they may have to
    wait for the script to execute are far better than the extra 200mb they
    would have to download for an uncompressed file. (Example, reduction
    from 420mb to 78mb), however I would like to be able to handle several
    of these operations occurring at once.

    Failing that.. looks like I'll have to write something to suit my needs.
    --
    --------------------
    Arkady Renko
    Network Admin
    To email me directly
    Remove all capitals
    and underscores from
    my posting address
  • petersprc@gmail.com

    #2
    Re: Creating Large zip files in php

    Hi,

    You can use command-line zip to do this. For example, to zip a file to
    stdout in unix:

    zip - file.txt

    Here's a function to zip and stream a file on the fly in PHP using the
    zip command:

    <?

    // This function will zip and transmit a file on the
    // fly using the zip command.
    //
    // Usage example:
    //
    // zipfly('/tmp/data.txt');

    function zipfly($path)
    {
    // Try to disable the browser cache

    header('Pragma: public');
    header('Expires : 0');
    header('Cache-Control: must-revalidate, post-check=0, pre-check=0');
    header('Cache-Control: private', false);

    // Run the zip command

    $zipErrorFile = tempnam('/tmp', 'zipfly-');

    $cmd = 'zip - ' . escapeshellarg( $path) .
    ' 2' . escapeshellarg( $zipErrorFile);

    $proc = popen($cmd, 'r') or die();

    // Stream the zip data

    if (is_resource($p roc) && !feof($proc)) {
    if (($data = fread($proc, 8192)) !== false && $data !== '') {
    header('Content-Transfer-Encoding: binary');
    header('Content-Type: application/zip');
    $fileName = preg_replace('/\..*?$/', '', basename($path) ) .
    '.zip';
    header('Content-Disposition: attachment; filename="' .
    "$fileName\";") ;
    echo $data;
    if (!feof($proc)) {
    fpassthru($proc );
    }
    }
    }

    // Get the exit code of the command

    $exitCode = pclose($proc);

    if ($exitCode != 0) {
    $msg = "Command \"$cmd\" failed with exit code $exitCode";
    $errors = is_file($zipErr orFile) ? file_get_conten ts($zipErrorFil e)
    : false;
    $msg .= $errors !== false ? ": $errors" : '.';
    trigger_error($ msg);
    }
    }

    ?>

    Best Regards,

    John Peters

    Arkady Renko wrote:
    Gday Guys
    >
    I'm attempting to create zip files on the fly for some highly
    compressible, yet very large files stored on my Web server. At present
    I'm using a class from the Zend library by Eric Mueller which I've
    modified to suit my purposes. The problem occurs when I have files of
    an arbitrarily large size.. say 100mb, whereby I end up with high
    memory usage. At the moment I'm using a workaround whereby the scripts
    may use a very large amount of memory and work on no time limit but
    obviously this is something that must not be done in a production
    environment.
    >
    What I'm hoping is that someone has written a class, or solved this by
    'streaming' a zip file as it's being created to the user to avoid high
    memory usage during the process. My understanding is that using the
    deflate algorithm, you can create sequential 'blocks' of compressed data
    which I imagine would play directly into the hands of what I'd want to
    achieve
    >
    Time is not an issue for my clients, as the 2 minutes they may have to
    wait for the script to execute are far better than the extra 200mb they
    would have to download for an uncompressed file. (Example, reduction
    from 420mb to 78mb), however I would like to be able to handle several
    of these operations occurring at once.
    >
    Failing that.. looks like I'll have to write something to suit my needs.
    --
    --------------------
    Arkady Renko
    Network Admin
    To email me directly
    Remove all capitals
    and underscores from
    my posting address

    Comment

    Working...