File() too slow

[RG]

I am trying to process a CSV file but am having trouble with my hosts
maximum execution time of 30 seconds.
This is how the script works at the moment.
User uploads their CSV file
The script goes through the file() and writes smaller chunk files.
The script then goes through processing the smaller files, populating the
database, deleting the processed file and refreshing itself, thus starting
again.

This system works for files up to 15000 rows, but I need to be able to
process larger files.
The bottleneck is with the initial splitting, since I use the file()
function to read the entire uploaded file.

Does anyone know a quicker way to split a file into smaller chunks.

TIA
RG

[Timo Henke]

Re: File() too slow

> This system works for files up to 15000 rows, but I need to be able to[color=blue]
> process larger files.
> The bottleneck is with the initial splitting, since I use the file()
> function to read the entire uploaded file.[/color]

file() is by far NOT a bottleneck at your present Problem as it is not
any slower than reading the using fread() and splitting it up afterwards.

The Bottleneck is (seems like) your Database compare / writing stuff.

It sounds like you read in eg. 15000 lines, break them down into chunks
and then proceed all the chunks line by line, checking the database,
comparing
the database data with your cvs and then deciding on what to do (delete,
insert
update ...)

At 15000 Entries THIS will make your Script timeout, not the file()
command.

If you do not believe it, try to manually read and explode it ...

regards

timo

[RG]

Re: File() too slow


"Timo Henke" <webmaster@fli7 e.de> wrote in message
news:bl0qhl$r0u $06$1@news.t-online.com...[color=blue][color=green]
> > This system works for files up to 15000 rows, but I need to be able to
> > process larger files.
> > The bottleneck is with the initial splitting, since I use the file()
> > function to read the entire uploaded file.[/color]
>
> file() is by far NOT a bottleneck at your present Problem as it is not
> any slower than reading the using fread() and splitting it up afterwards.
>
> The Bottleneck is (seems like) your Database compare / writing stuff.
>
> It sounds like you read in eg. 15000 lines, break them down into chunks
> and then proceed all the chunks line by line, checking the database,
> comparing
> the database data with your cvs and then deciding on what to do (delete,
> insert
> update ...)
>
> At 15000 Entries THIS will make your Script timeout, not the file()
> command.
>
> If you do not believe it, try to manually read and explode it ...
>
> regards
>
> timo
>[/color]


The initial splitting of the file is the bottleneck, I do not compare
anything in this procedure.
There is not a problem with the database comparing etc, it seems to take
about 2 seconds to do around 3000 mysql queries. This is fine.

I just need a quick way to initially split the large file into smaller
files.
TIA
RG

[Timo Henke]

Re: File() too slow

> The initial splitting of the file is the bottleneck, I do not compare[color=blue]
> anything in this procedure.
> There is not a problem with the database comparing etc, it seems to take
> about 2 seconds to do around 3000 mysql queries. This is fine.[/color]

lets talk about filesizes. I JUST tried it. Reading (and splitting) a 19MB
File
with 322000 Lines took 0.39843 seconds on My Machine:


<?php

list($usec, $sec) = explode(" ",microtime ());
$start =((float)$usec + (float)$sec);

$indata = file("cvs");

list($usec, $sec) = explode(" ",microtime ());
$end =((float)$usec + (float)$sec);

printf("%.5f",$ end-$start);

?>

This IS NOT a bottleneck i believe?

Timo

[RG]

Re: File() too slow


"Timo Henke" <webmaster@fli7 e.de> wrote in message
news:bl0rja$ul9 $05$1@news.t-online.com...[color=blue][color=green]
> > The initial splitting of the file is the bottleneck, I do not compare
> > anything in this procedure.
> > There is not a problem with the database comparing etc, it seems to take
> > about 2 seconds to do around 3000 mysql queries. This is fine.[/color]
>
> lets talk about filesizes. I JUST tried it. Reading (and splitting) a 19MB
> File
> with 322000 Lines took 0.39843 seconds on My Machine:
>
>
> <?php
>
> list($usec, $sec) = explode(" ",microtime ());
> $start =((float)$usec + (float)$sec);
>
> $indata = file("cvs");
>
> list($usec, $sec) = explode(" ",microtime ());
> $end =((float)$usec + (float)$sec);
>
> printf("%.5f",$ end-$start);
>
> ?>
>
> This IS NOT a bottleneck i believe?
>
> Timo
>[/color]

Lightning fast.
I think I've found the problem: upload_max_file size is 2M
It is dieing, very poorly.
You must have changed your php.ini?
Thanks for the pointers
RG

[Timo Henke]

Re: File() too slow

> Lightning fast.

yeehaa :-)
[color=blue]
> I think I've found the problem: upload_max_file size is 2M
> It is dieing, very poorly.
> You must have changed your php.ini?[/color]

jupp .. changed to 32M, because i often need to transfer bigger
data in our intranet.
[color=blue]
> Thanks for the pointers[/color]

hope it would work out well for you

regards

timo

[RG]

Re: File() too slow


"Timo Henke" <webmaster@fli7 e.de> wrote in message
news:bl0so7$vl4 $05$1@news.t-online.com...[color=blue][color=green]
> > Lightning fast.[/color]
>
> yeehaa :-)
>[color=green]
> > I think I've found the problem: upload_max_file size is 2M
> > It is dieing, very poorly.
> > You must have changed your php.ini?[/color]
>
> jupp .. changed to 32M, because i often need to transfer bigger
> data in our intranet.
>[color=green]
> > Thanks for the pointers[/color]
>
> hope it would work out well for you
>
> regards
>
> timo
>[/color]

I don't suppose there's a workaround for this?
RG

[RG]

Re: File() too slow


"Timo Henke" <webmaster@fli7 e.de> wrote in message
news:bl0so7$vl4 $05$1@news.t-online.com...[color=blue][color=green]
> > Lightning fast.[/color]
>
> yeehaa :-)
>[color=green]
> > I think I've found the problem: upload_max_file size is 2M
> > It is dieing, very poorly.
> > You must have changed your php.ini?[/color]
>
> jupp .. changed to 32M, because i often need to transfer bigger
> data in our intranet.
>[color=green]
> > Thanks for the pointers[/color]
>
> hope it would work out well for you
>
> regards
>
> timo
>[/color]

I don't suppose there's a workaround for this?
RG

[Eto Demerzel]

Re: File() too slow

In article <3f73f748$0$655 79$65c69314@mer cury.nildram.ne t>, RG's output
was...[color=blue][color=green][color=darkred]
> > > I think I've found the problem: upload_max_file size is 2M
> > > It is dieing, very poorly.
> > > You must have changed your php.ini?[/color]
> >
> > jupp .. changed to 32M, because i often need to transfer bigger
> > data in our intranet.[/color]
>
> I don't suppose there's a workaround for this?
> RG
>[/color]
IIRC - you can do something along the lines of:

ini_set("upload _max_filesize", "64M");


- I understand this will only change the max size for operations in that
particular script - not for the whole server/virtual server.

[RG]

Re: File() too slow


"Eto Demerzel" <eto.demerzel@f ijivillage.com> wrote in message
news:MPG.19de0a df1903d3b498971 b@news-text.blueyonder .co.uk...[color=blue]
> In article <3f73f748$0$655 79$65c69314@mer cury.nildram.ne t>, RG's output
> was...[color=green][color=darkred]
> > > > I think I've found the problem: upload_max_file size is 2M
> > > > It is dieing, very poorly.
> > > > You must have changed your php.ini?
> > >
> > > jupp .. changed to 32M, because i often need to transfer bigger
> > > data in our intranet.[/color]
> >
> > I don't suppose there's a workaround for this?
> > RG
> >[/color]
> IIRC - you can do something along the lines of:
>
> ini_set("upload _max_filesize", "64M");
>
>
> - I understand this will only change the max size for operations in that
> particular script - not for the whole server/virtual server.[/color]


Tried that out but it seems that the file is uploaded before the script is
executed which means, the file is dumped before the function is called.
I'm sure my host wont want to change these settings, Rackshack (cheap).
Looks like I'm gonna have to get some more expensive hosting.
Any suggestions: PHP with GD, Multiple MySQL databases, password protect
directories, 20gb month
Thanks
RG

[Ian.H]

Re: File() too slow

On Fri, 26 Sep 2003 10:22:39 +0100, RG wrote:
[color=blue][color=green]
>> jupp .. changed to 32M, because i often need to transfer bigger data in
>> our intranet.
>>[/color][/color]


[ snip ]

[color=blue]
> I don't suppose there's a workaround for this? RG[/color]


RG,

You can put a hidden field in a form:


<input type="hidden" name="MAX_UPLOA D_SIZE" value="33554432 " />


This should "do the biz" before the form is whizzed off to the server for
processing =)


HTH.



Regards,

Ian

--
Ian.H [Design & Development]
digiServ Network - Web solutions
www.digiserv.net | irc.digiserv.ne t | forum.digiserv. net
Programming, Web design, development & hosting.

[Ian.H]

Re: File() too slow

On Fri, 26 Sep 2003 10:29:15 +0000, Ian.H wrote:
[color=blue]
> <input type="hidden" name="MAX_UPLOA D_SIZE" value="33554432 " />[/color]


Oops.. that should be:


<input type="hidden" name="MAX_FILE_ SIZE" value="33554432 " />



Regards,

Ian

--
Ian.H [Design & Development]
digiServ Network - Web solutions
www.digiserv.net | irc.digiserv.ne t | forum.digiserv. net
Programming, Web design, development & hosting.

[RG]

Re: File() too slow


"Ian.H" <ian@WINDOZEdig iserv.net> wrote in message
news:pan.2003.0 9.26.09.30.10.1 90519@hybris.di giserv.net...[color=blue]
> On Fri, 26 Sep 2003 10:22:39 +0100, RG wrote:
>[color=green][color=darkred]
> >> jupp .. changed to 32M, because i often need to transfer bigger data in
> >> our intranet.
> >>[/color][/color]
>
>
> [ snip ]
>
>[color=green]
> > I don't suppose there's a workaround for this? RG[/color]
>
>
> RG,
>
> You can put a hidden field in a form:
>
>
> <input type="hidden" name="MAX_UPLOA D_SIZE" value="33554432 " />
>
>
> This should "do the biz" before the form is whizzed off to the server for
> processing =)
>
>
> HTH.
>
>
>
> Regards,
>
> Ian[/color]


Tried that and it didn't work.
Thanks anyway.
Anyone else?
RG

[color=blue]
>[/color]

[RG]

Re: File() too slow


"Ian.H" <ian@WINDOZEdig iserv.net> wrote in message
news:pan.2003.0 9.26.09.32.36.3 86900@hybris.di giserv.net...[color=blue]
> On Fri, 26 Sep 2003 10:29:15 +0000, Ian.H wrote:
>[color=green]
> > <input type="hidden" name="MAX_UPLOA D_SIZE" value="33554432 " />[/color]
>
>
> Oops.. that should be:
>
>
> <input type="hidden" name="MAX_FILE_ SIZE" value="33554432 " />
>
>
>
> Regards,
>[/color]


Tried the later too.
Still didn't work
Thanks though, I'm stuck here.
RG

[color=blue]
>[/color]

[Timo Henke]

Re: File() too slow

> Tried that and it didn't work.[color=blue]
> Thanks anyway.
> Anyone else?
> RG[/color]

Just another suggestion : try to upload the file in GZ Format (if
possible).

This would cut down transfer and filesize to a minimum and could easily
with NO TIME unpacked by PHPs internal gz functions.

My previously mentioned 19MB testfile got shrinked down to 351.164 bytes.

You get the point?

timo