File() too slow

[RG]

Re: File() too slow


"Timo Henke" <webmaster@fli7 e.de> wrote in message
news:bl13ds$iv1 $01$1@news.t-online.com...[color=blue][color=green]
> > Tried that and it didn't work.
> > Thanks anyway.
> > Anyone else?
> > RG[/color]
>
> Just another suggestion : try to upload the file in GZ Format (if
> possible).
>
> This would cut down transfer and filesize to a minimum and could easily
> with NO TIME unpacked by PHPs internal gz functions.
>
> My previously mentioned 19MB testfile got shrinked down to 351.164 bytes.
>
> You get the point?
>
> timo
>[/color]


Think I'm stuck, I can't trust clients with no knowledge to do this
Thanks though
RG

[Shawn Wilson]

Re: File() too slow

RG wrote:[color=blue]
>
> "Timo Henke" <webmaster@fli7 e.de> wrote in message
> news:bl13ds$iv1 $01$1@news.t-online.com...[color=green][color=darkred]
> > > Tried that and it didn't work.
> > > Thanks anyway.
> > > Anyone else?
> > > RG[/color]
> >
> > Just another suggestion : try to upload the file in GZ Format (if
> > possible).
> >
> > This would cut down transfer and filesize to a minimum and could easily
> > with NO TIME unpacked by PHPs internal gz functions.
> >
> > My previously mentioned 19MB testfile got shrinked down to 351.164 bytes.
> >
> > You get the point?
> >
> > timo
> >[/color]
>
> Think I'm stuck, I can't trust clients with no knowledge to do this
> Thanks though
> RG[/color]

You could probably write a tiny program in VB or something else that would split
a CSV file (file.csv) into 2MB chunks (file1.csv, file2.csv, file3.csv ...). It
would almost definitely be simple enough for your clients to use.

Shawn
--
Shawn Wilson
shawn@glassgian t.com

http://www.glassgiant.com

[RG]

Re: File() too slow


"Shawn Wilson" <shawn@glassgia nt.com> wrote in message
news:3F743043.F 40BDB6@glassgia nt.com...[color=blue]
> RG wrote:[color=green]
> >
> > "Timo Henke" <webmaster@fli7 e.de> wrote in message
> > news:bl13ds$iv1 $01$1@news.t-online.com...[color=darkred]
> > > > Tried that and it didn't work.
> > > > Thanks anyway.
> > > > Anyone else?
> > > > RG
> > >
> > > Just another suggestion : try to upload the file in GZ Format (if
> > > possible).
> > >
> > > This would cut down transfer and filesize to a minimum and could[/color][/color][/color]
easily[color=blue][color=green][color=darkred]
> > > with NO TIME unpacked by PHPs internal gz functions.
> > >
> > > My previously mentioned 19MB testfile got shrinked down to 351.164[/color][/color][/color]
bytes.[color=blue][color=green][color=darkred]
> > >
> > > You get the point?
> > >
> > > timo
> > >[/color]
> >
> > Think I'm stuck, I can't trust clients with no knowledge to do this
> > Thanks though
> > RG[/color]
>
> You could probably write a tiny program in VB or something else that would[/color]
split[color=blue]
> a CSV file (file.csv) into 2MB chunks (file1.csv, file2.csv, file3.csv[/color]
....). It[color=blue]
> would almost definitely be simple enough for your clients to use.
>
> Shawn[/color]


Thats not a bad idea. Would this be possible with Javascript? or maybe VB
Script?
Would ideally want the program to somehow interface with the import scripts,
so ideally, browser based.
Any ideas?
RG

[Shawn Wilson]

Re: File() too slow

RG wrote:[color=blue][color=green][color=darkred]
> > > > Just another suggestion : try to upload the file in GZ Format (if
> > > > possible).
> > > >
> > > > This would cut down transfer and filesize to a minimum and could[/color][/color]
> easily[color=green][color=darkred]
> > > > with NO TIME unpacked by PHPs internal gz functions.
> > > >
> > > > My previously mentioned 19MB testfile got shrinked down to 351.164[/color][/color]
> bytes.[color=green][color=darkred]
> > > >
> > > > You get the point?
> > > >
> > > > timo
> > > >
> > >
> > > Think I'm stuck, I can't trust clients with no knowledge to do this
> > > Thanks though
> > > RG[/color]
> >
> > You could probably write a tiny program in VB or something else that would[/color]
> split[color=green]
> > a CSV file (file.csv) into 2MB chunks (file1.csv, file2.csv, file3.csv[/color]
> ...). It[color=green]
> > would almost definitely be simple enough for your clients to use.
> >
> > Shawn[/color]
>
> Thats not a bad idea. Would this be possible with Javascript? or maybe VB
> Script?
> Would ideally want the program to somehow interface with the import scripts,
> so ideally, browser based.
> Any ideas?[/color]

JS, no for sure. VBScript - I doubt it, but I don't know VBScript. I doubt
you'll be able to do it from the browser. I think you'd almost have to write a
standalone executable and then use a browser to complete the upload normally.
It's kind of a clunky solve, but feasible. Alternately, you could tell them to
FTP it to a directory, then go to your program in a browser and select the file
to use from the FTP directory. Again, clunky... :(

Regards,
Shawn
--
Shawn Wilson
shawn@glassgian t.com

http://www.glassgiant.com

[Eto Demerzel]

Re: File() too slow

In article <3F747A12.D6D57 D9A@glassgiant. com>, Shawn Wilson's output
was...[color=blue][color=green]
> > Thats not a bad idea. Would this be possible with Javascript? or maybe VB
> > Script?
> > Would ideally want the program to somehow interface with the import scripts,
> > so ideally, browser based.
> > Any ideas?[/color]
>
> JS, no for sure. VBScript - I doubt it, but I don't know VBScript. I doubt
> you'll be able to do it from the browser. I think you'd almost have to write a
> standalone executable and then use a browser to complete the upload normally.
> It's kind of a clunky solve, but feasible. Alternately, you could tell them to
> FTP it to a directory, then go to your program in a browser and select the file
> to use from the FTP directory. Again, clunky... :(
>[/color]
This sounds like just the sort of thing java applets are useful for.

Perhaps have an applet in the web-page which selects the file, breaks it
into smaller chunks (and/or compresses), then calls a server-side script
to re-assemble the parts and then do whatever the script was supposed to
do with the file in the first place.
You might find some useful info at http://javaboutique.internet.com/ or
in comp.lang.java

[Brent Rieck]

Re: File() too slow

It sounds like you're using MySQL and if loading the data is still a
bottleneck, have you tried using the "LOAD DATA LOCAL INFILE ..." stuff?
I saw an order of magnitude improvement in import speed when I went
from manually splitting my CSVs to using LOAD DATA.

If you're also having problems uploading large files you'll also need to
set php's post_max_size and memory_limit (if it's enabled) to something
larger than the size of the largest file you expect.

RG wrote:[color=blue]
> Any suggestions: PHP with GD, Multiple MySQL databases, password protect
> directories, 20gb month[/color]

I've had good luck with pair.com.

--Brent

[Virgil Green]

Re: File() too slow

"RG" <Me@NotTellingY a.com> wrote in message
news:3f73fdc5$0 $65581$65c69314 @mercury.nildra m.net...[color=blue]
>
> "Eto Demerzel" <eto.demerzel@f ijivillage.com> wrote in message
> news:MPG.19de0a df1903d3b498971 b@news-text.blueyonder .co.uk...[color=green]
> > In article <3f73f748$0$655 79$65c69314@mer cury.nildram.ne t>, RG's output
> > was...[color=darkred]
> > > > > I think I've found the problem: upload_max_file size is 2M
> > > > > It is dieing, very poorly.
> > > > > You must have changed your php.ini?
> > > >
> > > > jupp .. changed to 32M, because i often need to transfer bigger
> > > > data in our intranet.
> > >
> > > I don't suppose there's a workaround for this?
> > > RG
> > >[/color]
> > IIRC - you can do something along the lines of:
> >
> > ini_set("upload _max_filesize", "64M");
> >
> >
> > - I understand this will only change the max size for operations in that
> > particular script - not for the whole server/virtual server.[/color]
>
>
> Tried that out but it seems that the file is uploaded before the script is
> executed which means, the file is dumped before the function is called.
> I'm sure my host wont want to change these settings, Rackshack (cheap).
> Looks like I'm gonna have to get some more expensive hosting.
> Any suggestions: PHP with GD, Multiple MySQL databases, password protect
> directories, 20gb month
> Thanks
> RG[/color]

I believe the 'ini_set("uploa d_max_filesize" , "64M");' needs to be set in
the script in which the FORM element is placed, not in the script where the
file is being processed.

- Virgil

[John Tkaczewski]

Re: File() too slow

There is already an exisitng product that has an applet that breaks
the files into smaller chunks and uploads those files to a server side
script (via Java Servlet) to re-assemble the file on the server side.
The product can also send the reconstructed file to an FTP server of
your choice. This method is great for Secure file transfers and to
by-pass proxies and firewalls on the client side.

For demos and to download an evaluation version
take a look at http://www.unlimitedftp.ca/uftps

Good Luck!

John


Eto Demerzel <eto.demerzel@f ijivillage.com> wrote in message news:<MPG.19e49 13443125b149897 1c@news-text.blueyonder .co.uk>...[color=blue]
> In article <3F747A12.D6D57 D9A@glassgiant. com>, Shawn Wilson's output
> was...[color=green][color=darkred]
> > > Thats not a bad idea. Would this be possible with Javascript? or maybe VB
> > > Script?
> > > Would ideally want the program to somehow interface with the import scripts,
> > > so ideally, browser based.
> > > Any ideas?[/color]
> >
> > JS, no for sure. VBScript - I doubt it, but I don't know VBScript. I doubt
> > you'll be able to do it from the browser. I think you'd almost have to write a
> > standalone executable and then use a browser to complete the upload normally.
> > It's kind of a clunky solve, but feasible. Alternately, you could tell them to
> > FTP it to a directory, then go to your program in a browser and select the file
> > to use from the FTP directory. Again, clunky... :(
> >[/color]
> This sounds like just the sort of thing java applets are useful for.
>
> Perhaps have an applet in the web-page which selects the file, breaks it
> into smaller chunks (and/or compresses), then calls a server-side script
> to re-assemble the parts and then do whatever the script was supposed to
> do with the file in the first place.
> You might find some useful info at http://javaboutique.internet.com/ or
> in comp.lang.java[/color]