Using AJAX to do inline, inpage file uploads

[JMB]

Hello,

I was wondering if anyone knew of any projects extending the inline upload
progress bar to utilize an inpage image uploader with bar, without having to
refresh or go to a seperate page, nor opening a second box for display of
the progress bar.

I had been using the MegaUpload that was adapted from Raditha's script at
http://www.raditha.com/upload.php . The MegaUpload script I have been using
takes the progress bar inpage, instead of opening a new page/box for the
display of the bar.

Anyhow, here is what I have been wondering. I have projects with large
amounts of 'application' usage and Web2.0 features, I guess you can say.
This includes dynamic form submissions, inline graphical interfaces,
realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
been able to nail is inline progress bar usage with uploaded images/files.
Does anyone know about any projects currently working on this very thing
that I could contribute to? Or, see some demo code to have a better idea.

I am guessing my request to the server should be a put, but then I would
need what from there modifed to handle this? As PHP sux for it, and the
closest I have gotten is using a modified verision of OB_ functions. The
Perl solution seems more feasible, but I would not know where to begin using
the async nature of AJAX, I think sending the file via PUT method, and the
form variables(if any), submitted during this display of the bar from a
seperate thread to the server for storage.

I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
posting here was a starting point.

Thanks in advance for direction to any/all projects, as well as suggestions,
help and comments.

[Sean Berry]

Re: Using AJAX to do inline, inpage file uploads


"JMB" <XXXXXX@XXXX.XX X> wrote in message
news:i1%tf.3384 1$az4.26548@trn dny03...[color=blue]
> Hello,
>
> I was wondering if anyone knew of any projects extending the inline upload
> progress bar to utilize an inpage image uploader with bar, without having
> to refresh or go to a seperate page, nor opening a second box for display
> of the progress bar.
>
> I had been using the MegaUpload that was adapted from Raditha's script at
> http://www.raditha.com/upload.php . The MegaUpload script I have been
> using takes the progress bar inpage, instead of opening a new page/box for
> the display of the bar.
>
> Anyhow, here is what I have been wondering. I have projects with large
> amounts of 'application' usage and Web2.0 features, I guess you can say.
> This includes dynamic form submissions, inline graphical interfaces,
> realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
> been able to nail is inline progress bar usage with uploaded images/files.
> Does anyone know about any projects currently working on this very thing
> that I could contribute to? Or, see some demo code to have a better idea.
>
> I am guessing my request to the server should be a put, but then I would
> need what from there modifed to handle this? As PHP sux for it, and the
> closest I have gotten is using a modified verision of OB_ functions. The
> Perl solution seems more feasible, but I would not know where to begin
> using the async nature of AJAX, I think sending the file via PUT method,
> and the form variables(if any), submitted during this display of the bar
> from a seperate thread to the server for storage.
>
> I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
> posting here was a starting point.
>
> Thanks in advance for direction to any/all projects, as well as
> suggestions, help and comments.
>[/color]

I just did something like this a few months ago and would be happy to share
the code. I have it working in both PERL and Python. The PERL is a
modified version of the MegaUpload project and the Python version is my own.
In either case I can send you the source for it. Do you also want the
javascript? That part is pretty straight forward.

Let me know via email and I will pass it along.

[VK]

Re: Using AJAX to do inline, inpage file uploads


JMB wrote:[color=blue]
> I have projects with large
> amounts of 'application' usage and Web2.0 features, I guess you can say.
> This includes dynamic form submissions, inline graphical interfaces,
> realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
> been able to nail is inline progress bar usage with uploaded images/files.
> Does anyone know about any projects currently working on this very thing
> that I could contribute to? Or, see some demo code to have a better idea.
>
> I am guessing my request to the server should be a put, but then I would
> need what from there modifed to handle this? As PHP sux for it, and the
> closest I have gotten is using a modified verision of OB_ functions. The
> Perl solution seems more feasible, but I would not know where to begin using
> the async nature of AJAX, I think sending the file via PUT method, and the
> form variables(if any), submitted during this display of the bar from a
> seperate thread to the server for storage.
>
> I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
> posting here was a starting point.[/color]

The question of upload bar is very old (since Netscape 3.0 at least).
It is one of Perpetuum Mobile's of the Web: it cannot be solved in the
given physical conditions but still many people are trying to invent it
instead of reading physics books :-)

The limitation is build into the HTTP upload model. Unlike FTP, HTTP
upload is a start/stop event w/o intermediary states: you have no idea
about uploading data until submission starts and you still have no idea
of the actual data to handle until the upload is done.
The second crutial limitation of HTTP upload is that the data is sent
in BASE64 encoding. It means that the actual submission size is *always
bigger* than the summary of file sizes to upload. The actual growth
depends heavily on the binary structure of files to upload. Roughly it
is no less than +20% and up to +100%. This is why all endless attempts
to make progress bar based on file(s) size + bandwidth speed failed so
miserably.

So in order to have a reliable progress bar for HTTP upload you have to
have *on client-side*:
1) A mechanics to know the exact upload size of files after BASE64
encoded.
2) Per packed-sent notification from HTTP upload module.

Neither of this is possible on the existing browsers. You still *can*
write your own ActiveX / plugin to take over the whole HTTP upload
functionality (or to have an FTP uploader right on the page). Nothing
you can do by just using server-side + slient-side scripting in any
shall perform combination.

P.S. Thermodynamic laws never stopped people from trying to invent a
perpetuum mobile. I'm affraid it is the case with the progress bar
neither. :-)

P.P.S. No current browser supports PUT method except W3C Amaya. Just to
save you from of one of dead branches.

[Duncan Booth]

Re: Using AJAX to do inline, inpage file uploads

VK wrote:
[color=blue]
> The second crutial limitation of HTTP upload is that the data is sent
> in BASE64 encoding. It means that the actual submission size is *always
> bigger* than the summary of file sizes to upload. The actual growth
> depends heavily on the binary structure of files to upload. Roughly it
> is no less than +20% and up to +100%. This is why all endless attempts
> to make progress bar based on file(s) size + bandwidth speed failed so
> miserably.
>[/color]
This is rubbish.

Base64 encoding always increases the size of the data by one third. It does
not depend on the 'binary structure' of the data: quite simply 3 input
bytes are encoded as 4 printable characters.

[Jim Ley]

Re: Using AJAX to do inline, inpage file uploads

On 2 Jan 2006 03:01:27 -0800, "VK" <schools_ring@y ahoo.com> wrote:
[color=blue]
>So in order to have a reliable progress bar for HTTP upload you have to
>have *on client-side*:
>1) A mechanics to know the exact upload size of files after BASE64
>encoded.[/color]

A progress bar could reasonably be done on non-base64 encoded sizes,
the only assumption would then be that the base64 encoding is
approximately linear - that would be a sufficiently accurate approach
for what is already a very rough ready visual indicator.
[color=blue]
>2) Per packed-sent notification from HTTP upload module.[/color]

Or a callback from the server announcing "I've got N bytes", indeed
this may well be more accurate as it counts bytes recieved not sent
(some may have gone missing...)

So all that is really missing from the entire system is the total
upload size. Certainly you need a seperate communication channel from
the server to the client - a script available socket or
xmlhttprequest, or an open iframe - to send the "I've got N bytes"
messages.

It's a relatively simple job in fact to have an "uploaded 10k, 20k,30k
etc." message, indeed I implemented it one about 2001.

Jim.

[VK]

Re: Using AJAX to do inline, inpage file uploads


Duncan Booth wrote:[color=blue]
> VK wrote:
>[color=green]
> > The second crutial limitation of HTTP upload is that the data is sent
> > in BASE64 encoding. It means that the actual submission size is *always
> > bigger* than the summary of file sizes to upload. The actual growth
> > depends heavily on the binary structure of files to upload. Roughly it
> > is no less than +20% and up to +100%. This is why all endless attempts
> > to make progress bar based on file(s) size + bandwidth speed failed so
> > miserably.
> >[/color]
> This is rubbish.
>
> Base64 encoding always increases the size of the data by one third. It does
> not depend on the 'binary structure' of the data: quite simply 3 input
> bytes are encoded as 4 printable characters.[/color]

My bad! I was thinking about zipping instead of base64-encoding. The
resulting file size is never less then source size + 30%
At the same time for MIME base64 it is never strictly equal to
binarySize+bina rySize/100*30. Because of additional service data is
approx 33%.
On 5-10-more files package this "approx" transforms a strict
calculation into guessing game.
Also there is such things as i) bandwidth fluctuations and ii) thread
priority management where you have no control whatsoever. The latter is
caused by the fact that a script is just another process within the
host process (browser). Periodically system will decide that it has
something more important to do and your script can wait till another
system tick (or two). On a long run it adds up significally.
The only reliable counter would be from a flag "another packet sent"
but you have none of such.

Still no one is forbidden to invent perpetuum mobiles and I expect to
see a lot of "perfectly working models" linked soon.

[Sean Berry]

Re: Using AJAX to do inline, inpage file uploads


"JMB" <XXXXXX@XXXX.XX X> wrote in message
news:i1%tf.3384 1$az4.26548@trn dny03...[color=blue]
> Hello,
>
> I was wondering if anyone knew of any projects extending the inline upload
> progress bar to utilize an inpage image uploader with bar, without having
> to refresh or go to a seperate page, nor opening a second box for display
> of the progress bar.
>
> I had been using the MegaUpload that was adapted from Raditha's script at
> http://www.raditha.com/upload.php . The MegaUpload script I have been
> using takes the progress bar inpage, instead of opening a new page/box for
> the display of the bar.
>
> Anyhow, here is what I have been wondering. I have projects with large
> amounts of 'application' usage and Web2.0 features, I guess you can say.
> This includes dynamic form submissions, inline graphical interfaces,
> realtime chat, realtime drawing/whiteboarding, etc. The thing I have not
> been able to nail is inline progress bar usage with uploaded images/files.
> Does anyone know about any projects currently working on this very thing
> that I could contribute to? Or, see some demo code to have a better idea.
>
> I am guessing my request to the server should be a put, but then I would
> need what from there modifed to handle this? As PHP sux for it, and the
> closest I have gotten is using a modified verision of OB_ functions. The
> Perl solution seems more feasible, but I would not know where to begin
> using the async nature of AJAX, I think sending the file via PUT method,
> and the form variables(if any), submitted during this display of the bar
> from a seperate thread to the server for storage.
>
> I know the uploader is PHP/Perl, but with it being an AJAX-savy script,
> posting here was a starting point.
>
> Thanks in advance for direction to any/all projects, as well as
> suggestions, help and comments.
>[/color]

JMB,

The posts above are quite right about Base64 encoding and size fluctuations.
However, in my case I had clients who wanted to have some sort of progress
indicator. When someone has to upload a 50 magabyte file for instance they
want to know that the upload is still in progress after 1 minute.

My program works by using the CONTENT_LENGTH of the request as the total
size. Then, using AJAX, it retrieves the current size of the file on the
server and creates a progress bar based on the that.

Is it 100% accurate? NO!

Does it satisfy the customer by providing them with exactly what they
wanted? YES!

And does it actually work with a fair amount of accuracy? YES!

[JMB]

Re: Using AJAX to do inline, inpage file uploads

Hello, and thank you for all of the replies...

However, in the applications I am currently using, I am doing an inpage
upload progress that is about 90% accurate with time and rate of tranfer, so
this is already covered. In fact, a 100k image to a 5mb to(we had reason
to) a 1.8GB file, all displayed near-correct size, time-to-fin and percent
done in all speed cases tried. You just needed a small modifier on readout,
and an incremental for timeFromStart of the upload.

What I am having problem with, is using the XMLHttpRequest object and
asyncronous actions to allow for a file to be placed onto the server, have a
filename callback/src designation to be placed back onto that page(I know
how to do this, so, once again, this is not a problem), but all the while,
maintain the ability of the app to display a progress bar while uploading is
going on in the background(my bane with letting the progress bar do it's
thing).

I know that it can be done, but I hit a wall, and am not having any luck
going around it...seems like my programming attempts on this are trying to
push me through it!:)

Thanks again, and I look forward to any more replies that the group may be
willing to send,
JMB

"VK" <schools_ring@y ahoo.com> wrote in message
news:1136204448 .536681.161370@ z14g2000cwz.goo glegroups.com.. .[color=blue]
>
> Duncan Booth wrote:[color=green]
>> VK wrote:
>>[color=darkred]
>> > The second crutial limitation of HTTP upload is that the data is sent
>> > in BASE64 encoding. It means that the actual submission size is *always
>> > bigger* than the summary of file sizes to upload. The actual growth
>> > depends heavily on the binary structure of files to upload. Roughly it
>> > is no less than +20% and up to +100%. This is why all endless attempts
>> > to make progress bar based on file(s) size + bandwidth speed failed so
>> > miserably.
>> >[/color]
>> This is rubbish.
>>
>> Base64 encoding always increases the size of the data by one third. It
>> does
>> not depend on the 'binary structure' of the data: quite simply 3 input
>> bytes are encoded as 4 printable characters.[/color]
>
> My bad! I was thinking about zipping instead of base64-encoding. The
> resulting file size is never less then source size + 30%
> At the same time for MIME base64 it is never strictly equal to
> binarySize+bina rySize/100*30. Because of additional service data is
> approx 33%.
> On 5-10-more files package this "approx" transforms a strict
> calculation into guessing game.
> Also there is such things as i) bandwidth fluctuations and ii) thread
> priority management where you have no control whatsoever. The latter is
> caused by the fact that a script is just another process within the
> host process (browser). Periodically system will decide that it has
> something more important to do and your script can wait till another
> system tick (or two). On a long run it adds up significally.
> The only reliable counter would be from a flag "another packet sent"
> but you have none of such.
>
> Still no one is forbidden to invent perpetuum mobiles and I expect to
> see a lot of "perfectly working models" linked soon.
>[/color]

[VK]

Re: Using AJAX to do inline, inpage file uploads


Sean Berry wrote:[color=blue]
> The posts above are quite right about Base64 encoding and size fluctuations.
> However, in my case I had clients who wanted to have some sort of progress
> indicator. When someone has to upload a 50 magabyte file for instance they
> want to know that the upload is still in progress after 1 minute.[/color]

50Mb over HTTP ?! That's at least 15.5MB of extra base64 trash on each
upload! How about environment protection (not talking about wasted
time) ? ;-)

There are plenty of FTP modules to plug in into your page for such huge
uploads.

And on relatively small uploads the "oopsy miscalculation" can easily
be up to 50% of the predicted time.

But it is indeed possible (and useful) to have an animated "currently
uploading" sign on the page. With a bit of trickery it can be done.

[Sean Berry]

Re: Using AJAX to do inline, inpage file uploads


"VK" <schools_ring@y ahoo.com> wrote in message
news:1136224974 .634191.130860@ g49g2000cwa.goo glegroups.com.. .[color=blue]
>
> Sean Berry wrote:[color=green]
>> The posts above are quite right about Base64 encoding and size
>> fluctuations.
>> However, in my case I had clients who wanted to have some sort of
>> progress
>> indicator. When someone has to upload a 50 magabyte file for instance
>> they
>> want to know that the upload is still in progress after 1 minute.[/color]
>
> 50Mb over HTTP ?! That's at least 15.5MB of extra base64 trash on each
> upload! How about environment protection (not talking about wasted
> time) ? ;-)
>[/color]

You are preaching to the choir with this argument.

The problem is end-users. Our clients for the most part do not deal with
(nor are they themselves) the smartest people on the planet. Heck, most of
them do not have what I would consider the basic aptitude needed for
survival... but they prove me wrong every day.
[color=blue]
> There are plenty of FTP modules to plug in into your page for such huge
> uploads.
>[/color]

In my particular case the upload area is a bit more complex and has users
with different access and abilities. It also needed to have added
functionalites like email notifications for new uploads, description
requirements and a whole bunch of other crap. But, trying to teach my
clients to teach their clients how to successfully upload a file via FTP
wouldn't have been my first choice anyway.[color=blue]
> And on relatively small uploads the "oopsy miscalculation" can easily
> be up to 50% of the predicted time.
>
> But it is indeed possible (and useful) to have an animated "currently
> uploading" sign on the page. With a bit of trickery it can be done.
>[/color]

As far as the progress bar, "people" like to have something really fun and
exciting like a progress bar that tells them that something is going on. At
first, I used an animation once the upload started by changing a static
image to an animated gif or something. However, I found that most "people"
didn't understand that when they are uploading a 50 megabyte file it may
take more than ten seconds and became frustrated thinking it "locked up
again". Anyway, what really matters - in my case anyway - is that the end
product does exactly what they, the brilliant people I get to deal with,
want it to do. It gives a visual indication through a recognizable and very
familiar way of how much progress has been made.

SMB

[VK]

Re: Using AJAX to do inline, inpage file uploads


Sean Berry wrote:[color=blue]
> You are preaching to the choir with this argument.
>
> The problem is end-users. Our clients for the most part do not deal with
> (nor are they themselves) the smartest people on the planet. Heck, most of
> them do not have what I would consider the basic aptitude needed for
> survival... but they prove me wrong every day.[/color]

The problem of "sub-zero" users is well known to me either :-)
<http://groups.google.c om/group/comp.lang.javas cript/browse_frm/thread/355b3f86ed2f11f a/17aa578042f0802 2>

But it is a wrong thinking IMHO to correlate "primitive user" with
"primitive solution". Mostly it's just opposite: a very "primitive
user" requires a very sophisticated solution - but with a very simple
interface covering all internal complexity. In the above linked case I
had to end up by designing a whole new set of pseudo-inputs paired with
<input type="hidden".. . for submission. On a normal run I would try to
avoid such pervertion. :-)

HTTP binary exchange is not a tool to replace FTP. It is merely a
convenience addon to a form for relatively small attachments. Besides
the file growth because of base64 format, it has a bounch of other
implications like buffer mechanics. See for example
<http://support.microso ft.com/default.aspx?sc id=kb;en-us;329781> I
don't think that we can talk about usability improvement if
upload/download takes 1hr instead of needed 10 min - but user gets an
animation for the entire hour :-)

There are FTP plugins like say Internet Transfer control from
Microsoft. One can use them to have FTP and emulate as close as needed
(if needed) the usual <input type="file"... interface.

But everyone is the king of his own solution - so it's all just
IMHighlyHO.

[PJ]

Re: Using AJAX to do inline, inpage file uploads

I have been using a component from http://www.abcupload.com/ for years.
After creating a unique id that is appended to the action attribute on the
form that does the post for the upload, I can then use this id in get
requests to get informatation on the progress of the upload. I perform
these get requests with XHR requests on the client side and handle the
requests on the server side, where I send the data on the progress of the
upload back in xml. The javascript updates an html progress bar, and byte
count / total bytes of the upload.

This "approx" "working model" handles 80 Gig ( yes, GIG ) uploads as
accurately as my eye is able to discern.

I love people that have looked into a solution, failed to find one, and thus
determine their problem is "impossible " to solve.


"VK" <schools_ring@y ahoo.com> wrote in message
news:1136204448 .536681.161370@ z14g2000cwz.goo glegroups.com.. .[color=blue]
>
> Duncan Booth wrote:[color=green]
> > VK wrote:
> >[color=darkred]
> > > The second crutial limitation of HTTP upload is that the data is sent
> > > in BASE64 encoding. It means that the actual submission size is[/color][/color][/color]
*always[color=blue][color=green][color=darkred]
> > > bigger* than the summary of file sizes to upload. The actual growth
> > > depends heavily on the binary structure of files to upload. Roughly it
> > > is no less than +20% and up to +100%. This is why all endless attempts
> > > to make progress bar based on file(s) size + bandwidth speed failed so
> > > miserably.
> > >[/color]
> > This is rubbish.
> >
> > Base64 encoding always increases the size of the data by one third. It[/color][/color]
does[color=blue][color=green]
> > not depend on the 'binary structure' of the data: quite simply 3 input
> > bytes are encoded as 4 printable characters.[/color]
>
> My bad! I was thinking about zipping instead of base64-encoding. The
> resulting file size is never less then source size + 30%
> At the same time for MIME base64 it is never strictly equal to
> binarySize+bina rySize/100*30. Because of additional service data is
> approx 33%.
> On 5-10-more files package this "approx" transforms a strict
> calculation into guessing game.
> Also there is such things as i) bandwidth fluctuations and ii) thread
> priority management where you have no control whatsoever. The latter is
> caused by the fact that a script is just another process within the
> host process (browser). Periodically system will decide that it has
> something more important to do and your script can wait till another
> system tick (or two). On a long run it adds up significally.
> The only reliable counter would be from a flag "another packet sent"
> but you have none of such.
>
> Still no one is forbidden to invent perpetuum mobiles and I expect to
> see a lot of "perfectly working models" linked soon.
>[/color]

[VK]

Re: Using AJAX to do inline, inpage file uploads


PJ wrote:[color=blue]
> I have been using a component from http://www.abcupload.com/ for years.
> After creating a unique id that is appended to the action attribute on the
> form that does the post for the upload, I can then use this id in get
> requests to get informatation on the progress of the upload. I perform
> these get requests with XHR requests on the client side and handle the
> requests on the server side, where I send the data on the progress of the
> upload back in xml. The javascript updates an html progress bar, and byte
> count / total bytes of the upload.
>
> This "approx" "working model" handles 80 Gig ( yes, GIG ) uploads as
> accurately as my eye is able to discern.
>
> I love people that have looked into a solution, failed to find one, and thus
> determine their problem is "impossible " to solve.[/color]

That is a subject substitution. I did not say that there were not ways
to have a status report for upload/download operations in browser. You
see such report in action every time you're downloading something from
the Web.

The problem is not solvable by using only default browser tools (HTML
elements and scripting) and server responces. If there is a need to
have status report at any price, you can do it by extending
browser/server core functionality. And you can do it without privacy
exposure like in the mentioned case. There are plenty of FTP Java
applets and ActiveX controls to embed onto your page.

P.S. 80 Gigs over FTP I hope? (I did not read abcupload.com proposal in
details).
Otherwise it would be an athem to uneffectiveness .