Intense CPU strain....suggestions

[cbmeeks]

I am writing YAPSS (Yet Another Photo Sharing Site).

Currently, the user can upload up to six pictures at once. When the
uploads are complete, I then have a script that will resize the pix to
different thumbnails for speed. However, imagine resizing 6 pictures
of 2300x2200 pixels in 6 different sizes (maintaining the best quality
I can) on a Pentium 3 450Mhz running Ubuntu and 384 megs of ram.

Seriously.

Obviously, this is a test server and the real deal will be more
powerful but I don't want my users sitting around waiting on
resizing. I would prefer to add the pictures to a queue for later
processing and allow the users to go do other things on the site.

My question is, how would you recommend I kick off this script and
allow the user to browse away? I've heard people suggest using an
iframe but that sounds kludgey. I've also considered using some type
of local XML-RPC server/client and have a script make a call. But
would the client then have to sit and wait for a response?

As you can see, I am on the planning stages at this point.

I do like the idea of having a server running waiting to accept resize-
requests. That way, he could easily be moved to dedicated hardware
later on.

Thoughts/opinions welcomed!

Thanks!

cbmeeks

http://www.signaldev.com

[Geoff Berrow]

Re: Intense CPU strain....sugge stions

Message-ID: <1180559482.648 398.209620@m36g 2000hse.googleg roups.comfrom
cbmeeks contained the following:

>Currently, the user can upload up to six pictures at once. When the
>uploads are complete, I then have a script that will resize the pix to
>different thumbnails for speed. However, imagine resizing 6 pictures
>of 2300x2200 pixels in 6 different sizes (maintaining the best quality
>I can) on a Pentium 3 450Mhz running Ubuntu and 384 megs of ram.
>
>Seriously.

Why not just upload the pics and leave resizing til the pics are fetched
for the first time? After that fetch from the filesystem.



--
Geoff Berrow (put thecat out to email)
It's only Usenet, no one dies.
My opinions, not the committee's, mine.
Simple RFDs http://www.ckdog.co.uk/rfdmaker/

[cbmeeks]

Re: Intense CPU strain....sugge stions

I thought about that. The problem is, that I want to allow very high
res uploads. If the user uploaded 20 pictures, and they went to a
gallery page for the first time, it would take a while to resize.
That could leave a bad first impression.



On May 30, 7:41 pm, Geoff Berrow <blthe...@ckdog .co.ukwrote:

Message-ID: <1180559482.648 398.209620@m36g 2000hse.googleg roups.comfrom
cbmeeks contained the following:
>

Currently, the user can upload up to six pictures at once. When the
uploads are complete, I then have a script that will resize the pix to
different thumbnails for speed. However, imagine resizing 6 pictures
of 2300x2200 pixels in 6 different sizes (maintaining the best quality
I can) on a Pentium 3 450Mhz running Ubuntu and 384 megs of ram.

>

Seriously.

>
Why not just upload the pics and leave resizing til the pics are fetched
for the first time? After that fetch from the filesystem.
>
--
Geoff Berrow (put thecat out to email)
It's only Usenet, no one dies.
My opinions, not the committee's, mine.
Simple RFDshttp://www.ckdog.co.uk/rfdmaker/

[Jerry Stuckle]

Re: Intense CPU strain....sugge stions

cbmeeks wrote:

I thought about that. The problem is, that I want to allow very high
res uploads. If the user uploaded 20 pictures, and they went to a
gallery page for the first time, it would take a while to resize.
That could leave a bad first impression.
>
>
>
On May 30, 7:41 pm, Geoff Berrow <blthe...@ckdog .co.ukwrote:

>Message-ID: <1180559482.648 398.209620@m36g 2000hse.googleg roups.comfrom
>cbmeeks contained the following:
>>

>>Currently, the user can upload up to six pictures at once. When the
>>uploads are complete, I then have a script that will resize the pix to
>>different thumbnails for speed. However, imagine resizing 6 pictures
>>of 2300x2200 pixels in 6 different sizes (maintaining the best quality
>>I can) on a Pentium 3 450Mhz running Ubuntu and 384 megs of ram.
>>Seriously.

>Why not just upload the pics and leave resizing til the pics are fetched
>for the first time? After that fetch from the filesystem.
>>
>--
>Geoff Berrow (put thecat out to email)
>It's only Usenet, no one dies.
>My opinions, not the committee's, mine.
>Simple RFDshttp://www.ckdog.co.uk/rfdmaker/

>
>

Well, you could kick off a batch job to do your resizing, then return to
the user. That way it can keep processing the images. The only thing
you'll have to handle is when the user wants to display his gallery
before it's ready.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[cbmeeks]

Re: Intense CPU strain....sugge stions

Well, you could kick off a batch job to do your resizing, then return to

the user. That way it can keep processing the images. The only thing
you'll have to handle is when the user wants to display his gallery
before it's ready.

That's exactly how I want to do it. I just don't know the best way to
execute it. I like the idea of having an XML-RPC server running but I
am concerned with security (as I should be).

So, let's say I have the following code (using CI):

....
$this->Upload->do_uploads() ; // handle the actual uploading.

$this->Photos->do_resizing( ); // handle the resizing

redirect('waiti ng');
.....

"Waiting" doesn't get kicked off until after do_resizing (sorry, new
to PHP). I'm sure there is a smarter way. I'd like to avoid pop-ups,
iframes, javascript, etc if possible.

I also thought about having the "do_resizin g" simple add some records
to a db and have a program running on the server full time that checks
the db frequently. But wouldn't that be wasteful? I like the idea of
programs laying dormant until something kicks it in the pants and says
"do something!" lol

Thanks

cbmeeks

http://www.signaldev.com

[Jerry Stuckle]

Re: Intense CPU strain....sugge stions

cbmeeks wrote:

>Well, you could kick off a batch job to do your resizing, then return to
>the user. That way it can keep processing the images. The only thing
>you'll have to handle is when the user wants to display his gallery
>before it's ready.

>
>
That's exactly how I want to do it. I just don't know the best way to
execute it. I like the idea of having an XML-RPC server running but I
am concerned with security (as I should be).
>
So, let's say I have the following code (using CI):
>
...
$this->Upload->do_uploads() ; // handle the actual uploading.
>
$this->Photos->do_resizing( ); // handle the resizing
>
redirect('waiti ng');
....
>
"Waiting" doesn't get kicked off until after do_resizing (sorry, new
to PHP). I'm sure there is a smarter way. I'd like to avoid pop-ups,
iframes, javascript, etc if possible.
>
I also thought about having the "do_resizin g" simple add some records
to a db and have a program running on the server full time that checks
the db frequently. But wouldn't that be wasteful? I like the idea of
programs laying dormant until something kicks it in the pants and says
"do something!" lol
>
Thanks
>
cbmeeks

http://www.signaldev.com

>
>
>
>

First of all, the redirect won't happen until the code reaches there -
which is after your resizing. So that's normal operation.

You could have a cron job kicked off every few minutes to resize the
pictures, but again, you'll run into the problem that after uploading
the user won't be able to see the pictures he's just uploaded - at least
in various sizes.

You could use shell_exec() to start off a background process, i.e.
(assuming Unix/Linux):

shell_exec('noh up myprog file1 file2 file3 2/dev/null &');

nohup says to execute a command without hangups and no output, 2>
/dev/null throws away any error messages, and the '&' at the end says
run in the background.

You'll still run into a problem that the resizing might not occur as
fast as the user wants to display it; you'll have to be able to handle
this in your display routines.

Alternatively, you could do something like:

<body>
echo 'Resizing your images, please wait...<br>';
flush();
$this->Upload->do_uploads() ; // handle the actual uploading.
$this->Photos->do_resizing( ); // handle the resizing
echo 'Done!. <a href="nextpage. php">Click here to continue</a>';
</body>

The flush() works with most browsers if you keep the html simple - it
forces the output to the browser and the browser displays it. But if
you start to get fancy (i.e. tables, etc.), many browsers will wait for
the end of the element to actually display.

The only problem with this is, once you've sent *any* output to the
browser, you can't redirect them to another page via php (you could use
an html redirect). That's why the link.

Just some ideas.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[cbmeeks]

Re: Intense CPU strain....sugge stions

Thanks for the ideas!

I think I might have a somewhat complex system that might work.

First of all, when pictures are uploaded, I record the information in
the db. I could add a column that says "is_ready" so that they
wouldn't show in the galleries and the resize could mark them ready.

I'm really leaning towards and XML-RPC system. The XML-RPC server
could sit and wait for requests. The client ("do_upload" ) could send
a request when uploading is finished. That should take just a second
or so...hopefully.

The request could contain information on what to resize.

The XML-RPC server then could instruct the resize script to run and
then the script could mark the image ready.

I would need to make sure I knew how to run a robust XML-RPC server
capable of handling many requests.

I know this whole system seems complicated but I am trying to think
scalability. The XML-RPC server could easily be moved to any machine.

Thanks for suggestions guys.

cbmeeks

http://www.signaldev.com

On May 31, 11:40 am, Jerry Stuckle <jstuck...@attg lobal.netwrote:

cbmeeks wrote:

Well, you could kick off a batch job to do your resizing, then return to
the user. That way it can keep processing the images. The only thing
you'll have to handle is when the user wants to display his gallery
before it's ready.

>

That's exactly how I want to do it. I just don't know the best way to
execute it. I like the idea of having an XML-RPC server running but I
am concerned with security (as I should be).

>

So, let's say I have the following code (using CI):

>

...
$this->Upload->do_uploads() ; // handle the actual uploading.

>

$this->Photos->do_resizing( ); // handle the resizing

>

redirect('waiti ng');
....

>

"Waiting" doesn't get kicked off until after do_resizing (sorry, new
to PHP). I'm sure there is a smarter way. I'd like to avoid pop-ups,
iframes, javascript, etc if possible.

>

I also thought about having the "do_resizin g" simple add some records
to a db and have a program running on the server full time that checks
the db frequently. But wouldn't that be wasteful? I like the idea of
programs laying dormant until something kicks it in the pants and says
"do something!" lol

>

Thanks

>

cbmeeks
http://www.signaldev.com

>
First of all, the redirect won't happen until the code reaches there -
which is after your resizing. So that's normal operation.
>
You could have a cron job kicked off every few minutes to resize the
pictures, but again, you'll run into the problem that after uploading
the user won't be able to see the pictures he's just uploaded - at least
in various sizes.
>
You could use shell_exec() to start off a background process, i.e.
(assuming Unix/Linux):
>
shell_exec('noh up myprog file1 file2 file3 2/dev/null &');
>
nohup says to execute a command without hangups and no output, 2>
/dev/null throws away any error messages, and the '&' at the end says
run in the background.
>
You'll still run into a problem that the resizing might not occur as
fast as the user wants to display it; you'll have to be able to handle
this in your display routines.
>
Alternatively, you could do something like:
>
<body>
echo 'Resizing your images, please wait...<br>';
flush();
$this->Upload->do_uploads() ; // handle the actual uploading.
$this->Photos->do_resizing( ); // handle the resizing
echo 'Done!. <a href="nextpage. php">Click here to continue</a>';
</body>
>
The flush() works with most browsers if you keep the html simple - it
forces the output to the browser and the browser displays it. But if
you start to get fancy (i.e. tables, etc.), many browsers will wait for
the end of the element to actually display.
>
The only problem with this is, once you've sent *any* output to the
browser, you can't redirect them to another page via php (you could use
an html redirect). That's why the link.
>
Just some ideas.
>
--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstuck...@attgl obal.net
=============== ===