image uploads and version control?

[lawpoop@gmail.com]

I'm developing a php website that I have under subversion version
control. I'm working on an image upload functionality, and in the
middle of it I realized that any files that a user uploads will not be
under version control, and if I checkout or export the site from
version control, and deploy it, it won't bring any of the uploaded
files with it.

I'm looking at php subversion functions, but the manual says that
they're experimental, and my webhost hasn't deployed them.

What's a workaround for this? I'm thinking that I'll have to have a
parallel site for image hosting. For example, I'll have
images.website. com, hosted under a separate filesystem directory, and
then in the actual website that a user would peruse, any img src would
reference images.website. com . When I accept my uploads, I'll do my
filesystem copy to the images filesystem link.

Thoughts?

[Charles Calvert]

Re: image uploads and version control?

On Fri, 10 Oct 2008 12:25:59 -0700 (PDT), lawpoop@gmail.c om wrote in
<36d92320-f3bf-4428-b6c4-ffb95bdd1869@s5 0g2000hsb.googl egroups.com>:

>I'm developing a php website that I have under subversion version
>control. I'm working on an image upload functionality, and in the
>middle of it I realized that any files that a user uploads will not be
>under version control, and if I checkout or export the site from
>version control, and deploy it, it won't bring any of the uploaded
>files with it.

Why would you want it to? This is user data, not data that is
actually part of the application. In any case, why would users be
interacting with a copy of the application that is a Subversion
working copy?

I think that I must have misunderstood, because it sounds like you
want to use Subversion as a packaging and distribution mechanism, and
that's not what it's for.

[snip]
--
Charles Calvert | Web-site Design/Development
Celtic Wolf, Inc. | Software Design/Development
http://www.celticwolf.com/ | Data Conversion
(703) 580-0210 | Project Management

[Willem Bogaerts]

Re: image uploads and version control?

>I'm developing a php website that I have under subversion version

>control. I'm working on an image upload functionality, and in the
>middle of it I realized that any files that a user uploads will not be
>under version control, and if I checkout or export the site from
>version control, and deploy it, it won't bring any of the uploaded
>files with it.

>
Why would you want it to? This is user data, not data that is
actually part of the application.

Exactly. Subversion is there to facilitate rolling new versions out and
develop them. What you probably want is a good backup facility. The fact
that subversion never forgets anything makes it a less-than-optimal
backup system. Off course it was never *meant* as a backup system.

Best regards,
--
Willem Bogaerts

Application smith
Kratz B.V.

http://www.kratz.nl/

[lawpoop@gmail.com]

Re: image uploads and version control?

On Oct 13, 12:06 am, Charles Calvert <cb...@yahoo.co mwrote:

I think that I must have misunderstood, because it sounds like you
want to use Subversion as a packaging and distribution mechanism, and
that's not what it's for.

Well, here's the problem, as I perceive it:

Say my website is served from a directory called /home/user/website .

Users of the website need to be able to upload images, which in the
current incarnation of the website I specified to be served /home/user/
website/images. This directory currently holds all images served with
the site, not just user uploaded images.

My 'packaging and distribution' practice currently consists of 'svn
export http://svn-host/website/trunk', and then linking that to the
website directory, e.g. 'ln -s website-trunk-revision-189 /home/user/
website'.

So the problem is, unless user uploaded images were put under version
control, which I would have to do by migrating the image from the live
to the development site, and then putting it under version control,
the user uploaded images wouldn't be in an exported revision of the
website.

So my thought would be rather than try to include them in version
control, move the functionality of accepting and serving user uploaded
images from a subdirectory of the website to its own domain, and just
not keep it under version control.

So when I export the website, it won't include any uploaded files,
unless they were put under version control.

[lawpoop@gmail.com]

Re: image uploads and version control?

On Oct 13, 4:04 am, Willem Bogaerts <w.bogae...@kra tz.nlwrote:

>
Exactly. Subversion is there to facilitate rolling new versions out and
develop them. What you probably want is a good backup facility. The fact
that subversion never forgets anything makes it a less-than-optimal
backup system. Off course it was never *meant* as a backup system.
>

Well, I guess I don't know how others roll out websites from
subversion, but I'm just using 'export', and then creating a symlink
of the svn export to the webhost directory. So the problem is, if I'm
serving user-uploaded images out of a subdirectory in the website
( e.g. /user/home/website/user-uploaded-images ), unless new uploads
are put under version control, they would not exist in an export when
I roll out a new release of the website. So I guess if you want to
call that 'backup' functionality, I see that side of it, but to me, it
isn't any different than just making sure all the pieces of your
project exist when you do an export, which I think might be an
implicit functionality of version control.

When I use svn export to roll out a new version of the site, I could
copy the user uploaded images into the image directory, but that adds
an extra step to releasing a new site, and I like to keep things
simple. I have enough to worry about when I roll out a new version. So
my thought now for user uploaded images is simply not to serve them
from within the web directory, but put up a whole other host that's
not under version control, and serve them from there.

[Jerry Stuckle]

Re: image uploads and version control?

lawpoop@gmail.c om wrote:

On Oct 13, 4:04 am, Willem Bogaerts <w.bogae...@kra tz.nlwrote:
>

>Exactly. Subversion is there to facilitate rolling new versions out and
>develop them. What you probably want is a good backup facility. The fact
>that subversion never forgets anything makes it a less-than-optimal
>backup system. Off course it was never *meant* as a backup system.
>>

>
Well, I guess I don't know how others roll out websites from
subversion, but I'm just using 'export', and then creating a symlink
of the svn export to the webhost directory. So the problem is, if I'm
serving user-uploaded images out of a subdirectory in the website
( e.g. /user/home/website/user-uploaded-images ), unless new uploads
are put under version control, they would not exist in an export when
I roll out a new release of the website. So I guess if you want to
call that 'backup' functionality, I see that side of it, but to me, it
isn't any different than just making sure all the pieces of your
project exist when you do an export, which I think might be an
implicit functionality of version control.
>
When I use svn export to roll out a new version of the site, I could
copy the user uploaded images into the image directory, but that adds
an extra step to releasing a new site, and I like to keep things
simple. I have enough to worry about when I roll out a new version. So
my thought now for user uploaded images is simply not to serve them
from within the web directory, but put up a whole other host that's
not under version control, and serve them from there.
>
>

As others have indicated - the problem is you're using subversion for
something other than what it's intended for. What you need is a good
backup strategy, not a version control system.

But in any case, this has absolutely nothing to do with PHP, so this is
the wrong newsgroup, anyway.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Charles Calvert]

Re: image uploads and version control?

On Mon, 13 Oct 2008 06:23:48 -0700 (PDT), lawpoop@gmail.c om wrote in
<7edb880b-1c8e-4060-b59c-e2ecd2d0d296@v7 2g2000hsv.googl egroups.com>:

>On Oct 13, 12:06 am, Charles Calvert <cb...@yahoo.co mwrote:
>

>I think that I must have misunderstood, because it sounds like you
>want to use Subversion as a packaging and distribution mechanism, and
>that's not what it's for.

>
>Well, here's the problem, as I perceive it:
>
>Say my website is served from a directory called /home/user/website .

Are you on a shared host? If not, then it should be /var/www/website
or something similar. I'll assume that you're on a shared host.

>Users of the website need to be able to upload images, which in the
>current incarnation of the website I specified to be served /home/user/
>website/images. This directory currently holds all images served with
>the site, not just user uploaded images.
>
>My 'packaging and distribution' practice currently consists of 'svn
>export http://svn-host/website/trunk', and then linking that to the
>website directory, e.g. 'ln -s website-trunk-revision-189 /home/user/
>website'.

I would just get the desired revision from svn and copy the source
files to the production directory. You might need to clean out the
production directory first if you've removed any files.

>So the problem is, unless user uploaded images were put under version
>control, which I would have to do by migrating the image from the live
>to the development site, and then putting it under version control,
>the user uploaded images wouldn't be in an exported revision of the
>website.

You're making this too complex. Subversion is for controling
revisions of your source code and related files (e.g. design docs),
nothing else. Once you put the application into production and have
begun accumulating user data, just copy the source files from a
working directory to the production directory using cp -r or a shell
script.
--
Charles Calvert | Web-site Design/Development
Celtic Wolf, Inc. | Software Design/Development
http://www.celticwolf.com/ | Data Conversion
(703) 580-0210 | Project Management

[lawpoop@gmail.com]

Re: image uploads and version control?

On Oct 13, 10:38 am, Jerry Stuckle <jstuck...@attg lobal.netwrote:

>
As others have indicated - the problem is you're using subversion for
something other than what it's intended for.  What you need is a good
backup strategy, not a version control system.

I'm using subversion for version control. If my website has an image
that's part of the website, it goes under version control, like any
other file that's part of the website. When I say it belongs under
version control, I mean that "Version 3.x has X version of this logo,
Version 4.x has another version".

If versioning images and other binary files are not the purview of
version control, then why does subversion have functionality for
versioning binary images, such as locking, etc.?

I'm not using it for 'backup' in the sense of "Oops! I deleted this
file accidentally. I need it to get it back, so let me go into the
backup and retrieve a copy of it. What is the definition of "backup"
that you're using here?

I don't see how this is a backup issue rather than a version control
issue. I think you might argure the point that this might be a
packaging and distribution issue as Charles Calvert did earlier, but
in this case, it isn't a c++ solution that requires a build
environment -- here, "svn export" *is* the packaging and distribution.
If you're using php and svn, what extra packaging and distribution
would you need for a site that's not released to anybody else, or
installed on any other systems?

[lawpoop@gmail.com]

Re: image uploads and version control?

On Oct 13, 11:30 am, Charles Calvert <cb...@yahoo.co mwrote:

On Mon, 13 Oct 2008 06:23:48 -0700 (PDT), lawp...@gmail.c om wrote in
<7edb880b-1c8e-4060-b59c-e2ecd2d0d...@v7 2g2000hsv.googl egroups.com>:
>

On Oct 13, 12:06 am, Charles Calvert <cb...@yahoo.co mwrote:

>

I think that I must have misunderstood, because it sounds like you
want to use Subversion as a packaging and distribution mechanism, and
that's not what it's for.

>

Well, here's the problem, as I perceive it:

>

Say my website is served from a directory called /home/user/website .

>
Are you on a shared host?  If not, then it should be /var/www/website
or something similar.  I'll assume that you're on a shared host.
>

Users of the website need to be able to upload images, which in the
current incarnation of the website I specified to be served /home/user/
website/images. This directory currently holds all images served with
the site, not just user uploaded images.

>

My 'packaging and distribution' practice currently consists of 'svn
exporthttp://svn-host/website/trunk', and then linking that to the
website directory, e.g. 'ln -s website-trunk-revision-189 /home/user/
website'.

>
I would just get the desired revision from svn and copy the source
files to the production directory.  You might need to clean out the
production directory first if you've removed any files.

Well, I can do all that in one fell swoop by creating a symbolic link.
I don't have to worry about whether or not I've deleted anything in
version control that I then need to remove in the production; it's
already done for me. If I had to delete files what were already
deleted in version control, then in my mind, I'm doing manual labor
that the computer should be doing anyway.

Also, by using symbolic links, I can deploy, test, and fallback to the
old directory by re-linking very easily if I need to.

Finally, by using an adequately named directory to symlink to, I know
what version I have deployed as production. If /home/user/website/
production is a symlink to /home/user/website/trunk-R255, then there's
a pretty good chance that it's revision 255 that is currently in
production. If I were just copying files into the website directory, I
would have to have another way to know or guess what revision was
actually deployed. I could reconcile my svn history against my bash
history, and try to extraploate what version is currently deployed,
but that takes some leg works and concluding. Of course, both ways are
subject to failure if you don't practice your procedures properly, but
I think symlinking is harder to screw up.

So, three good reasons to use svn exports in and symbolic linking to
deploy a website.

You're making this too complex. Subversion is for controling
revisions of your source code and related files (e.g. design docs),
nothing else. Once you put the application into production and have
begun accumulating user data, just copy the source files from a
working directory to the production directory using cp -r or a shell
script.

Well, I didn't *intentionally* make it to complex; my assumptions
about website design and structure sort of led me to the place that
I'm in now. :)

I'm making a dymanimc website where users can input data. The source
code of the website is under version control, and the user-generated
data is stored in MySQL. So far, so good. If I needed to re-generate
the website for any reason ( backup, new webhost, errant rm -rf ), all
I needed was a recent export from subversion and the latest MySQL
dump.

So now I think I'm beginning to understand the philosophy. In the
past, I could get away with keeping all the website images under /
images, or some similar structure. But now that I'm accepting user
data that I can't or shouldn't put into MySQL, I need to find a third
place to store things. I liked the simplicity of having the whole
website existing as svn export + mysqldump, so I was hoping there was
a way I could keep this new kind of data within that system. But I
can't. Now, the website must be mysqldump + svn + some other thing.

[Michael Fesser]

Re: image uploads and version control?

..oO(lawpoop@gm ail.com)

>On Oct 13, 11:30 am, Charles Calvert <cb...@yahoo.co mwrote:

>On Mon, 13 Oct 2008 06:23:48 -0700 (PDT), lawp...@gmail.c om wrote in
>>

>My 'packaging and distribution' practice currently consists of 'svn
>exporthttp://svn-host/website/trunk', and then linking that to the
>website directory, e.g. 'ln -s website-trunk-revision-189 /home/user/
>website'.

>>
>I would just get the desired revision from svn and copy the source
>files to the production directory.  You might need to clean out the
>production directory first if you've removed any files.

>
>Well, I can do all that in one fell swoop by creating a symbolic link.
>I don't have to worry about whether or not I've deleted anything in
>version control that I then need to remove in the production; it's
>already done for me. If I had to delete files what were already
>deleted in version control, then in my mind, I'm doing manual labor
>that the computer should be doing anyway.

That's what synchronizing is for. The computer can easily check which
files have been modified or deleted from the working copy and upload all
required changes to the production server. There are various tools
available for this task.

>Also, by using symbolic links, I can deploy, test, and fallback to the
>old directory by re-linking very easily if I need to.

This should happen on a testing server. You hardly need multiple
versions on the production server.

>Finally, by using an adequately named directory to symlink to, I know
>what version I have deployed as production. If /home/user/website/
>production is a symlink to /home/user/website/trunk-R255, then there's
>a pretty good chance that it's revision 255 that is currently in
>production. If I were just copying files into the website directory, I
>would have to have another way to know or guess what revision was
>actually deployed.

You could use tags on your repository and then simply assume that it's
always the latest tagged version which goes production. When you reach
the next stable release, tag it and deploy it. Quite easy.

Micha

[lawpoop@gmail.com]

Re: image uploads and version control?

On Oct 13, 12:38 pm, Michael Fesser <neti...@gmx.de wrote:

.oO(lawp...@gma il.com)

>

Well, I can do all that in one fell swoop by creating a symbolic link.
I don't have to worry about whether or not I've deleted anything in
version control that I then need to remove in the production; it's
already done for me. If I had to delete files what were already
deleted in version control, then in my mind, I'm doing manual labor
that the computer should be doing anyway.

>
That's what synchronizing is for. The computer can easily check which
files have been modified or deleted from the working copy and upload all
required changes to the production server. There are various tools
available for this task.

I suppose there are tools for this. Care to give me a sales pitch on
any of them? Right now an svn export does everything I want, with no
risks of the downsides of copying or synchronizing.
And the deployment tool is built right into the version control
system. Why do I need an extra tool here?

>

Also, by using symbolic links, I can deploy, test, and fallback to the
old directory by re-linking very easily if I need to.

>
This should happen on a testing server. You hardly need multiple
versions on the production server.

That would always work if the development and production environments
were identical, but in my experience, they're not. Code that passes a
test in the development environment might still have problems when you
roll it out in production. And then that happens, you'll want to roll
it back *really really badly*. :)

The situation I've been involved in before was where we made a change
to the database structure in development which we didn't replicate in
the production database. The code was developed and tested in
production against a new table structure, and it passed all its test.
Then when we went to roll it out, SQL errors all over the place. It
was quite a relief to be able to roll back the website with a simple
command. ( BTW, I'd really like a smart version control system for our
table structure, one that could do ALTER TABLEs appropriately when
deploying).

In our situation, we have a development server, and a production
server with a staging hostname and a production hostname. Once a
feature pasts testing on development, it goes out to staging. Staging
is on the production server and connects to the production database.
We do 'pre-live' testing there.

The great part about this is the seamless deployment from stating to
production. Our initial svn export gets linked to the staging
hostname. E.g.:
$ls
website-release-320
staging.website .com -website-release-320/
website.com -website-release-320/
$svn export http://svn-host/website/releases/release-345 ./website-
release-345
$rm -rf stating.website .com; ln -s ./website-release-345
staging.website .com
$ls
website-release-320
website-release-345
staging.website .com -website-release-345/
website.com -website-release-320/

Where staging.website .com was previously linked to ./website-
release-320 . Now the new code really is in the production
environment. There shouldn't be any problem in production that
wouldn't be apparent here. So, after we've done the testing at
staging, we do:

$rm -rf website.com; ln -s website-release-320 website.com
$ls
website-release-320
website-release-345
staging.website .com -website-release-345/
website.com -website-release-345/

The deployment of the new code to live takes milliseconds.

We can let the 320 revision hang around for a while; it's not being
served, and it's a "known good configuration" of the website
( provided it doesn't reference any outdated database structures :)
that we can rollback to if needed.

Finally, by using an adequately named directory to symlink to, I know
what version I have deployed as production. If /home/user/website/
production is a symlink to /home/user/website/trunk-R255, then there's
a pretty good chance that it's revision 255 that is currently in
production. If I were just copying files into the website directory, I
would have to have another way to know or guess what revision was
actually deployed.

>
You could use tags on your repository and then simply assume that it's
always the latest tagged version which goes production. When you reach
the next stable release, tag it and deploy it. Quite easy.

That's a good idea. We're suing subversion, which doesn't have tags,
but the manual says you get the same functionality by doing a release
branch.

Knowing myself and my computing habits, I wouldn't like to assume that
a directory is the latest tagged released; it may not be. ( My most
difficult bash-head-against-wall troubleshooting cases are always
faulty assumptions ). By creating a directory named trunk-R273 or
release-R503, then there's a greater likelihood that that directory
actually *contains* revision 503. Not that there's anything magical in
naming it as such, but hopefully when I'm improperly naming a
directory ( e.g. svn export http://svn-host/releases/R502 ./website-
release-R498 ), the cognitive dissonance becomes apparent, and I stop
and say "Hey, what am I doing? This is *wrong*!" For me at least,
there's a lesser likelihood of that happening when I'm doing export to
the generic directory name that we've been using for years.

[Charles Calvert]

Re: image uploads and version control?

On Mon, 13 Oct 2008 08:58:12 -0700 (PDT), lawpoop@gmail.c om wrote in
<125c3f5e-df0d-4cf0-9329-9813802bc663@k7 g2000hsd.google groups.com>:

>On Oct 13, 11:30 am, Charles Calvert <cb...@yahoo.co mwrote:

>On Mon, 13 Oct 2008 06:23:48 -0700 (PDT), lawp...@gmail.c om wrote in
><7edb880b-1c8e-4060-b59c-e2ecd2d0d...@v7 2g2000hsv.googl egroups.com>:
>>

>On Oct 13, 12:06 am, Charles Calvert <cb...@yahoo.co mwrote:

>>

>I think that I must have misunderstood, because it sounds like you
>want to use Subversion as a packaging and distribution mechanism, and
>that's not what it's for.

>>

>Well, here's the problem, as I perceive it:

[snip]

>Users of the website need to be able to upload images, which in the
>current incarnation of the website I specified to be served /home/user/
>website/images. This directory currently holds all images served with
>the site, not just user uploaded images.

>>

>My 'packaging and distribution' practice currently consists of 'svn
>exporthttp://svn-host/website/trunk', and then linking that to the
>website directory, e.g. 'ln -s website-trunk-revision-189 /home/user/
>website'.

>>
>I would just get the desired revision from svn and copy the source
>files to the production directory.  You might need to clean out the
>production directory first if you've removed any files.

>
>Well, I can do all that in one fell swoop by creating a symbolic link.
>I don't have to worry about whether or not I've deleted anything in
>version control that I then need to remove in the production; it's
>already done for me. If I had to delete files what were already
>deleted in version control, then in my mind, I'm doing manual labor
>that the computer should be doing anyway.
>
>Also, by using symbolic links, I can deploy, test, and fallback to the
>old directory by re-linking very easily if I need to.
>
>Finally, by using an adequately named directory to symlink to, I know
>what version I have deployed as production. If /home/user/website/
>production is a symlink to /home/user/website/trunk-R255, then there's
>a pretty good chance that it's revision 255 that is currently in
>production.

This was the part that I was missing. It sounded like the working
copy was being used for production. You're not doing that. What you
are doing sounds reasonable.

[snip]

>You're making this too complex. Subversion is for controling
>revisions of your source code and related files (e.g. design docs),
>nothing else. Once you put the application into production and have
>begun accumulating user data, just copy the source files from a
>working directory to the production directory using cp -r or a shell
>script.

>
>Well, I didn't *intentionally* make it to complex; my assumptions
>about website design and structure sort of led me to the place that
>I'm in now. :)
>
>I'm making a dymanimc website where users can input data. The source
>code of the website is under version control, and the user-generated
>data is stored in MySQL. So far, so good. If I needed to re-generate
>the website for any reason ( backup, new webhost, errant rm -rf ), all
>I needed was a recent export from subversion and the latest MySQL
>dump.

Check. That's the same strategy I use.

>So now I think I'm beginning to understand the philosophy. In the
>past, I could get away with keeping all the website images under /
>images, or some similar structure.

Assuming that they are part of the application and not user content,
that makes perfect sense. After all, version control can work for
images too.

>But now that I'm accepting user data that I can't or shouldn't put
>into MySQL, I need to find a third place to store things. I liked
>the simplicity of having the whole website existing as svn export +
>mysqldump, so I was hoping there was a way I could keep this new
>kind of data within that system. But I can't. Now, the website must
>be mysqldump + svn + some other thing.

Right. You could combine the mysqldump and the back-up of the
extradatabase user content into a single script and just tar
everything up together into a single back-up file. That should be a
fairly simple shell script, perhaps with the revision number as a
parameter that can be appended to the name of the tarball, much like
you're doing with the directories.
--
Charles Calvert | Web-site Design/Development
Celtic Wolf, Inc. | Software Design/Development
http://www.celticwolf.com/ | Data Conversion
(703) 580-0210 | Project Management

[AqD]

Re: image uploads and version control?

On Oct 11, 3:25 am, lawp...@gmail.c om wrote:

I'm developing a php website that I have under subversion version
control. I'm working on an image upload functionality, and in the
middle of it I realized that any files that a user uploads will not be
under version control, and if I checkout or export the site from
version control, and deploy it, it won't bring any of the uploaded
files with it.
>
I'm looking at php subversion functions, but the manual says that
they're experimental, and my webhost hasn't deployed them.
>
What's a workaround for this? I'm thinking that I'll have to have a
parallel site for image hosting. For example, I'll have
images.website. com, hosted under a separate filesystem directory, and
then in the actual website that a user would peruse, any img src would
reference images.website. com . When I accept my uploads, I'll do my
filesystem copy to the images filesystem link.
>
Thoughts?

it could be done. Just call the "svn" commands in your code.

But it sucks in my experience to use large binary files (hundreds of
KBs) with subversion. The status (scanning / detecting) and exporting/
updating are painfully slow compared to direct copy.