(Forewarning, most of these problems and solutions come from being the
only developer in a 1 server department with no budget, few resources,
unresponsive IT, and non-technical managers, so thats where I'm coming
from.)
(Additionally, this may or may not fit in this group, but I know
there's bright people here, and it is largely PHP development centric)
Any time I've done web development, I've always been plagued by some of
the intrinsic differences between it and 'classical' development. I've
solved most of the problems and simply want to share/get feedback, but
at the same time I'm wondering how you guys solve these problems,
namely:
- Fast speed of development and deployment
- Shared codebase
- Layer seperation (logic, business, presentation, dev, staging,
production, etc)
- Tool support (editors and ide's that satisfy me)
Now, the ideal environment, in my opinion anyways, would allow me (or a
team) to:
- Do development, staging, and release all without having to worry
about the other.
- Identify dependencies between projects and code bases.
- Code versioning
- API documentation
- Obviously: easily write the code, ie: a good editor.
- Receive bug reports easily
Some of the solutions to these problems and environment requirements
are as simple as good programming etiquette and a bit of discipline,
but other's are not, also, allow me to clarify a bit.
When I originally started, this elaborate process was the really simple
way we all started with: rename index.php to index1.php, modify
index1.php, make sure it works, overwrite index.php with the new one.
Then it developed into having a vhost for development (but no seperate
sql server - oh the nightmare and white knuckle rollercoaster ride of
having to make structural changes to a live database), and finally I
setup this elaborate staging/dev/release system which has made my life
much easier.
Development/Staging/Production:
Ideally, I would like to be able to independently develop my code
(under the same conditions as productions, ie: path, php variables,
server settings, etc), merge it with everyone else's changes, then put
this into a staging environment for regression testing and QA, then
finally release into the production environment. (Ideally,
development, staging, and production would all be on seperate computers
so that dev/staging can be hosed wheen needed, but this isn't always
possible.)
This is relatively easy to do with Apache, Subversion, and a couple
shell scripts:
- Setup a development server with per-user directories in Apache and
rewrite user directories to a vhost. Ideally, it would be to a
subdomain host, but sometimes this isn't possible (because IT refuses
to setup subdomain DNS entries, for example), in which case you can
redirect to different port numbers. These per-user directories provide
each developer with his own space in which to run the project, and,
with .htaccess files, change server settings as may be necessary during
development.
- Setup a Subversion server with a post-commit hook that updates the
development server with the latest version (as a working copy,
preferablly), so that everyone can see what the 'final' version looks
like.
- Create a staging server in Apache and write a shell script to export
the specified revision to it and run automated tests. Revision intead
of tagged version in case you needed to roll back a release and tests
here because development can happen very rapidly and this is the 'QA'
phase.
- Create a production server in Apache and write a shell script to
export the specified tagged version, as well as a shell script to apply
patches. Ideally, no person should have to touch the production by
hand, except to alter config files (which don't change much, once
setup)
* A few notes about this setup:
* Config Files: There have to be many types of config files. One for
the user (this isn't versioned) so he can override settings as
necessary without changing the versioned config file. One for the
development server (this isn't versioned as well) so it can override
necessary settings. One for the production server. The production
server config file conditionally includes the user and dev config files
if they exist and the globals are appropriately define()'d. A problem
I haven't quite solved is that production configs will be overwritten,
so the shell script (that copies from subversion to the server) needs
to detect when it has changed to save the old one and notify the person
(ala Gentoo's emerge)
* In my situation i'm the only developer, so i skipped the staging
server.
* In cases where you don't have a dedicated SQL server for each of
development and production, I've found that creating a development
account with access on dev_%
* This doesn't quite work for handling SQL changes. If a table was
added, modified, etc, then releases must be manually compared and notes
checked for changes to the .sql defintion files, then the live database
updated accordingly
* Fittingly, data is backed up three times a day (morning, noon and
night - when you come back from lunch and your boss is asking if the
database has an 'undo,' they're delighted when you say it can go back a
few hours), as well as before every live release. With a small-medium
sized database this isn't really that much after it has been
compressed. Every day is maybe 3 megabytes, rotated monthly.
* This sounds like a lot of overkill and more work for just a single
dev, but I work every other day (just a student, part time), and it is
much easier to compare previous version releases in viewVC, update the
live website (portions not handled by the script, apache changes,
permissions, etc), then ssh in and run ./goLive.sh. Subversion also
helps you remember what you wrote the other day when you come in with a
hangover or are otherwise burned out.
Identifying dependencies between projects:
No software I know of does this, really, except grep. The goal is to
identify other places that might be affected by changes in common code
(utilities.php, user authentication code, etc). Grep isn't so bad at
it, really, but it would be nice to have a tool that did this, ya know?
Code Versioning:
I talked about this above. Essentially a system to track all changes
to code. I use Subversion and am quite happy with it; some use CVS,
whatever floats your boat. This is also much more reliable than
relying on, say, dreamweaver to tell you when something has been
changed or checked out or some such. It would be nice to have per-file
comments in commit logs, though, especially for web development where
changes can be minor.
Bug Reports:
I'd love to have something like Bugzilla, but my managers and users
would be far too intimidated by such a beast. Currently, I have a
simple 2 input form (subject/text) they fill out that gets emailed
directly to me. It works well as I have my email client flag, tag, and
bag them to specific folders and priorities. But:
- That wouldn't work too well if it was more than me
- My bosses (hell, sometimes me) like to know if it was fixed,
recieved, etc, and I find it a tedious and time consuming to reply
'thanks' to every bug report and 'done' to every completed item. I'd
like to be able to say 'go to intranet.com/status to see where I am and
what i'm up to.'
- It is also harder to track and identify bugs when they're simply
text in an inbox, too. Seeing a lot of similar bug reports might help
me improve my own writing and processes.
- I also would rather say 'Fixed Bug #14729' than 'Fixed issue with
auto select not submitting when user entered malformatted data into the
wrong field' in change logs. Makes me sound cool and I feel like i'm
getting somethign done. Plus, management likes seeing a slew of 'Fixed
Bug #whatever' instead of technical jargon they don't understand.
API Documentation:
This is really a matter of discipline while coding: remembering to
write the javadoc syntax. The API can be autodocumented with the
post-commit hook on commits. Ideally, it would be nice to integrate
this into a wiki to preserve code changes and api documentation
changes, as well as provide a means of easily editing and commenting
the docs.
Tool Support (Editors):
I've yet to find an editor that satifies me fully. Everything has
something nice, but they are all lacking something I really want.
Ideally, it would:
- Support Subversion (or CVS, as the case may be)
- Save a backup/second copy of the file to another location every save
(1: network drives over VPN can be slow, causeing the editor to lag, 2:
in the event the remote data is damaged). This would also mean it may
have to upload via ftp or other access method.
- Live syntax checking of the file (ala Eclipse, where it underlines
the errors like a spell check). This also reinforces the need to save
a backup because a network over a VPN can be very slow sometimes,
causing the editor to lag severely.
- Syntax Highlighting, the best thing since forever.
- Autocomplete/Intellisense of native PHP functions as well as user
defined functions.
- JavaDoc tooltip for variables and functions
- Regular expression searching of files *that display in a seperate
area* (ala Dreamweaver, NOT like visual studio, where it flags the
line)
- Jump to functionality. Visual studio has *alphabetized* dropdowns
that help you jump between classes, functions, and methods within a
file, and it is incredibly helpful. Dreamweaver has this...but its not
alphabetized, incredibly frustrating. It is also nice to have right
click menu's that let you do this (as in VS)
- Format code, because I'm a format-nazi. Not as you type mind you,
that's annoying.
- Be able to open include()'d files *with path checking* with right
click. Typically i set the php_include path to search the standard
includes directory, avoiding use of include($dir/common.php). VS.php
has this, but its broken.
- A second code window built in. It may sound odd, but sometimes I
want to look at two seperate files, or two portions of the same file
within the same window to visually compare or use as reference. And
not all of us have two monitors. VS can do this, I think, as can
Dreamweaver (kinda, but only within the same file and the view resets
randomly)
- A file browsing pane if it includes the shell's right click menu or
some really nifty functionality. Namely, subversion controls and 'open
with...' Most of the time I have a file explorer window open for one
reason or another, anyways.
* No editor does all this, none that I know of, they are all missing
something. VS.php has great syntax and checking, but horrible
deployment options. Dreamweaver will use ftp, and has great
HTML/Javascript support, but lacks satisfactory PHP support (at least,
after i've used vs.php and Eclipse). Then again, I bet vim does - but
i want to use my mouse, not memorize another hundred thousand commands
and keypresses. Web devs have to do that enough between php, sql,
javascript, html, shell, config files, and god knows what else.
Layer Seperation:
I know there are template languages and such out there, but I've never
gotten a chance to use them much. How well they work I don't really
know. But with things like AJAX autocomplete, it would be nice to
seperate config, core functionality, and presentation so a simple
autocomplete box doesn't need to have the server load the entire class
and subsequent includes.
Anyways, that leaves us with only a few questions and problems i've not
quite worked out:
- Managing config files appropriately (so new, blank ones don't
overwrite older configured ones)
- Keeping the database structure in sync with the .sql files
- Finding a kick ass editor that doesn't cost a bunch of money (cough,
VS: $1500+), and does enough to make the job easier (I honestly
consider DW the baseline as it does everything just good enough to keep
me happy).
and finally:
- How does everyone deal with the problems of web development? My
solution works, but it is somewhat ragtag and hacked together. Insight
is welcome.
only developer in a 1 server department with no budget, few resources,
unresponsive IT, and non-technical managers, so thats where I'm coming
from.)
(Additionally, this may or may not fit in this group, but I know
there's bright people here, and it is largely PHP development centric)
Any time I've done web development, I've always been plagued by some of
the intrinsic differences between it and 'classical' development. I've
solved most of the problems and simply want to share/get feedback, but
at the same time I'm wondering how you guys solve these problems,
namely:
- Fast speed of development and deployment
- Shared codebase
- Layer seperation (logic, business, presentation, dev, staging,
production, etc)
- Tool support (editors and ide's that satisfy me)
Now, the ideal environment, in my opinion anyways, would allow me (or a
team) to:
- Do development, staging, and release all without having to worry
about the other.
- Identify dependencies between projects and code bases.
- Code versioning
- API documentation
- Obviously: easily write the code, ie: a good editor.
- Receive bug reports easily
Some of the solutions to these problems and environment requirements
are as simple as good programming etiquette and a bit of discipline,
but other's are not, also, allow me to clarify a bit.
When I originally started, this elaborate process was the really simple
way we all started with: rename index.php to index1.php, modify
index1.php, make sure it works, overwrite index.php with the new one.
Then it developed into having a vhost for development (but no seperate
sql server - oh the nightmare and white knuckle rollercoaster ride of
having to make structural changes to a live database), and finally I
setup this elaborate staging/dev/release system which has made my life
much easier.
Development/Staging/Production:
Ideally, I would like to be able to independently develop my code
(under the same conditions as productions, ie: path, php variables,
server settings, etc), merge it with everyone else's changes, then put
this into a staging environment for regression testing and QA, then
finally release into the production environment. (Ideally,
development, staging, and production would all be on seperate computers
so that dev/staging can be hosed wheen needed, but this isn't always
possible.)
This is relatively easy to do with Apache, Subversion, and a couple
shell scripts:
- Setup a development server with per-user directories in Apache and
rewrite user directories to a vhost. Ideally, it would be to a
subdomain host, but sometimes this isn't possible (because IT refuses
to setup subdomain DNS entries, for example), in which case you can
redirect to different port numbers. These per-user directories provide
each developer with his own space in which to run the project, and,
with .htaccess files, change server settings as may be necessary during
development.
- Setup a Subversion server with a post-commit hook that updates the
development server with the latest version (as a working copy,
preferablly), so that everyone can see what the 'final' version looks
like.
- Create a staging server in Apache and write a shell script to export
the specified revision to it and run automated tests. Revision intead
of tagged version in case you needed to roll back a release and tests
here because development can happen very rapidly and this is the 'QA'
phase.
- Create a production server in Apache and write a shell script to
export the specified tagged version, as well as a shell script to apply
patches. Ideally, no person should have to touch the production by
hand, except to alter config files (which don't change much, once
setup)
* A few notes about this setup:
* Config Files: There have to be many types of config files. One for
the user (this isn't versioned) so he can override settings as
necessary without changing the versioned config file. One for the
development server (this isn't versioned as well) so it can override
necessary settings. One for the production server. The production
server config file conditionally includes the user and dev config files
if they exist and the globals are appropriately define()'d. A problem
I haven't quite solved is that production configs will be overwritten,
so the shell script (that copies from subversion to the server) needs
to detect when it has changed to save the old one and notify the person
(ala Gentoo's emerge)
* In my situation i'm the only developer, so i skipped the staging
server.
* In cases where you don't have a dedicated SQL server for each of
development and production, I've found that creating a development
account with access on dev_%
* This doesn't quite work for handling SQL changes. If a table was
added, modified, etc, then releases must be manually compared and notes
checked for changes to the .sql defintion files, then the live database
updated accordingly
* Fittingly, data is backed up three times a day (morning, noon and
night - when you come back from lunch and your boss is asking if the
database has an 'undo,' they're delighted when you say it can go back a
few hours), as well as before every live release. With a small-medium
sized database this isn't really that much after it has been
compressed. Every day is maybe 3 megabytes, rotated monthly.
* This sounds like a lot of overkill and more work for just a single
dev, but I work every other day (just a student, part time), and it is
much easier to compare previous version releases in viewVC, update the
live website (portions not handled by the script, apache changes,
permissions, etc), then ssh in and run ./goLive.sh. Subversion also
helps you remember what you wrote the other day when you come in with a
hangover or are otherwise burned out.
Identifying dependencies between projects:
No software I know of does this, really, except grep. The goal is to
identify other places that might be affected by changes in common code
(utilities.php, user authentication code, etc). Grep isn't so bad at
it, really, but it would be nice to have a tool that did this, ya know?
Code Versioning:
I talked about this above. Essentially a system to track all changes
to code. I use Subversion and am quite happy with it; some use CVS,
whatever floats your boat. This is also much more reliable than
relying on, say, dreamweaver to tell you when something has been
changed or checked out or some such. It would be nice to have per-file
comments in commit logs, though, especially for web development where
changes can be minor.
Bug Reports:
I'd love to have something like Bugzilla, but my managers and users
would be far too intimidated by such a beast. Currently, I have a
simple 2 input form (subject/text) they fill out that gets emailed
directly to me. It works well as I have my email client flag, tag, and
bag them to specific folders and priorities. But:
- That wouldn't work too well if it was more than me
- My bosses (hell, sometimes me) like to know if it was fixed,
recieved, etc, and I find it a tedious and time consuming to reply
'thanks' to every bug report and 'done' to every completed item. I'd
like to be able to say 'go to intranet.com/status to see where I am and
what i'm up to.'
- It is also harder to track and identify bugs when they're simply
text in an inbox, too. Seeing a lot of similar bug reports might help
me improve my own writing and processes.
- I also would rather say 'Fixed Bug #14729' than 'Fixed issue with
auto select not submitting when user entered malformatted data into the
wrong field' in change logs. Makes me sound cool and I feel like i'm
getting somethign done. Plus, management likes seeing a slew of 'Fixed
Bug #whatever' instead of technical jargon they don't understand.
API Documentation:
This is really a matter of discipline while coding: remembering to
write the javadoc syntax. The API can be autodocumented with the
post-commit hook on commits. Ideally, it would be nice to integrate
this into a wiki to preserve code changes and api documentation
changes, as well as provide a means of easily editing and commenting
the docs.
Tool Support (Editors):
I've yet to find an editor that satifies me fully. Everything has
something nice, but they are all lacking something I really want.
Ideally, it would:
- Support Subversion (or CVS, as the case may be)
- Save a backup/second copy of the file to another location every save
(1: network drives over VPN can be slow, causeing the editor to lag, 2:
in the event the remote data is damaged). This would also mean it may
have to upload via ftp or other access method.
- Live syntax checking of the file (ala Eclipse, where it underlines
the errors like a spell check). This also reinforces the need to save
a backup because a network over a VPN can be very slow sometimes,
causing the editor to lag severely.
- Syntax Highlighting, the best thing since forever.
- Autocomplete/Intellisense of native PHP functions as well as user
defined functions.
- JavaDoc tooltip for variables and functions
- Regular expression searching of files *that display in a seperate
area* (ala Dreamweaver, NOT like visual studio, where it flags the
line)
- Jump to functionality. Visual studio has *alphabetized* dropdowns
that help you jump between classes, functions, and methods within a
file, and it is incredibly helpful. Dreamweaver has this...but its not
alphabetized, incredibly frustrating. It is also nice to have right
click menu's that let you do this (as in VS)
- Format code, because I'm a format-nazi. Not as you type mind you,
that's annoying.
- Be able to open include()'d files *with path checking* with right
click. Typically i set the php_include path to search the standard
includes directory, avoiding use of include($dir/common.php). VS.php
has this, but its broken.
- A second code window built in. It may sound odd, but sometimes I
want to look at two seperate files, or two portions of the same file
within the same window to visually compare or use as reference. And
not all of us have two monitors. VS can do this, I think, as can
Dreamweaver (kinda, but only within the same file and the view resets
randomly)
- A file browsing pane if it includes the shell's right click menu or
some really nifty functionality. Namely, subversion controls and 'open
with...' Most of the time I have a file explorer window open for one
reason or another, anyways.
* No editor does all this, none that I know of, they are all missing
something. VS.php has great syntax and checking, but horrible
deployment options. Dreamweaver will use ftp, and has great
HTML/Javascript support, but lacks satisfactory PHP support (at least,
after i've used vs.php and Eclipse). Then again, I bet vim does - but
i want to use my mouse, not memorize another hundred thousand commands
and keypresses. Web devs have to do that enough between php, sql,
javascript, html, shell, config files, and god knows what else.
Layer Seperation:
I know there are template languages and such out there, but I've never
gotten a chance to use them much. How well they work I don't really
know. But with things like AJAX autocomplete, it would be nice to
seperate config, core functionality, and presentation so a simple
autocomplete box doesn't need to have the server load the entire class
and subsequent includes.
Anyways, that leaves us with only a few questions and problems i've not
quite worked out:
- Managing config files appropriately (so new, blank ones don't
overwrite older configured ones)
- Keeping the database structure in sync with the .sql files
- Finding a kick ass editor that doesn't cost a bunch of money (cough,
VS: $1500+), and does enough to make the job easier (I honestly
consider DW the baseline as it does everything just good enough to keep
me happy).
and finally:
- How does everyone deal with the problems of web development? My
solution works, but it is somewhat ragtag and hacked together. Insight
is welcome.
Comment