Continuously Growing Data

[bgold12]

I'm curious about how websites deal with the problem of continuously
growing data. For example, there are many forums that preserve all
posts for many years back, and are continuously receiving and storing
new posts all the time. No website has access to an infinite amount of
storage space, so:

What kind of buffer (i.e. space available) do they like to have, or in
other words, how much time do they have before they would run out of
space based on the rate of data growth and storage space?

What do they do when they are running out of space on a particular
machine; how do they increase their storage space?

Is maintaining this kind of data stream (like a forum) feasible for a
small-scale website, like one that I might want to host from my own
computer (with about 20 GB of space left) or from a cheap web host
(that probably wouldn't offer me any more than something like 60 GB)?

Thanks,

bgold12

[Chris F.A. Johnson]

Re: Continuously Growing Data

On 2008-08-15, bgold12 wrote:

I'm curious about how websites deal with the problem of continuously
growing data. For example, there are many forums that preserve all
posts for many years back, and are continuously receiving and storing
new posts all the time. No website has access to an infinite amount of
storage space, so:
>
What kind of buffer (i.e. space available) do they like to have, or in
other words, how much time do they have before they would run out of
space based on the rate of data growth and storage space?
>
What do they do when they are running out of space on a particular
machine; how do they increase their storage space?
>
Is maintaining this kind of data stream (like a forum) feasible for a
small-scale website, like one that I might want to host from my own
computer (with about 20 GB of space left) or from a cheap web host
(that probably wouldn't offer me any more than something like 60 GB)?

As the average size of a forum post is not likely to be more than
5KB, you could store 200,000 posts in 1GB. How many posts do you
expect to get?

--
Chris F.A. Johnson <http://cfaj.freeshell. org>
=============== =============== =============== =============== =======
Author:
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)

[Steve Swift]

Re: Continuously Growing Data

bgold12 wrote:

I'm curious about how websites deal with the problem of continuously
growing data. For example, there are many forums that preserve all
posts for many years back, and are continuously receiving and storing
new posts all the time. No website has access to an infinite amount of
storage space, so:

I've been running a website that gathers data since the early 1990's.
Over time it has had to be upgraded, but not because it ever ran out of
space, but because it needed to get faster to cope with increasing
numbers of users and more complex interactions. One side affect of
those upgrades (about 6) is that nearly each time the newer server came
with more disk space, simply because disk size seem to follow Moore's
law (well, the popular misrepresentati on).

Most of the data collected by websites has had to have been typed by a
person at some time or other, and that puts an upper limit on how fast
it can accumulate. Next time you read something on a website, pick one
word, and try to imagine the person typing it.


--
Steve Swift

http://www.swiftys.org.uk/swifty.html

http://www.ringers.org.uk

[Sherm Pendley]

Re: Continuously Growing Data

bgold12 <bgold12@gmail. comwrites:

I'm curious about how websites deal with the problem of continuously
growing data.

Start small, and don't buy more hardware until it actually becomes a
problem for your current system. That will take a *lot* longer than
your marketing department hopes for.

Along with the Aeron chairs, a very common mistake in the dot-bomb era
of the 90s was investing in hardware that could handle 1000x the load
a new site can realistically expect. I call it the "server of dreams"
mentality - "if you build it, they'll come."

sherm--

--
My blog: http://shermspace.blogspot.com
Cocoa programming in Perl: http://camelbones.sourceforge.net