"proper" way to download several hundred text files from a web-site ?

[Bill Woodruff]

As an exercise I wrote a small C# program to download several hundred text files of guitar tab music by parsing the home page of the
site and retrieving all the links to the music text files and using 'WebRequest and a StreamReader to download them.

I followed up the call to WebRequest.Crea te with a call to Application.DoE vents, and called Application.DoE vents() again after the
call to ReadToEnd on the StreamReader.

What I was looking for was some kind of delay mechanism to ensure that I was not making requests to the site too frequently. Is
there any way to ask a site something like : "Can I have another helping please, or would you rather I come back later ?."

Appreciate any clues.

thanks, Bill

[Mark]

Re: "proper&qu ot; way to download several hundred text files from a web-site ?

"Bill Woodruff" <billw@dotscien ce.com> wrote in
news:#xjxIXTUDH A.1324@TK2MSFTN GP11.phx.gbl:
[color=blue]
> What I was looking for was some kind of delay mechanism to ensure that
> I was not making requests to the site too frequently. Is there any way
> to ask a site something like : "Can I have another helping please, or
> would you rather I come back later ?."
>
> Appreciate any clues.
>
> thanks, Bill[/color]

The RFC says no more than 4 connections to any given HTTP 1.0 server at any
given moment, and no more than 2 to any given HTTP 1.1 server at any
moment. As long as you don't break that rule (and I think that the
WebRequest class respects IE's settings, so if you haven't changed them
with some sort of "accellerat or", those are defaults) you should be fine.

If you want to introduce more delay, then Thread.Sleep() is always good, as
long as you don't do it in UI threads (but you wouldn't do work in UI
threads, would you? that would be bad).

Mark