WebRequest from behind a proxy

[james.dixon@dewr.gov.au]

Hi

I have been struggling with what should be a simple thing. I want to
crawl the web for specific links, and save any html and files that meet
my criteria to my hard drive.

I thought that WebRequest/WebResponse would be the best way to go,
using a proxy. The page that I am returned is just a proxy generated
file, and not the source code. Is there any way around this?

As a workaround, I used the AxWebBrowser and an mshtml document. this
works well, except when I come to downloading files. As stated above,
the easy options don't work - WebRequest/WebRespose return a 404:File
not Found error (which you would expect, given that it doesn't get to
the html), webclient doesn't work behind a proxy AND after doing some
searching, it is proving difficult to programmaticall y download linked
files using AxWebbrowser.

So, my questions to the smart people out there include:

a) is there any way to get the webrequest/webresponse objects working
behind a proxy when only proxy generated source is returned to the
WebRequest object 9and not the file source code) - could this be
anything to do with proxy authorisation?;

b) is there a way to programatticall y download pdf files using
AxWebbrowser?

Grateful for any advice.

Cheers

James

[Michael C]

Re: WebRequest from behind a proxy

<james.dixon@de wr.gov.au> wrote in message
news:1135046790 .883636.69180@g 14g2000cwa.goog legroups.com...[color=blue]
> b) is there a way to programatticall y download pdf files using
> AxWebbrowser?
>
> Grateful for any advice.[/color]

Can't you just use the TcpClient class to get the page? The syntax is very
simple and you get the full text of the page back unprocessed, so you can do
what you like with it. All you need to do is send:

GET /pagename.htm HTTP/1.1
HOST: nameofhost.com

then 2 crlfs.

Michael

[Yosh]

Re: WebRequest from behind a proxy

Does this help?

http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q301102&ID=KB;EN-US;Q301102&SD=MSDN

Yosh
<james.dixon@de wr.gov.au> wrote in message
news:1135046790 .883636.69180@g 14g2000cwa.goog legroups.com...[color=blue]
> Hi
>
> I have been struggling with what should be a simple thing. I want to
> crawl the web for specific links, and save any html and files that meet
> my criteria to my hard drive.
>
> I thought that WebRequest/WebResponse would be the best way to go,
> using a proxy. The page that I am returned is just a proxy generated
> file, and not the source code. Is there any way around this?
>
> As a workaround, I used the AxWebBrowser and an mshtml document. this
> works well, except when I come to downloading files. As stated above,
> the easy options don't work - WebRequest/WebRespose return a 404:File
> not Found error (which you would expect, given that it doesn't get to
> the html), webclient doesn't work behind a proxy AND after doing some
> searching, it is proving difficult to programmaticall y download linked
> files using AxWebbrowser.
>
> So, my questions to the smart people out there include:
>
> a) is there any way to get the webrequest/webresponse objects working
> behind a proxy when only proxy generated source is returned to the
> WebRequest object 9and not the file source code) - could this be
> anything to do with proxy authorisation?;
>
> b) is there a way to programatticall y download pdf files using
> AxWebbrowser?
>
> Grateful for any advice.
>
> Cheers
>
> James
>[/color]

[james.dixon@dewr.gov.au]

Re: WebRequest from behind a proxy

Thanks Yosh - yes I had seen that and tried webrequest etc, but
couldn't get through.

Michael - could you provide more information (or links to more
information) on how to set a simple TCPClient up - haven't done it
before.

Thanks

James

Yosh wrote:
[color=blue]
> Does this help?
>
> http://support.microsoft.com/default...301102&SD=MSDN
>
> Yosh
> <james.dixon@de wr.gov.au> wrote in message
> news:1135046790 .883636.69180@g 14g2000cwa.goog legroups.com...[color=green]
> > Hi
> >
> > I have been struggling with what should be a simple thing. I want to
> > crawl the web for specific links, and save any html and files that meet
> > my criteria to my hard drive.
> >
> > I thought that WebRequest/WebResponse would be the best way to go,
> > using a proxy. The page that I am returned is just a proxy generated
> > file, and not the source code. Is there any way around this?
> >
> > As a workaround, I used the AxWebBrowser and an mshtml document. this
> > works well, except when I come to downloading files. As stated above,
> > the easy options don't work - WebRequest/WebRespose return a 404:File
> > not Found error (which you would expect, given that it doesn't get to
> > the html), webclient doesn't work behind a proxy AND after doing some
> > searching, it is proving difficult to programmaticall y download linked
> > files using AxWebbrowser.
> >
> > So, my questions to the smart people out there include:
> >
> > a) is there any way to get the webrequest/webresponse objects working
> > behind a proxy when only proxy generated source is returned to the
> > WebRequest object 9and not the file source code) - could this be
> > anything to do with proxy authorisation?;
> >
> > b) is there a way to programatticall y download pdf files using
> > AxWebbrowser?
> >
> > Grateful for any advice.
> >
> > Cheers
> >
> > James
> >[/color][/color]

[Michael C]

Re: WebRequest from behind a proxy

<james.dixon@de wr.gov.au> wrote in message
news:1135051120 .670560.231020@ g49g2000cwa.goo glegroups.com.. .[color=blue]
> Thanks Yosh - yes I had seen that and tried webrequest etc, but
> couldn't get through.
>
> Michael - could you provide more information (or links to more
> information) on how to set a simple TCPClient up - haven't done it
> before.[/color]

Note the slash after the GET is the page you're requesting, in this case the
default page for MS.

TcpClient client = new TcpClient();
client.Connect( "www.microsoft. com", 80);
NetworkStream stream = client.GetStrea m();
byte[] data = System.Text.ASC IIEncoding.ASCI I.GetBytes("GET /
HTTP/1.1\r\nHOST: microsoft.com\r \n\r\n");
stream.Write(da ta, 0, data.Length);
data = new byte[256];
int len = 0;
do
{
len = stream.Read(dat a, 0, 256);
Console.WriteLi ne(System.Text. ASCIIEncoding.A SCII.GetString( data, 0, len));
}while(len == 256);
stream.Close();
client.Close();

Michael

[james.dixon@dewr.gov.au]

Re: WebRequest from behind a proxy

Thanks Michael - that code worked beautifully. I have also found
another proxy server I can use which allows me to use WebRequest and
WebResponse, so case closed ...