Code below ----
I've asked a similar question on this forum earlier. This is a slightly different situation.
Previous Question ----
I'm trying to save some specific web pages to disk as text files. I searched the Internet and found a basic example which I changed to fit my needs.
I tested it out first on a single URL and it worked fine. Now when I incorporate an array of URL's, it fails to work. The first "responseFromSe rver"
properly retrieves and displays "http://finance.yahoo.c om" (or any other valid URL for the first webrequest.crea te), but when I use the urls from the
array in the for/next structure, I never get a responseFromSer ver. My guess is that there must be some kind of initialization to the WebRequest,
Stream, or WebResponse.
Current Question ----
After changing some code, I now get responses returned to me. BUT, the pages I am going after have JavaScript in them to build the retrieve data,
tables, layout, etc. What I get back is the JavaScript code but none of the data I'm looking for. I thought the problem was an initialization of
some object, but now I believe that the actual page is not being built by the time my program is ready for the streamReader. I may be all wrong on
that assumption, but I need someone to steer me in the right direction.
Even checking EndOfStream doesn't wait for the page to be built.
Thanks,
Hexman
P.S. I can run the program repeatedly and on some runs, SOME of the pages are comeplete! So to me that points me to the building of the requested
pages.
Newer Code ----
--------------------------------------------------------------------------------------------
This code replaces the processing of the Old Code. Changes: Try-Catch, ReadLine
rather than ReadToEnd, URL from a datatable. None of these changes should affect the
problem area of the program.
Try
Dim request As HttpWebRequest = WebRequest.Crea te(URL)
Dim response As HttpWebResponse = request.GetResp onse()
Dim reader As StreamReader = New StreamReader(re sponse.GetRespo nseStream())
strToWrite = ""
Do While reader.EndOfStr eam = False
str = reader.ReadLine ()
strToWrite = strToWrite & str
Loop
strOutFile = OutFN(Idx)
File.WriteAllTe xt(strOutFile, strToWrite)
reader.Close()
response.Close( )
Catch ex As WebException
Console.WriteLi ne(ex.Message)
If ex.Status = WebExceptionSta tus.ProtocolErr or Then
Console.WriteLi ne("Status Code : {0}", CType(ex.Respon se, HttpWebResponse ).StatusCode)
Console.WriteLi ne("Status Description : {0}", CType(ex.Respon se, HttpWebResponse ).StatusDescrip tion)
End If
Catch ex As Exception
Console.WriteLi ne(ex.Message)
End Try
End If
Old Code ----
--------------------------------------------------------------------------------------------
Imports System
Imports System.IO
Imports System.Net
Imports System.Net.WebC lient
Imports System.Windows. Forms
Public Class Form1
Private Sub btnProcess_Clic k(ByVal sender As System.Object, ByVal e As System.EventArg s) Handles btnProcess.Clic k
Dim x As Integer
Dim aryUrls(15) As String
aryUrls(0) = "http://finance.yahoo.c om/q/bc?s=DOW&t=1d"
aryUrls(1) = "http://finance.yahoo.c om/q/bc?s=IBM&t=6m"
aryUrls(2) = "http://finance.yahoo.c om/q/bc?s=MSFT&t=6m"
aryUrls(3) = "http://biz.yahoo.com/top.html"
Dim request As WebRequest = WebRequest.Crea te("http://finance.yahoo.c om/")
Dim response As WebResponse = request.GetResp onse()
Dim dataStream As Stream = response.GetRes ponseStream()
Dim reader As New StreamReader(da taStream)
Dim responseFromSer ver As String = reader.ReadToEn d()
MsgBox(response FromServer) <=== Shows finance.yahoo.c om
For x = 0 To 3
request = WebRequest.Crea te(aryUrls(x))
dataStream = response.GetRes ponseStream()
responseFromSer ver = reader.ReadToEn d()
MsgBox(response FromServer) <=== Shows nothing
Next
reader.Close()
response.Close( )
End Sub
I've asked a similar question on this forum earlier. This is a slightly different situation.
Previous Question ----
I'm trying to save some specific web pages to disk as text files. I searched the Internet and found a basic example which I changed to fit my needs.
I tested it out first on a single URL and it worked fine. Now when I incorporate an array of URL's, it fails to work. The first "responseFromSe rver"
properly retrieves and displays "http://finance.yahoo.c om" (or any other valid URL for the first webrequest.crea te), but when I use the urls from the
array in the for/next structure, I never get a responseFromSer ver. My guess is that there must be some kind of initialization to the WebRequest,
Stream, or WebResponse.
Current Question ----
After changing some code, I now get responses returned to me. BUT, the pages I am going after have JavaScript in them to build the retrieve data,
tables, layout, etc. What I get back is the JavaScript code but none of the data I'm looking for. I thought the problem was an initialization of
some object, but now I believe that the actual page is not being built by the time my program is ready for the streamReader. I may be all wrong on
that assumption, but I need someone to steer me in the right direction.
Even checking EndOfStream doesn't wait for the page to be built.
Thanks,
Hexman
P.S. I can run the program repeatedly and on some runs, SOME of the pages are comeplete! So to me that points me to the building of the requested
pages.
Newer Code ----
--------------------------------------------------------------------------------------------
This code replaces the processing of the Old Code. Changes: Try-Catch, ReadLine
rather than ReadToEnd, URL from a datatable. None of these changes should affect the
problem area of the program.
Try
Dim request As HttpWebRequest = WebRequest.Crea te(URL)
Dim response As HttpWebResponse = request.GetResp onse()
Dim reader As StreamReader = New StreamReader(re sponse.GetRespo nseStream())
strToWrite = ""
Do While reader.EndOfStr eam = False
str = reader.ReadLine ()
strToWrite = strToWrite & str
Loop
strOutFile = OutFN(Idx)
File.WriteAllTe xt(strOutFile, strToWrite)
reader.Close()
response.Close( )
Catch ex As WebException
Console.WriteLi ne(ex.Message)
If ex.Status = WebExceptionSta tus.ProtocolErr or Then
Console.WriteLi ne("Status Code : {0}", CType(ex.Respon se, HttpWebResponse ).StatusCode)
Console.WriteLi ne("Status Description : {0}", CType(ex.Respon se, HttpWebResponse ).StatusDescrip tion)
End If
Catch ex As Exception
Console.WriteLi ne(ex.Message)
End Try
End If
Old Code ----
--------------------------------------------------------------------------------------------
Imports System
Imports System.IO
Imports System.Net
Imports System.Net.WebC lient
Imports System.Windows. Forms
Public Class Form1
Private Sub btnProcess_Clic k(ByVal sender As System.Object, ByVal e As System.EventArg s) Handles btnProcess.Clic k
Dim x As Integer
Dim aryUrls(15) As String
aryUrls(0) = "http://finance.yahoo.c om/q/bc?s=DOW&t=1d"
aryUrls(1) = "http://finance.yahoo.c om/q/bc?s=IBM&t=6m"
aryUrls(2) = "http://finance.yahoo.c om/q/bc?s=MSFT&t=6m"
aryUrls(3) = "http://biz.yahoo.com/top.html"
Dim request As WebRequest = WebRequest.Crea te("http://finance.yahoo.c om/")
Dim response As WebResponse = request.GetResp onse()
Dim dataStream As Stream = response.GetRes ponseStream()
Dim reader As New StreamReader(da taStream)
Dim responseFromSer ver As String = reader.ReadToEn d()
MsgBox(response FromServer) <=== Shows finance.yahoo.c om
For x = 0 To 3
request = WebRequest.Crea te(aryUrls(x))
dataStream = response.GetRes ponseStream()
responseFromSer ver = reader.ReadToEn d()
MsgBox(response FromServer) <=== Shows nothing
Next
reader.Close()
response.Close( )
End Sub
Comment