httpWebRequest through Proxy

[retroviz]

Hi there.

I have written a screen scraping application (both web based and windows forms) in vb.net. When testing on a public broadband link it works fine. However it fails at work due to our proxy server.

To authenticate in the asp version, I just added:

<defaultProxy useDefaultCredentials="true">

to my web.config - this works fine.

However, I need to integrate this with an existing windows form app (vb6) so need to get the windows form version working.

I am struggling to achieve the same settings (as per web.config) directly within my code. This is what I am trying to use so far:

Dim mywebRequest As HttpWebRequest = TryCast(WebRequest.Create("http://www.thedomain.com/login.aspx"), HttpWebRequest)

mywebRequest.Proxy = New WebProxy("http://myproxyserver")
mywebRequest.Proxy.Credentials = CredentialCache.DefaultCredentials
mywebRequest.Credentials = CredentialCache.DefaultCredentials

As the page I am trying to access requires authentication, I first capture the viewstate of the page. Once I have this I can then simulate a post of my login credentials and use a streamreader to retrieve the contents of the page.

Click here for a C# tutorial.

The thing I dont understand is that the first part of the code appears to work (and passes through the proxy). It is only when i try and close the contents of the response:

mywebRequest.GetResponse().Close()

that I get the following error:

The remote server returned an error: (407) Proxy Authentication Required.

Could someone advise me as to whether I am defining the proxy correctly?

Thanks.

Ben

[retroviz]

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

[Frinavale]

Just to let everyone know I got this working in the end. Made a schoolboy error as I had reinitiated the web request but did not add the proxy information for that instance.

If anyone would like a copy of the complete code (to authenticate then screen scrape) then sent me a message

I'm glad you got it working!
Could you please share the solution so others can learn as well.

Thanks a lot!

-Frinny

[retroviz]

Well I am glad to help. For some tutorials on httpwebrequest and persisting viewstate for sites that require authentication I would seriously recommend:

http://odetocode.com/Articles/162.aspx (c# version of my code)

and

http://aspnet.4guysfromrolla.com/art...spx#postadlink (great article covering the basics of the httpwebrequest class by Scott Mitchell.

I do have a C# version of the following code if you require (as originally created in this then needed to write in vb.net). This also works as a windows form application or a asp.net app.

Create a new class called prices.vb. Insert the following code (comments explain it pretty well):

[CODE=vbnet]Imports System
Imports System.Net
Imports System.IO
Imports System.Text
Imports System.Web
Imports System.Text.Reg ularExpressions
Imports Microsoft.Visua lBasic
Namespace PageGetter
Public Class prices
Public targetURL As String
Private Function ExtractViewStat e(ByVal s As String) As String
Dim viewStateNameDe limiter As String = "__VIEWSTAT E"
Dim valueDelimiter As String = "value="""


Dim viewStateNamePo sition As Integer = s.IndexOf(viewS tateNameDelimit er)
Dim viewStateValueP osition As Integer = s.IndexOf(value Delimiter, viewStateNamePo sition)

Dim viewStateStartP osition As Integer = viewStateValueP osition + valueDelimiter. Length
Dim viewStateEndPos ition As Integer = s.IndexOf("""", viewStateStartP osition)

Return HttpUtility.Url EncodeUnicode(s .Substring(view StateStartPosit ion, viewStateEndPos ition - viewStateStartP osition))
End Function

Public Function GetPrices(ByVal targetUrl As String) As String
*************** *************** *************** *************** *************** ******
'only need this if you are behind a proxy
Dim pxy As New WebProxy("http://yourproxyaddres s:0000")
pxy.Credentials = CredentialCache .DefaultCredent ials
*************** *************** *************** *************** *************** ******
' first, request the login form to get the viewstate value
Dim mywebRequest As HttpWebRequest = TryCast(WebRequ est.Create("htt p://somedomain.com/TheLogOnCheck.a spx"), HttpWebRequest)

mywebRequest.Pr oxy = pxy

'Set the timeout to 1 second (or 1,000 milliseconds)
mywebRequest.Ti meout = 1000


Try

Dim responseReader As New StreamReader(my webRequest.GetR esponse().GetRe sponseStream())
Dim responseData As String = responseReader. ReadToEnd()

responseReader. Close()

' extract the viewstate value and build out POST data
Dim viewState As String = ExtractViewStat e(responseData)
Dim postData As String = [String].Format("__VIEW STATE={0}&TUser Name={1}&TPassw ord={2}&_ctl0%3 AContent%3AbtnL ogon=Logon&__PR EVIOUSPAGE=cys_ as-zp6tmeXXlc07Fgg KJUKD96k3RyL8XY HQ-U3I1&__EVENTVAL IDATION=%2FwEWA wLV3qjMAQKL2pbe CALf%2B9ffB29vW whgfdAvHzzk%2F% 2BqB%2BKkddRGi" , viewState, "yourname@domai n.co.uk", "password")


' have a cookie container ready to receive the forms auth cookie
Dim cookies As New CookieContainer ()

' now post to the login form
mywebRequest = TryCast(WebRequ est.Create("htt p://somedomain.com/TheLogOnCheck.a spx"), HttpWebRequest)

mywebRequest.Pr oxy = pxy 'only if behind proxy (see above)

mywebRequest.Me thod = "POST"
mywebRequest.Co ntentType = "applicatio n/x-www-form-urlencoded"
mywebRequest.Co okieContainer = cookies

' write the form values into the request message
Dim requestWriter As New StreamWriter(my webRequest.GetR equestStream())
requestWriter.W rite(postData)
requestWriter.C lose()

' we don't need the contents of the response, just the cookie it issues
mywebRequest.Ge tResponse().Clo se()

' now we can send out cookie along with a request for the protected page
mywebRequest = TryCast(WebRequ est.Create(targ etUrl), HttpWebRequest)
mywebRequest.Co okieContainer = cookies
responseReader = New StreamReader(my webRequest.GetR esponse().GetRe sponseStream())


' and read the response
responseData = responseReader. ReadToEnd()
responseReader. Close()

'Here we set up our Regular expression to snatch what's between the tags we want in our html source
Dim regex As New Regex("<!-- main table -->((.|" & Chr(10) & ")*?)<!-- / main table -->", RegexOptions.Ig noreCase)

'Here we apply our regular expression to our string using the
'Match object.
Dim oM As Match = regex.Match(res ponseData)

Return oM.Value

Catch wex As WebException
'Something went wrong in the HTTP request! See if it was a timeout problem
If wex.Status = WebExceptionSta tus.Timeout Then
Return ("<font color=red>The httpWebRequest has timed out. Please contact the helpdesk</font>")
Else
Return ("<font color=red>FAILE D TO CONNECT<br />Status: " & wex.Status & " Message: " & wex.Message & "</font>")
End If
End Try


End Function
End Class
End Namespace[/CODE]

In my windows form I then used the following code to extract (scrape) the page contents (puts the retrieved html into a webbrowser control):

[CODE=vbnet] Private Sub myForm_Load(ByV al sender As System.Object, ByVal e As System.EventArg s) Handles MyBase.Load
Dim myPrice As New prices
Dim temp As String = myPrice.GetPric es("http://www.somedomain. org/thepageyouwant. aspx")
WebBrowser1.Doc umentText = temp
End Sub[/CODE]

If you have an asp.net application you can do exactly the same but on your page_load event:
[CODE=vbnet]Dim futuresprice As New prices
Dim temp As String = futuresprice.Ge tPrices("http://www.somedomain. org/thepageyouwant. aspx")
Reponse.Write(t emp)[/CODE]

Just some points worth noting. The viewstate value that you build your post data with will not necessarily be the same as mine. What you need to do is download a tool called Fiddler (http://www.fiddler2.com/fiddler2). Open it up and log into your target site. Have a look in the session inspector of the page that is used to authenticate the log in (mine was called logoncheck.aspx ) and you can see the complete viewstate value. You can then ammend the code to suit the site your requirements. Also I cannot guarantee that his will work for everyone's proxy servers.
Finally credit must be given to Scott Mitchell and Scott Allen as it is their tutorials I used to produce this application.

Hope this was of help.
Ben Foster
<email removed>

[Frinavale]

Wow!
Thank you for providing your solution.

-Frinny