How to retrieve data from a website

[rinu mariam johnson]

I want to retrieve data from a website based on a file number.But i have to do this programmaticall y using vb.net and finally have to store the results in a csv file.
In my web application,i will give the file number in a textbox(through GUI) and the site address(through vb.net code) and when i click the search button,the csv output should be obtained.
I am fed up with this problem for the last 2 weeks..can anybody pls help me in this situation.

Thanks & Regards,
rinu

[anijos]

You can use a datatable to save the data first and later convert it into .csv format.

[rinu mariam johnson]

hi anijos,
Thanx for ur quick reply..but i didnt get the first part itself..i.e,i am not able to access the site through vb.net code(i dont want to open the site in my page)and also i am not getting the search results of the particular file no. in my page

Hope u got it..pls help me

[anijos]

Rinu,

what do u mean by serach result for file no? i dont understand that part.. can you explain a bit more?

[rinu mariam johnson]

hi anijos,
surely..i'll explain that in detail..my intention is to give one traffic file no(e.g.vehiclen umber)in one textbox in my webapplication. ..on clicking search button it should give the same results as the results from the website(in the third party website...there is a provision for entering the file no..and on search..it will display the results in that site )

I dont want to take the website each and everytime,inste ad,i'm giving the website address in my vb.net code..and based on the file no. which i'm changing in the textbox,the output should vary..

Did u get it?

[anijos]

Hi Rinu,

I still dont under stand your application. you said something about a third party website rt? can u send me the url for that? may be that will allow me to understand your problem better.

what i've understood till now is,

In a web site when you give a parameter like vehicle no and search, you are getting the details. you want to implement that locally. is my understanding correct?

Thanks
AniJos

[rinu mariam johnson]

Hi anijos
wat u understood is right.yes,i hav to get search results based on the input vehicle no(file no) and finally i hav to store the results in a csv file(i want to do this programmaticall y in my web applicationusin g vb.net or c#).
Pls pardon me for not exposing the url bcos i dont hav the permission for that.Hope u understood my situation.

If any idea is there,pls help me.
regards
rinu

[Frinavale]

Are you having problems retrieving the information?
Or are you having problems storing the information retrieved into a CVS file?

-Frinny

[rinu mariam johnson]

Hi,
I didnt get the first part itself..(retrie ving the information from the website)
I tried a lot with datascraping code,but it didnt work.
I got the html of the website,after that, am not able to get the input tag of the concerned textbox(using getElementsByTa gName)
i'm stuck from that onwards.
hope u guys can help me

regards
rinu

[Frinavale]

Go to the website and look at the HTML code to find out what the name of the input field is.

You're going to have to use the HttpWebRequest to request the page's content (using the GetResponse() method). From there you'll have to parse through the HTML (returned to you in a HttpWebResponse object) and find the input field.

Please look into how to retrieve web content and investigate how to parse that information before continuing to attempt to write anything to a CVS file.

Taking smaller steps will keep you from getting overwhelmed by the problem...and it will make it easier for us to answer your questions.

-Frinny

[rinu mariam johnson]

Hi,
I used HttpWebRequest to request the content of the website and is returned in a HttpWebResponse object.
I got the html of the website,but my problem is,i'm not able to parse through the html of the website(not getting the match of the concerned textbox' input tag).

This code is working properly for website address like "http://www.abc.com"
but can you give me solution for such like (http://www.abc.com/index.php?name=abc&id=123) for user authentication as u give? how can I get the contents from such types of pages.. (I am not able to expose the exact website address..www.abc.com is just given for example)

I am attaching the code :
[code=vbnet]
Dim str As String = ""
Dim uri As New Uri("http://www.abc.com/index.php?name= abc&id=123")
Dim request As HttpWebRequest = DirectCast(WebR equest.Create(u ri.AbsoluteUri) , HttpWebRequest)
request.AllowAu toRedirect = True
request.Maximum AutomaticRedire ctions = 5
request.KeepAli ve = True
request.Timeout = &H7530

Dim response As HttpWebResponse = Nothing
Try
response = DirectCast(requ est.GetResponse , HttpWebResponse )
Catch exception1 As WebException
literal1.Text = exception1.Mess age
End Try


Dim reader As StreamReader = Nothing
Try
Dim encoding As New UTF8Encoding
reader = New StreamReader(re sponse.GetRespo nseStream, encoding)
str = reader.ReadToEn d
literal1.Text = str
Catch exception As Exception
literal1.Text = exception.Messa ge
End Try
[/code]
Thanx
Rinu

[anijos]

Try

request.Credent ials = new NetworkCredenti al(username, password);

AniJos

[Frinavale]

Try

request.Credent ials = new NetworkCredenti al(username, password);

AniJos

:) This has already been covered, from what I understand NetworkCredenti als don't work with Forms Authentication.

[rinu mariam johnson]

Hi..
I am still fed up with the problem,as I said earlier,i got the html of the website,but i'm not able to get the match for that input tag from this html,I tried the code below:
[code=vbnet]
HTMLDoc = wbBrowser.Docum ent

Dim iHTMLCol As IHTMLElementCol lection
Dim iHTMLEle As IHTMLElement
''HTMLDoc.focus ()

iHTMLCol = HTMLDoc.getElem entsByTagName(" input") ' Type the Traffic file no in the TrfNo text box

'(when i am trying to get HTMLDoc.getElem entsByTagName(" input").length. .it is returning 0,eventhough so many input tags r there)
For Each iHTMLEle In iHTMLCol
If Not iHTMLEle.getAtt ribute("name") Is Nothing Then str = iHTMLEle.getAtt ribute("name"). ToString
If str = "TrfNo" Then
iHTMLEle.setAtt ribute("value", 1234)
End If
Exit For
End If
Next


' Press the submit button

For Each iHTMLEle In iHTMLCol
If Not iHTMLEle.getAtt ribute("name") Is Nothing Then

If iHTMLEle.outerH TML = "<INPUT type=SUBMIT" & _
"name=btnG" & _
"value=Sear ch> " Then
'document.forms[0].submitBttn.cli ck()
iHTMLEle.click( )
Exit For
End If
End If
Next

Do
Loop Until Not wbBrowser.Busy[/code]

[Ramk]

Plz use [code] tags for when you are posting code.

Try using this:

 
HtmlDocument.GetFilteredHtmlTags()