Well I have just been having a really hard time lately and am hoping for a new thought on some trouble that I am having programmaticall y accessing my companies website to download my paycheck.
The problem lies in the levels or security that they use. On the surface there is (as expected) a user name and password, then you are in. Simple enough huh :) But on the other side there is a certificate which I think I am adding correctly and multiple cookies (but what is really throwing me, and as far as I understand is there is no way to deal with this) is the JavaScript level of determining where to send you.
I have been able to only get the front page and no further using HttpWebRequest/Response. I am adding my local certificate file, my credentials, adding a cookie container, etc..... I'm just lost at this point. Even if I try to directly access the final URL of the page that I want I just get bumped back to the front page. I have examined the headers as I log in and let me tell you that once you hit 'log in' you are redirected multiple times and the final capture is quite long. I'm quite sure that they are actively trying to prevent scraping. The company is quite paranoid about security.
I am able to access the site via a webBrowser control and even set up an automated way to get me where I want to go but it's slower that I would like. Is there a way to convert (cast) a webBrowser connection into a HttpWebRequest/Response so I can programmaticall y navigate once I'm through the front door?
I can offer some of my code if you feel that it is necessary but I'm pretty sure it's just impossible to do this without the webBrowser control.
So the base question is simply; Is there always a way to access a site programmaticall y or are there some sites that this is just not possible?
The problem lies in the levels or security that they use. On the surface there is (as expected) a user name and password, then you are in. Simple enough huh :) But on the other side there is a certificate which I think I am adding correctly and multiple cookies (but what is really throwing me, and as far as I understand is there is no way to deal with this) is the JavaScript level of determining where to send you.
I have been able to only get the front page and no further using HttpWebRequest/Response. I am adding my local certificate file, my credentials, adding a cookie container, etc..... I'm just lost at this point. Even if I try to directly access the final URL of the page that I want I just get bumped back to the front page. I have examined the headers as I log in and let me tell you that once you hit 'log in' you are redirected multiple times and the final capture is quite long. I'm quite sure that they are actively trying to prevent scraping. The company is quite paranoid about security.
I am able to access the site via a webBrowser control and even set up an automated way to get me where I want to go but it's slower that I would like. Is there a way to convert (cast) a webBrowser connection into a HttpWebRequest/Response so I can programmaticall y navigate once I'm through the front door?
I can offer some of my code if you feel that it is necessary but I'm pretty sure it's just impossible to do this without the webBrowser control.
So the base question is simply; Is there always a way to access a site programmaticall y or are there some sites that this is just not possible?
Comment