Difference between page source and what's being displayed

**acoder** · Sep 14 '08, 07:39 PM

For JavaScript, you would have to eval it or you could add it to the main web page using dynamic script tags.

Depending on browser/add-ons, you can see the generated source in some browsers.

**vegetable21** · Sep 15 '08, 06:04 PM

Ok Thanks,

Can you provide an example of how i can evaluate it in my context?

I can view the source in my browser but that's not the problem, i'm missing some! The items displayed in the browser differ to what's in the source code.

I need the output of the JavaScript functions displayed in the view source.

**rnd me** · Sep 16 '08, 07:29 AM

if you want to display content that is provided by running javascript, you will have to run the javascript.

this will prove difficult in C#.

you might be able to use a webbrowser control to do your scraping.

you might also be able to evaluate the code a jscript, but you wouldn't have any DOM methods in that case, and your dynamic content would probably not work.

in short, you really can't do this.

you may be able to engineer a workaround, but it would likely take a lot longer than implementing another route to your solution.

**vegetable21** · Sep 16 '08, 06:27 PM

OK, thanks.

I'll try explaining what i'm trying to do and see if someone can suggest an alternate method!

Code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml">
  <head>
    <meta http-equiv="content-type" content="text/html; charset=utf-8"/>
    <title>Test</title>
    <script type="text/javascript">
	var word = ["H", "e", "l", "l", "o", "W", "o", "r", "l", "d"];
    function initialize() {
		for (i = 0; i < word.length; i++) {
        	document.getElementById("message").innerHTML += word[i];
		}
    }  
    </script>
  </head>

  <body onload="initialize()" >
    <div id="message"></div>
  </body>
</html>

Imagine the source code above saved in test.html.

Now if you view it in a browser you can clearly see "Hello World" on the page.
But if you click on view source then all you see is the above code.
Now heres my problem: if you did a search of the scraped source code from this website for "Hello World" it would not be found.
We can clearly see it on the web page, but not the source code as "Hello World" is only created at runtime.
How can i scrap "Hello World" from this website? (I need the dynamic output!)

[The above is just a simplification of what i'm trying to do, obviously i'm using regex to scrap data that changes all the time and has nothing to do with the above, but the principle is identical]

Hope this is clear enough for someone to suggest a solution, i don't mind if it uses a different approach / technology all together!

Thanks

Andy

**rnd me** · Sep 16 '08, 08:39 PM

if my proposals sound complicated, that's because they are.
screen-scraping has always been complex, hence the push toward rss.
dynamic content only serves to greatly complicate things further.

it sounds as though you want to access the data you can see in a browser.
it would only make sense then to use a browser in your solution.

unfortunately, browsers don't like sharing web-pages with javascript code.
if you solution is to be coded in javascript (the natural language of browsers), you will run into the "same-origin policy".

this means that you will not only need a browser, but a browser navigated to the site you want to scrape.

you would then run your scraping code in the context of that site.

how the heck can you run your code on someone else's site?

two suggestions:
1. greasemonkey, which is also capable of cross-domain transfers. you code a greasemonkey script file, and greasemonkey automatically injects the code into an associated webpage every time the page is visited. coupled with firebug, this makes a simple yet robust screen-scraping solution. main downside is that running your application mean navigating to the page in firefox, rather than simply executing an exe.

2. MS's webbrowser control. using a browser as an object in another language like VB or C# allows injection of code into any particular site though a series of public API properties. the current source can be read and manipulated with this interface. the main advantage to this approach is that you can do the data processing in either javascript, or the host language of your project. the disadvantage is the complexity and rigidness of a hard-coded solution tailored to a single site/design.

there may be other solutions as well:

- john resig recently emulated a browser's DOM entirely in javascript running on rhino. you may be able to harness the updates using a model like his.

-you might be able to download and reverse-engineer the javascript files to grab the apropos data from the script block. this would hinge largely on the complexity of the update script.

-you could use ASP jscript and a msHTMLDocument object to meld the changes to the document and capture the result. this avoids the same origin problems that plague browser solutions.

in conclusion, it can be done, but it won't be easy. i like the greasemonkey scenario for its simplicity and available documentation. if you can live with running firefox to grab your updates, this is certainly the path of least resistance.

Difference between page source and what's being displayed

Difference between page source and what's being displayed

Comment

Comment

Comment

Comment

Comment