Example Script to parse web page links and extract data?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • livin

    #1

    Example Script to parse web page links and extract data?

    I'm hoping someone knows of an example script I can see to help me build
    mine.

    I'm looking for an easy way to automate the below web site browsing and pull
    the data I'm searching for.
    Here's steps it needs to accomplish...

    1) login to the site (windows dialog when hitting web page) *optional*

    2) Choose menu link from ASP page (script shows/hides menu items depending
    on mouseover) *optional*

    3) Basic Search Form and enter zip code or city to pull all the data.

    4) After search, table shows many links (hundreds sometimes) to the actual
    data I need.
    Links are this format... <a href="javascrip t:GetAgent('AA0 59')">

    5) Each link opens new window with table providing required data.
    The URLs that each href opens is this...
    http://armls.marketlinx.com/Roster/S...sp?PubID=AA059 where the
    PubID is record I need.

    Table format looks like this:

    <tr>

    <td bgcolor="#C0C0C 0" align="center">

    <a href="javascrip t:GetAgent('MA1 42')">

    <font face="Arial" size="2">6</font></a></td>

    <td><font face="Arial" size="2">

    <a href="javascrip t:GetAgent('MA1 42')">Alaze</a><br></font></td>

    <td><font face="Arial" size="2">Mark <br></font></td>


    <td><font face="Arial" size="2">MA142</font><br>

    </td>

    <td><font face="Arial" size="2">

    <a href="javascrip t:GetBroker('CO LD56')">Banker Success
    Realty</a><br></font></td>

    <td>COLD56</td>

    <td><font face="Arial" size="2"><scrip t LANGUAGE="javas cript">

    <!--

    writePhoneNumbe r('480-999-9999');

    //--></script></td>

    </tr>





  • Steven Bethard

    #2
    Re: Example Script to parse web page links and extract data?

    livin wrote:[color=blue]
    > I'm looking for an easy way to automate the below web site browsing and pull
    > the data I'm searching for.[/color]

    This is a task that BeautifulSoup[1] is usually good for.
    [color=blue]
    > 4) After search, table shows many links (hundreds sometimes) to the actual
    > data I need.
    > Links are this format... <a href="javascrip t:GetAgent('AA0 59')">
    >
    > 5) Each link opens new window with table providing required data.
    > The URLs that each href opens is this...
    > http://armls.marketlinx.com/Roster/S...sp?PubID=AA059 where the
    > PubID is record I need.[/color]

    I'm not entirely sure I got your problem description right, but I think
    points 4 and 5 would look something like:

    base_url = 'http://armls.marketlin x.com/.../Member.asp?PubI D=AA059'
    html = urllib.urlopen( base_url).read( )
    soup = BeautifulSoup.B eautifulSoup(ht ml)

    link_matcher = re.compile(r'ja vascript:GetAge nt('[^']*')
    for link_elem in soup('a', {'href': link_matcher}):
    ...

    HTH,

    STeVe

    [1] http://www.crummy.com/software/BeautifulSoup/

    Comment

    Working...