Using python to create a web crawler/ spider

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Billy Bob
    New Member
    • Jun 2010
    • 1

    Using python to create a web crawler/ spider

    Hello everyone, I thank you for your time. I am completely new to the field so I apologize for any ignorance. I am trying to write a program using python that will go into a university web-page and retrieve all the ISBNs for books being used the following semester that fit certain criteria. I believe Python can do this but like I said I know almost nothing about computer programming. I believe this can be VERY easily done from what I've seen. If someone could just give me a starting point or at least let me know if Python can do this. Again, thank you for your time!!!
  • dwblas
    Recognized Expert Contributor
    • May 2008
    • 626

    #2
    Since you have no idea, I would suggest using something like links


    use "links -dump www.URL" to download and save as a text file which you can then parse and extract whatever data you want.

    Comment

    • Glenton
      Recognized Expert Contributor
      • Nov 2008
      • 391

      #3
      Python can almost certainly do this.

      You'll need the urllib and urllib2 libraries, and possibly the regular expression library. And a bunch of hours. Good luck!

      Comment

      Working...