I am working on a python webcrawler, that will extract all links from an
html page, and add them to a queue, The problem I am having is building
absolute links from relative links, as there are so many different types of
relative links. If I just append the relative links to the current url, some
websites will send it into a never-ending loop.
What I am looking for is a regexp that will extract the root url from any
url string I pass to it, such as
'http://example.com/stuff/stuff/morestuff/index.html'
Regexp = http:example.co m
'http://anotherexample. com/stuff/index.php
Regexp = 'http://anotherexample. com/
'http://example.com/stuff/stuff/
Regext = 'http://example.com'
html page, and add them to a queue, The problem I am having is building
absolute links from relative links, as there are so many different types of
relative links. If I just append the relative links to the current url, some
websites will send it into a never-ending loop.
What I am looking for is a regexp that will extract the root url from any
url string I pass to it, such as
'http://example.com/stuff/stuff/morestuff/index.html'
Regexp = http:example.co m
'http://anotherexample. com/stuff/index.php
Regexp = 'http://anotherexample. com/
'http://example.com/stuff/stuff/
Regext = 'http://example.com'
Comment