Python and Unicode

[Suresh Iyengar]

Hello,

I want to fetch a web page and parse links in that. I am using the foll. code

file =urllib.urlopen("file:///home/suresh/html_parser/Category:Sports.html")
content = file.read()
# Process the page.

But since the page contains UTF-8 content, its not able to parse it properly. But, if I save the page locally, then its able to parse. How to handle this problem,

Thanks

[YarrOfDoom]

If I understand your problem, this should help:

If you have data in a specific encoding and want to produce a corresponding Unicode string from it, you can use the unicode() function with the encoding name as the second argument.

>>> unicode('\xc3\x a4\xc3\xb6\xc3\ xbc', 'utf-8')
u'\xe4\xf6\xfc'

If this is not what you're looking for, you can probably find your answer here.