URL 'special character' replacements

[Claude Henchoz]

Hi guys

I have a huge list of URLs. These URLs all have ASCII codes for special
characters, like "%20" for a space or "%21" for an exclamation mark.

I've already googled quite some time, but I have not been able to find
any elegant way on how to replace these with their 'real' counterparts
(" " and "!").

Of course, I could just replace(), but that seems to be a lot of work.

Thanks for any help.

Cheers, Claude

[Duncan Booth]

Re: URL 'special character' replacements

Claude Henchoz wrote:
[color=blue]
> I have a huge list of URLs. These URLs all have ASCII codes for special
> characters, like "%20" for a space or "%21" for an exclamation mark.
>
> I've already googled quite some time, but I have not been able to find
> any elegant way on how to replace these with their 'real' counterparts
> (" " and "!").
>
> Of course, I could just replace(), but that seems to be a lot of work.
>[/color]

urllib.unquote( ) or urllib.unquote_ plus() as appropriate:

unquote( string)

Replace "%xx" escapes by their single-character equivalent.
Example: unquote('/%7Econnolly/') yields '/~connolly/'.


unquote_plus( string)

Like unquote(), but also replaces plus signs by spaces, as required for
unquoting HTML form values.

[Fredrik Lundh]

Re: URL 'special character' replacements

Claude Henchoz wrote:
[color=blue]
> I have a huge list of URLs. These URLs all have ASCII codes for special
> characters, like "%20" for a space or "%21" for an exclamation mark.
>
> I've already googled quite some time, but I have not been able to find
> any elegant way on how to replace these with their 'real' counterparts
> (" " and "!").
>
> Of course, I could just replace(), but that seems to be a lot of work.[/color]
[color=blue][color=green][color=darkred]
>>> import urllib
>>> urllib.unquote( "http://docs.python.org/lib/module-urllib.html%20% 21")[/color][/color][/color]
'http://docs.python.org/lib/module-urllib.html !'

</F>

[Tim N. van der Leeuw]

Re: URL 'special character' replacements

My outline for a solution would be:

- Use StringIO or cStringIO for reading the original URLs character for
character, and to build the result URLs character for character

- When you read a '%' then read the next 2 character (should be
digits!!!) and create a new string with them
- The numbers like '20' etc. are hexadecimal values, meaning integers
with base 16.
Get the actual int-value like this:
code_int = int(code_str, 16)
- Convert to character as: code_chr = chr(code_int)
- Write this character to the output cStringIO buffer
- When the whole URL is done, do getvalue() to get the string of the
new URL and close the cStringIO buffer.

Is that sufficiently comprehensible? Or still too convoluted for you?

(PS: I researched doing it the manual way, 'the hard way'. However,
there are plenty of libraries in Python for all sorts of internet
stuff. Perhaps urllib or urllib2 already has the functionality that you
need -- didn't look it up)

cheers,

--Tim

[Brett g Porter]

Re: URL 'special character' replacements

Claude Henchoz wrote:[color=blue]
> Hi guys
>
> I have a huge list of URLs. These URLs all have ASCII codes for special
> characters, like "%20" for a space or "%21" for an exclamation mark.
>
> I've already googled quite some time, but I have not been able to find
> any elegant way on how to replace these with their 'real' counterparts
> (" " and "!").
>
> Of course, I could just replace(), but that seems to be a lot of work.
>
> Thanks for any help.
>
> Cheers, Claude
>[/color]

The standard library module 'urllib' gies you two choices, depending on
the exact behavior you'd like:

http://www.python.org/doc/2.3.2/lib/module-urllib.html

unquote(string)
Replace "%xx" escapes by their single-character equivalent.

Example: unquote('/%7Econnolly/') yields '/~connolly/'.

unquote_plus(st ring)
Like unquote(), but also replaces plus signs by spaces, as required
for unquoting HTML form values.


--
// Today's Oblique Strategy (© Brian Eno/Peter Schmidt):
// Accretion
// Brett g Porter * BgPorter@acm.or g

[Claude Henchoz]

Re: URL 'special character' replacements

Thanks guys, I like the urllib solution. Stupid me, looked at urllib
reference, but thought that "quote" and "unquote" deal with
_&_n_b_s_p_;_ style entities.

[Richie Hindle]

Re: URL 'special character' replacements


[Claude][color=blue]
> I have a huge list of URLs. These URLs all have ASCII codes for special
> characters, like "%20" for a space or "%21" for an exclamation mark.[/color]

You need urllib.unquote:
[color=blue][color=green][color=darkred]
>>> import urllib
>>> help(urllib.unq uote)[/color][/color][/color]
Help on function unquote in module urllib:

unquote(s)
unquote('abc%20 def') -> 'abc def'.

--
Richie Hindle
richie@entrian. com