encoding problems (é and è)

**Christoph Zwerschke** · Mar 23 '06, 04:05 PM

Re: encoding problems (é and è)

bussiere bussiere wrote:[color=blue]
> hi i'am making a program for formatting string,
> i've added :
> #!/usr/bin/python
> # -*- coding: utf-8 -*-
>
> in the begining of my script but
>
> str = str.replace('Ç' , 'C')
> ...
> doesn't work it put me " and , instead of remplacing é by E[/color]

Are your sure your script and your input file *is* actually encoded with
utf-8? If it does not work as expected, it is probably latin-1, just
like your posting. Try changing the coding to latin-1. Does it work now?

-- Christoph

**Larry Bates** · Mar 23 '06, 04:05 PM

Re: encoding problems (é and è)

Seems to work fine for me.
[color=blue][color=green][color=darkred]
>>> x="éÇ"
>>> x=x.replace('é' ,'E')[/color][/color][/color]
'E\xc7'[color=blue][color=green][color=darkred]
>>> x=x.replace('Ç' ,'C')
>>> x[/color][/color][/color]
'E\xc7'[color=blue][color=green][color=darkred]
>>> x=x.replace('Ç' ,'C')
>>> x[/color][/color][/color]
'EC'

You should also be able to use .upper() method to
uppercase everything in the string in a single statement:

tstr=ligneA.upp er()

Note: you should never use 'str' as a variable as
it will mask the built-in str function.

-Larry Bates

bussiere bussiere wrote:[color=blue]
> hi i'am making a program for formatting string,
> or
> i've added :
> #!/usr/bin/python
> # -*- coding: utf-8 -*-
>
> in the begining of my script but
>
> str = str.replace('Ç' , 'C')
> str = str.replace('é' , 'E')
> str = str.replace('É' , 'E')
> str = str.replace('è' , 'E')
> str = str.replace('È' , 'E')
> str = str.replace('ê' , 'E')
>
>
> doesn't work it put me " and , instead of remplacing é by E
>
>
> if someone have an idea it could be great
>
> regards
> Bussiere
> ps : i've added the whole script under :
>
>
>
>
>
>
> _______________ _______________ _______________ _______________ ______________
>
>
>
>
> #!/usr/bin/python
> # -*- coding: utf-8 -*-
> import fileinput, glob, string, sys, os, re
>
> fichA=raw_input ("Entrez le nom du fichier d'entree : ")
> print ("\n")
> fichC=raw_input ("Entrez le nom du fichier de sortie : ")
> print ("\n")
> normalisation1 = raw_input("Norm aliser les adresses 1 (ex : Avenue->
> AV) (O/N) ou A pour tout normaliser \n")
> normalisation1 = normalisation1. upper()
>
> if normalisation1 != "A":
> print ("\n")
> normalisation2 = raw_input("Norm aliser les civilités (ex :
> Docteur-> DR) (O/N) \n")
> normalisation2 = normalisation2. upper()
> print ("\n")
> normalisation3 = raw_input("Norm aliser les Adresses 2 (ex :
> Place-> PL) (O/N) \n")
> normalisation3 = normalisation3. upper()
>
>
> normalisation4 = raw_input("Norm aliser les caracteres / et - (ex :
> / -> ) (O/N) \n" )
> normalisation4 = normalisation4. upper()
>
> if normalisation1 == "A":
> normalisation1 = "O"
> normalisation2 = "O"
> normalisation3 = "O"
> normalisation4 = "O"
>
>
> fiA=open(fichA, "r")
> fiC=open(fichC, "w")
>
>
> compteur = 0
>
> while 1:
>
> ligneA=fiA.read line()
>
>
>
> if ligneA == "":
>
> break
>
> if ligneA != "":
> str = ligneA
> str = str.replace('a' , 'A')
> str = str.replace('b' , 'B')
> str = str.replace('c' , 'C')
> str = str.replace('d' , 'D')
> str = str.replace('e' , 'E')
> str = str.replace('f' , 'F')
> str = str.replace('g' , 'G')
> str = str.replace('h' , 'H')
> str = str.replace('i' , 'I')
> str = str.replace('j' , 'J')
> str = str.replace('k' , 'K')
> str = str.replace('l' , 'L')
> str = str.replace('m' , 'M')
> str = str.replace('n' , 'N')
> str = str.replace('o' , 'O')
> str = str.replace('p' , 'P')
> str = str.replace('q' , 'Q')
> str = str.replace('r' , 'R')
> str = str.replace('s' , 'S')
> str = str.replace('t' , 'T')
> str = str.replace('u' , 'U')
> str = str.replace('v' , 'V')
> str = str.replace('w' , 'W')
> str = str.replace('x' , 'X')
> str = str.replace('y' , 'Y')
> str = str.replace('z' , 'Z')
>
> str = str.replace('ç' , 'C')
> str = str.replace('Ç' , 'C')
> str = str.replace('é' , 'E')
> str = str.replace('É' , 'E')
> str = str.replace('è' , 'E')
> str = str.replace('È' , 'E')
> str = str.replace('ê' , 'E')
> str = str.replace('Ê' , 'E')
> str = str.replace('ë' , 'E')
> str = str.replace('Ë' , 'E')
> str = str.replace('ä' , 'A')
> str = str.replace('Ä' , 'A')
> str = str.replace('à' , 'A')
> str = str.replace('À' , 'A')
> str = str.replace('Á' , 'A')
> str = str.replace('Â' , 'A')
> str = str.replace('Ä' , 'A')
> str = str.replace('Ã' , 'A')
> str = str.replace('â' , 'A')
> str = str.replace('Ä' , 'A')
> str = str.replace('ï' , 'I')
> str = str.replace('Ï' , 'I')
> str = str.replace('î' , 'I')
> str = str.replace('Î' , 'I')
> str = str.replace('ô' , 'O')
> str = str.replace('Ô' , 'O')
> str = str.replace('ö' , 'O')
> str = str.replace('Ö' , 'O')
> str = str.replace('Ú' ,'U')
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
>
>
>
> if normalisation1 == "O":
> str = str.replace('AV ENUE', 'AV')
> str = str.replace('BO ULEVARD', 'BD')
> str = str.replace('FA UBOURG', 'FBG')
> str = str.replace('GE NERAL', 'GAL')
> str = str.replace('CO MMANDANT', 'CMDT')
> str = str.replace('MA RECHAL', 'MAL')
> str = str.replace('PR ESIDENT', 'PRDT')
> str = str.replace('SA INT', 'ST')
> str = str.replace('SA INTE', 'STE')
> str = str.replace('LO TISSEMENT', 'LOT')
> str = str.replace('RE SIDENCE', 'RES')
> str = str.replace('IM MEUBLE', 'IMM')
> str = str.replace('IM EUBLE', 'IMM')
> str = str.replace('BA TIMENT', 'BAT')
>
> if normalisation2 == "O":
> str = str.replace('MO NSIEUR', 'M')
> str = str.replace('MR ', 'M')
> str = str.replace('MA DAME', 'MME')
> str = str.replace('MA DEMOISELLE', 'MLLE')
> str = str.replace('DO CTEUR', 'DR')
> str = str.replace('PR OFESSEUR', 'PR')
> str = str.replace('MO NSEIGNEUR', 'MGR')
> str = str.replace('M ME','MME')
>
>
> if normalisation3 == "O":
> str = str.replace('PL ACE', 'PL')
> str = str.replace('IM PASSE', 'IMP')
> str = str.replace('ES PLANADE', 'ESP')
> str = str.replace('RO ND POINT', 'RPT')
> str = str.replace('RO UTE', 'RTE')
> str = str.replace('PA SSAGE', 'PAS')
> str = str.replace('SQ UARE', 'SQ')
> str = str.replace('AL LEE', 'ALL')
> str = str.replace('ES CALIER', 'ESC')
> str = str.replace('ET AGE', 'ETG')
> str = str.replace('PO RTE', 'PTE')
> str = str.replace('AP PARTEMENT', 'APT')
> str = str.replace('AP ARTEMENT', 'APT')
> str = str.replace('AV ENUE', 'AV')
> str = str.replace('BO ULEVARD', 'BD')
> str = str.replace('ZO NE D ACTIVITE', 'ZA')
> str = str.replace('ZO NE D ACTIVITEE', 'ZA')
> str = str.replace('ZO NE D AMENAGEMENT CONCERTE', 'ZAC')
> str = str.replace('ZO NE D AMENAGEMENT CONCERTEE', 'ZAC')
> str = str.replace('ZO NE INDUSTRELLE', 'ZI')
> str = str.replace('CE NTRE COMMERCIAL', 'CCAL')
> str = str.replace('CE NTRE', 'CTRE')
> str = str.replace('C. CIAL','CCAL')
> str = str.replace('CT RE CIAL','CCAL')
> str = str.replace('CT RE CCAL','CCAL')
> str = str.replace('GA LERIE','GAL')
> str = str.replace('MA RTYR', 'M')
> str = str.replace('AN CIENS', 'AC')
> str = str.replace('AN CIEN', 'AC')
> str = str.replace('RE VEREND PERE','R P')
>
> if normalisation4 == "O":
> str = str.replace(';\ "', ' ')
> str = str.replace('\" ', ' ')
> str = str.replace('\' ', ' ')
> str = str.replace('-', ' ')
> str = str.replace(',' , ' ')
> str = str.replace('\\ ', ' ')
> str = str.replace('\/', ' ')
> str = str.replace('&' , ' ')
> str = str.replace('%' , ' ')
> str = str.replace('*' , ' ')
> str = str.replace(' ', ' ')
> str = str.replace('.' , ' ')
> str = str.replace('_' , ' ')
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
> str = str.replace('?' , ' ')
> str = str.replace('%' , ' ')
> str = str.replace('|' , ' ')
>
>
>
>
>
>
>
>
>
>
>
>
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
> fiC.write(str)
> compteur += 1
> print compteur, "\n"
>
>
> print "FINIT"
> fiA.close()
> fiC.close()[/color]

**John Machin** · Mar 23 '06, 09:25 PM

Re: encoding problems (é and è)

On 23/03/2006 10:07 PM, bussiere bussiere wrote:[color=blue]
> hi i'am making a program for formatting string,
> or
> i've added :
> #!/usr/bin/python
> # -*- coding: utf-8 -*-
>
> in the begining of my script but
>
> str = str.replace('Ç' , 'C')
> str = str.replace('é' , 'E')
> str = str.replace('É' , 'E')
> str = str.replace('è' , 'E')
> str = str.replace('È' , 'E')
> str = str.replace('ê' , 'E')
>
>
> doesn't work it put me " and , instead of remplacing é by E
>
>
> if someone have an idea it could be great[/color]

Hi, I've added some comments below ... I hope they help.
Cheers,
John
[color=blue]
>
> regards
> Bussiere
> ps : i've added the whole script under :
> _______________ _______________ _______________ _______________ ______________[/color]
[snip][color=blue]
>
> if ligneA != "":
> str = ligneA
> str = str.replace('a' , 'A')[/color]
[snip][color=blue]
> str = str.replace('z' , 'Z')
>
> str = str.replace('ç' , 'C')
> str = str.replace('Ç' , 'C')
> str = str.replace('é' , 'E')
> str = str.replace('É' , 'E')
> str = str.replace('è' , 'E')[/color]
[snip][color=blue]
> str = str.replace('Ú' ,'U')[/color]

You can replace ALL of this upshifting and accent removal in one blow by
using the string translate() method with a suitable table.
[color=blue]
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')
> str = str.replace(' ', ' ')[/color]

The standard Python idiom for normalising whitespace is
strg = ' '.join(strg.spl it())
[color=blue][color=green][color=darkred]
>>> strg = ' ALLO BUSSIERE\tCA VA? '
>>> strg.split()[/color][/color][/color]
['ALLO', 'BUSSIERE', 'CA', 'VA?'][color=blue][color=green][color=darkred]
>>> ' '.join(strg.spl it())[/color][/color][/color]
'ALLO BUSSIERE CA VA?'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

[snip][color=blue]
> if normalisation2 == "O":
> str = str.replace('MO NSIEUR', 'M')
> str = str.replace('MR ', 'M')[/color]

You need to be very careful with this approach. You are changing EVERY
occurrence of "MR" in the string, not just where it is a whole "word"
meaning "Monsieur".
Copnstructed example of what can go wrong:[color=blue][color=green][color=darkred]
>>> strg = 'MR IMRE NAGY, 123 PRIMROSE STREET, SHAMROCK VALLEY'
>>> strg.replace('M R', 'M')[/color][/color][/color]
'M IME NAGY, 123 PRIMOSE STREET, SHAMOCK VALLEY'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

A real, non-constructed history lesson: A certain database indicated
duplicate records by having the annotation "DUP" in the surname field
e.g. "SMITH DUP". Fortunately it was detected in testing that the
so-called clean-up was causing DUPLESSIS to become PLESSIS and DUPRAT to
become RAT!

Two points here: (1) Split up your strings into "words" or "tokens".
Using strg.split() is a start but you may need something more
sophisticated e.g. "-" as an additional token separator. (2) Instead of
writing out all those lines of code, consider putting those
substitutions in a dictionary:

title_substitut ion = {
'MONSIEUR': 'M',
'MR': 'M',
'MADAME': 'MME',
# etc
}
Next level of improvement is to read that stuff from a file.
[snip][color=blue]
>
> if normalisation4 == "O":
> str = str.replace(';\ "', ' ')
> str = str.replace('\" ', ' ')
> str = str.replace('\' ', ' ')
> str = str.replace('-', ' ')
> str = str.replace(',' , ' ')
> str = str.replace('\\ ', ' ')
> str = str.replace('\/', ' ')
> str = str.replace('&' , ' ')[/color]
[snip]
Again, consider the string translate() method.
Also, consider that some of those characters may have some meaning that
you perhaps shouldn't blow away e.g. compare 'SMITH & WESSON' with
'SMITH ET WESSON' :-)

**Peter Otten** · Mar 23 '06, 09:45 PM

Re: encoding problems (é and è)

John Machin wrote:
[color=blue]
> You can replace ALL of this upshifting and accent removal in one blow by
> using the string translate() method with a suitable table.[/color]

Only if you convert to unicode first or if your data maintains 1 byte == 1
character, in particular it is not UTF-8.

Peter

**John Machin** · Mar 23 '06, 10:45 PM

Re: encoding problems (é and è)

On 24/03/2006 8:36 AM, Peter Otten wrote:[color=blue]
> John Machin wrote:
>[color=green]
>>You can replace ALL of this upshifting and accent removal in one blow by
>>using the string translate() method with a suitable table.[/color]
>
> Only if you convert to unicode first or if your data maintains 1 byte == 1
> character, in particular it is not UTF-8.
>[/color]

I'm sorry, I forgot that there were people who are unaware that
variable-length gizmos like UTF-8 and various legacy CJK encodings are
for storage & transmission, and are better changed to a
one-character-per-storage-unit representation before *ANY* data
processing is attempted.

:-)
Unicode? I'm just a benighted Anglo from the a**-end of the globe; who
am I to be preaching Unicode to a European?
(-:

**Duncan Booth** · Mar 24 '06, 09:25 AM

Re: encoding problems (é and è)

Peter Otten wrote:
[color=blue][color=green]
>> You can replace ALL of this upshifting and accent removal in one blow
>> by using the string translate() method with a suitable table.[/color]
>
> Only if you convert to unicode first or if your data maintains 1 byte
> == 1 character, in particular it is not UTF-8.
>[/color]

There's a nice little codec from Skip Montaro for removing accents from
latin-1 encoded strings. It also has an error handler so you can convert
from unicode to ascii and strip all the accents as you do so:

http://orca.mojam.com/~skip/python/latscii.py

[color=blue][color=green][color=darkred]
>>> import latscii
>>> import htmlentitydefs
>>> print u'\u00c9'.encod e('ascii','repl acelatscii')[/color][/color][/color]
E[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

So Bussiere could replace a large chunk of his code with:

ligneA = ligneA.decode(I NPUTENCODING).e ncode('ascii', 'replacelatscii ')
ligneA = ligneA.upper()

INPUTENCODING is 'utf8' unless (one possible explanation for his problem)
his files are actually in some different encoding.

Unfortunately, just as I finished writing this I discovered that the
latscii module isn't as robust as I thought, it blows up on consecutive
accented characters.

:(

**Peter Otten** · Mar 24 '06, 11:45 AM

Re: encoding problems (é and è)

Duncan Booth wrote:
[color=blue]
> There's a nice little codec from Skip Montaro for removing accents from
> latin-1 encoded strings. It also has an error handler so you can convert
> from unicode to ascii and strip all the accents as you do so:
>
> http://orca.mojam.com/~skip/python/latscii.py
>[color=green][color=darkred]
>>>> import latscii
>>>> import htmlentitydefs
>>>> print u'\u00c9'.encod e('ascii','repl acelatscii')[/color][/color]
> E[color=green][color=darkred]
>>>>[/color][/color]
>
> So Bussiere could replace a large chunk of his code with:
>
> ligneA = ligneA.decode(I NPUTENCODING).e ncode('ascii',
> 'replacelatscii ') ligneA = ligneA.upper()
>
> INPUTENCODING is 'utf8' unless (one possible explanation for his problem)
> his files are actually in some different encoding.
>
> Unfortunately, just as I finished writing this I discovered that the
> latscii module isn't as robust as I thought, it blows up on consecutive
> accented characters.
>
> :([/color]

You made me look into it -- and I found that reusing the decoding map as the
encoding map lets you write
[color=blue][color=green][color=darkred]
>>> u"Élève ééé".encode("la tscii")[/color][/color][/color]
'Eleve eee'

without relying on the faulty error handler. I tried to fix the handler,
too:
[color=blue][color=green][color=darkred]
>>> u"Élève ééé".encode("as cii", "replacelatscii ")[/color][/color][/color]
'Eleve eee'[color=blue][color=green][color=darkred]
>>> g = u"\N{GREEK CAPITAL LETTER GAMMA}"
>>> (u"möglich ähnlich üblich ááá" + g*3).encode("as cii", "replacelatscii ")[/color][/color][/color]
'moglich ahnlich ublich aaa???'

No real testing was performed.

Peter

--- latscii_old.py 2006-03-24 11:45:22.580588 520 +0100
+++ latscii.py 2006-03-24 11:48:13.191651 696 +0100
@@ -141,7 +141,7 @@

### Encoding Map

-encoding_map = codecs.make_ide ntity_dict(rang e(256))
+encoding_map = decoding_map

### From Martin Blais
@@ -166,9 +166,9 @@
## ustr.encode('as cii', 'replacelatscii ')
##
def latscii_error( uerr ):
- key = ord(uerr.object[uerr.start:uerr .end])
+ key = ord(uerr.object[uerr.start])
try:
- return unichr(decoding _map[key]), uerr.end
+ return unichr(decoding _map[key]), uerr.start + 1
except KeyError:
handler = codecs.lookup_e rror('replace')
return handler(uerr)

**John Machin** · Mar 24 '06, 11:55 AM

Re: encoding problems (é and è)

On 24/03/2006 8:11 PM, Duncan Booth wrote:[color=blue]
> Peter Otten wrote:
>
>[color=green][color=darkred]
>>>You can replace ALL of this upshifting and accent removal in one blow
>>>by using the string translate() method with a suitable table.[/color]
>>
>>Only if you convert to unicode first or if your data maintains 1 byte
>>== 1 character, in particular it is not UTF-8.
>>[/color]
>
>
> There's a nice little codec from Skip Montaro for removing accents from[/color]

For the benefit of those who may read only this far, it is NOT nice.
[color=blue]
> latin-1 encoded strings. It also has an error handler so you can convert
> from unicode to ascii and strip all the accents as you do so:
>
> http://orca.mojam.com/~skip/python/latscii.py
>
>[color=green][color=darkred]
>>>>import latscii
>>>>import htmlentitydefs
>>>>print u'\u00c9'.encod e('ascii','repl acelatscii')[/color][/color]
>
> E
>
>
> So Bussiere could replace a large chunk of his code with:[/color]

Could, but definitely shouldn't.
[color=blue]
>
> ligneA = ligneA.decode(I NPUTENCODING).e ncode('ascii', 'replacelatscii ')
> ligneA = ligneA.upper()
>
> INPUTENCODING is 'utf8' unless (one possible explanation for his problem)
> his files are actually in some different encoding.
>
> Unfortunately, just as I finished writing this I discovered that the
> latscii module isn't as robust as I thought, it blows up on consecutive
> accented characters.
>
> :(
>[/color]
Some of the transformations are a little unfortunate :-(
0x00d0: ord('D'), # Ð
0x00f0: ord('o'), # ð
Icelandic capital eth becomes D, OK; but the small letter becomes o!!!
The Icelandic thorn letters become P & p (based on physical appearance),
when they should become Th and th.
The German letter Eszett (00DF) becomes B (appearance) when it should be ss.
Creating alphabetics out of punctuation is scarcely something that
bussiere should be interested in:
0x00a2: ord('c'), # ¢
0x00a4: ord('o'), # ¤
0x00a5: ord('Y'), # ¥
0x00a7: ord('S'), # §
0x00a9: ord('c'), # ©
0x00ae: ord('R'), # ®
0x00b6: ord('P'), # ¶

**Peter Otten** · Mar 24 '06, 12:55 PM

Re: encoding problems (é and è)

John Machin wrote:
[color=blue]
> 0x00d0: ord('D'), # Ð
> 0x00f0: ord('o'), # ð
> Icelandic capital eth becomes D, OK; but the small letter becomes o!!![/color]

I see information flow from Iceland is a bit better than from Armenia :-)
[color=blue]
> Some of the transformations are a little unfortunate :-([/color]

The OP, as you pointed out in your first post in this thread, has more
pressing problems with his normalization approach.

Lastly, even if all went well, turning a list of French addresses into an
ascii-uppercase graveyard would be a sad thing to do...

Peter

**Walter Dörwald** · Mar 24 '06, 04:05 PM

Re: encoding problems (X and X)

Duncan Booth wrote:
[color=blue]
> [...]
> Unfortunately, just as I finished writing this I discovered that the
> latscii module isn't as robust as I thought, it blows up on consecutive
> accented characters.
>
> :([/color]

Replace the error handler with this (untested) and it should work with
consecutive accented characters:

def latscii_error( uerr ):
v = []
for c in uerr.object[uerr.start:uerr .end]
key = ord(c)
try:
v.append(unichr (decoding_map[key]))
except KeyError:
v.append(u"?")
return (u"".join(v), uerr.end)
codecs.register _error('replace latscii', latscii_error)

Bye,
Walter Dörwald

**John Machin** · Mar 24 '06, 07:05 PM

Re: encoding problems (é and è)

On 24/03/2006 11:44 PM, Peter Otten wrote:[color=blue]
> John Machin wrote:
>
>[color=green]
>>0x00d0: ord('D'), # Ð
>>0x00f0: ord('o'), # ð
>>Icelandic capital eth becomes D, OK; but the small letter becomes o!!![/color]
>
>
> I see information flow from Iceland is a bit better than from Armenia :-)[/color]

No information flow needed. Capital letter BLAH -> D and small letter
BLAH -> o should trigger one's palpable nonsense detector for *any* BLAH.
[color=blue]
>
>[color=green]
>>Some of the transformations are a little unfortunate :-([/color]
>
>
> The OP, as you pointed out in your first post in this thread, has more
> pressing problems with his normalization approach.
>
> Lastly, even if all went well, turning a list of French addresses into an
> ascii-uppercase graveyard would be a sad thing to do...[/color]

Oh indeed. Not only sad, but incredibly stupid. I fervently hope and
trust that such a normalisation is intended only for fuzzy matching
purposes. I can't imagine that anyone would contemplate writing the
output to storage for any reason other than logging or for regression
testing. Update it back to the database? Do you know anyone who would do
that??

**Fredrik Lundh** · Mar 24 '06, 07:45 PM

Re: encoding problems (é and è)

John Machin wrote:
[color=blue]
> Some of the transformations are a little unfortunate :-([/color]

here's a slightly silly way to map a unicode string to its "unaccented "
version:

###

import unicodedata, sys

CHAR_REPLACEMEN T = {
0xc6: u"AE", # LATIN CAPITAL LETTER AE
0xd0: u"D", # LATIN CAPITAL LETTER ETH
0xd8: u"OE", # LATIN CAPITAL LETTER O WITH STROKE
0xde: u"Th", # LATIN CAPITAL LETTER THORN
0xdf: u"ss", # LATIN SMALL LETTER SHARP S
0xe6: u"ae", # LATIN SMALL LETTER AE
0xf0: u"d", # LATIN SMALL LETTER ETH
0xf8: u"oe", # LATIN SMALL LETTER O WITH STROKE
0xfe: u"th", # LATIN SMALL LETTER THORN
}

class unaccented_map( dict):

def mapchar(self, key):
ch = self.get(key)
if ch is not None:
return ch
ch = unichr(key)
try:
ch = unichr(int(unic odedata.decompo sition(ch).spli t()[0], 16))
except (IndexError, ValueError):
ch = CHAR_REPLACEMEN T.get(key, ch)
# uncomment the following line if you want to remove remaining
# non-ascii characters
# if ch >= u"\x80": return None
self[key] = ch
return ch

if sys.version >= "2.5":
__missing__ = mapchar
else:
__getitem__ = mapchar

assert isinstance(myst ring, unicode)

print mystring.transl ate(unaccented_ map())

###

if the source string is not unicode, you can use something like

s = mystring.decode ("iso-8859-1")
s = s.translate(una ccented_map())
s = s.encode("ascii ", "ignore")

(this works well for characters in the latin-1 range, at least. no
guarantees for other character ranges)

</F>

**Serge Orlov** · Mar 25 '06, 05:35 AM

Re: encoding problems (é and è)

Jean-Paul Calderone wrote:[color=blue]
> On Fri, 24 Mar 2006 09:33:19 +1100, John Machin <sjmachin@lexic on.net> wrote:[color=green]
> >On 24/03/2006 8:36 AM, Peter Otten wrote:[color=darkred]
> >> John Machin wrote:
> >>
> >>>You can replace ALL of this upshifting and accent removal in one blow by
> >>>using the string translate() method with a suitable table.
> >>
> >> Only if you convert to unicode first or if your data maintains 1 byte == 1
> >> character, in particular it is not UTF-8.
> >>[/color]
> >
> >I'm sorry, I forgot that there were people who are unaware that
> >variable-length gizmos like UTF-8 and various legacy CJK encodings are
> >for storage & transmission, and are better changed to a
> >one-character-per-storage-unit representation before *ANY* data
> >processing is attempted.[/color]
>
> Unfortunately, unicode only appears to solve this problem in a sane manner.[/color]

What problem do you mean? Loose matching is solved by unicode in a sane
manner, it is described in the unicode collation algorithm.

Serge.

encoding problems (é and è)

encoding problems (é and è)

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment