utf - string translation

[hg]

Hi,

I'm bringing over a thread that's going on on f.c.l.python.

The point was to get rid of french accents from words.

We noticed that len('à') != len('a') and I found the hack below to fix
the "problem" ... yet I do not understand - especially since 'à' is
included in the extended ASCII table, and thus can be stored in one byte.

Any clue ?

hg





# -*- coding: utf-8 -*-
import string

def convert(mot):
print len(mot)
print mot[0]
print '%x' % ord(mot[1])
table =
string.maketran s('àâäéèêëîïôöù üû','\x00a\x00a \x00a\x00e\x00e \x00e\x00e\x00i \x00i\x00o\x00o \x00u\x00u\x00u ')

return mot.translate(t able).replace(' \x00','')


c = 'àbôö a '
print convert(c)

[Fredrik Lundh]

Re: utf - string translation

hg wrote:

We noticed that len('à') != len('a')

sounds odd.

>>len('à') == len('a')

True

are you perhaps using an UTF-8 editor?

to keep your sanity, no matter what editor you're using, I recommend
adding a coding directive to the source file, and using *only* Unicode
string literals for non-ASCII text.

or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):

# -*- coding: utf-8 -*-

and use

u'<text>'

for all non-ASCII literals.

</F>

[hg]

Re: utf - string translation

Fredrik Lundh wrote:

hg wrote:
>

>We noticed that len('à') != len('a')

>
sounds odd.
>

>>>len('à') == len('a')

True
>
are you perhaps using an UTF-8 editor?
>
to keep your sanity, no matter what editor you're using, I recommend
adding a coding directive to the source file, and using *only* Unicode
string literals for non-ASCII text.
>
or in other words, put this at the top of your file (where "utf-8" is
whatever your editor/system is using):
>
# -*- coding: utf-8 -*-
>
and use
>
u'<text>'
>
for all non-ASCII literals.
>
</F>
>

Hi,

The problem is that:

# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')

returns 1 then 2

and string.maketran s(str1, str2) requires that len(str1) == len(str2)

hg

[hg]

Re: utf - string translation

hg wrote:

Fredrik Lundh wrote:

>hg wrote:
>>

>>We noticed that len('à') != len('a')

>sounds odd.
>>

>>>>len('à') == len('a')

>True
>>
>are you perhaps using an UTF-8 editor?
>>
>to keep your sanity, no matter what editor you're using, I recommend
>adding a coding directive to the source file, and using *only* Unicode
>string literals for non-ASCII text.
>>
>or in other words, put this at the top of your file (where "utf-8" is
>whatever your editor/system is using):
>>
> # -*- coding: utf-8 -*-
>>
>and use
>>
> u'<text>'
>>
>for all non-ASCII literals.
>>
></F>
>>

>
Hi,
>
The problem is that:
>
# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')
>
returns 1 then 2
>
and string.maketran s(str1, str2) requires that len(str1) == len(str2)
>
hg
>
>
>
>
>

PS: I'm running this under Idle

[Duncan Booth]

Re: utf - string translation

hg <hg@nospam.comw rote:

>or in other words, put this at the top of your file (where "utf-8" is
>whatever your editor/system is using):
>>
> # -*- coding: utf-8 -*-
>>
>and use
>>
> u'<text>'
>>
>for all non-ASCII literals.
>>
></F>
>>

>
Hi,
>
The problem is that:
>
# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')
>
returns 1 then 2

And if you do what was suggested and write:

# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')

then you get:

1
1

[hg]

Re: utf - string translation

Duncan Booth wrote:

hg <hg@nospam.comw rote:
>

>>or in other words, put this at the top of your file (where "utf-8" is
>>whatever your editor/system is using):
>>>
>> # -*- coding: utf-8 -*-
>>>
>>and use
>>>
>> u'<text>'
>>>
>>for all non-ASCII literals.
>>>
>></F>
>>>

>Hi,
>>
>The problem is that:
>>
># -*- coding: utf-8 -*-
>import string
>print len('a')
>print len('à')
>>
>returns 1 then 2

>
And if you do what was suggested and write:
>
# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')
>
then you get:
>
1
1

OK,

How would you handle the string.maketran s then ?

hg

[Fredrik Lundh]

Re: utf - string translation

hg wrote:

How would you handle the string.maketran s then ?

maketrans works on bytes, not characters. what makes you think that you
can use maketrans if you haven't gotten the slightest idea what's in the
string?

if you want to get rid of accents in a Unicode string, you can do the
approaches described here

http://www.peterbe.com/plog/unicode-to-ascii

or here

http://effbot.org/zone/unicode-convert.htm

which both works on any Unicode string.

</F>

[hg]

Re: utf - string translation

Fredrik Lundh wrote:

hg wrote:
>

>How would you handle the string.maketran s then ?

>
maketrans works on bytes, not characters. what makes you think that you
can use maketrans if you haven't gotten the slightest idea what's in the
string?
>
if you want to get rid of accents in a Unicode string, you can do the
approaches described here
>

http://www.peterbe.com/plog/unicode-to-ascii

>
or here
>

http://effbot.org/zone/unicode-convert.htm

>
which both works on any Unicode string.
>
</F>
>

Thanks

[John Machin]

Re: utf - string translation

hg wrote:

Duncan Booth wrote:

hg <hg@nospam.comw rote:

>or in other words, put this at the top of your file (where "utf-8" is
>whatever your editor/system is using):
>>
> # -*- coding: utf-8 -*-
>>
>and use
>>
> u'<text>'
>>
>for all non-ASCII literals.
>>
></F>
>>
Hi,
>
The problem is that:
>
# -*- coding: utf-8 -*-
import string
print len('a')
print len('à')
>
returns 1 then 2

And if you do what was suggested and write:

# -*- coding: utf-8 -*-
import string
print len(u'a')
print len(u'à')

then you get:

1
1

Some general comments:

1. There has been at least one thread on the subject of ripping accents
off Latin1 characters in the last 3 or 4 months. Try Google.

2. About your earlier problem, when len(thing1) != len(thing2):
In that and similar situations, it can be *very* useful to use this
technique:
print repr(thing1), type(thing1)
print repr(thing2), type(thing2)
Go back now and try it out!

OK,
>
How would you handle the string.maketran s then ?
>

I suggest that you first read the documentation on the str and unicode
"translate" methods.
You can obtain this quickly at the interactive prompt by doing
help(''.transla te)
and
help(u''.transl ate)
respectively.

Next steps:

Is your *real* data (not the examples you were hard-coding earlier)
encoded (latin1, utf8) in str objects or is it in unicode objects?
After reading previous posts my head is spinning & I'm not going to
guess; you determine it yourself.

[pseudocode -- blend of Pythonic & Knuthian styles]
if latin1: (A) you can use string.maketran s and str.translate
immediately.

elif unicode: (B) either (1) encode to latin1; goto (A) or (2) use
unicode.transla te with do-it-yourself mapping

elif utf8: decode to unicode; goto (B)

else: ???

HTH,
John

[Dan]

Re: utf - string translation

Thank you for your answers.

In fact, I'm getting start with Python.

I was looking for transform a text through elementary cryptographic
processes (Vigenère).
The initial text is in a file, and my system is under UTF-8 by default
(Ubuntu)

[John Machin]

Re: utf - string translation

Dan wrote:

Thank you for your answers.
>
In fact, I'm getting start with Python.

That was a good decision. Welcome!

>
I was looking for transform a text through elementary cryptographic
processes (Vigenère).

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The initial text is in a file, and my system is under UTF-8 by default
(Ubuntu)

Your system being "under UTF-8" does give you some clue, I suppose. Do
find the time to locate some data with accents and do print(repr(data ))
as I suggested, to *verify* what you've got.

Don't guess. Different underlying representations can look the same
when rendered on your screen. Don't rely on what sysadmins tell you.
Peculiar things can happen, e.g.

me: How is your data encoded?
them: XYZese [a language]
me: I'll try again; Are you using encoding A or encoding B?
them: We've heard A mentioned; what's an encoding anyway?
[snip long explanation plus investigation of what locales [plural] had
been used when configuring their workstations and servers]
them: OK, so there's more than one way of representing XYZese on a
computer. That might explain why the government regulatory authority
for our industry is very sad [to put it mildly] about not being able to
read our monthly filings!!!

Cheers,
John

[David H Wild]

Re: utf - string translation

In article <1164232741.215 312.322990@k70g 2000cwa.googleg roups.com>,
John Machin <sjmachin@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

The difference there, though, is a punctuation character, not an accent.

--
David Wild using RISC OS on broadband

[John Machin]

Re: utf - string translation


David H Wild wrote:

In article <1164232741.215 312.322990@k70g 2000cwa.googleg roups.com>,
John Machin <sjmachin@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

>
The difference there, though, is a punctuation character, not an accent.
>

I did say "difference s in punctuation or accents". Yes, the only
example I could recall OTTOMH was a difference in punctuation --
according to legend, a fly-spot IIRC :-)

[Klaas]

Re: utf - string translation

David H Wild wrote:

In article <1164232741.215 312.322990@k70g 2000cwa.googleg roups.com>,
John Machin <sjmachin@lexic on.netwrote:

So why do you want to strip off accents? The history of communication
has several examples of significant difference in meaning caused by
minute differences in punctuation or accents including one of which you
may have heard: a will that could be read (in part) as either "a chacun
d'eux million francs" or "a chacun deux million francs" with the
remainder to a 3rd party.

>
The difference there, though, is a punctuation character, not an accent.

It's not too hard to imagine an accentual difference, eg:

Le soldat protège avec le fusil --the soldier protects with the gun
Le soldat protégé avec le fusil --the soldier who is protected by
the gun (perhaps a cannon)

Contrived example, I realize, but there are scads of such instances.
(Caveat: my french is also very rusty).

-Mike

[Fredrik Lundh]

Re: utf - string translation

Klaas wrote:

It's not too hard to imagine an accentual difference, eg:

especially in languages where certain combinations really are distinct
letters, not just letters with accents or silly marks.

I have a Swedish children's book somewhere, in which some characters are
harassed by a big ugly monster who carries a sign around his neck that
says "Monster".

the protagonist ends up adding two dots to that sign, turning it into
"Mönster" (meaning "model", in the "model citizen" sense), and all ends
well.

just imagine that story in reverse.

</F>