problems with  character

[jdonnell]

I have a mysql database with characters like   » in it. I'm
trying to write a python script to remove these, but I'm having a
really hard time.

These strings are coming out as type 'str' not 'unicode' so I tried to
just

record[4].replace('Â', '')

but this does nothing. However the following code works

#!/usr/bin/python

s = 'aaaaa  aaa'
print type(s)
print s
print s.find('Â')

This returns
<type 'str'>
aaaaa  aaa
6

The other odd thing is that the  character shows up as two spaces if
I print it to the terminal from mysql, but it shows up as  when I
print from the simple script above.
What am I doing wrong?

[deelan]

Re: problems with  character

jdonnell wrote:[color=blue]
> I have a mysql database with characters like   » in it. I'm
> trying to write a python script to remove these, but I'm having a
> really hard time.[/color]

use the "hammer" recipe. i'm using it to create URL-friendly
fragment from latin-1 album titles:

<http://aspn.activestat e.com/ASPN/Cookbook/Python/Recipe/251871>
(check the last comment, "a cleaner solution"
for a better implementation) .

it basically hammers down accented chars like à and Â
to the most near ASCII representation.

since you receive string data as str from mysql
object first convert them as unicode with:

u = unicode('Â', 'latin-1')

then feed u to the hammer function (the fix_unicode at the
end).

HTH,
deelan

--
"Però è bello sapere che, di questi tempi spietati, almeno
un mistero sopravvive: l'età di Afef Jnifen." -- dagospia.com

[Claudio Grondi]

Re: problems with  character

>>s = 'aaaaa  aaa'[color=blue][color=green]
>>What am I doing wrong?[/color][/color]

First get rid of characters not allowed
in Python code.
Replace  with appropriate escape
sequence: /x## where ## is the
hexadecimal code of the ASCII
character.

Claudio

[Claudio Grondi]

Re: problems with  character


"Claudio Grondi" <claudio.grondi @freenet.de> schrieb im Newsbeitrag
news:3abdeiF67q q7uU1@individua l.net...[color=blue][color=green][color=darkred]
> >>s = 'aaaaa  aaa'
> >>What am I doing wrong?[/color][/color]
>
> First get rid of characters not allowed
> in Python code.
> Replace  with appropriate escape
> sequence: /x## where ## is the (should be \x##)
> hexadecimal code of the ASCII
> character.
>
> Claudio[/color]

i.e. probably instead of 'aaaaa  aaa'
'aaaaa \xC2 aaa'
In my ASCII table 'Â' is '\xC2'

Claudio

[Do Re Mi chel La Si Do]

Re: problems with  character

aaaaa  aaa'
0123456


It's OK

[Do Re Mi chel La Si Do]

Re: problems with  character

And this run OK for me :

s = 'aaaaa  aaa'
print s
print s.replace('Â', '')

[Marc 'BlackJack' Rintsch]

Re: problems with  character

In <1111521139.657 563.55410@o13g2 000cwo.googlegr oups.com>, jdonnell wrote:
[color=blue]
> I have a mysql database with characters like   » in it. I'm
> trying to write a python script to remove these, but I'm having a
> really hard time.
>
> [...]
>
> The other odd thing is that the  character shows up as two spaces if
> I print it to the terminal from mysql, but it shows up as  when I
> print from the simple script above.
> What am I doing wrong?[/color]

Is it possible that your DB stores strings UTF-8 encoded? The
byte sequence '\xc2\xa0' which displays as 'Â ' in latin-1 encoding is a
non breakable space character.

Ciao,
Marc 'BlackJack' Rintsch

[John Roth]

Re: problems with  character

I had this problem recently. It turned out that something
had encoded a unicode string into utf-8. When I found
the culprit and fixed the underlying design issue, it went away.

John Roth



"jdonnell" <jaydonnell@gma il.com> wrote in message
news:1111521139 .657563.55410@o 13g2000cwo.goog legroups.com...
I have a mysql database with characters like   » in it. I'm
trying to write a python script to remove these, but I'm having a
really hard time.

These strings are coming out as type 'str' not 'unicode' so I tried to
just

record[4].replace('Â', '')

but this does nothing. However the following code works

#!/usr/bin/python

s = 'aaaaa  aaa'
print type(s)
print s
print s.find('Â')

This returns
<type 'str'>
aaaaa  aaa
6

The other odd thing is that the  character shows up as two spaces if
I print it to the terminal from mysql, but it shows up as  when I
print from the simple script above.
What am I doing wrong?

[Bengt Richter]

Re: Re: problems with  character

On Tue, 22 Mar 2005 20:09:55 -0600, "John Roth" <newsgroups@jhr othjr.com> wrote:
[color=blue]
>I had this problem recently. It turned out that something
>had encoded a unicode string into utf-8. When I found
>the culprit and fixed the underlying design issue, it went away.
>
>John Roth
>
>
>
>"jdonnell" <jaydonnell@gma il.com> wrote in message
>news:111152113 9.657563.55410@ o13g2000cwo.goo glegroups.com.. .
>I have a mysql database with characters like   » in it. I'm
>trying to write a python script to remove these, but I'm having a
>really hard time.
>
>These strings are coming out as type 'str' not 'unicode' so I tried to
>just
>
>record[4].replace('Â', '')
>
>but this does nothing. However the following code works
>
>#!/usr/bin/python
>
>s = 'aaaaa  aaa'
>print type(s)
>print s
>print s.find('Â')
>
>This returns
><type 'str'>
>aaaaa  aaa
>6
>
>The other odd thing is that the  character shows up as two spaces if
>I print it to the terminal from mysql, but it shows up as  when I
>print from the simple script above.
>What am I doing wrong?
>[/color]
What encodings are involved?

This is from idle on windows, which seems to display latin-1 source ok:
----[color=blue][color=green][color=darkred]
>>> "Latin-1:»\n".decode( 'latin-1')[/color][/color][/color]
u'Latin-1:\xc2\xbb\n'[color=blue][color=green][color=darkred]
>>> "Latin-1:»\n".decode( 'latin-1').encode('cp4 37', 'replace')[/color][/color][/color]
'Latin-1:?\xaf\n'[color=blue][color=green][color=darkred]
>>> "Latin-1:»\n".decode( 'latin-1').encode('cp4 37', 'ignore')[/color][/color][/color]
'Latin-1:\xaf\n'[color=blue][color=green][color=darkred]
>>> u'Latin-1:\xc2\xbb\n'.e ncode('cp437',' replace')[/color][/color][/color]
'Latin-1:?\xaf\n'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
----
Now this is in an NT4 console windows with code page 437:

----[color=blue][color=green][color=darkred]
>>> u'Latin-1:\xc2\xbb\n'.e ncode('cp437',' replace')[/color][/color][/color]
'Latin-1:?\xaf\n'[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.stdout.writ e(u'Latin-1:\xc2\xbb\n'.e ncode('cp437',' replace'))[/color][/color][/color]
Latin-1:?»
----

Notice that the interactive output does a repr that creates the \xaf, but
the character is available and can be written non-repr'd via sys.stdout.writ e.

For the heck of it:
[color=blue][color=green][color=darkred]
>>> sys.stdout.writ e(u'Latin-1:\xc2\xbb\n'.e ncode('cp437',' xmlcharrefrepla ce'))[/color][/color][/color]
Latin-1:&#194;»

I don't know if this is going to get through to your screen ;-)

Regards,
Bengt Richter

[jdonnell]

Re: problems with  character

Thanks for all the replies. I just got in to work so I haven't tried
any of them yet. I see that I wasn't as clear as I should have been so
I'll clarify a little. I'm grabbing some data from msn's rss feed.
Here's an example.

http://search.msn.com/results.aspx?q=domain+name&format=rss&FORM=ZZRE

The string ' all domain name extensions » Good' is where I have a
problem. The
' »' shows up as '  »' when I write it to a file or stick
it in mysql. I did a hex dump and this is what I see.

jay@localhost:~/scripts> cat test.txt
extensions » Good
jay@localhost:~/scripts> xxd test.txt
0000000: 6578 7465 6e73 696f 6e73 20c2 a020 c2a0 extensions .. ..
0000010: 20c2 bb20 476f 6f64 0a .. Good

One thing that jumps out is that two of the Â's are c2a0, but one of
them is c2bb. Well, those are the details since I wasn't clear before.

[Marc 'BlackJack' Rintsch]

Re: Re: problems with  character

In <1111599099.854 099.60300@g14g2 000cwa.googlegr oups.com>, jdonnell wrote:
[color=blue]
> Thanks for all the replies. I just got in to work so I haven't tried
> any of them yet. I see that I wasn't as clear as I should have been so
> I'll clarify a little. I'm grabbing some data from msn's rss feed.
> Here's an example.
> http://search.msn.com/results.aspx?q...=rss&FORM=ZZRE[/color]

Then you are getting UTF-8 encoded strings.
[color=blue]
> The string ' all domain name extensions » Good' is where I have a
> problem. The
> ' »' shows up as '  »' when I write it to a file or stick
> it in mysql. I did a hex dump and this is what I see.
>
> jay@localhost:~/scripts> cat test.txt
> extensions » Good
> jay@localhost:~/scripts> xxd test.txt
> 0000000: 6578 7465 6e73 696f 6e73 20c2 a020 c2a0 extensions .. ..
> 0000010: 20c2 bb20 476f 6f64 0a .. Good
>
> One thing that jumps out is that two of the Â's are c2a0, but one of
> them is c2bb. Well, those are the details since I wasn't clear before.[/color]

That are two no-break spaces and a '»' character::

In [42]: import unicodedata

In [43]: unicodedata.nam e('\xc2\xa0'.de code('utf-8'))
Out[43]: 'NO-BREAK SPACE'

In [44]: unicodedata.nam e('\xc2\xbb'.de code('utf-8'))
Out[44]: 'RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK'

Ciao,
Marc 'BlackJack' Rintsch

[jdonnell]

Re: problems with  character

Thanks everyone, I got it working earlier this morning using deelan's
suggestion. I modified the code in his link so that it removes rather
than replaces the characters.

Also, this was my first experience with unicode and what confused me is
that I was thinking of a unicode object as an encoding, but it's not.
It's just a series of bytes and you later tell it to use a specific
encoding like utf-8 or latin-1. Thanks again for all the help.

[John Machin]

Re: problems with  character

On Tue, 22 Mar 2005 21:39:30 -0000, "Claudio Grondi"
<claudio.grondi @freenet.de> wrote:

[color=blue]
>In my ASCII table 'Â' is '\xC2'[/color]

You've got an *ASCII* table that includes that??

I hope you paid for it in Confederate dollars or czarist roubles --
that's about what such a table would be worth.