Clean "Durty" strings

[Ulysse]

Hello,

I need to clean the string like this :

string =
"""
bonne mentalit&eacute ; mec!:) \n <br>bon pour
info moi je suis un serial posteur arceleur dictateur ^^*
\n <br>mais pour avoir des resultats probant il
faut pas faire les mariolles, comme le &quot;fondateur &quot; de bvs
krew \n
mais pour avoir des resultats probant il faut pas faire les mariolles,
comme le &quot;fondateur &quot; de bvs krew \n
"""

into :
bonne mentalité mec!:) bon pour info moi je suis un serial posteur
arceleur dictateur ^^* mais pour avoir des resultats probant il faut
pas faire les mariolles, comme le "fondateur" de bvs krew
mais pour avoir des resultats probant il faut pas faire les mariolles,
comme le "fondateur" de bvs krew

To do this I wold like to use only strandard librairies.

Thanks

[Diez B. Roggisch]

Re: Clean &quot;Durty&quo t; strings

Ulysse wrote:

Hello,
>
I need to clean the string like this :
>
string =
"""
bonne mentalit&eacute ; mec!:) \n <br>bon pour
info moi je suis un serial posteur arceleur dictateur ^^*
\n <br>mais pour avoir des resultats probant il
faut pas faire les mariolles, comme le &quot;fondateur &quot; de bvs
krew \n
mais pour avoir des resultats probant il faut pas faire les mariolles,
comme le &quot;fondateur &quot; de bvs krew \n
"""
>
into :
bonne mentalité mec!:) bon pour info moi je suis un serial posteur
arceleur dictateur ^^* mais pour avoir des resultats probant il faut
pas faire les mariolles, comme le "fondateur" de bvs krew
mais pour avoir des resultats probant il faut pas faire les mariolles,
comme le "fondateur" de bvs krew

The obvious way that has been suggested to you at other places is to use
BeautifulSoup.

To do this I wold like to use only strandard librairies.

Then you need to reprogram what BeautifulSoup does. Happy hacking!

Diez

[rzed]

Re: Clean &quot;Durty&quo t; strings

"Diez B. Roggisch" <deets@nospam.w eb.dewrote in
news:57bt02F2c9 fpuU2@mid.uni-berlin.de:

Ulysse wrote:
>

>Hello,
>>
>I need to clean the string like this :
>>
>string =
>"""
>bonne mentalit&eacute ; mec!:) \n <br>bon
>pour info moi je suis un serial posteur arceleur dictateur ^^*
>\n <br>mais pour avoir des resultats
>probant il faut pas faire les mariolles, comme le
>&quot;fondateu r&quot; de bvs krew \n
>mais pour avoir des resultats probant il faut pas faire les
>mariolles, comme le &quot;fondateur &quot; de bvs krew \n
>"""
>>
>into :
>bonne mentalité mec!:) bon pour info moi je suis un serial
>posteur arceleur dictateur ^^* mais pour avoir des resultats
>probant il faut pas faire les mariolles, comme le "fondateur"
>de bvs krew mais pour avoir des resultats probant il faut pas
>faire les mariolles, comme le "fondateur" de bvs krew

>
The obvious way that has been suggested to you at other places
is to use BeautifulSoup.
>

>To do this I wold like to use only strandard librairies.

>
Then you need to reprogram what BeautifulSoup does. Happy
hacking!
>

If the OP is constrained to standard libraries, then it may be a
question of defining what should be done more clearly. The extraneous
spaces can be removed by tokenizing the string and rejoining the
tokens. Replacing portions of a string with equivalents is standard
stuff. It might be preferable to create a function that will accept
lists of from and to strings and translate the entire string by
successively applying the replacements. From what I've seen so far,
that would be all the OP needs for this task. It might take a half-
dozen lines of code, plus the from/to table definition.

--
rzed

[Diez B. Roggisch]

Re: Clean &quot;Durty&quo t; strings

>

If the OP is constrained to standard libraries, then it may be a
question of defining what should be done more clearly. The extraneous
spaces can be removed by tokenizing the string and rejoining the
tokens. Replacing portions of a string with equivalents is standard
stuff. It might be preferable to create a function that will accept
lists of from and to strings and translate the entire string by
successively applying the replacements. From what I've seen so far,
that would be all the OP needs for this task. It might take a half-
dozen lines of code, plus the from/to table definition.

The OP had <br>-tags in his text. Which is _more_ than a half dozen lines of
code to clean up. Because your simple replacement-approach won't help here:

<br>foo <brbar </br>

Which is perfectly legal HTML, but nasty to parse.

Diez

[irstas@gmail.com]

Re: Clean &quot;Durty&quo t; strings

On Apr 2, 4:05 pm, "Diez B. Roggisch" <d...@nospam.we b.dewrote:

If the OP is constrained to standard libraries, then it may be a
question of defining what should be done more clearly. The extraneous
spaces can be removed by tokenizing the string and rejoining the
tokens. Replacing portions of a string with equivalents is standard
stuff. It might be preferable to create a function that will accept
lists of from and to strings and translate the entire string by
successively applying the replacements. From what I've seen so far,
that would be all the OP needs for this task. It might take a half-
dozen lines of code, plus the from/to table definition.

>
The OP had <br>-tags in his text. Which is _more_ than a half dozen lines of
code to clean up. Because your simple replacement-approach won't help here:
>
<br>foo <brbar </br>
>
Which is perfectly legal HTML, but nasty to parse.
>
Diez

But it could be that he just wants all HTML tags to disappear, like in
his example. A code like this might be sufficient then: re.sub(r'<[^>]
+>', '', s). For whitespace, re.sub(r'\s+', ' ', s). For XML
characters like &eacute;, re.sub(r'&(\w+) ;', lambda mo:
unichr(htmlenti tydefs.name2cod epoint[mo.group(1)]), s) and
re.sub(r'&#(\d+ );', lambda mo: unichr(int(mo.g roup(1))), s). That's it
pretty much.

I'd like to see how this transformation can be done with
BeautifulSoup. Well, the last two regexps can be replaced with this:

unicode(Beautif ulStoneSoup(s,c onvertEntities= BeautifulStoneS oup.HTML_ENTITI ES).contents[0])

[Marc 'BlackJack' Rintsch]

Re: Clean &quot;Durty&quo t; strings

In <1175530649.060 784.147900@d57g 2000hsg.googleg roups.com>, irstas wrote:

I'd like to see how this transformation can be done with
BeautifulSoup. Well, the last two regexps can be replaced with this:
>
unicode(Beautif ulStoneSoup(s,c onvertEntities= BeautifulStoneS oup.HTML_ENTITI ES).contents[0])

Completely without regular expressions:

def main():
soup = BeautifulSoup(s ource, convertEntities =BeautifulSoup. HTML_ENTITIES)
print ' '.join(''.join( soup(text=True) ).split())

Ciao,
Marc 'BlackJack' Rintsch

[Michael Hoffman]

Re: Clean &quot;Durty&quo t; strings

irstas@gmail.co m wrote:

But it could be that he just wants all HTML tags to disappear, like in
his example. A code like this might be sufficient then: re.sub(r'<[^>]
+>', '', s).

Won't work for, say, this:

<img src="src" alt="<text>">
--
Michael Hoffman

[irstas@gmail.com]

Re: Clean &quot;Durty&quo t; strings

On Apr 2, 10:08 pm, Michael Hoffman <cam.ac...@mh39 1.invalidwrote:

irs...@gmail.co m wrote:

But it could be that he just wants all HTML tags to disappear, like in
his example. A code like this might be sufficient then: re.sub(r'<[^>]
+>', '', s).

>
Won't work for, say, this:
>
<img src="src" alt="<text>">
--
Michael Hoffman

True, but is that legal? I think the alt attribute needs to use &lt;
and &gt;. Although I know what you're going to reply. That
BeautifulSoup probably parses it even if it's invalid HTML. And I'd
say that I agree, using BeautifulSoup is a better solution than custom
regexps.

[rzed]

Re: Clean &quot;Durty&quo t; strings

"Diez B. Roggisch" <deets@nospam.w eb.dewrote in
news:57cddlF2bs qdbU2@mid.uni-berlin.de:

>>
>If the OP is constrained to standard libraries, then it may be
>a question of defining what should be done more clearly. The
>extraneous spaces can be removed by tokenizing the string and
>rejoining the tokens. Replacing portions of a string with
>equivalents is standard stuff. It might be preferable to create
>a function that will accept lists of from and to strings and
>translate the entire string by successively applying the
>replacements . From what I've seen so far, that would be all the
>OP needs for this task. It might take a half- dozen lines of
>code, plus the from/to table definition.

>
The OP had <br>-tags in his text. Which is _more_ than a half
dozen lines of code to clean up. Because your simple
replacement-approach won't help here:
>
<br>foo <brbar </br>
>
Which is perfectly legal HTML, but nasty to parse.

Well, as I said, given the input the OP supplied, it's not even
necessary to parse it. It isn't clear what the true desired
operation is, but this seems to meet the criteria given:

<code -- the string 's' is wrapped nastily, but ...>
s ="""\
bonne mentalit&eacute ; mec!:) \n <br>bon
pour
info moi je suis un serial posteur arceleur dictateur ^^*
\n <br>mais pour avoir des resultats
probant il
faut pas faire les mariolles, comme le &quot;fondateur &quot; de
bvs
krew \n
mais pour avoir des resultats probant il faut pas faire les
mariolles,
comme le &quot;fondateur &quot; de bvs krew \n"""

fromlist = ['<br>', '&eacute;', '&quot;']
tolist = ['', 'é', '"' ]


def withReplacement s( s, flist,tlist ):
for ix, f in enumerate(flist ):
t = tlist[ix]
s = s.replace( f,t )
return s

print withReplacement s(' '.join(s.split( )),fromlist,tol ist)

</code>

If the question is about efficiency or robustness or generality,
then that's another set of issues, but that's for the 1.1 version
to handle.

--
rzed