Replacing words from strings except 'and' / 'or' / 'and not'

**Diez B. Roggisch** · Jul 18 '05, 06:04 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

import sets
KEYWORDS = sets.Set(['and', 'or', 'not'])

query = "test and testing and not perl or testit or example"

def decorate(w):
if w in KEYWORDS:
return w
return "*%s*" % w

query = " ".join([decorate(w.stri p()) for w in query.split()])

--
Regards,

Diez B. Roggisch

**Thomas Guettler** · Jul 18 '05, 06:04 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Am Thu, 25 Nov 2004 15:43:53 +0100 schrieb Nico Grubert:
[color=blue]
> Hi there,
>
> Background of this question is:
> I want to convert all words <word> except 'and' / 'or' / 'and not' from
> a string into '*<word>*'.[/color]

You can give re.sub() a function

import re
ignore=["and", "not", "or"]
test="test and testing and not perl or testit or example"
def repl(match):
word=match.grou p(1)
if word in ignore:
return word
else:
return "*%s*" % word
print re.sub(r'(\w+)' , repl, test)

Result: *test* and *testing* and not *perl* or *testit* or *example*

HTH,
Thomas

**Jean Brouwers** · Jul 18 '05, 06:05 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Just a comment. The w.strip() call in the last line is superfluous in
this particular case. The items in the list resulting from the
query.split() call will be stripped already. Example,
[color=blue][color=green][color=darkred]
>>> "a b c".split()[/color][/color][/color]
['a', 'b', 'c']

/Jean Bouwers

In article <co4s42$mkb$04$ 1@news.t-online.com>, Diez B. Roggisch
<deetsNOSPAM@we b.de> wrote:
[color=blue]
> import sets
> KEYWORDS = sets.Set(['and', 'or', 'not'])
>
> query = "test and testing and not perl or testit or example"
>
> def decorate(w):
> if w in KEYWORDS:
> return w
> return "*%s*" % w
>
> query = " ".join([decorate(w.stri p()) for w in query.split()])[/color]

**Mitja** · Jul 18 '05, 06:05 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

On Thu, 25 Nov 2004 15:43:53 +0100, Nico Grubert <nicogrubert@ar cor.de>
wrote:
[color=blue]
> Example:
> I have the following string: "test and testing and not perl or testit or
> example"
>
> I want to convert this string to:
> '*test*' and '*testing*' and not '*perl*' or '*testit*' or '*example*'[/color]

A compact, though not too readable a solution:

foo="test and testing and not perl or testit or example"

' '.join([
("'*"+w+"*'" ,w)[w in ('and','or')]
for w in foo.split()
]).replace("and '*not*'","and not")

--
Mitja

**Peter Maas** · Jul 18 '05, 06:05 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Diez B. Roggisch schrieb:[color=blue]
> import sets
> KEYWORDS = sets.Set(['and', 'or', 'not'])
>
> query = "test and testing and not perl or testit or example"
>
> def decorate(w):
> if w in KEYWORDS:
> return w
> return "*%s*" % w
>
> query = " ".join([decorate(w.stri p()) for w in query.split()])[/color]

Is there a reason to use sets here? I think lists will do as well.

--
-------------------------------------------------------------------
Peter Maas, M+R Infosysteme, D-52070 Aachen, Tel +49-241-93878-0
E-mail 'cGV0ZXIubWFhc0 BtcGx1c3IuZGU=\ n'.decode('base 64')
-------------------------------------------------------------------

**Peter Otten** · Jul 18 '05, 06:05 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Peter Maas wrote:
[color=blue]
> Diez B. Roggisch schrieb:[color=green]
>> import sets
>> KEYWORDS = sets.Set(['and', 'or', 'not'])
>>
>> query = "test and testing and not perl or testit or example"
>>
>> def decorate(w):
>> if w in KEYWORDS:
>> return w
>> return "*%s*" % w
>>
>> query = " ".join([decorate(w.stri p()) for w in query.split()])[/color]
>
> Is there a reason to use sets here? I think lists will do as well.[/color]

Sets represent the concept better, and large lists will significantly slow
down the code (linear vs constant time). Unfortunately, as 2.3's Set is
implemented in Python, you'll have to wait for the 2.4 set builtin to see
the effect for small lists/sets. In the meantime, from a performance point
of view, a dictionary fares best:

$cat contains.py
from sets import Set

# we need more items than in KEYWORDS above for Set
# to even meet the performance of list :-(
alist = dir([])
aset = Set(alist)
adict = dict.fromkeys(a list)

$timeit.py -s"from contains import alist, aset, adict" "'not' in alist"
100000 loops, best of 3: 2.21 usec per loop
$timeit.py -s"from contains import alist, aset, adict" "'not' in aset"
100000 loops, best of 3: 2.2 usec per loop
$timeit.py -s"from contains import alist, aset, adict" "'not' in adict"
1000000 loops, best of 3: 0.337 usec per loop

Peter

**Peter Hansen** · Jul 18 '05, 06:07 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Peter Maas wrote:[color=blue]
> Diez B. Roggisch schrieb:[color=green]
>> import sets
>> KEYWORDS = sets.Set(['and', 'or', 'not'])
>>...
>> def decorate(w):
>> if w in KEYWORDS:
>> return w
>> return "*%s*" % w
>>[/color]
> Is there a reason to use sets here? I think lists will do as well.[/color]

Sets are implemented using dictionaries, so the "if w in KEYWORDS"
part would be O(1) instead of O(n) as with lists...

(I.e. searching a list is a brute-force operation, whereas
sets are not.)

-Peter

**Skip Montanaro** · Jul 18 '05, 06:07 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

[color=blue][color=green][color=darkred]
>> > Is there a reason to use sets here? I think lists will do as well.[/color]
>>
>> Sets are implemented using dictionaries, so the "if w in KEYWORDS"
>> part would be O(1) instead of O(n) as with lists...
>>
>> (I.e. searching a list is a brute-force operation, whereas
>> sets are not.)[/color][/color]

Jp> And yet... using sets here is slower in every possible case:
...
Jp> This is a pretty clear example of premature optimization.

I think the set concept is correct. The keywords of interest are best
thought of as an unordered collection. Lists imply some ordering (or at
least that potential). Premature optimization would have been realizing
that scanning a short list of strings was faster than testing for set
membership and choosing to use lists instead of sets.

Skip

**John Machin** · Jul 18 '05, 06:08 PM

Re: Replacing words from strings except 'and' / 'or' / 'and not'

Skip Montanaro <skip@pobox.com > wrote in message news:<mailman.6 853.1101656845. 5135.python-list@python.org >...[color=blue][color=green][color=darkred]
> >> > Is there a reason to use sets here? I think lists will do as well.
> >>
> >> Sets are implemented using dictionaries, so the "if w in KEYWORDS"
> >> part would be O(1) instead of O(n) as with lists...
> >>
> >> (I.e. searching a list is a brute-force operation, whereas
> >> sets are not.)[/color][/color]
>
> Jp> And yet... using sets here is slower in every possible case:
> ...
> Jp> This is a pretty clear example of premature optimization.
>
> I think the set concept is correct. The keywords of interest are best
> thought of as an unordered collection. Lists imply some ordering (or at
> least that potential). Premature optimization would have been realizing
> that scanning a short list of strings was faster than testing for set
> membership and choosing to use lists instead of sets.
>
> Skip[/color]

Jp scores extra points for pre-maturity by not trying out version 2.4,
by not reading the bit about sets now being built-in, based on dicts,
dicts being one of the timbot's optimise-the-snot-out-of targets ...
herewith some results from a box with a 1.4Ghz Athlon chip running
Windows 2000:

C:\junk>\python 24\python \python24\lib\t imeit.py -s "from sets import
Set; x = Set(['and', 'or', 'not'])" "None in x"
1000000 loops, best of 3: 1.81 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "from sets import
Set; x = Set(['and', 'or', 'not'])" "None in x"
1000000 loops, best of 3: 1.77 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = set(['and',
'or', 'not'])" "None in x"
1000000 loops, best of 3: 0.29 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = set(['and',
'or', 'not'])" "None in x"
1000000 loops, best of 3: 0.289 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = ['and',
'or', 'not']" "None in x"
1000000 loops, best of 3: 0.804 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = ['and',
'or', 'not']" "None in x"
1000000 loops, best of 3: 0.81 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "from sets import
Set; x = Set(['and', 'or', 'not'])" "'and' in x"
1000000 loops, best of 3: 1.69 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = set(['and',
'or', 'not'])" "'and' in x"
1000000 loops, best of 3: 0.243 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = set(['and',
'or', 'not'])" "'and' in x"
1000000 loops, best of 3: 0.245 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = ['and',
'or', 'not']" "'and' in x"
1000000 loops, best of 3: 0.22 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = ['and',
'or', 'not']" "'and' in x"
1000000 loops, best of 3: 0.22 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = set(['and',
'or', 'not'])" "'not' in x"
1000000 loops, best of 3: 0.257 usec per loop

C:\junk>\python 24\python \python24\lib\t imeit.py -s "x = ['and',
'or', 'not']" "'not' in x"
1000000 loops, best of 3: 0.34 usec per loop

tee hee ...

Replacing words from strings except 'and' / 'or' / 'and not'

Replacing words from strings except 'and' / 'or' / 'and not'

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment