regular expression

**Bengt Richter** · Jul 18 '05, 11:06 PM

Re: regular expression

On Sat, 26 Mar 2005 02:07:15 GMT, aaron <asteele@berkel ey.edu> wrote:
[color=blue]
>dear readers,
>
>given a string, suppose i wanted to do the following:
>- replace all periods with colons, except for periods with a digit to
>the right and left of it.
>
>for example, given:
>'375 mi. south of U.C.B. is 3.4 degrees warmer'
>
>would be changed to:
>"375 mi: south of U:C:B: is 3.4 degrees warmer'
>
>i was thinking that a regular expression might do the trick. here's what
>i tried:
>!----------------------------------------------------------------------!
>Python 2.4.1c1
>[GCC 3.2.3 20030502 (Red Hat Linux 3.2.3-49)] on linux2[color=green][color=darkred]
> >>> import re
> >>> pattern = re.compile(r'(? !\d)[.](?!\d)')
> >>> pattern.sub(':' , '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color][/color]
>'375 mi: south of U:C:B is 3.4 degrees warmer:'
>!----------------------------------------------------------------------!
>
>so this works, but not in the following case:
>!----------------------------------------------------------------------![color=green][color=darkred]
> >>> pattern.sub(':' , '.3')[/color][/color]
>'.3'
>!----------------------------------------------------------------------!
>
>but going the other direction works:
>!----------------------------------------------------------------------![color=green][color=darkred]
> >>> pattern.sub(':' , '3.')[/color][/color]
>'3:'
>!----------------------------------------------------------------------!
>
>any thoughts?
>[/color]
Brute force the exceptional case that happens at the start of the line?
[color=blue][color=green][color=darkred]
>>> import re
>>> pattern = re.compile(r'^[.]|(?!\d)[.](?!\d)')
>>> pattern.sub(':' , '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color][/color][/color]
'375 mi: south of U:C:B is 3.4 degrees warmer:'[color=blue][color=green][color=darkred]
>>> pattern.sub(':' , '.3')[/color][/color][/color]
':3'[color=blue][color=green][color=darkred]
>>> pattern.sub(':' , '3.')[/color][/color][/color]
'3:'

Seems like an asymmetry in re's handling of (?!\d) after the last char vs before first though.

Regards,
Bengt Richter

**Peter Hansen** · Jul 18 '05, 11:06 PM

Re: regular expression

Bengt Richter wrote:[color=blue]
> On Sat, 26 Mar 2005 02:07:15 GMT, aaron <asteele@berkel ey.edu> wrote:[color=green][color=darkred]
>>>>>pattern.su b(':', '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color]
>>'375 mi: south of U:C:B is 3.4 degrees warmer:'
>>
>>so this works, but not in the following case:[color=darkred]
>>>>>pattern.su b(':', '.3')[/color]
>>[/color]
> Brute force the exceptional case that happens at the start of the line?
>[color=green][color=darkred]
> >>> import re
> >>> pattern = re.compile(r'^[.]|(?!\d)[.](?!\d)')
> >>> pattern.sub(':' , '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color][/color]
> '375 mi: south of U:C:B is 3.4 degrees warmer:'[color=green][color=darkred]
> >>> pattern.sub(':' , '.3')[/color][/color]
> ':3'[color=green][color=darkred]
> >>> pattern.sub(':' , '3.')[/color][/color]
> '3:'[/color]

Be careful... the OP has assumed something that isn't true,
and Bengt's fix isn't sufficient:
[color=blue][color=green][color=darkred]
>>> import re
>>> s = 'x.3'
>>> pattern = re.compile(r'^[.]|(?!\d)[.](?!\d)')
>>> pattern.sub(':' , '.3')[/color][/color][/color]
':3'[color=blue][color=green][color=darkred]
>>> pattern.sub(':' , s)[/color][/color][/color]
'x.3'

So the OP's "this works" comment was wrong.

Suggestion: whip up a variety of automated test cases and
make sure you run them all whenever you make changes to
this code...

(No, I don't have a solution to the continuing problem,
other than to wonder whether the input data really requires
all these edge cases to be handled properly.)

-Peter

**Bengt Richter** · Jul 18 '05, 11:06 PM

Re: regular expression

On Fri, 25 Mar 2005 23:54:32 -0500, Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
>Bengt Richter wrote:[color=green]
>> On Sat, 26 Mar 2005 02:07:15 GMT, aaron <asteele@berkel ey.edu> wrote:[color=darkred]
>>>>>>pattern.s ub(':', '375 mi. south of U.C.B is 3.4 degrees warmer.')
>>>'375 mi: south of U:C:B is 3.4 degrees warmer:'
>>>
>>>so this works, but not in the following case:
>>>>>>pattern.s ub(':', '.3')
>>>[/color]
>> Brute force the exceptional case that happens at the start of the line?
>>[color=darkred]
>> >>> import re
>> >>> pattern = re.compile(r'^[.]|(?!\d)[.](?!\d)')
>> >>> pattern.sub(':' , '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color]
>> '375 mi: south of U:C:B is 3.4 degrees warmer:'[color=darkred]
>> >>> pattern.sub(':' , '.3')[/color]
>> ':3'[color=darkred]
>> >>> pattern.sub(':' , '3.')[/color]
>> '3:'[/color]
>
>Be careful... the OP has assumed something that isn't true,
>and Bengt's fix isn't sufficient:
>[color=green][color=darkred]
> >>> import re
> >>> s = 'x.3'
> >>> pattern = re.compile(r'^[.]|(?!\d)[.](?!\d)')
> >>> pattern.sub(':' , '.3')[/color][/color]
>':3'[color=green][color=darkred]
> >>> pattern.sub(':' , s)[/color][/color]
>'x.3'
>
>So the OP's "this works" comment was wrong.
>
>Suggestion: whip up a variety of automated test cases and
>make sure you run them all whenever you make changes to
>this code...
>
>(No, I don't have a solution to the continuing problem,
>other than to wonder whether the input data really requires
>all these edge cases to be handled properly.)
>[/color]
Goes to show you ;-/ Do we need more tests than these?
[color=blue][color=green][color=darkred]
>>> import re
>>> pattern = re.compile(r'[.](?!\d)|(?<!\d)[.]')
>>> print pattern.sub(':' , '375 mi. south of U.C.B is 3.4 degrees warmer.')[/color][/color][/color]
375 mi: south of U:C:B is 3.4 degrees warmer:[color=blue][color=green][color=darkred]
>>> for s,ss in ((s,pattern.sub (':', s)) for s in ('%s%s.%s%s'%(s p1,c1,c2,sp2)[/color][/color][/color]
... for sp1 in ('', ' ')
... for c1 in ('', 'x', '3')
... for c2 in ('', 'x', '3')
... for sp2 in ('', ' '))):
... print '%10r => %r' %(s,ss)
...
'.' => ':'
'. ' => ': '
'.x' => ':x'
'.x ' => ':x '
'.3' => ':3'
'.3 ' => ':3 '
'x.' => 'x:'
'x. ' => 'x: '
'x.x' => 'x:x'
'x.x ' => 'x:x '
'x.3' => 'x:3'
'x.3 ' => 'x:3 '
'3.' => '3:'
'3. ' => '3: '
'3.x' => '3:x'
'3.x ' => '3:x '
'3.3' => '3.3'
'3.3 ' => '3.3 '
' .' => ' :'
' . ' => ' : '
' .x' => ' :x'
' .x ' => ' :x '
' .3' => ' :3'
' .3 ' => ' :3 '
' x.' => ' x:'
' x. ' => ' x: '
' x.x' => ' x:x'
' x.x ' => ' x:x '
' x.3' => ' x:3'
' x.3 ' => ' x:3 '
' 3.' => ' 3:'
' 3. ' => ' 3: '
' 3.x' => ' 3:x'
' 3.x ' => ' 3:x '
' 3.3' => ' 3.3'
' 3.3 ' => ' 3.3 '

Regards,
Bengt Richter

**Peter Hansen** · Jul 18 '05, 11:07 PM

Re: regular expression

Bengt Richter wrote:[color=blue]
> On Fri, 25 Mar 2005 23:54:32 -0500, Peter Hansen <peter@engcorp. com> wrote:[color=green]
>>Suggestion: whip up a variety of automated test cases and
>>make sure you run them all whenever you make changes to
>>this code...[/color]
>
> Goes to show you ;-/ Do we need more tests than these?[/color]
[snip loads of tests]

Hmm... if I were doing this for real, not only would the
tests actually *tell* me when there was a failure, but
I would also throw in a few more cases involving larger
strings that more closely represent the expected real
inputs (i.e. using some numbers like 3.1415 and using
some strings that have the periods as punctuation such
as in the OP's original first "test case"). That way
if, during maintenance, somebody changes the algorithm
significantly, I'll be confident that it still covers
the broader set of cases, as well as the (exhaustive?)
set you've defined, which appear at first glance to
cover all the possible combinations of x and 3 and .
that might happen...

I'd also probably be generating most of the existing
test cases automatically, just to be sure I've got
100% coverage. Are you sure you didn't leave out one?
And what about, say, ".." or ".x3."? :-)

-Peter

**Paul McGuire** · Jul 18 '05, 11:08 PM

Re: regular expression

Aaron -

Here's a pyparsing approach (requires latest 1.3 pyparsing version).
It may not be as terse or fast as your regexp, but it may be easier to
maintain.

By defining floatNum ahead of DOT in the scanner definition, you
specify the dot-containing expressions that you do *not* want to have
dots converted to colons.

-- Paul

=============== ====
from pyparsing import Word,Literal,re placeWith, Combine, nums

DOT = Literal(".").se tParseAction( replaceWith(":" ) )
floatNum = Combine( Word(nums) + "." + Word(nums) )

scanner = floatNum | DOT

testdata = "'375 mi. south of U.C.B is 3.4 degrees warmer."

print scanner.transfo rmString( testdata )
=============== ====
prints out:
'375 mi: south of U:C:B is 3.4 degrees warmer:

regular expression

regular expression

Comment

Comment

Comment

Comment

Comment