Flexable Collating (feedback please)

**Ron Adam** · Oct 18 '06, 06:55 AM

Re: Flexable Collating (feedback please)

Fixed...

Changed the collate() function to return None the same as sort() since it is an
in place collate.

A comment in _test() doctests was reversed. CAPS_FIRST option puts words
beginning with capitals before, not after, words beginning with lower case of
the same letter.

It seems I always find a few obvious glitches right after I post something. ;-)

Cheers,
Ron

**georgeryoung@gmail.com** · Oct 18 '06, 11:35 AM

Re: Flexable Collating (feedback please)

On Oct 18, 2:42 am, Ron Adam <r...@ronadam.c omwrote:

I put together the following module today and would like some feedback on any
obvious problems. Or even opinions of weather or not it is a good approach.

,,,
def __call__(self, a, b):
""" This allows the Collate class work as a sort key.

USE: list.sort(key=C ollate(flags))
"""
return cmp(self.transf orm(a), self.transform( b))

You document _call__ as useful for the "key" keyword to sort, but you
implement it for the "cmp" keyword. The "key" allows much better
performance, since it's called only once per value. Maybe just :
return self.transform( a)

-- George

**Ron Adam** · Oct 18 '06, 02:25 PM

Re: Flexable Collating (feedback please)

georgeryoung@gm ail.com wrote:

>
On Oct 18, 2:42 am, Ron Adam <r...@ronadam.c omwrote:

>I put together the following module today and would like some feedback on any
>obvious problems. Or even opinions of weather or not it is a good approach.

,,,
def __call__(self, a, b):
""" This allows the Collate class work as a sort key.
>
USE: list.sort(key=C ollate(flags))
"""
return cmp(self.transf orm(a), self.transform( b))
>
You document _call__ as useful for the "key" keyword to sort, but you
implement it for the "cmp" keyword. The "key" allows much better
performance, since it's called only once per value. Maybe just :
return self.transform( a)
>
-- George
>

Thanks, I changed it to the following...

def __call__(self, a):
""" This allows the Collate class work as a sort key.

USE: list.sort(key=C ollate(flags))
"""
return self.transform( a)

And also changed the sort call here ...

def collate(slist, flags=0):
""" Collate list of strings in place.
"""
slist.sort(key= Collate(flags)) <<<

Today I'll do some performance tests to see how much faster it is for moderate
sized lists.

Cheers,
Ron

**bearophileHUGS@lycos.com** · Oct 18 '06, 08:35 PM

Re: Flexable Collating (feedback please)

This part of code uses integer "constants" to be or-ed (or added):

CAPS_FIRST = 1
NUMERICAL = 2
HYPHEN_AS_SPACE = 4
UNDERSCORE_AS_S PACE = 8
IGNORE_LEADING_ WS = 16
COMMA_IN_NUMERA LS = 32

....

def __init__(self, flag):
self.flag = flag
def transform(self, s):
""" Transform a string for collating.
"""
if self.flag & CAPS_FIRST:
s = s.swapcase()
if self.flag & HYPHEN_AS_SPACE :
s = s.replace('-', ' ')
if self.flag & UNDERSCORE_AS_S PACE:
s = s.replace('_', ' ')
if self.flag & IGNORE_LEADING_ WS:
s = s.strip()
if self.flag & NUMERICAL:
if self.flag & COMMA_IN_NUMERA LS:

This is used in C, but maybe for Python other solutions may be better.
I can see some different (untested) solutions:

1)

def selfassign(self , locals):
# Code from web.py, modified.
for key, value in locals.iteritem s():
if key != 'self':
setattr(self, key, value)

def __init__(self,
caps_first=Fals e,
hyphen_as_space =False,
underscore_as_s pace=False,
ignore_leading_ ws=False,
numerical=False ,
comma_in_numera ls=False):
selfassign(self , locals())

def transform(self, s):
if self.caps_first :
...

Disadvangages: if a flag is added/modified, the code has to be modified
in two places.

2)

def __init__(self, **kwds):
self.lflags = [k for k,v in kwds.items() if v]
def transform(self, s):
if "caps_first " in self.lflags:
...

This class can be created with 1 instead of Trues, to shorten the code.

Disadvantages: the user of this class has to read from the class
doctring or from from the docs the list of possible flags (and such
docs can be out of sync from the code).

3)

Tkinter (Tcl) shows that sometimes strings are better than int
constants (like using "left" instead of tkinter.LEFT, etc), so this is
another possibile solution:

def __init__(self, flags=""):
self.lflags = flags.lower().s plit()
def transform(self, s):
if "caps_first " in self.lflags:
...

An example of calling this class:

.... = Collate("caps_f irst hyphen_as_space numerical")

I like this third (nonstandard) solution enough.

Bye,
bearophile

**Ron Adam** · Oct 18 '06, 08:45 PM

Re: Flexible Collating (feedback please)

I made a number of changes ... (the new version is listed below)

These changes also resulted in improving the speed by about 3 times when all
flags are specified.

Collating now takes about 1/3 (or less) time. Although it is still quite a bit
slower than a bare list.sort(), that is to be expected as collate is locale
aware and does additional transformations on the data which you would need to do
anyways. The tests where done with Unicode strings as well.

Changed the flag types from integer values to a list of named strings. The
reason for this is it makes finding errors easier and you can examine the flags
attribute and get a readable list of flags.

A better regular expression for separating numerals. It now separates numerals
in the middle of the string.

Changed flag COMMA_IN_NUMERA LS to IGNORE_COMMAS, This was how it was implemented.

Added flag PERIOD_AS_COMMA S

This lets you collate decimal separated numbers correctly such as version
numbers and internet address's. It also prevents numerals from being
interpreted as floating point or decimal.

It might make more since to implement it as PERIOD_IS_SEPAR ATOR. Needed?

Other minor changes to doc strings and tests were made.

Any feedback is welcome.

Cheers,
Ron

"""
Collate.py

A general purpose configurable collate module.

Collation can be modified with the following keywords:

CAPS_FIRST -Aaa, aaa, Bbb, bbb
HYPHEN_AS_SPACE -Don't ignore hyphens
UNDERSCORE_AS_S PACE -Underscores as white space
IGNORE_LEADING_ WS -Disregard leading white space
NUMERICAL -Digit sequences as numerals
IGNORE_COMMAS -Allow commas in numerals
PERIOD_AS_COMMA S -Periods can separate numerals.

* See doctests for examples.

Author: Ron Adam, ron@ronadam.com

"""
__version__ = '0.02 (pre-alpha) 10/18/2006'

import re
import locale
import string

locale.setlocal e(locale.LC_ALL , '') # use current locale settings

# The above line may change the string constants from the string
# module. This may have unintended effects if your program
# assumes they are always the ascii defaults.

CAPS_FIRST = 'CAPS_FIRST'
HYPHEN_AS_SPACE = 'HYPHEN_AS_SPAC E'
UNDERSCORE_AS_S PACE = 'UNDERSCORE_AS_ SPACE'
IGNORE_LEADING_ WS = 'IGNORE_LEADING _WS'
NUMERICAL = 'NUMERICAL'
IGNORE_COMMAS = 'IGNORE_COMMAS'
PERIOD_AS_COMMA S = 'PERIOD_AS_COMM AS'

class Collate(object) :
""" A general purpose and configurable collator class.
"""
def __init__(self, flags=[]):
self.flags = flags
self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)
self.txtable = []
if HYPHEN_AS_SPACE in flags:
self.txtable.ap pend(('-', ' '))
if UNDERSCORE_AS_S PACE in flags:
self.txtable.ap pend(('_', ' '))
if PERIOD_AS_COMMA S in flags:
self.txtable.ap pend(('.', ','))
if IGNORE_COMMAS in flags:
self.txtable.ap pend((',', ''))
self.flags = flags

def transform(self, s):
""" Transform a string for collating.
"""
if not self.flags:
return locale.strxfrm( s)
for a, b in self.txtable:
s = s.replace(a, b)
if IGNORE_LEADING_ WS in self.flags:
s = s.strip()
if CAPS_FIRST in self.flags:
s = s.swapcase()
if NUMERICAL in self.flags:
slist = self.numrex.spl it(s)
for i, x in enumerate(slist ):
try:
slist[i] = float(x)
except:
slist[i] = locale.strxfrm( x)
return slist
return locale.strxfrm( s)

def __call__(self, a):
""" This allows the Collate class work as a sort key.

USE: list.sort(key=C ollate(flags))
"""
return self.transform( a)

def collate(slist, flags=[]):
""" Collate list of strings in place.
"""
slist.sort(key= Collate(flags). transform)

def collated(slist, flags=[]):
""" Return a collated list of strings.
"""
return sorted(slist, key=Collate(fla gs).transform)

def _test():
"""
DOC TESTS AND EXAMPLES:

Sort (and sorted) normally order all words beginning with caps
before all words beginning with lower case.

>>t = ['tuesday', 'Tuesday', 'Monday', 'monday']
>>sorted(t) # regular sort

['Monday', 'Tuesday', 'monday', 'tuesday']

Locale collation puts words beginning with caps after words
beginning with lower case of the same letter.

>>collated(t)

['monday', 'Monday', 'tuesday', 'Tuesday']

The CAPS_FIRST option can be used to put all words beginning
with caps before words beginning in lowercase of the same letter.

>>collated(t, [CAPS_FIRST])

['Monday', 'monday', 'Tuesday', 'tuesday']

The HYPHEN_AS_SPACE option causes hyphens to be equal to space.

>>t = ['a-b', 'b-a', 'aa-b', 'bb-a']
>>collated(t)

['aa-b', 'a-b', 'b-a', 'bb-a']

>>collated(t, [HYPHEN_AS_SPACE])

['a-b', 'aa-b', 'b-a', 'bb-a']

The IGNORE_LEADING_ WS and UNDERSCORE_AS_S PACE options can be
used together to improve ordering in some situations.

>>t = ['sum', '__str__', 'about', ' round']
>>collated(t)

[' round', '__str__', 'about', 'sum']

>>collated(t, [IGNORE_LEADING_ WS])

['__str__', 'about', ' round', 'sum']

>>collated(t, [UNDERSCORE_AS_S PACE])

[' round', '__str__', 'about', 'sum']

>>collated(t, [IGNORE_LEADING_ WS, UNDERSCORE_AS_S PACE])

['about', ' round', '__str__', 'sum']

The NUMERICAL option orders sequences of digits as numerals.

>>t = ['a5', 'a40', '4abc', '20abc', 'a10.2', '13.5b', 'b2']
>>collated(t, [NUMERICAL])

['4abc', '13.5b', '20abc', 'a5', 'a10.2', 'a40', 'b2']

The IGNORE_COMMAS option prevents commas from seperating numerals.

>>t = ['a5', 'a4,000', '500b', '100,000b']
>>collated(t, [NUMERICAL, IGNORE_COMMAS])

['500b', '100,000b', 'a5', 'a4,000']

The PERIOD_AS_COMMA S option can be used to sort version numbers
and other decimal seperated numbers correctly.

>>t = ['5.1.1', '5.10.12','5.2. 2', '5.2.19' ]
>>collated(t, [NUMERICAL, PERIOD_AS_COMMA S])

['5.1.1', '5.2.2', '5.2.19', '5.10.12']

Collate also can be done in place by using collate() instead of
collated().

>>t = ['Fred', 'Ron', 'Carol', 'Bob']
>>collate(t)
>>t

['Bob', 'Carol', 'Fred', 'Ron']

"""
import doctest
doctest.testmod ()

if __name__ == '__main__':
_test()

**Ron Adam** · Oct 18 '06, 09:05 PM

Re: Flexable Collating (feedback please)

Thanks, But I fixed it already. (almost) ;-)

I think I will use strings as you suggest, and verify they are valid so a type
don't go though silently.

I ended up using string based option list. I agree a space separated string is
better and easier from a user point of view.

The advantage of the list is it can be iterated without splitting first. But
that's a minor thing. self.options = options.lower() .split(' ') fixes that easily.

Once I'm sure it's not going to get any major changes I'll post this as a
recipe. I think it's almost there.

Cheers and thanks,
Ron

bearophileHUGS@ lycos.com wrote:

This part of code uses integer "constants" to be or-ed (or added):
>
CAPS_FIRST = 1
NUMERICAL = 2
HYPHEN_AS_SPACE = 4
UNDERSCORE_AS_S PACE = 8
IGNORE_LEADING_ WS = 16
COMMA_IN_NUMERA LS = 32
>
...
>
def __init__(self, flag):
self.flag = flag
def transform(self, s):
""" Transform a string for collating.
"""
if self.flag & CAPS_FIRST:
s = s.swapcase()
if self.flag & HYPHEN_AS_SPACE :
s = s.replace('-', ' ')
if self.flag & UNDERSCORE_AS_S PACE:
s = s.replace('_', ' ')
if self.flag & IGNORE_LEADING_ WS:
s = s.strip()
if self.flag & NUMERICAL:
if self.flag & COMMA_IN_NUMERA LS:
>
This is used in C, but maybe for Python other solutions may be better.
I can see some different (untested) solutions:
>
1)
>
def selfassign(self , locals):
# Code from web.py, modified.
for key, value in locals.iteritem s():
if key != 'self':
setattr(self, key, value)
>
def __init__(self,
caps_first=Fals e,
hyphen_as_space =False,
underscore_as_s pace=False,
ignore_leading_ ws=False,
numerical=False ,
comma_in_numera ls=False):
selfassign(self , locals())
>
def transform(self, s):
if self.caps_first :
...
>
Disadvangages: if a flag is added/modified, the code has to be modified
in two places.
>
>
2)
>
def __init__(self, **kwds):
self.lflags = [k for k,v in kwds.items() if v]
def transform(self, s):
if "caps_first " in self.lflags:
...
>
This class can be created with 1 instead of Trues, to shorten the code.
>
Disadvantages: the user of this class has to read from the class
doctring or from from the docs the list of possible flags (and such
docs can be out of sync from the code).
>
>
3)
>
Tkinter (Tcl) shows that sometimes strings are better than int
constants (like using "left" instead of tkinter.LEFT, etc), so this is
another possibile solution:

I think maybe this is better, but I need to verify the flags so typos don't go
though silently.

def __init__(self, flags=""):
self.lflags = flags.lower().s plit()
def transform(self, s):
if "caps_first " in self.lflags:
...
>
An example of calling this class:
>
... = Collate("caps_f irst hyphen_as_space numerical")
>
I like this third (nonstandard) solution enough.
>
Bye,
bearophile
>

**Ron Adam** · Oct 18 '06, 10:05 PM

Re: Flexable Collating (feedback please)

This is how I changed it...

(I edited out the test and imports for posting here.)

locale.setlocal e(locale.LC_ALL , '') # use current locale settings

class Collate(object) :
""" A general purpose and configurable collator class.
"""
options = [ 'CAPS_FIRST', 'NUMERICAL', 'HYPHEN_AS_SPAC E',
'UNDERSCORE_AS_ SPACE', 'IGNORE_LEADING _WS',
'IGNORE_COMMAS' , 'PERIOD_AS_COMM AS' ]

def __init__(self, flags=""):
if flags:
flags = flags.upper().s plit()
for value in flags:
if value not in self.options:
raise ValueError, 'Invalid option: %s' % value
self.txtable = []
if 'HYPHEN_AS_SPAC E' in flags:
self.txtable.ap pend(('-', ' '))
if 'UNDERSCORE_AS_ SPACE' in flags:
self.txtable.ap pend(('_', ' '))
if 'PERIOD_AS_COMM AS' in flags:
self.txtable.ap pend(('.', ','))
if 'IGNORE_COMMAS' in flags:
self.txtable.ap pend((',', ''))
self.flags = flags
self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)

def transform(self, s):
""" Transform a string for collating.
"""
if not self.flags:
return locale.strxfrm( s)
for a, b in self.txtable:
s = s.replace(a, b)
if 'IGNORE_LEADING _WS' in self.flags:
s = s.strip()
if 'CAPS_FIRST' in self.flags:
s = s.swapcase()
if 'NUMERICAL' in self.flags:
slist = self.numrex.spl it(s)
for i, x in enumerate(slist ):
try:
slist[i] = float(x)
except:
slist[i] = locale.strxfrm( x)
return slist
return locale.strxfrm( s)

def __call__(self, a):
""" This allows the Collate class to be used as a sort key.

USE: list.sort(key=C ollate(flags))
"""
return self.transform( a)

def collate(slist, flags=[]):
""" Collate list of strings in place.
"""
slist.sort(key= Collate(flags). transform)

def collated(slist, flags=[]):
""" Return a collated list of strings.
"""
return sorted(slist, key=Collate(fla gs).transform)

**bearophileHUGS@lycos.com** · Oct 18 '06, 10:35 PM

Re: Flexible Collating (feedback please)

Ron Adam:

Insted of:

def __init__(self, flags=[]):
self.flags = flags
self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)
self.txtable = []
if HYPHEN_AS_SPACE in flags:
self.txtable.ap pend(('-', ' '))
if UNDERSCORE_AS_S PACE in flags:
self.txtable.ap pend(('_', ' '))
if PERIOD_AS_COMMA S in flags:
self.txtable.ap pend(('.', ','))
if IGNORE_COMMAS in flags:
self.txtable.ap pend((',', ''))
self.flags = flags

I think using a not mutable flags default is safer, this is an
alternative (NOT tested!):

numrex = re.compile(r'[\d\.]* | \D*', re.LOCALE|re.VE RBOSE)
dflags = {"hyphen_as_spa ce": ('-', ' '),
"underscore_as_ space": ('_', ' '),
"period_as_comm as": ('_', ' '),
"ignore_commas" : (',', ''),
...
}

def __init__(self, flags=()):
self.flags = [fl.strip().lowe r() for fl in flags]
self.txtable = []
df = self.__class__. dflags
for flag in self.flags:
if flag in df:
self.txtable.ap pend(df[flag])
...

This is just an idea, it surely has some problems that have to be
fixed.

Bye,
bearophile

**Gabriel Genellina** · Oct 18 '06, 10:35 PM

Re: Flexable Collating (feedback please)

At Wednesday 18/10/2006 03:42, Ron Adam wrote:

>I put together the following module today and would like some feedback on any
>obvious problems. Or even opinions of weather or not it is a good approach.
if self.flag & CAPS_FIRST:
s = s.swapcase()

This is just coincidental; it relies on (lowercase)<(up percase) on
the locale collating sequence, and I don't see why it should be always so.

if self.flag & IGNORE_LEADING_ WS:
s = s.strip()

This ignores trailing ws too. (lstrip?)

if self.flag & NUMERICAL:
if self.flag & COMMA_IN_NUMERA LS:
rex =
re.compile('^(\ d*\,?\d*\.?\d*) (\D*)(\d*\,?\d* \.?\d*)',
>re.LOCALE)
else:
rex = re.compile('^(\ d*\.?\d*)(\D*)( \d*\.?\d*)', re.LOCALE)
slist = rex.split(s)
for i, x in enumerate(slist ):
if self.flag & COMMA_IN_NUMERA LS:
x = x.replace(',', '')
try:
slist[i] = float(x)
except:
slist[i] = locale.strxfrm( x)
return slist
return locale.strxfrm( s)

You should try to make this part a bit more generic. If you are
concerned about locales, do not use "comma" explicitely. In other
countries 10*100=1.000 - and 1,234 is a fraction between 1 and 2.

The NUMERICAL option orders leading and trailing digits as numerals.
>

>>t = ['a5', 'a40', '4abc', '20abc', 'a10.2', '13.5b', 'b2']
>>collated(t, NUMERICAL)

['4abc', '13.5b', '20abc', 'a5', 'a10.2', 'a40', 'b2']

From the name "NUMERICAL" I would expect this sorting: b2, 4abc, a5,
a10.2, 13.5b, 20abc, a40 (that is, sorting as numbers only).
Maybe GROUP_NUMBERS.. . but I dont like that too much either...

--
Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

Yahoo

http://www.yahoo.com.ar/respuestas

**Ron Adam** · Oct 19 '06, 12:25 AM

Re: Flexable Collating (feedback please)

Gabriel Genellina wrote:

At Wednesday 18/10/2006 03:42, Ron Adam wrote:
>

>I put together the following module today and would like some feedback
>on any
>obvious problems. Or even opinions of weather or not it is a good
>approach.
> if self.flag & CAPS_FIRST:
> s = s.swapcase()

>
This is just coincidental; it relies on (lowercase)<(up percase) on the
locale collating sequence, and I don't see why it should be always so.

The LC_COLLATE structure (in the python.exe C code I think) controls the order
of upper and lower case during collating. I don't know if there is anyway to
examine it unfortunately.

If there was a way to change the LC_COLLATE structure, I wouldn't need to resort
to tricks like s.swapcase(). But without that info, I don't know of another way.

Maybe changing the CAPS_FIRST to REVERSE_CAPS_OR DER would do?

> if self.flag & IGNORE_LEADING_ WS:
> s = s.strip()

I'm not sure if this would make any visible difference. It might determine order
of two strings where they are the same, but one has white space at the end the
other doesn't.

They run at the same speed either way, so I'll go ahead and change it. Thanks.

This ignores trailing ws too. (lstrip?)
>

> if self.flag & NUMERICAL:
> if self.flag & COMMA_IN_NUMERA LS:
> rex =
>re.compile('^( \d*\,?\d*\.?\d* )(\D*)(\d*\,?\d *\.?\d*)',
>re.LOCALE)
> else:
> rex = re.compile('^(\ d*\.?\d*)(\D*)( \d*\.?\d*)',
>re.LOCALE)
> slist = rex.split(s)
> for i, x in enumerate(slist ):
> if self.flag & COMMA_IN_NUMERA LS:
> x = x.replace(',', '')
> try:
> slist[i] = float(x)
> except:
> slist[i] = locale.strxfrm( x)
> return slist
> return locale.strxfrm( s)

>
You should try to make this part a bit more generic. If you are
concerned about locales, do not use "comma" explicitely. In other
countries 10*100=1.000 - and 1,234 is a fraction between 1 and 2.

See the most recent version of this I posted. It is a bit more generic.

news://news.cox.net:11 9/PNxZg.6714$fl.4 591@dukeread08

Maybe a 'comma_is_decim al' option?

Options are cheep so it's no problem to add them as long as they make sense. ;-)

These options are what I refer to as mid-level options. The programmer does
still need to know something about the data they are collating. They may still
need to do some preprocessing even with this, but maybe not as much.

In a higher level collation routine, I think you would just need to specify a
named sort type, such as 'dictionary', 'directory', 'enventory' and it would set
the options and accordingly. The problem with that approach is the higher level
definitions may be different depending on locale or even the field it is used in.

> The NUMERICAL option orders leading and trailing digits as numerals.
>>

> >>t = ['a5', 'a40', '4abc', '20abc', 'a10.2', '13.5b', 'b2']
> >>collated(t, NUMERICAL)

> ['4abc', '13.5b', '20abc', 'a5', 'a10.2', 'a40', 'b2']

>
From the name "NUMERICAL" I would expect this sorting: b2, 4abc, a5,
a10.2, 13.5b, 20abc, a40 (that is, sorting as numbers only).
Maybe GROUP_NUMBERS.. . but I dont like that too much either...

How about 'VALUE_ORDERING ' ?

The term I've seen before is called natural ordering, but that is more general
and can include date, roman numerals, as well as other type.

Cheers,
Ron

**Gabriel Genellina** · Oct 19 '06, 01:55 AM

Re: Flexable Collating (feedback please)

At Wednesday 18/10/2006 21:36, Ron Adam wrote:

if self.flag & CAPS_FIRST:
s = s.swapcase()

This is just coincidental; it relies on (lowercase)<(up percase) on the
locale collating sequence, and I don't see why it should be always so.

>
>The LC_COLLATE structure (in the python.exe C code I think) controls
>the order
>of upper and lower case during collating. I don't know if there is anyway to
>examine it unfortunately.

LC_COLLATE is just a #define'd constant. I don't know how to examine
the collating definition, either.

>If there was a way to change the LC_COLLATE structure, I wouldn't
>need to resort
>to tricks like s.swapcase(). But without that info, I don't know of
>another way.
>
>Maybe changing the CAPS_FIRST to REVERSE_CAPS_OR DER would do?

At least it's a more accurate name.
There is an indirect way: test locale.strcoll( "A","a") and see how
they get sorted. Then define options CAPS_FIRST, LOWER_FIRST
accordingly. But maybe it's too much trouble...

You should try to make this part a bit more generic. If you are
concerned about locales, do not use "comma" explicitely. In other
countries 10*100=1.000 - and 1,234 is a fraction between 1 and 2.

>
>See the most recent version of this I posted. It is a bit more generic.
>
news://news.cox.net:11 9/PNxZg.6714$fl.4 591@dukeread08
>
>Maybe a 'comma_is_decim al' option?

I'd prefer to use the 'decimal_point' and 'thousands_sep' from the
locale information. That would be more coherent with the locale usage
along your module.

>Options are cheep so it's no problem to add them as long as they
>make sense. ;-)
>
>These options are what I refer to as mid-level options. The programmer does
>still need to know something about the data they are
>collating. They may still
>need to do some preprocessing even with this, but maybe not as much.
>
>In a higher level collation routine, I think you would just need to specify a
>named sort type, such as 'dictionary', 'directory', 'enventory' and
>it would set
>the options and accordingly. The problem with that approach is the
>higher level
>definitions may be different depending on locale or even the field
>it is used in.

Sure. But your module is a good starting point for building a more
high-level procedure.

The NUMERICAL option orders leading and trailing digits as numerals.
>
>>t = ['a5', 'a40', '4abc', '20abc', 'a10.2', '13.5b', 'b2']
>>collated(t, NUMERICAL)
['4abc', '13.5b', '20abc', 'a5', 'a10.2', 'a40', 'b2']

From the name "NUMERICAL" I would expect this sorting: b2, 4abc, a5,
a10.2, 13.5b, 20abc, a40 (that is, sorting as numbers only).
Maybe GROUP_NUMBERS.. . but I dont like that too much either...

>
>How about 'VALUE_ORDERING ' ?
>
>The term I've seen before is called natural ordering, but that is
>more general
>and can include date, roman numerals, as well as other type.

Sometimes that's the hard part, finding a name which is concise,
descriptive, and accurately reflects what the code does. A good name
should make obvious what it is used for (being these option names, or
class names, or method names...) but in this case it may be difficult
to find a good one. So users will have to read the documentation (a
good thing, anyway!)

--
Gabriel Genellina
Softlab SRL

_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

Yahoo

http://www.yahoo.com.ar/respuestas

**Leo Kislov** · Oct 19 '06, 10:15 AM

Re: Flexible Collating (feedback please)

Ron Adam wrote:

locale.setlocal e(locale.LC_ALL , '') # use current locale settings

It's not current locale settings, it's user's locale settings.
Application can actually use something else and you will overwrite
that. You can also affect (unexpectedly to the application)
time.strftime() and C extensions. So you should move this call into the
_test() function and put explanation into the documentation that
application should call locale.setlocal e

self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)

[snip]

if NUMERICAL in self.flags:
slist = self.numrex.spl it(s)
for i, x in enumerate(slist ):
try:
slist[i] = float(x)
except:
slist[i] = locale.strxfrm( x)

I think you should call locale.atof instead of float, since you call
re.compile with re.LOCALE.

Everything else looks fine. The biggest missing piece is support for
unicode strings.

-- Leo.

**Ron Adam** · Oct 19 '06, 12:45 PM

Re: Flexible Collating (feedback please)

Leo Kislov wrote:

Ron Adam wrote:
>

>locale.setloca le(locale.LC_AL L, '') # use current locale settings

>
It's not current locale settings, it's user's locale settings.
Application can actually use something else and you will overwrite
that. You can also affect (unexpectedly to the application)
time.strftime() and C extensions. So you should move this call into the
_test() function and put explanation into the documentation that
application should call locale.setlocal e

I'll experiment with this a bit, I was under the impression that local.strxfrm
needed the locale set for it to work correctly.

Maybe it would be better to have two (or more) versions? A string, unicode, and
locale version or maybe add an option to __init__ to choose the behavior?
Multiple versions seems to be the approach of pre-py3k. Although I was trying
to avoid that.

Sigh, of course issues like this is why it is better to have a module to do this
with. If it was as simple as just calling sort() I wouldn't have bothered. ;-)

> self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)

>
[snip]
>

> if NUMERICAL in self.flags:
> slist = self.numrex.spl it(s)
> for i, x in enumerate(slist ):
> try:
> slist[i] = float(x)
> except:
> slist[i] = locale.strxfrm( x)

>
I think you should call locale.atof instead of float, since you call
re.compile with re.LOCALE.

I think you are correct, but it seems locale.atof() is a *lot* slower than
float(). :(

Here's the local.atof() code.

def atof(string,fun c=float):
"Parses a string as a float according to the locale settings."
#First, get rid of the grouping
ts = localeconv()['thousands_sep']

if ts:
string = string.replace( ts, '')
#next, replace the decimal point with a dot
dd = localeconv()['decimal_point']
if dd:
string = string.replace( dd, '.')
#finally, parse the string
return func(string)

I could set ts and dd in __init__ and just do the replacements in the try...

if NUMERICAL in self.flags:
slist = self.numrex.spl it(s)
for i, x in enumerate(slist ):
if x: # slist may contain null strings
if self.ts:
xx = x.replace(self. ts, '') # remove thousands sep
if self.dd:
xx = xx.replace(self .dd, '.') # replace decimal point
try:
slist[i] = float(xx)
except:
slist[i] = locale.strxfrm( x)

How does that look?

It needs a fast way to determine if x is a number or a string. Any suggestions?

Everything else looks fine. The biggest missing piece is support for
unicode strings.

This was the reason for using locale.strxfrm. It should let it work with unicode
strings from what I could figure out from the documents.

Am I missing something?

Thanks,
Ron

**Ron Adam** · Oct 19 '06, 02:25 PM

Re: Flexible Collating (feedback please)

bearophileHUGS@ lycos.com wrote:

Ron Adam:
>
Insted of:
>
def __init__(self, flags=[]):
self.flags = flags
self.numrex = re.compile(r'([\d\.]*|\D*)', re.LOCALE)
self.txtable = []
if HYPHEN_AS_SPACE in flags:
self.txtable.ap pend(('-', ' '))
if UNDERSCORE_AS_S PACE in flags:
self.txtable.ap pend(('_', ' '))
if PERIOD_AS_COMMA S in flags:
self.txtable.ap pend(('.', ','))
if IGNORE_COMMAS in flags:
self.txtable.ap pend((',', ''))
self.flags = flags
>
I think using a not mutable flags default is safer, this is an
alternative (NOT tested!):
>
numrex = re.compile(r'[\d\.]* | \D*', re.LOCALE|re.VE RBOSE)
dflags = {"hyphen_as_spa ce": ('-', ' '),
"underscore_as_ space": ('_', ' '),
"period_as_comm as": ('_', ' '),
"ignore_commas" : (',', ''),
...
}
>
def __init__(self, flags=()):
self.flags = [fl.strip().lowe r() for fl in flags]
self.txtable = []
df = self.__class__. dflags
for flag in self.flags:
if flag in df:
self.txtable.ap pend(df[flag])
...
>
This is just an idea, it surely has some problems that have to be
fixed.

I think the 'if's are ok since there are only a few options that need to be
handled by them.

I'm still trying to determine what options are really needed. I can get the
thousand separator and decimal character from local.localconv () function. So
ignore_commas isn't needed I think. And maybe change period_as_comma s to period
_as_sep and then split on periods before comparing.

I also want it to issue exceptions when the Collate object is created if invalid
options are specified. That makes finding problems much easier. The example
above doesn't do that, it accepts them silently. That was one of the reasons I
went to named constants at first.

How does this look?

numrex = re.compile(r'([\d\.]* | \D*)', re.LOCALE|re.VE RBOSE)
options = ( 'CAPS_FIRST', 'NUMERICAL', 'HYPHEN_AS_SPAC E',
'UNDERSCORE_AS_ SPACE', 'IGNORE_LEADING _WS',
'IGNORE_COMMAS' , 'PERIOD_AS_COMM AS' )
def __init__(self, flags=""):
if flags:
flags = flags.upper().s plit()
for value in flags:
if value not in self.options:
raise ValueError, 'Invalid option: %s' % value
self.txtable = []
if 'HYPHEN_AS_SPAC E' in flags:
self.txtable.ap pend(('-', ' '))
if 'UNDERSCORE_AS_ SPACE' in flags:
self.txtable.ap pend(('_', ' '))
if 'PERIOD_AS_COMM AS' in flags:
self.txtable.ap pend(('.', ','))
if 'IGNORE_COMMAS' in flags:
self.txtable.ap pend((',', ''))
self.flags = flags

So you can set an option strings as...

import collate as C

collateopts = \
""" caps_first
hyphen_as_space
numerical
ignore_commas
"""
colatedlist = C.collated(some list, collateopts)

A nice advantage with an option string is you don't have to prepend all your
options with the module name. But you do have to validate it.

Cheers,
Ron

Flexable Collating (feedback please)

Flexable Collating (feedback please)

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment