regexps with unicode-aware characterclasses?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Stefan Rank

    #1

    regexps with unicode-aware characterclasses?

    Hi all,

    in a python re pattern, how do I match all unicode uppercase characters
    (in a unicode string/in a utf-8 string)?

    I know that there is string.uppercas e/.lowercase which are
    'locale-aware', but I don't think there is a "all locales" locale.

    I know that there is a re.U switch that makes \w match all unicode word
    characters, but there are no subclasses of that ([[:upper:]] or
    preferably \u).
    Or is there a module/extension to get that?

    There is the module unicodedata, but it has no unicodedata.upp ercase
    that would correspond to string.uppercas e.

    <wishful thinking>

    re.compile('|'. join([x.encode('utf8' ) for x in unicode.upperca se]))

    or::

    re.compile('(?u )[[:upper:]]')

    or::

    re.compile('(?u )\u')

    for the latter two, to work on utf-8 strings, would I have to set the
    defaultencoding to utf-8?

    </wishful thinking>

  • Martin v. Löwis

    #2
    Re: regexps with unicode-aware characterclasse s?

    Stefan Rank wrote:[color=blue]
    > <wishful thinking>
    >
    > re.compile('|'. join([x.encode('utf8' ) for x in unicode.upperca se]))[/color]

    This would (almost) work, but it would be terribly inefficient (time
    linear to the number of alternatives). You can realistically do

    uppers = [u'[']
    for i in range(sys.maxun icode):
    c = unichr(i)
    if c.isupper(): uppers.append(c )
    uppers.append(u ']')
    uppers = u"".join(uppers )
    uppers_re = re.compile(uppe rs)

    Compiling this expression is quite expensive; matching it is fairly
    efficient (time independent of the number of characters in the class).
    To save startup cost, consider pickling the compiled expression.

    (syntax note: this only works because none of the characters special
    to a RE class (]-^\) is an uppercase letter; otherwise, escaping might
    be needed)
    [color=blue]
    > for the latter two, to work on utf-8 strings, would I have to set the
    > defaultencoding to utf-8?[/color]

    For Unicode things, you should avoid using byte strings - especially
    when it comes to regular expressions. Use Unicode strings instead.

    Regards,
    Martin

    Comment

    Working...