python regex character group matches

[christopher taylor]

hello python-list!

the other day, i was trying to match unicode character sequences that
looked like this:

\\uAD0X...

my issue, is that the pattern i used was returning:

[ '\\uAD0X', '\\u1BF3', ... ]

when i expected:

[ '\\uAD0X\\u1BF3 ', ]

the code looks something like this:

pat = re.compile("(\\ \u[0-9A-F]{4})+", re.UNICODE|re.L OCALE)
#print pat.findall(txt _line)
results = pat.finditer(tx t_line)

i ran the pattern through a couple of my colleagues and they were all
in agreement that my pattern should have matched correctly.

is this a simple case of a messed up regex or am i not using the regex
api correctly?

cheers,

ct

[Marc 'BlackJack' Rintsch]

Re: python regex character group matches

On Wed, 17 Sep 2008 09:27:47 -0400, christopher taylor wrote:

the other day, i was trying to match unicode character sequences that
looked like this:
>
\\uAD0X...
>
my issue, is that the pattern i used was returning:
>
[ '\\uAD0X', '\\u1BF3', ... ]
>
when i expected:
>
[ '\\uAD0X\\u1BF3 ', ]
>
the code looks something like this:
>
pat = re.compile("(\\ \u[0-9A-F]{4})+", re.UNICODE|re.L OCALE) #print
pat.findall(txt _line)
results = pat.finditer(tx t_line)
>
i ran the pattern through a couple of my colleagues and they were all in
agreement that my pattern should have matched correctly.

Correctly for what input? And the examples above are not matching (no
pun intended) the regular expression. `pat` doesn't match '\\uAD0X'
because there's no 'X' in the character class. BTW: Are you sure you
need or want the `re.UNICODE` flag?

Ciao,
Marc 'BlackJack' Rintsch

[Steven D'Aprano]

Re: python regex character group matches

On Wed, 17 Sep 2008 09:27:47 -0400, christopher taylor wrote:

hello python-list!
>
the other day, i was trying to match unicode character sequences that
looked like this:
>
\\uAD0X...

It is not clear what this is supposed to be. Is that matching a literal
pair of backslashes, or a single escaped backslash, or a single unicode
character with codepoint AD0X, or what?

If I read it *literally*, then you're trying to match:

backslash backslash lowercase-u uppercase-A uppercase-D zero uppercase-X

Is that what you intended to match?

my issue, is that the pattern i used was returning:
>
[ '\\uAD0X', '\\u1BF3', ... ]

Unless you are using Python 3, I see that you aren't actually dealing
with Unicode strings, you're using byte strings. Is that deliberate?

when i expected:
>
[ '\\uAD0X\\u1BF3 ', ]

I make that to be a string of length 12. Is that what you are expecting?

>>len('\\uAD0X\ \u1BF3')

12

the code looks something like this:
>
pat = re.compile("(\\ \u[0-9A-F]{4})+", re.UNICODE|re.L OCALE)
#print pat.findall(txt _line)
results = pat.finditer(tx t_line)

First point: I don't think the UNICODE flag does what you think it does.
It redefines the meaning of special escape sequences \b etc. Since you
aren't using any special escape sequences, I'm going to guess that you
think it turns your search string into Unicode. It doesn't. (Apologies in
advance if I guessed wrong.) I don't think you need either the UNICODE or
LOCALE flag for this search: they don't seem to have any effect.

Secondly: you will generally save yourself a lot of trouble when writing
regexes to use raw strings, because backslashes in the regex engine clash
with backslashes in Python strings. But there's a gotcha: backslash
escapes behave differently in ordinary strings and the re engine.

In an ordinary string, the sequence backslash-char is treated as a
literal backslash-char if it isn't a special escape. So:

>>len('\t') # special escape

1

>>len('\u') # not a special escape

2

But that's not the case in the re engine! As the Fine Manual says:

The special sequences consist of "\" and a character from
the list below. If the ordinary character is not on the
list, then the resulting RE will match the second character.
For example, \$ matches the character "$".

http://docs.python.org/lib/re-syntax.html

So all of these match the same thing:
re.compile('\\u ')
re.compile(r'\u ')
re.compile('u')

To match a literal backslash-u, you need to escape the backslash before
the engine joins it to the u: r'\\u'.

Putting it all together again:

pat = re.compile(r"(\ \u[0-9A-F]{4})+")

will probably do what you want, assuming I have guessed what you want
correctly!



--
Steven