sre is broken in SuSE 9.2

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Fredrik Lundh

    #16
    Re: sre is broken in SuSE 9.2

    Serge Orlov wrote:
    [color=blue][color=green][color=darkred]
    >> >>> re.compile(ur'\ w+', re.U).findall(u '\xb5\xba\xe4\u 0430')[/color]
    >> [u'\xb5\xba\xe4\ u0430'][/color]
    >
    > I can't find the strict definition of isalpha, but I believe average
    > C program shouldn't care about the current locale alphabet, so isalpha
    > is a union of all supported characters in all alphabets[/color]

    nope. isalpha() depends on the locale, as does all other ctype functions
    (this also applies to wctype, on some platforms).

    </F>



    Comment

    • Martin v. Löwis

      #17
      Re: sre is broken in SuSE 9.2

      Serge Orlov wrote:[color=blue]
      > To summarize the discussion: either it's a bug in glibc or there is an
      > option to specify modern POSIX locale. POSIX locale consist of
      > characters from the portable character set, unicode is certainly
      > portable.[/color]

      Yes, but U+00E4 is not in the portable character set. The portable
      character set is defined here:



      Regards,
      Martin

      Comment

      • Serge Orlov

        #18
        Re: sre is broken in SuSE 9.2

        "Martin v. Löwis" wrote:[color=blue]
        > Serge Orlov wrote:[color=green]
        > > To summarize the discussion: either it's a bug in glibc or there[/color]
        > is an[color=green]
        >> option to specify modern POSIX locale. POSIX locale consist of
        >> characters from the portable character set, unicode is certainly
        >> portable.[/color]
        >
        > Yes, but U+00E4 is not in the portable character set. The portable
        > character set is defined here:
        >
        > http://www.opengroup.org/onlinepubs/...d/charset.html[/color]

        Thanks for the link. They write (in 1997 or earlier ?):

        The wide-character value for each member of the Portable
        Character Set will equal its value when used as the lone character
        in an integer character constant. Wide-character codes for other
        characters are locale- and *implementation-dependent*

        Emphasis is mine. So how many libc implementations with
        non-unicode wide-character codes do we have in 2005?
        I'm really interested to know.

        Serge.


        Comment

        • Serge Orlov

          #19
          Re: sre is broken in SuSE 9.2

          Fredrik Lundh wrote:[color=blue]
          > Serge Orlov wrote:
          >[color=green][color=darkred]
          >>>>>> re.compile(ur'\ w+', re.U).findall(u '\xb5\xba\xe4\u 0430')
          >>>>>> [u'\xb5\xba\xe4\ u0430'][/color]
          >>
          >> I can't find the strict definition of isalpha, but I believe average
          >> C program shouldn't care about the current locale alphabet, so
          >> isalpha is a union of all supported characters in all alphabets[/color]
          >
          > btw, what does isalpha have to do with this example?[/color]

          It has to do with this thread. u'\xe4'.isalpha () returns false in
          Suse. It's in the same boat as \w

          Serge.


          Comment

          • Serge Orlov

            #20
            Re: sre is broken in SuSE 9.2

            Fredrik Lundh wrote:[color=blue]
            > Serge Orlov wrote:
            >[color=green][color=darkred]
            >>>>>> re.compile(ur'\ w+', re.U).findall(u '\xb5\xba\xe4\u 0430')
            >>>>>> [u'\xb5\xba\xe4\ u0430'][/color]
            >>
            >> I can't find the strict definition of isalpha, but I believe average
            >> C program shouldn't care about the current locale alphabet, so
            >> isalpha is a union of all supported characters in all alphabets[/color]
            >
            > nope. isalpha() depends on the locale, as does all other ctype
            > functions (this also applies to wctype, on some platforms).[/color]

            I mean "all supported characters in all alphabets [in the current
            locale]". For example in ru_RU.koi8-r isalpha should return
            true for characters in English and Russian alphabets. In
            ru_RU.koi8-u -- for characters in English, Russia and Ukrain
            alphabets, in ru_RU.utf-8 -- for all supported by the implementation
            alphabetic characters in unicode. IMHO iswalpha in POSIX
            locale can return true for all alphabetic characters in unicode
            instead of being limited by English alphabet.

            Serge.

            true in


            Comment

            • Fredrik Lundh

              #21
              Re: sre is broken in SuSE 9.2

              Serge Orlov wrote:
              [color=blue]
              > The wide-character value for each member of the Portable
              > Character Set will equal its value when used as the lone character
              > in an integer character constant. Wide-character codes for other
              > characters are locale- and *implementation-dependent*
              >
              > Emphasis is mine.[/color]

              the relevant part for this thread is *locale-*. if wctype depends on the
              locale, it cannot be used for generic build. (custom interpreters are an-
              other thing, but they shouldn't be shipped as "python").

              </F>



              Comment

              • Denis S. Otkidach

                #22
                Re: sre is broken in SuSE 9.2

                On Fri, 11 Feb 2005 18:49:53 +0100
                "Fredrik Lundh" <fredrik@python ware.com> wrote:
                [color=blue][color=green][color=darkred]
                > >> >>> re.compile(ur'\ w+', re.U).findall(u '\xb5\xba\xe4\u 0430')
                > >> [u'\xb5\xba\xe4\ u0430'][/color]
                > >
                > > I can't find the strict definition of isalpha, but I believe average
                > > C program shouldn't care about the current locale alphabet, so isalpha
                > > is a union of all supported characters in all alphabets[/color]
                >
                > btw, what does isalpha have to do with this example?[/color]

                The same problem is with isalpha. In most distributions:[color=blue][color=green][color=darkred]
                >>> for c in u'\xb5\xba\xe4\ u0430': print c.isalpha(),[/color][/color][/color]
                ....
                True True True True

                And in SuSE 9.2:[color=blue][color=green][color=darkred]
                >>> for c in u'\xb5\xba\xe4\ u0430': print c.isalpha(),[/color][/color][/color]
                ....
                False False False False

                --
                Denis S. Otkidach
                http://www.python.ru/ [ru]

                Comment

                • Denis S. Otkidach

                  #23
                  Re: sre is broken in SuSE 9.2

                  On Sat, 12 Feb 2005 09:42:41 +0100
                  "Fredrik Lundh" <fredrik@python ware.com> wrote:
                  [color=blue]
                  > the relevant part for this thread is *locale-*. if wctype depends on
                  > the locale, it cannot be used for generic build. (custom interpreters
                  > are an- other thing, but they shouldn't be shipped as "python").[/color]

                  You are right. But isalpha behavior looks strange for me anyway: why
                  cyrillic character '\u0430' is recognized as alpha one for de_DE locale,
                  but is not for C?

                  --
                  Denis S. Otkidach
                  http://www.python.ru/ [ru]

                  Comment

                  • Martin v. Löwis

                    #24
                    Re: sre is broken in SuSE 9.2

                    Serge Orlov wrote:[color=blue]
                    > Emphasis is mine. So how many libc implementations with
                    > non-unicode wide-character codes do we have in 2005?[/color]

                    Solaris has supported 2-byte wchar_t implementations for many
                    years, and so I believe did HP-UX and AIX.

                    ISO C99 defines a constant __STDC_ISO_1064 6__ which an
                    implementation can use to indicate that wchar_t uses
                    Unicode (aka ISO 10646) in all locales. Very few
                    implementations define this constant at this time, though.

                    Regards,
                    Martin

                    Comment

                    • Martin v. Löwis

                      #25
                      Re: sre is broken in SuSE 9.2

                      Denis S. Otkidach wrote:[color=blue]
                      > You are right. But isalpha behavior looks strange for me anyway: why
                      > cyrillic character '\u0430' is recognized as alpha one for de_DE locale,
                      > but is not for C?[/color]

                      In glibc, all "real" locales are based on
                      /usr/share/locale/i18n/locales/i18n, e.g. for de_DE through

                      LC_CTYPE
                      copy "i18n"

                      i18n includes U+0430 as a character, through

                      lower /
                      ....
                      % TABLE 11 CYRILLIC/
                      <U0430>..<U045F >;<U0461>..(2). .<U047F>;/

                      This makes U+0430 a letter in all locales including i18n
                      (unless locally overridden). This entire approach apparently
                      is based on ISO 14652, which, in section 4.3.3, introduces
                      the "i18n" LC_CTYPE category.

                      Why the C locale does not use i18n, I don't know. Most likely,
                      the intention is that the "C" locale works without any
                      additional data files - you should ask the glibc developers.
                      OTOH, there is a definition file POSIX for what appears
                      to be the POSIX locale.

                      I'd like to point out that this implementation is potentially
                      in violation of ISO 14652; annex A.2.2 says that the notion
                      of a POSIX locale is replaced with the i18n FDCC-set. So
                      accordingly, I would expect that i18n is used in POSIX as
                      well - see for yourself that it isn't in glibc 2.3.2.

                      Again, I suggest to ask the glibc developers as to why
                      this is so.

                      Regards,
                      Martin

                      Comment

                      Working...