distutils & OS X universal binaries

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 8 '07, 03:05 PM

Re: distutils & OS X universal binaries

One proposed fix is to make the endian variable code dynamically change

at run time.

I would advise against that. Endianness depdency should be resolved at
compile time, with appropriate conditional compilation. Endianness won't
change at run-time (and no, not even for a fat binary - the x86 code
will always "see" the same endianness, and so will the ppc code).

Is there a way to get distutils to pass different macros/definitions to
the separate compilations.

No. distutils only invokes the compiler a single time, not multiple
times, for a specific universal object file. The gcc driver then invokes
different cc1 backends repeatedly.

Failing that does anyone know off hand exactly how this problem is
supposed to be handled? Ie if there are multiple compiles what
distinguishes them. My understanding is that up to 4 different binaries
are being squashed together in these universal binaries.

In the specific case, just use the WORDS_BIGENDIAN macro defined in
pyconfig.h; it will be defined if the target is bigendian, and
undefined otherwise. In the case of a universal build, it will be
undefined in the x86 compiler invocation, and defined in the ppc
invocation.

If you are curious as to how it arranges that, read the source.

Regards,
Martin

**Christian Heimes** · Dec 8 '07, 03:15 PM

Re: distutils & OS X universal binaries

Robin Becker wrote:

A user reports problems with one of our extensions when running the
intel compiled extension on ppc and vice versa. He is building the
extension as a universal binary. Although the intel compiled version
runs fine it displays a known bug when run on a ppc.

Have you reported the problem at http://bugs.python.org/? A minimal
example could help us to fix the problem.

Christian

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 8 '07, 03:35 PM

Re: distutils & OS X universal binaries

>A user reports problems with one of our extensions when running the

>intel compiled extension on ppc and vice versa. He is building the
>extension as a universal binary. Although the intel compiled version
>runs fine it displays a known bug when run on a ppc.

>
Have you reported the problem at http://bugs.python.org/? A minimal
example could help us to fix the problem.

At first, I also thought that Robin suggested that there is a problem
with Python. Upon re-reading, I now believe he rather sees the bug
in the reportlabs code, and is asking for an approach to solve it there.

Regards,
Martin

**Robin Becker** · Dec 8 '07, 04:05 PM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:

>>A user reports problems with one of our extensions when running the
>>intel compiled extension on ppc and vice versa. He is building the
>>extension as a universal binary. Although the intel compiled version
>>runs fine it displays a known bug when run on a ppc.

>Have you reported the problem at http://bugs.python.org/? A minimal
>example could help us to fix the problem.

>
At first, I also thought that Robin suggested that there is a problem
with Python. Upon re-reading, I now believe he rather sees the bug
in the reportlabs code, and is asking for an approach to solve it there.
>.....

Yes that's right. Unfortunately this problem doesn't arise in the python
interface, but in libart_lgpl which we depend on. I will look at the
pyconfig.h code to see how our definition should be put into libart's .h
config file. Presumably I can then remove the calculated definition we
have in our setup.py script or at least override it in the right way.

PIL may also have a similar problem as the 1.1.6 setup.py script also
defines WORDS_BIGENDIAN like this

if struct.unpack(" h", "\0\1")[0] == 1:
defs.append(("W ORDS_BIGENDIAN" , None))

probably I borrowed/stole this as we have something very similar in our
setup.py.
--
Robin Becker

**Christian Heimes** · Dec 8 '07, 04:25 PM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:

At first, I also thought that Robin suggested that there is a problem
with Python. Upon re-reading, I now believe he rather sees the bug
in the reportlabs code, and is asking for an approach to solve it there.

I saw your posting after I sent mine. The gmane web interface is slow
and sluggish today.

The macro WORDS_BIGENDIAN isn't mentioned in the docs. The docs sure
need some extra information how to create universal binaries and how to
write endian safe C code. I'm going to create a GHOP task.

Christian

**Robin Becker** · Dec 8 '07, 04:55 PM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:
..........

>
In the specific case, just use the WORDS_BIGENDIAN macro defined in
pyconfig.h; it will be defined if the target is bigendian, and
undefined otherwise. In the case of a universal build, it will be
undefined in the x86 compiler invocation, and defined in the ppc
invocation.
>
If you are curious as to how it arranges that, read the source.

.......

OK I read the source and am slightly puzzled by the code in
pyconfig.h.in which reads

#ifdef __BIG_ENDIAN__
#define WORDS_BIGENDIAN 1
#else
#ifndef __LITTLE_ENDIAN __
#undef WORDS_BIGENDIAN
#endif
#endif

I'm puzzled why WORDS_BIGENDIAN is undefined if both __BIG_ENDIAN__ and
__LITTLE_ENDIAN __ are undefined. Surely in that case WORDS_BIGENDIAN
should be left alone (if it is already defined). If there's a compiler
for a bigendian architecture which doesn't define the gcc macros the we
seem to get the wrong result.
--
Robin Becker

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 8 '07, 05:15 PM

Re: distutils & OS X universal binaries

#ifdef __BIG_ENDIAN__

#define WORDS_BIGENDIAN 1
#else
#ifndef __LITTLE_ENDIAN __
#undef WORDS_BIGENDIAN
#endif
#endif
>
>
I'm puzzled why WORDS_BIGENDIAN is undefined if both __BIG_ENDIAN__ and
__LITTLE_ENDIAN __ are undefined. Surely in that case WORDS_BIGENDIAN
should be left alone (if it is already defined). If there's a compiler
for a bigendian architecture which doesn't define the gcc macros the we
seem to get the wrong result.

No. pyconfig.h.in gets processed by configure into pyconfig.h; configure
replaces all #undef lines with appropriate #define lines if the macro
got define in configure. The autoconf macro AC_C_BIGENDIAN performs
a configure-time check. So

- if the compiler either defines __BIG_ENDIAN__ or __LITTLE_ENDIAN __,
that is taken for granted.
- otherwise, the configure-time value is used

On your normal big-endian compiler (e.g. SPARC), it's the
configure-time value that makes WORDS_BIGENDIAN defined.

HTH,
Martin

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 8 '07, 05:15 PM

Re: distutils & OS X universal binaries

PIL may also have a similar problem as the 1.1.6 setup.py script also

defines WORDS_BIGENDIAN like this
>
if struct.unpack(" h", "\0\1")[0] == 1:
defs.append(("W ORDS_BIGENDIAN" , None))
>
probably I borrowed/stole this as we have something very similar in our
setup.py.

All such checks are broken for fat binaries. Fat binaries essentially
are a form of cross-compilation, and in cross-compilation, thou shalt
not infer target system properties by looking at the host system.

(IOW, autoconf is, in principle, also broken for fat binaries. Indeed,
although the current solution for WORDS_BIGENDIAN is good for ppc
vs. x86, many of the other configure-time detected properties are
incorrect for ppc vs. ppc64 or x86 vs. amd64, such as SIZEOF_LONG)

Regards,
Martin

**Robin Becker** · Dec 8 '07, 05:35 PM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:
........

>I'm puzzled why WORDS_BIGENDIAN is undefined if both __BIG_ENDIAN__ and
>__LITTLE_ENDIA N__ are undefined. Surely in that case WORDS_BIGENDIAN
>should be left alone (if it is already defined). If there's a compiler
>for a bigendian architecture which doesn't define the gcc macros the we
>seem to get the wrong result.

>
No. pyconfig.h.in gets processed by configure into pyconfig.h; configure
replaces all #undef lines with appropriate #define lines if the macro
got define in configure. The autoconf macro AC_C_BIGENDIAN performs
a configure-time check. So

>
- if the compiler either defines __BIG_ENDIAN__ or __LITTLE_ENDIAN __,
that is taken for granted.
- otherwise, the configure-time value is used
>
On your normal big-endian compiler (e.g. SPARC), it's the
configure-time value that makes WORDS_BIGENDIAN defined.

.......

OK I need to use something a bit more complex then; I figure this should
work

#if defined(__BIG_E NDIAN__) || defined(__LITTL E_ENDIAN__)
# ifdef __BIG_ENDIAN__
# ifdef WORDS_BIGENDIAN
# undef WORDS_BIGENDIAN
# endif
# define WORDS_BIGENDIAN 1
# else
# ifdef __LITTLE_ENDIAN __
# ifdef WORDS_BIGENDIAN
# undef WORDS_BIGENDIAN
# endif
# endif
# endif
#endif

--
Robin Becker

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 8 '07, 05:55 PM

Re: distutils & OS X universal binaries

OK I need to use something a bit more complex then; I figure this should

work
>
#if defined(__BIG_E NDIAN__) || defined(__LITTL E_ENDIAN__)
# ifdef __BIG_ENDIAN__
# ifdef WORDS_BIGENDIAN
# undef WORDS_BIGENDIAN
# endif
# define WORDS_BIGENDIAN 1
# else
# ifdef __LITTLE_ENDIAN __
# ifdef WORDS_BIGENDIAN
# undef WORDS_BIGENDIAN
# endif
# endif
# endif
#endif

I don't understand. If you assume that either __BIG_ENDIAN__ or
__LITTLE_ENDIAN __ is defined, anyway - just use that!

If neither is defined, you are still lost, unless you use pyconfig.h,
in which case, you don't need anything of that.

Regards,
Martin

**Robin Becker** · Dec 8 '07, 06:25 PM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:

>OK I need to use something a bit more complex then; I figure this should
>work
>>
>#if defined(__BIG_E NDIAN__) || defined(__LITTL E_ENDIAN__)
># ifdef __BIG_ENDIAN__
># ifdef WORDS_BIGENDIAN

.........

>#endif

>
I don't understand. If you assume that either __BIG_ENDIAN__ or
__LITTLE_ENDIAN __ is defined, anyway - just use that!
>
If neither is defined, you are still lost, unless you use pyconfig.h,
in which case, you don't need anything of that.

I prefer to continue using WORDS_BIGENDIAN so fewer changes need to be
made to the code. It just makes resynching with the upstream code
easier. If neither are defined we get to use the definition from
setup.py if it's needed.
--
Robin Becker

**=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=** · Dec 9 '07, 10:25 AM

Re: distutils & OS X universal binaries

I prefer to continue using WORDS_BIGENDIAN so fewer changes need to be

made to the code. It just makes resynching with the upstream code
easier. If neither are defined we get to use the definition from
setup.py if it's needed.

Ok. Still, I would write it as

#if defined(__LITTL E_ENDIAN__)
#undef WORDS_BIGENDIAN
#elif defined(__BIG_E NDIAN__)
#undef WORDS_BIGENDIAN
#define WORDS_BIGENDIAN 1
#endif

Regards,
Martin

**Robin Becker** · Dec 9 '07, 10:55 AM

Re: distutils & OS X universal binaries

Martin v. Löwis wrote:

>I prefer to continue using WORDS_BIGENDIAN so fewer changes need to be
>made to the code. It just makes resynching with the upstream code
>easier. If neither are defined we get to use the definition from
>setup.py if it's needed.

>
>
Ok. Still, I would write it as
>
#if defined(__LITTL E_ENDIAN__)
#undef WORDS_BIGENDIAN
#elif defined(__BIG_E NDIAN__)
#undef WORDS_BIGENDIAN
#define WORDS_BIGENDIAN 1
#endif
>
Regards,
Martin

I'm never sure if undef gives an error if the variable isn't defined,
but that is cleaner

thanks for the assistance
--
Robin Becker

**Robin Becker** · Dec 10 '07, 02:55 PM

Re: distutils & OS X universal binaries

Robin Becker wrote:
.........

>Ok. Still, I would write it as
>>
>#if defined(__LITTL E_ENDIAN__)
> #undef WORDS_BIGENDIAN
>#elif defined(__BIG_E NDIAN__)
> #undef WORDS_BIGENDIAN
> #define WORDS_BIGENDIAN 1
>#endif
>>
>Regards,
>Martin

I'm never sure if undef gives an error if the variable isn't defined,
but that is cleaner
>
thanks for the assistance

Just like to say my testee/victim reports great success with the above snippet
inserted into the right config file. Thanks to all.
--
Robin Becker

distutils & OS X universal binaries

distutils & OS X universal binaries

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment