eggs considered harmful

[Harry George]

....at least around here.

I run a corporate Open Source Software Toolkit, which makes hundreds
of libraries and apps available to thousands of technical employees.
The rules are that a) a very few authorized downloaders obtain
tarballs and put them in a depot and b) other users get tarballs from
the depot and build from source.

Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.

Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

These are unacceptable behaviors. I am therefore dropping ZODB3, and
am considering dropping TurboGears and ZSI. If the egg paradigm
spreads, yet more packages will be dropped (or will never get a chance
to compete for addition).

I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well. You
can have dependencies, as long as they are documented and can be
obtained by separate manual download.

Thanks for listening.

--
Harry George
PLM Engineering Architecture

[John J. Lee]

Re: eggs considered harmful

Harry George <harry.g.george @boeing.comwrit es:
[...]

These are unacceptable behaviors. I am therefore dropping ZODB3, and
am considering dropping TurboGears and ZSI. If the egg paradigm
spreads, yet more packages will be dropped (or will never get a chance
to compete for addition).
>
I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well. You
can have dependencies, as long as they are documented and can be
obtained by separate manual download.

1. Given the presumptuous tone of your own message, I guess I'm not in
danger of coming across as more rude than you when I point out that
your requirements are just that: your own. The rest of the world
won't *always* bend over backwards to support just exactly what you'd
most prefer.

2. You can run your own private egg repository. IIRC, it's as simple
as a directory of eggs and a plain old web server with directory
listings turned on. You then run easy_install -f URL package_name
instead of easy_install package_name . The distutils-sig archives
will have more on this.

3. Alternatively, you could create bundled packages that include
dependencies (perhaps zc.buildout can do that for you, even? not sure)


John

[Robert Kern]

Re: eggs considered harmful

Harry George wrote:

...at least around here.
>
I run a corporate Open Source Software Toolkit, which makes hundreds
of libraries and apps available to thousands of technical employees.
The rules are that a) a very few authorized downloaders obtain
tarballs and put them in a depot and b) other users get tarballs from
the depot and build from source.
>
Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.
>
Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

Have you considered establishing a policy that these setuptools-using packages
should be installed using the --single-version-externally-managed option to the
install command? This does not check for dependencies.

Alternately, you can provide a company repository of the tarballs and their
depedencies tarballs. Your users can use the easy_install option --find-links to
point to that URL such that they do not have to go outside of the firewall to
install everything.

These are unacceptable behaviors. I am therefore dropping ZODB3, and
am considering dropping TurboGears and ZSI. If the egg paradigm
spreads, yet more packages will be dropped (or will never get a chance
to compete for addition).

I'm sorry to hear that.

I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well. You
can have dependencies, as long as they are documented and can be
obtained by separate manual download.

Given the options I outlined above, you can easily satisfy these requirements
for the vast majority of setuptools-using packages that are out there. There are
a handful of packages that only distribute the eggs and not the source tarballs,
but those are rare.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Ben Finney]

Setuptools, build and install dependencies (was: eggs considered harmful)

Harry George <harry.g.george @boeing.comwrit es:

Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.
>
Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

If you provide the build and install script with all the dependencies
already present (in the current directory), my experience is that
setuptools does not do any network actions.

--
\ "Self-respect: The secure feeling that no one, as yet, is |
`\ suspicious." -- Henry L. Mencken |
_o__) |
Ben Finney

[Harry George]

Re: eggs considered harmful

jjl@pobox.com (John J. Lee) writes:

Harry George <harry.g.george @boeing.comwrit es:
[...]

These are unacceptable behaviors. I am therefore dropping ZODB3, and
am considering dropping TurboGears and ZSI. If the egg paradigm
spreads, yet more packages will be dropped (or will never get a chance
to compete for addition).

I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well. You
can have dependencies, as long as they are documented and can be
obtained by separate manual download.

>
1. Given the presumptuous tone of your own message, I guess I'm not in
danger of coming across as more rude than you when I point out that
your requirements are just that: your own. The rest of the world
won't *always* bend over backwards to support just exactly what you'd
most prefer.
>

You deleted the "...at least here", which was intended to make clear I
was NOT speaking for the world at large, though possibly for a large
chunk of corporate life. Also, this wasn't out of the lbue. I ha ve
previously discussed this with several development teasm privately,
but the trend appears to be accelerating

2. You can run your own private egg repository. IIRC, it's as simple
as a directory of eggs and a plain old web server with directory
listings turned on. You then run easy_install -f URL package_name
instead of easy_install package_name . The distutils-sig archives
will have more on this.

Again, not speaking for anyone else: With 500 OSS packages, all of
which play by the same tarball rules, we don't have resources to
handle eggs differently.

>
3. Alternatively, you could create bundled packages that include
dependencies (perhaps zc.buildout can do that for you, even? not sure)
>

No resources for special handling.

>
John

--
Harry George
PLM Engineering Architecture

[Harry George]

Re: eggs considered harmful

Robert Kern <robert.kern@gm ail.comwrites:

Harry George wrote:

...at least around here.

I run a corporate Open Source Software Toolkit, which makes hundreds
of libraries and apps available to thousands of technical employees.
The rules are that a) a very few authorized downloaders obtain
tarballs and put them in a depot and b) other users get tarballs from
the depot and build from source.

Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.

Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

>
Have you considered establishing a policy that these setuptools-using packages
should be installed using the --single-version-externally-managed option to the
install command? This does not check for dependencies.

I didn't know that one. I'll try it. Thanks.

>
Alternately, you can provide a company repository of the tarballs and their
depedencies tarballs. Your users can use the easy_install option --find-links to
point to that URL such that they do not have to go outside of the firewall to
install everything.
>

This is a possibility. The tarballs can be seen in a directory
listing. They are in different subdirs (for different "bundles" of
functionality), so I'll need -f to look several places.

These are unacceptable behaviors. I am therefore dropping ZODB3, and
am considering dropping TurboGears and ZSI. If the egg paradigm
spreads, yet more packages will be dropped (or will never get a chance
to compete for addition).

>
I'm sorry to hear that.

Me too. We worked long and hard to get Python established as a
standard language for corporate systems development, we have a host of
projects that need ZSI, and I look forward to making further inroads
into C++, Java, and VB development camps. Didn't really need a
roadblock at this point.

>

I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well. You
can have dependencies, as long as they are documented and can be
obtained by separate manual download.

>
Given the options I outlined above, you can easily satisfy these requirements
for the vast majority of setuptools-using packages that are out there. There are
a handful of packages that only distribute the eggs and not the source tarballs,
but those are rare.
>

I agree pure eggs are rare. The fact that they increased this past
quarter was what concerned me. ZODB even looks like a normal tarball,
builds ok, but uses a easy-install-style lookup during install.

--
Robert Kern
>
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
>

--
Harry George
PLM Engineering Architecture

[Harry George]

Re: Setuptools, build and install dependencies (was: eggs considered harmful)

Ben Finney <bignose+hate s-spam@benfinney. id.auwrites:

Harry George <harry.g.george @boeing.comwrit es:
>

Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.

Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

>
If you provide the build and install script with all the dependencies
already present (in the current directory), my experience is that
setuptools does not do any network actions.
>
--
\ "Self-respect: The secure feeling that no one, as yet, is |
`\ suspicious." -- Henry L. Mencken |
_o__) |
Ben Finney

Thanks for the idea. It doesn't work so well in our context, since
many dependencies are installed long before a particular egg is
attempted.

We need to know the dependencies, install them in dependency order,
and expect the next package to find them. "configure" does this for
hundreds of packages. cmake, scons, and others also tackle this
problem. Python's old setup.py seems to be able to do it.

However, as I understand it, setuptools can't detect previously
installed python packages if they were not installed via eggs. Thus,
my ZSI install was failing on "PyXML>=8.3 ", even though PyXML 8.4 is
installed. I can't afford to drag copies of all the dependent source
tarballs into an egg's currdir just so it can find them. (We have 6 GB
of tarballs -- who knows how much untarred source that would be.)

I just found hints that you should not attempt to install ZSI form
tarball, but should rather install from an egg. So I was able to
install ZSI for py2.4.

Unfortunately, that means I would have to carry
python-version-dependent renditions of every egg. We have people
running on py23, py24, and py25, thus tripling the number of
tarballs/eggs to manage. This is the very reason we went to a
*source* based repository.

--
Harry George
PLM Engineering Architecture

[Robert Kern]

Re: Setuptools, build and install dependencies

Harry George wrote:

We need to know the dependencies, install them in dependency order,
and expect the next package to find them. "configure" does this for
hundreds of packages. cmake, scons, and others also tackle this
problem. Python's old setup.py seems to be able to do it.

No, generic setup.py scripts don't do anything of that kind.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Harry George]

Re: Setuptools, build and install dependencies

Robert Kern <robert.kern@gm ail.comwrites:

Harry George wrote:
>

>We need to know the dependencies, install them in dependency order,
>and expect the next package to find them. "configure" does this for
>hundreds of packages. cmake, scons, and others also tackle this
>problem. Python's old setup.py seems to be able to do it.

>
No, generic setup.py scripts don't do anything of that kind.
>

Ok, setup.py itself may not do the work, but from the end users'
perspective it works that way. Setup.py runs a configure and a make,
which in turn find the right already-installed libraries. The point
is, setup.py plays well in such an environment.

--
Robert Kern
>
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
>

--
Harry George
PLM Engineering Architecture

[Christopher Arndt]

Re: eggs considered harmful

On 21 Jun., 14:10, Harry George <harry.g.geo... @boeing.comwrot e:

I've asked before, and I'll ask again: If you are doing a Python
project, please make a self-sufficient tarball available as well.

Alomost all projects I know of that provide eggs, also have a CVS or
SVN repository. Just download a tagged release and then use "python
setup.py <whatever>" or "easy_insta ll ." in the checkout. easy_install
can even do the checkout for you.

You can have dependencies, as long as they are documented and can be
obtained by separate manual download.

Eggs document dependencies better (i.e with version numbers) than most
other projects do, through the "install_requir es" argument to the
"setup()" call in "setup.py". In an egg, this list is found in *-egg-
info/requires.txt.

Ok, setup.py itself may not do the work, but from the end users'
perspective it works that way. Setup.py runs a configure and a make,
which in turn find the right already-installed libraries. The point
is, setup.py plays well in such an environment.

Configure etc. may be able to detect an installed version number of a
package/module because they include scripts to check for those. IMHO
it's silly to place the burden for checking for version numbers on the
developer who wants to distribute an app. The package/module should
provide a standard way to query the version number itself. This is
exactly one of things that setuptools is about.

Chris

[Benji York]

Re: eggs considered harmful

On Jun 21, 8:10 am, Harry George <harry.g.geo... @boeing.comwrot e:
[snip description of unacceptable behaviors]

These are unacceptable behaviors. I am therefore dropping ZODB3

If you have bugs to report against ZODB, I sugest posting to zodb-dev
(http://mail.zope.org/mailman/listinfo/zodb-dev).
--
Benji York

[John J. Lee]

Re: eggs considered harmful

Harry George <harry.g.george @boeing.comwrit es:

jjl@pobox.com (John J. Lee) writes:

[...]

>2. You can run your own private egg repository. IIRC, it's as simple
>as a directory of eggs and a plain old web server with directory
>listings turned on. You then run easy_install -f URL package_name
>instead of easy_install package_name . The distutils-sig archives
>will have more on this.

>
Again, not speaking for anyone else: With 500 OSS packages, all of
which play by the same tarball rules, we don't have resources to
handle eggs differently.

You said earlier:

The rules are that a) a very few authorized downloaders obtain
tarballs and put them in a depot and b) other users get tarballs from
the depot and build from source.

Not sure how this differs significantly "from running a repository",
in the sense I use it above.


John

[Fuzzyman]

Re: eggs considered harmful

On Jun 21, 1:10 pm, Harry George <harry.g.geo... @boeing.comwrot e:

...at least around here.
>
I run a corporate Open Source Software Toolkit, which makes hundreds
of libraries and apps available to thousands of technical employees.
The rules are that a) a very few authorized downloaders obtain
tarballs and put them in a depot and b) other users get tarballs from
the depot and build from source.
>
Historically, python packages played well in this context. Install
was a simple download, untar, setup.py build/install.
>
Eggs and with other setuptools-inspired install processes break this
paradigm. The tarballs are incomplete in the first place. The builds
sometimes wander off to the internet looking for more downloads. The
installs sometimes wander off to the internet looking for
compatibility conditions. (Or rather they try to do so and fail
because I don't let themn through the firewall.)

I understand your situation and I have some misgivings myself. It
reminds me of the time when I worked in a 'corporate environment' and
I was trying to install a Perl application to get round the internet
blocking.

The application (localproxy - very good) was *intended* to be
installed via CPAN for tracking requirements - which didn't work
behind our proxy firewall. Although the project author (a very
technical guy) knew the direct dependencies, some of these had
dependencies. He *didn't know* the full dependency set for his
project.

Eventually, through trial and error (and a lot of help from the
author) I was able to get it working. But it was painful.

My guess is that a lot of the world's computers are behind firewalls
or proxies that preclude automatic dependency resolution.

*However*, there is a very good reason why setuptools and eggs are
gaining in popularity (and will continue to do so). For the majority
of users eggs are just *so damned convenient*. Being able to do
``easy_install some_project`` and have it just work is fantastic.

There are probably ways round this. For most non-esoteric eggs it
should be possible to create an ordinary installation tarball from an
egg. If you do easy_install of a project into a bare Python
installation (a VM instance for example) then you should be able to
see which dependencies are fetched.

If this is too much then I fear that you may be SOL...

Fuzzyman

http://www.voidspace.org.uk/ironpython/index.shtml

[John J. Lee]

Re: eggs considered harmful

Harry George <harry.g.george @boeing.comwrit es:

Robert Kern <robert.kern@gm ail.comwrites:

[...]

This is a possibility. The tarballs can be seen in a directory
listing. They are in different subdirs (for different "bundles" of
functionality), so I'll need -f to look several places.

One possibility here is to have a script maintain symlinks (or have it
otherwise appropriately configure a web server).

[...]

I agree pure eggs are rare. The fact that they increased this past
quarter was what concerned me. ZODB even looks like a normal tarball,
builds ok, but uses a easy-install-style lookup during install.

All setuptools-based packages work this way: they have a setup.py that
(roughly) imports the setup function from setuptools rather than
distutils.


John

[Paul Boddie]

Re: eggs considered harmful

Fuzzyman wrote:

>
I understand your situation and I have some misgivings myself. It
reminds me of the time when I worked in a 'corporate environment' and
I was trying to install a Perl application to get round the internet
blocking.
>
The application (localproxy - very good) was *intended* to be
installed via CPAN for tracking requirements - which didn't work
behind our proxy firewall.

Sounds like an "interestin g" bootstrapping issue to me.

[...]

My guess is that a lot of the world's computers are behind firewalls
or proxies that preclude automatic dependency resolution.

I'd argue that mechanisms already exist for automatic upgrades even in
restricted environments, and we're not always talking about "big
bucks" corporate solutions, either. Indeed, the more established GNU/
Linux distributions seem to have had the required flexibility of
dependency resolution *and* not requiring an "always on" connection to
the Internet for quite some time - for obvious reasons if you consider
how long they've been going.

*However*, there is a very good reason why setuptools and eggs are
gaining in popularity (and will continue to do so). For the majority
of users eggs are just *so damned convenient*. Being able to do
``easy_install some_project`` and have it just work is fantastic.

Sure. But being able to install any software (not just eggs via the
Package Index, or Perl software via CPAN, or...) with dependency
resolution isn't alien to a lot of people. Again, it's time to look at
established practice rather than pretend it doesn't exist:

http://mail.python.org/pipermail/python-dev/2006-November/070101.html

Paul