Odd files; just left behind?

[Robin Becker]

I see a folder .python-eggs in my home directory on one of our servers with
various .so files

~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

are these just left behind from some install process?
--
Robin Becker

[John J. Lee]

Re: Odd files; just left behind?

Robin Becker <robin@reportla b.comwrites:

I see a folder .python-eggs in my home directory on one of our servers
with various .so files
>
~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>
are these just left behind from some install process?

Ah, it's about the resource extraction feature of setuptools. It's
possible for zipped eggs to contain resources that have to be
extracted to the filesystem. It puts them in the directory you saw
(or somewhere else -- see the docstring in my other reply).

http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction

http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction

John

[Robin Becker]

Re: Odd files; just left behind?

John J. Lee wrote:

Robin Becker <robin@reportla b.comwrites:
>

>I see a folder .python-eggs in my home directory on one of our servers
>with various .so files
>>
>~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>>
>are these just left behind from some install process?

>
Ah, it's about the resource extraction feature of setuptools. It's
possible for zipped eggs to contain resources that have to be
extracted to the filesystem. It puts them in the directory you saw
(or somewhere else -- see the docstring in my other reply).
>

http://peak.telecommunity.com/DevCenter/PkgResources#resource-extraction

http://peak.telecommunity.com/DevCen...rce-extraction

......
Thanks John, but why not clean up?

--
Robin Becker

[Bruno Desthuilliers]

Re: Odd files; just left behind?

Robin Becker a écrit :

John J. Lee wrote:
>

>Robin Becker <robin@reportla b.comwrites:
>>

>>I see a folder .python-eggs in my home directory on one of our servers
>>with various .so files
>>>
>>~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>>>
>>>
>>are these just left behind from some install process?

>>
>>
>Ah, it's about the resource extraction feature of setuptools. It's
>possible for zipped eggs to contain resources that have to be
>extracted to the filesystem. It puts them in the directory you saw
>(or somewhere else -- see the docstring in my other reply).
>>
>http://peak.telecommunity.com/DevCen...rce-extraction
>http://peak.telecommunity.com/DevCen...rce-extraction
>>

>
.....
Thanks John, but why not clean up?
>

Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
extracted from the egg to be usable. Where they are extracted is IIRC
configurable...

[Robin Becker]

Re: Odd files; just left behind?

Bruno Desthuilliers wrote:

Robin Becker a écrit :

>John J. Lee wrote:
>>

>>Robin Becker <robin@reportla b.comwrites:
>>>
>>>I see a folder .python-eggs in my home directory on one of our servers
>>>with various .so files
>>>>
>>>~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>>>>
>>>>
>>>are these just left behind from some install process?
>>>
>>Ah, it's about the resource extraction feature of setuptools. It's
>>possible for zipped eggs to contain resources that have to be
>>extracted to the filesystem. It puts them in the directory you saw
>>(or somewhere else -- see the docstring in my other reply).
>>>
>>http://peak.telecommunity.com/DevCen...rce-extraction
>>http://peak.telecommunity.com/DevCen...rce-extraction
>>>

>.....
>Thanks John, but why not clean up?
>>

Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
extracted from the egg to be usable. Where they are extracted is IIRC
configurable...

You're right! These files aren't temporary? It seems strange that the python
files live with their associated lib/site-packages and the .so files are
somewhere entirely different.

Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright" , "credits" or "license" for more information.

>>import MySQLdb
>>print MySQLdb.__file_ _

/myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc

>>import _mysql
>>print _mysql.__file__

/myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

>>>

it would seem simpler to have the .so files inside the site-packages and there's
the question of why this folder has to be obfuscated (name starts with .). Even
if these files are "resources" why should they be assumed to belong to the user?
I could make the PYTHON folder world usable, but that won't help if I make all
other things in myhome private. Surely code resources belong in the same area as
the code that uses them.

Just another nonsense from the eggworld
--
Robin Becker

[Bruno Desthuilliers]

Re: Odd files; just left behind?

Robin Becker a écrit :

Bruno Desthuilliers wrote:

>Robin Becker a écrit :

>>John J. Lee wrote:
>>>
>>>Robin Becker <robin@reportla b.comwrites:
>>>>
>>>>I see a folder .python-eggs in my home directory on one of our servers
>>>>with various .so files
>>>>>
>>>>~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>>>>>
>>>>>
>>>>are these just left behind from some install process?
>>>>
>>>Ah, it's about the resource extraction feature of setuptools. It's
>>>possible for zipped eggs to contain resources that have to be
>>>extracted to the filesystem. It puts them in the directory you saw
>>>(or somewhere else -- see the docstring in my other reply).
>>>>
>>>http://peak.telecommunity.com/DevCen...rce-extraction
>>>>
>>>http://peak.telecommunity.com/DevCen...rce-extraction
>>>>
>>.....
>>Thanks John, but why not clean up?
>>>

>Does it needs to be "cleaned up" ? .so files (shared libs) *need* to
>be extracted from the egg to be usable. Where they are extracted is
>IIRC configurable...

You're right! These files aren't temporary?

It seems strange that the
python files live with their associated lib/site-packages and the .so
files are somewhere entirely different.

IIRC, where .so files are extracted is configurable using
PYTHON_EGG_CACH E (cf the first link above).

Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright" , "credits" or "license" for more information.

>>import MySQLdb
>>print MySQLdb.__file_ _

/myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc
>

>>import _mysql
>>print _mysql.__file__

/myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

>>>

>
it would seem simpler to have the .so files inside the site-packages and
there's the question of why this folder has to be obfuscated (name
starts with .). Even if these files are "resources" why should they be
assumed to belong to the user?

Notice that you did install your lib/python in /myhome, not in /usr[/local]

(snip)

Just another nonsense from the eggworld

Or just another example of not reading the FineManual(tm) ?

[Robin Becker]

Re: Odd files; just left behind?

Bruno Desthuilliers wrote:
.........

> >>>

>>
>it would seem simpler to have the .so files inside the site-packages and
>there's the question of why this folder has to be obfuscated (name
>starts with .). Even if these files are "resources" why should they be
>assumed to belong to the user?

>
Notice that you did install your lib/python in /myhome, not in /usr[/local]

yes, so should I then assume that the root installs will put all the .so files
in /usr/local/lib/.python-eggs?

In fact when I look at my root installed pythons they seem quite happy to put
the _mysql.so underneath site-packages. I suspect that if I just installed from
source I'd get that behaviour, but the egg stuff is different.

>
(snip)

>Just another nonsense from the eggworld

>
Or just another example of not reading the FineManual(tm) ?

well I didn't write the installer or the associated egg setup. The egg things
are supposed to do the right thing, but perhaps they're too difficult for the
egg layers.
--
Robin Becker

[Bruno Desthuilliers]

Re: Odd files; just left behind?

Robin Becker a écrit :

Bruno Desthuilliers wrote:
........

>> >>>
>>>
>>it would seem simpler to have the .so files inside the site-packages
>>and there's the question of why this folder has to be obfuscated
>>(name starts with .). Even if these files are "resources" why should
>>they be assumed to belong to the user?

>>
>Notice that you did install your lib/python in /myhome, not in
>/usr[/local]

>
yes, so should I then assume that the root installs will put all the .so
files in /usr/local/lib/.python-eggs?
>
In fact when I look at my root installed pythons they seem quite happy
to put the _mysql.so underneath site-packages. I suspect that if I just
installed from source I'd get that behaviour, but the egg stuff is
different.
>

>>
>(snip)

>>Just another nonsense from the eggworld

>>
>Or just another example of not reading the FineManual(tm) ?

>
well I didn't write the installer or the associated egg setup. The egg
things are supposed to do the right thing, but perhaps they're too
difficult for the egg layers.

Can't tell. So far, I've had no problem with eggs, so...

[Robert Kern]

Re: Odd files; just left behind?

Robin Becker wrote:

Bruno Desthuilliers wrote:

>Robin Becker a écrit :

>>John J. Lee wrote:
>>>
>>>Robin Becker <robin@reportla b.comwrites:
>>>>
>>>>I see a folder .python-eggs in my home directory on one of our servers
>>>>with various .so files
>>>>>
>>>>~/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so
>>>>>
>>>>>
>>>>are these just left behind from some install process?
>>>Ah, it's about the resource extraction feature of setuptools. It's
>>>possible for zipped eggs to contain resources that have to be
>>>extracted to the filesystem. It puts them in the directory you saw
>>>(or somewhere else -- see the docstring in my other reply).
>>>>
>>>http://peak.telecommunity.com/DevCen...rce-extraction
>>>http://peak.telecommunity.com/DevCen...rce-extraction
>>>>
>>.....
>>Thanks John, but why not clean up?
>>>

>Does it needs to be "cleaned up" ? .so files (shared libs) *need* to be
>extracted from the egg to be usable. Where they are extracted is IIRC
>configurable.. .

You're right! These files aren't temporary?

They're a cache.

It seems strange that the python
files live with their associated lib/site-packages and the .so files are
somewhere entirely different.
>
Python 2.3.6 (#1, Jun 7 2007, 10:44:52)
[GCC 3.4.4 [FreeBSD] 20050518] on freebsd6
Type "help", "copyright" , "credits" or "license" for more information.

>>import MySQLdb
>>print MySQLdb.__file_ _

/myhome/PYTHON/lib/python2.3/site-packages/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg/MySQLdb/__init__.pyc

>>import _mysql
>>print _mysql.__file__

/myhome/.python-eggs/MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg-tmp/_mysql.so

>>>

>
it would seem simpler to have the .so files inside the site-packages and there's
the question of why this folder has to be obfuscated (name starts with .). Even
if these files are "resources" why should they be assumed to belong to the user?

Because they are unpacked at runtime by the user that imported the module.
Usually, they won't have write access to site-packages.

I could make the PYTHON folder world usable, but that won't help if I make all
other things in myhome private. Surely code resources belong in the same area as
the code that uses them.

Well, you can't import a .so from a zip file. Thus they need to be placed
somewhere else. Don't worry about making ~/.python-eggs world readable. When
other users import the package, they will have the .so's unpacked, too.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[John J. Lee]

Re: Odd files; just left behind?

Robin Becker <robin@reportla b.comwrites:

Bruno Desthuilliers wrote:
........

>> >>>
>>>
>>it would seem simpler to have the .so files inside the
>>site-packages and there's the question of why this folder has to be
>>obfuscated (name starts with .). Even if these files are
>>"resources" why should they be assumed to belong to the user?

>>
>Notice that you did install your lib/python in /myhome, not in /usr[/local]

>
yes, so should I then assume that the root installs will put all the
.so files in /usr/local/lib/.python-eggs?

No no -- the extraction of these .so files happens at *runtime*, so it
has to write the files somewhere it has permission to write to. It's
like RL's zipapp resource extraction feature in fact, except the
resource extraction happens at runtime. See those web pages I pointed
you at before.

In fact when I look at my root installed pythons they seem quite happy
to put the _mysql.so underneath site-packages. I suspect that if I
just installed from source I'd get that behaviour, but the egg stuff
is different.

Not sure without all the details, but I suspect those are non-egg
installs.


John

[Robert Kern]

Re: Odd files; just left behind?

Robin Becker wrote:

Robert Kern wrote:
.......

>They're a cache.

>
they're actually the files that get used by the installed extension in
this case (MySQLdb).

Yes. They are extracted from the zipfile at runtime and left there so they don't
have to be extracted again. That's why I called it a cache. I didn't mean
anything else by the term.

..........

>>it would seem simpler to have the .so files inside the site-packages and there's
>>the question of why this folder has to be obfuscated (name starts with .). Even
>>if these files are "resources" why should they be assumed to belong to the user?

>Because they are unpacked at runtime by the user that imported the module.
>Usually, they won't have write access to site-packages.

>
in this particular case the python being used was installed with a user
prefix eg /myhome/PYTHON so everything belongs to the user; extensions
installed by the same user can surely be installed in those folders.
Upon checking I find that the same installs which produced the
~/.python-eggs files also managed to put the eggs like
MySQL_python-1.2.2-py2.3-freebsd-6.1-SECURITY-i386.egg inside the
various site-packages directories.
>
I see no reason why the binary resources shouldn't be unpacked to
site-packages at install time.

Then do so: Use the -Z argument to easy_install such that it always unpacks the
zip file. If you don't, then the egg will be left zipped, and the resources
extracted at runtime. Zipped eggs have benefits over the unzipped eggs in that
they decrease the amount of import overhead of having many items in your sys.path.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco