Idea: PYTHONPATH_VER

[Pete Shinners]

I've been working on systems that have multiple versions of python.
People are also trying to manage the compilation of external extension
modules. The problem is there is no easy way to control the separate
modules for each python version.

I keep thinking having a separate PYTHONPATH environment variable for
each version of Python would really make life easier. I have heard Perl
started doing this sometime ago and it is in use on these same machines.

At startup Python 2.4 would prepend PYTHONPATH24 to the regular path.
Easy to do, likely just slip into the "site.py". Would there be any
problems trying to get something like this added for 2.4?

[Peter Hansen]

Re: Idea: PYTHONPATH_VER

Pete Shinners wrote:
[color=blue]
> I've been working on systems that have multiple versions of python.
> People are also trying to manage the compilation of external extension
> modules. The problem is there is no easy way to control the separate
> modules for each python version.
>
> I keep thinking having a separate PYTHONPATH environment variable for
> each version of Python would really make life easier. I have heard Perl
> started doing this sometime ago and it is in use on these same machines.[/color]

I'm not entirely clear on the need for all this version stuff
people keep talking about, but wouldn't the effect of the above
be about the same as using a .pth file in the appropriate place
for each different Python version on your machine? Then you
can pick up different versions of different things at will
depending on which version of Python is run.

Having missed the discussion preceding, I'm probably off the mark,
but in my experience just about everything that can be solved with
PYTHONPATH can be handled with a .pth file as well, and I haven't
had any need to use PYTHONPATH for quite some time as a result.

-Peter

[Fernando Perez]

Re: Idea: PYTHONPATH_VER

Peter Hansen wrote:
[color=blue]
> Having missed the discussion preceding, I'm probably off the mark,
> but in my experience just about everything that can be solved with
> PYTHONPATH can be handled with a .pth file as well, and I haven't
> had any need to use PYTHONPATH for quite some time as a result.[/color]

How do you handle the fact that .pth files are only read from certain places,
and not others? This problem has me currently rather stuck with gross hacks:
packages like Numeric, which rely on a .pth file instead of being a 'true'
python package (with __init__.py), are very problematic. I am trying to share
Numeric from an NFS mounted directory for multiple clients, located
at /usr/local/lib/python. It was installed there via 'setup.py install
--home=/usr/local'. The problem is that python does NOT scan this directory
for .pth files, even if it is listed in PYTHONPATH. .pth files only have an
effect for directories in sys.prefix, I think.

I have actually come to HATE with a vengeance packages which rely on .pth files,
because of this behavior of python of not including them for anything in
PYTHONPATH. So I would really appreciate pointers from someone who has
successfully solved this, since it's quite likely that I'm just misusing the
system.

Thanks in advance,

Fernando

[Peter Hansen]

Re: Idea: PYTHONPATH_VER

Fernando Perez wrote:
[color=blue]
> Peter Hansen wrote:[color=green]
>>Having missed the discussion preceding, I'm probably off the mark,
>>but in my experience just about everything that can be solved with
>>PYTHONPATH can be handled with a .pth file as well, and I haven't
>>had any need to use PYTHONPATH for quite some time as a result.[/color]
>
> How do you handle the fact that .pth files are only read from certain places,
> and not others? This problem has me currently rather stuck with gross hacks:[/color]

I confess to never having had any particular troubles with it, other
than in trying to use it for adding special other directories during
unit testing, where I had to do "import site; site.addsitedir ('.')"
or something close to that at the top of my xxx_test.py file.
[color=blue]
> packages like Numeric, which rely on a .pth file instead of being a 'true'
> python package (with __init__.py), are very problematic.[/color]

Sorry, haven't used it.
[color=blue]
> --home=/usr/local'. The problem is that python does NOT scan this directory
> for .pth files, even if it is listed in PYTHONPATH. .pth files only have an
> effect for directories in sys.prefix, I think.[/color]

Actually, I believe it's recursive, in that any directories added during
..pth processing are themselves scanned for .pth files and the new ones
added, and so on.
[color=blue]
> I have actually come to HATE with a vengeance packages which rely on .pth files,
> because of this behavior of python of not including them for anything in
> PYTHONPATH. So I would really appreciate pointers from someone who has
> successfully solved this, since it's quite likely that I'm just misusing the
> system.[/color]

I'm not sure I've seen a package which *relied* on it, except for
pywin32, and it seems to work very nicely and the .pth file (which
I just had to verify is still there) is pretty much invisible.

I might have some better idea how to deal with it if I'd ever had
trouble with it. One thing to note: I do most development on Windows.
Perhaps, for some reason, the problems are lesser there. (Which would
be a little surprising, but given Linux' issues with installing
software, perhaps not entirely unlikely.)

-Peter

[Fernando Perez]

Re: Idea: PYTHONPATH_VER

Peter Hansen wrote:
[color=blue]
> Fernando Perez wrote:
>[color=green]
>> Peter Hansen wrote:[color=darkred]
>>>Having missed the discussion preceding, I'm probably off the mark,
>>>but in my experience just about everything that can be solved with
>>>PYTHONPATH can be handled with a .pth file as well, and I haven't
>>>had any need to use PYTHONPATH for quite some time as a result.[/color]
>>
>> How do you handle the fact that .pth files are only read from certain places,
>> and not others? This problem has me currently rather stuck with gross hacks:[/color][/color]

[snip][color=blue]
> I might have some better idea how to deal with it if I'd ever had
> trouble with it. One thing to note: I do most development on Windows.
> Perhaps, for some reason, the problems are lesser there. (Which would
> be a little surprising, but given Linux' issues with installing
> software, perhaps not entirely unlikely.)[/color]

Thanks, Peter. I might start a separate thread asking about this, with a
clearer description of my problem. I don't think it's a linux issue, but
rather one of dealing with when python loads (or not) .pth files. As I said,
it may well just be my misunderstandin g of how the system really works.

Regards,

Fernando

[Peter Hansen]

Re: Idea: PYTHONPATH_VER

Fernando Perez wrote:
[color=blue]
> Thanks, Peter. I might start a separate thread asking about this, with a
> clearer description of my problem. I don't think it's a linux issue, but
> rather one of dealing with when python loads (or not) .pth files. As I said,
> it may well just be my misunderstandin g of how the system really works.[/color]

Check the source in site.py first, then, as it is fairly straight-
forward about what it does.

-Peter

[Fernando Perez]

Re: Idea: PYTHONPATH_VER

Peter Hansen wrote:
[color=blue]
> Check the source in site.py first, then, as it is fairly straight-
> forward about what it does.[/color]

Thanks for the pointer, I didn't know what part of python was responsible for
these manipulations. Knowing where to look, it will be much easier now to deal
with this issue.

Regards,

Fernando

[David M. Cooke]

Re: Idea: PYTHONPATH_VER

At some point, Fernando Perez <fperez528@yaho o.com> wrote:
[color=blue]
> Peter Hansen wrote:
>[color=green]
>> Having missed the discussion preceding, I'm probably off the mark,
>> but in my experience just about everything that can be solved with
>> PYTHONPATH can be handled with a .pth file as well, and I haven't
>> had any need to use PYTHONPATH for quite some time as a result.[/color]
>
> How do you handle the fact that .pth files are only read from certain places,
> and not others? This problem has me currently rather stuck with gross hacks:
> packages like Numeric, which rely on a .pth file instead of being a 'true'
> python package (with __init__.py), are very problematic. I am trying to share
> Numeric from an NFS mounted directory for multiple clients, located
> at /usr/local/lib/python. It was installed there via 'setup.py install
> --home=/usr/local'. The problem is that python does NOT scan this directory
> for .pth files, even if it is listed in PYTHONPATH. .pth files only have an
> effect for directories in sys.prefix, I think.[/color]

Debian uses the following patch to site.py to look in /usr/local also:

--- /usr/lib/python2.3/site.py 2004-05-13 15:55:47.000000 000 -0400
+++ Lib/site.py 2004-03-30 20:46:01.000000 000 -0500
@@ -161,10 +156,10 @@
if reset:
_dirs_in_sys_pa th = None

-prefixes = [os.path.join(sy s.prefix, "local"), sys.prefix]
+prefixes = [sys.prefix]
sitedir = None # make sure sitedir is initialized because of later 'del'
if sys.exec_prefix != sys.prefix:

You can also make a sitecustomize.p y that does the adding of the
various directories (it's imported at the end of site.py).

## sitecustomize.p y
import site
site.addsitedir ('/usr/local')
[color=blue]
> I have actually come to HATE with a vengeance packages which rely on .pth files,
> because of this behavior of python of not including them for anything in
> PYTHONPATH. So I would really appreciate pointers from someone who has
> successfully solved this, since it's quite likely that I'm just misusing the
> system.[/color]

Bleh, yes. .pth files are mostly a hack for packages that *should be*
actual Python packages.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca