module import search path strangeness

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • tow
    #1

    module import search path strangeness

    I have a python script (part of a django application, if it makes any
    difference) which is exhibiting the following behaviour:

    import my_module # succeeds
    imp.find_module ("my_module" ) # fails, raising ImportError

    which is completely baffling me. According to sys.path, both should
    fail; the directory containing my_module is not in sys.path (though
    the my_module directory itself is). More puzzlingly, printing out
    my_module.__fil e__ gives:

    /home/tow/test/my_module/../my_module/__init__.pyc

    I don't really understand what the ".." is doing in there.

    Can someone explain what I'm missing here, it's got me stumped.

    Toby
    Tags: None
    • Gabriel Genellina
      #2
      Re: module import search path strangeness

      En Mon, 11 Aug 2008 19:19:19 -0300, tow <toby.o.h.white @googlemail.com >
      escribi�:
      I have a python script (part of a django application, if it makes any
      difference) which is exhibiting the following behaviour:
      >
      import my_module # succeeds
      imp.find_module ("my_module" ) # fails, raising ImportError
      >
      which is completely baffling me. According to sys.path, both should
      fail; the directory containing my_module is not in sys.path (though
      the my_module directory itself is).
      my_module is not a module but a package, right? Else I don't understand
      the above statement.
      More puzzlingly, printing out
      my_module.__fil e__ gives:
      >
      /home/tow/test/my_module/../my_module/__init__.pyc
      >
      I don't really understand what the ".." is doing in there.
      >
      Can someone explain what I'm missing here, it's got me stumped.
      Perhaps you have ".." in sys.path? And the current directory happens to be
      /home/tow/test/my_module?

      --
      Gabriel Genellina

        Comment

        • tow
          #3
          Re: module import search path strangeness

          On Aug 12, 4:59 am, "Gabriel Genellina" <gagsl-...@yahoo.com.a r>
          wrote:
          En Mon, 11 Aug 2008 19:19:19 -0300, tow <toby.o.h.wh... @googlemail.com 
          escribi :
          >
          I have a python script (part of a django application, if it makes any
          difference) which is exhibiting the following behaviour:
          >
          import my_module # succeeds
          imp.find_module ("my_module" ) # fails, raising ImportError
          >
          which is completely baffling me. According to sys.path, both should
          fail; the directory containing my_module is not in sys.path (though
          the my_module directory itself is).
          >
          my_module is not a module but a package, right? Else I don't understand  
          the above statement.
          Sorry, sloppy terminology on my part, yes, my_module is a package.
          More puzzlingly, printing out
          my_module.__fil e__ gives:
          >
          /home/tow/test/my_module/../my_module/__init__.pyc
          >
          I don't really understand what the ".." is doing in there.
          >
          Can someone explain what I'm missing here, it's got me stumped.
          >
          Perhaps you have ".." in sys.path? And the current directory happens to be  
          /home/tow/test/my_module?
          The current directory is actually a subdirectory of /home/tow/test/
          my_module,
          but ".." is not in sys.path (nor is anything containing "..") In any
          case, if
          it were a matter of sys.path, surely the imp.find_module call above
          should succeed?

          Basically, I had thought that import and imp.find_module used exactly
          the same
          search path, but the above example shows that at least in this
          circumstance they
          don't; import is picking up additional search paths from somewhere -
          what am I missing?

          Toby

            Comment

            • Peter Otten
              #4
              Re: module import search path strangeness

              tow wrote:
              >I have a python script (part of a django application, if it makes any
              difference) which is exhibiting the following behaviour:
              >>
              import my_module # succeeds
              imp.find_module ("my_module" ) # fails, raising ImportError
              >>
              which is completely baffling me. According to sys.path, both should
              fail; the directory containing my_module is not in sys.path (though
              the my_module directory itself is).
              The current directory is actually a subdirectory of /home/tow/test/
              my_module,
              but ".." is not in sys.path (nor is anything containing "..") In any
              Basically, I had thought that import and imp.find_module used exactly
              the same
              search path, but the above example shows that at least in this
              circumstance they
              don't; import is picking up additional search paths from somewhere -
              what am I missing?
              Grepping through the django source finds

              ../trunk/django/core/management/__init__.py:
              sys.path.append (os.path.join(p roject_director y, os.pardir))

              sys.path could be changed as a side effect of an import, so I ask you to
              verify (again) that import and imp.find_module () see the same sys.path.

              assert all(".." not in p for p in sys.path)
              import my_module

              assert all(".." not in p for p in sys.path)
              imp.find_module ("my_module" )


              Peter

                Comment

                • tow
                  #5
                  Re: module import search path strangeness

                  On Aug 12, 9:56 am, Peter Otten <__pete...@web. dewrote:
                  tow wrote:
                  Basically, I had thought that import and imp.find_module used exactly
                  the same
                  search path, but the above example shows that at least in this
                  circumstance they
                  don't; import is picking up additional search paths from somewhere -
                  what am I missing?
                  >
                  Grepping through the django source finds
                  >
                  ./trunk/django/core/management/__init__.py:  
                  sys.path.append (os.path.join(p roject_director y, os.pardir))
                  Hmm. It turns out that that is indeed the issue, but in a way that
                  wasn't immediately obvious to me. Looking at it in more context:

                  sys.path.append (os.path.join(p roject_director y, os.pardir))
                  project_module = __import__(proj ect_name, {}, {}, [''])
                  sys.path.pop()

                  sys.path is extended, the project module is imported, then the
                  additional path is dropped from sys.path.
                  And if I comment out those sys.path manipulations, I get the result I
                  expect later on.

                  What I think is happening is that in that stanza, project_module
                  (which is my_module in my case) is
                  imported from the altered sys.path. sys.path is then put back, but
                  python now knows about my_module.

                  As a result when my_module is imported again elsewhere, python already
                  knows about it, so no searching
                  of sys.path is done (which is good, because it wouldn't be found on
                  the current sys.path), and the
                  import apparently succeeds, though in fact it's effectively
                  (actually?) a no-op.

                  However, imp.find_module ("my_module" ) forces a search of the sys.path,
                  and thus fails.

                  Or at least, that is what I surmise, given that I see

                  import my_module # succeeds
                  imp.find_module ("my_module" ) # fails, raising ImportError

                  So, to answer my original question, the difference in search behaviour
                  between "import" and "imp.find_modul e" is that the former might not
                  look at sys.path at all if the module has already been loaded, while
                  the latter will only search on the current sys.path.

                  Am I right?

                  Toby

                    Comment

                    • Peter Otten
                      #6
                      Re: module import search path strangeness

                      tow wrote:
                      On Aug 12, 9:56 am, Peter Otten <__pete...@web. dewrote:
                      >tow wrote:
                      >
                      Basically, I had thought that import and imp.find_module used exactly
                      the same
                      search path, but the above example shows that at least in this
                      circumstance they
                      don't; import is picking up additional search paths from somewhere -
                      what am I missing?
                      >>
                      >Grepping through the django source finds
                      >>
                      >./trunk/django/core/management/__init__.py:
                      >sys.path.appen d(os.path.join( project_directo ry, os.pardir))
                      >
                      Hmm. It turns out that that is indeed the issue, but in a way that
                      wasn't immediately obvious to me. Looking at it in more context:
                      >
                      sys.path.append (os.path.join(p roject_director y, os.pardir))
                      project_module = __import__(proj ect_name, {}, {}, [''])
                      sys.path.pop()
                      Ouch.
                      So, to answer my original question, the difference in search behaviour
                      between "import" and "imp.find_modul e" is that the former might not
                      look at sys.path at all if the module has already been loaded, while
                      the latter will only search on the current sys.path.
                      >
                      Am I right?
                      Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
                      before it falls back to the more costly alternatives.

                      Peter

                        Comment

                        • Sion Arrowsmith
                          #7
                          Re: module import search path strangeness

                          Peter Otten <__peter__@web. dewrote:
                          >tow wrote:
                          > sys.path.append (os.path.join(p roject_director y, os.pardir))
                          > project_module = __import__(proj ect_name, {}, {}, [''])
                          > sys.path.pop()
                          >Ouch.
                          I presume that "Ouch" is in consideration of what might happen if
                          the subject of the __import__ also calls sys.path.append ...?

                          --
                          \S -- siona@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
                          "Frankly I have no feelings towards penguins one way or the other"
                          -- Arthur C. Clarke
                          her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

                            Comment

                            • tow
                              #8
                              Re: module import search path strangeness

                              On Aug 12, 4:59 pm, Peter Otten <__pete...@web. dewrote:
                              tow wrote:
                              On Aug 12, 9:56 am, Peter Otten <__pete...@web. dewrote:
                              tow wrote:
                              >
                              Basically, I had thought that import and imp.find_module used exactly
                              the same
                              search path, but the above example shows that at least in this
                              circumstance they
                              don't; import is picking up additional search paths from somewhere -
                              what am I missing?
                              >
                              Grepping through the django source finds
                              >
                              ./trunk/django/core/management/__init__.py:
                              sys.path.append (os.path.join(p roject_director y, os.pardir))
                              >
                              Hmm. It turns out that that is indeed the issue, but in a way that
                              wasn't immediately obvious to me. Looking at it in more context:
                              >
                                  sys.path.append (os.path.join(p roject_director y, os.pardir))
                                  project_module = __import__(proj ect_name, {}, {}, [''])
                                  sys.path.pop()
                              >
                              Ouch.
                              >
                              So, to answer my original question, the difference in search behaviour
                              between "import" and "imp.find_modul e" is that the former might not
                              look at sys.path at all if the module has already been loaded, while
                              the latter will only search on the current sys.path.
                              >
                              Am I right?
                              >
                              Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
                              before it falls back to the more costly alternatives.
                              >
                              Peter
                              Thanks, at least I understand what's going on now.

                              Toby

                                Comment

                                • Peter Otten
                                  #9
                                  Re: module import search path strangeness

                                  Sion Arrowsmith wrote:
                                  Peter Otten <__peter__@web. dewrote:
                                  >>tow wrote:
                                  >> sys.path.append (os.path.join(p roject_director y, os.pardir))
                                  >> project_module = __import__(proj ect_name, {}, {}, [''])
                                  >> sys.path.pop()
                                  >>Ouch.
                                  >
                                  I presume that "Ouch" is in consideration of what might happen if
                                  the subject of the __import__ also calls sys.path.append ...?
                                  Yes ;)

                                  Peter

                                    Comment

                                    Previous template Next
                                    Working...