threading support in python

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Paul Rubin

    #16
    Re: threading support in python

    Steve Holden <steve@holdenwe b.comwrites:
    You write as though the GIL was invented to get in the programmer's
    way, which is quite wrong. It's there to avoid deep problems with
    thread interaction. Languages that haven't bitten that bullet can bite
    you in quite nasty ways when you write threaded applications.
    And yet, Java programmers manage to write threaded applications all
    day long without getting bitten (once they're used to the issues),
    despite usually being less skilled than Python programmers ;-).
    Contrary to your apparent opinion, the GIL has nothing to do with
    reference-counting.
    I think it does, i.e. one of the GIL's motivations was to protect the
    management of reference counts in CPython, which otherwise wasn't
    thread-safe. The obvious implementation of Py_INCREF has a race
    condition, for example. The GIL documentation at



    describes this in its very first paragraph.
    Will the madness never end?
    >
    This reveals an opinion of the development team that's altogether too
    low. I believe the GIL was introduced for good reasons.
    The GIL was an acceptable tradeoff when it was first created in the
    previous century. First of all, it gave a way to add threads to the
    existing, non-threadsafe CPython implementation without having to
    rework the old code too much. Second, Python was at that time
    considered a "scripting language" and there was less concern about
    writing complex apps in it, especially multiprocessing apps. Third,
    multiprocessor computers were themselves exotic, so people who wanted
    to program them probably had exotic problems that they were willing to
    jump through hoops to solve.

    These days, even semi-entry-level consumer laptop computers have dual
    core CPU's, and quad Opteron boxes (8-way multiprocessing using X2
    processors) are quite affordable for midrange servers or engineering
    workstations, and there's endless desire to write fancy server apps
    completely in Python. There is no point paying for all that
    multiprocessor hardware if your programming language won't let you use
    it. So, Python must punt the GIL if it doesn't want to keep
    presenting undue obstacles to writing serious apps on modern hardware.

    Comment

    • Felipe Almeida Lessa

      #17
      Re: threading support in python

      4 Sep 2006 19:19:24 -0700, Sandra-24 <sandravandale@ yahoo.com>:
      If there was a mod_dotnet I wouldn't be using
      CPython anymore.
      I guess you won't be using then: http://www.mono-project.com/Mod_mono

      --
      Felipe.

      Comment

      • Sandra-24

        #18
        Re: threading support in python


        Steve Holden wrote:
        Quite right too. You haven't even sacrificed a chicken yet ...
        Hopefully we don't get to that point.
        You write as though the GIL was invented to get in the programmer's way,
        which is quite wrong. It's there to avoid deep problems with thread
        interaction. Languages that haven't bitten that bullet can bite you in
        quite nasty ways when you write threaded applications.
        I know it was put there because it is meant to be a good thing.
        However, it gets in my way. I would be perfectly happy if it were gone.
        I've never written code that assumes there's a GIL. I always write my
        code with all shared writable objects protected by locks. It's far more
        portable, and a good habit to get into. You realize that because of the
        GIL, they were discussing (and may have already implemented) Java style
        synchronized dictionaries and lists for IronPython simply because
        python programmers just assume they are thread safe thanks to the GIL.
        I always hated that about Java. If you want to give me thread safe
        collections, fine, they'll be nice for sharing between threads, but
        don't make me use synchronized collections for single-threaded code.
        You'll notice the newer Java collections are not synchronized, it would
        seem I'm not alone in that opinion.
        Contrary to your apparent opinion, the GIL has nothing to do with
        reference-counting.
        Actually it does. Without the GIL reference counting is not thread
        safe. You have to synchronize all reference count accesses, increments,
        and decrements because you have no way of knowing which objects get
        shared across threads. I think with Python's current memory management,
        the GIL is the lesser evil.

        I'm mostly writing this to provide a different point of view, many
        people seem to think (previously linked blog) that there is no downside
        to the GIL, and that's just not true. However, I don't expect that the
        GIL can be safely removed from CPython. I also think that it doesn't
        matter because projects like IronPython and PyPy are very likely the
        way of the future for Python anyway. Once you move away from C there
        are so many more things you can do.
        I think the suggestion was rather that abandoning Python because of the
        GIL might be premature optimisation. But since you appear to be sticking
        with it, that might have been unnecessary advice.
        I would never abandon Python, and I hold the development team in very
        high esteem. That doesn't mean there's a few things (like the GIL, or
        super) that I don't like. But overall they've done an excellent job on
        the 99% of things the've got right. I guess we don't say that enough.

        I might switch from CPython sometime to another implementation, but it
        won't be because of the GIL. I'm very fond of the .net framework as a
        library, and I'd also rather write performance critical code in C# than
        C (who wouldn't?) I'm also watching PyPy with interest.

        -Sandra

        Comment

        • Bryan Olson

          #19
          Re: threading support in python

          bayerj wrote:
          Then you can use POSH [1] to share data and objects.
          Do you use POSH? How well does it work with current Python?
          Any major gotchas?

          I think POSH looks like a great thing to have, but the latest
          version is an alpha from over three years ago. Also, it only
          runs on *nix systems.


          --
          --Bryan

          Comment

          • Sandra-24

            #20
            Re: threading support in python

            Felipe Almeida Lessa wrote:
            4 Sep 2006 19:19:24 -0700, Sandra-24 <sandravandale@ yahoo.com>:
            If there was a mod_dotnet I wouldn't be using
            CPython anymore.
            >
            I guess you won't be using then: http://www.mono-project.com/Mod_mono
            >
            Oh I'm aware of that, but it's not what I'm looking for. Mod_mono just
            lets you run ASP.NET on Apache. I'd much rather use Python :) Now if
            there was a way to run IronPython on Apache I'd be interested.

            -Sandra

            Comment

            • skip@pobox.com

              #21
              Re: threading support in python


              SandraHowever, I don't expect that the GIL can be safely removed from
              SandraCPython.

              It was removed at one point in the dim, dark past (circa Python 1.4) on an
              experimental basis. Aside from the huge amount of work, it resulted in
              significantly lower performance for single-threaded apps (that is, the
              common case). Maybe more effort should have been put in at that time to
              improve performance, but that didn't happen. Much more water has gone under
              the bridge at this point, so extracting the GIL from the core would be
              correspondingly more difficult.

              Skip

              Comment

              • km

                #22
                Re: threading support in python

                Hi all,
                And yet, Java programmers manage to write threaded applications all
                day long without getting bitten (once they're used to the issues),
                despite usually being less skilled than Python programmers ;-).
                These days, even semi-entry-level consumer laptop computers have dual
                core CPU's, and quad Opteron boxes (8-way multiprocessing using X2
                processors) are quite affordable for midrange servers or engineering
                workstations, and there's endless desire to write fancy server apps
                completely in Python. There is no point paying for all that
                multiprocessor hardware if your programming language won't let you use
                it. So, Python must punt the GIL if it doesn't want to keep
                presenting undue obstacles to writing serious apps on modern hardware.
                True
                GIL implementation must have got its own good causes as it it designed
                but as language evolves its very essential that one increases the
                scope such that it fits into many usage areas(eg. scientific
                applications using multiprocessors etc.).

                In the modern scientific age where
                __multiprocesso r_execution_env ironment__ is quite common, i feel there
                is a need to rethink abt the introduction of true parallelization
                capabilities in python.
                I know many of my friends who didnot choose python for obvious reasons
                of the nature of thread execution in the presence of GIL which means
                that one is wasting sophisticated hardware resources.


                ############### ############### ############
                if __name__ == ''__multiproces sor_execution_e nvironment__':
                for python_version in range(python2.4 .x, python3.x, x):

                if python_version. GIL:

                print 'unusable for computation intensive multiprocessor
                architecture'

                else:
                print cmp(python,java )
                ############### ############### ##############

                regards,
                KM

                Comment

                • Bryan Olson

                  #23
                  Re: threading support in python

                  Paul Rubin wrote:
                  "sjdevnull@yaho o.com" <sjdevnull@yaho o.comwrites:
                  >If it's read/write data or you're not on a Unix platform, you can use
                  >shared memory to shared it between many processes.
                  >>
                  >Threads are way overused in modern multiexecution programming. The
                  >decision on whether to use processes or threads should come down to
                  >whether you want to share everything, or whether you have specific
                  >pieces of data you want to share.
                  >
                  Shared memory means there's a byte vector (the shared memory region)
                  accessible to multiple processes. The processes don't use the same
                  machine addresses to reference the vector. Any data structures
                  (e.g. those containing pointers) shared between the processes have to
                  be marshalled in and out of the byte vector instead of being accessed
                  normally.
                  I think it's even worse. The standard Python library offers
                  shared memory, but not cross-process locks. Sharing read-write
                  memory looks like an automatic race condition. I guess one could
                  implement one of the primitive spin-lock based mutual exclusion
                  algorithms, but I think even that would depend on non-portable
                  assumptions about cache consistency.


                  --
                  --Bryan

                  Comment

                  • Richard Brodie

                    #24
                    Re: threading support in python


                    "km" <srikrishnamoha n@gmail.comwrot e in message
                    news:mailman.20 .1157468962.527 9.python-list@python.org ...
                    I know many of my friends who did not choose python for obvious reasons
                    of the nature of thread execution in the presence of GIL which means
                    that one is wasting sophisticated hardware resources.
                    It would probably be easier to find smarter friends than to remove the
                    GIL from Python.


                    Comment

                    • km

                      #25
                      Re: threading support in python

                      True, since smartness is a comparison, my friends who have chosen java
                      over python for considerations of a true threading support in a
                      language are smarter, which makes me a dumbo ! :-)

                      KM


                      On 9/5/06, Richard Brodie <R.Brodie@rl.ac .ukwrote:
                      >
                      "km" <srikrishnamoha n@gmail.comwrot e in message
                      news:mailman.20 .1157468962.527 9.python-list@python.org ...
                      >
                      I know many of my friends who did not choose python for obvious reasons
                      of the nature of thread execution in the presence of GIL which means
                      that one is wasting sophisticated hardware resources.
                      >
                      It would probably be easier to find smarter friends than to remove the
                      GIL from Python.
                      >
                      >
                      --

                      >

                      Comment

                      • Richard Brodie

                        #26
                        Re: threading support in python


                        "km" <srikrishnamoha n@gmail.comwrot e in message
                        news:mailman.21 .1157471042.527 9.python-list@python.org ...
                        True, since smartness is a comparison, my friends who have chosen java
                        over python for considerations of a true threading support in a
                        language are smarter, which makes me a dumbo ! :-)
                        No, but I think you making unwise assumptions about performance.
                        You have to ask yourself: is Amdahl's law really hurting me?

                        In some situations Python could no doubt benefit from fine grained
                        locking. However, it's likely that scientific programming is not typically
                        one of them, because most of the heavy lifting is done in C or C++
                        extensions which can run in parallel if they release the GIL. Or you
                        are going to use a compute farm, and fork as many worker processes
                        as you have cores.

                        You might find these slides from SciPy 2004 interesting:







                        Comment

                        • Steve Holden

                          #27
                          Re: threading support in python

                          skip@pobox.com wrote:
                          SandraHowever, I don't expect that the GIL can be safely removed from
                          SandraCPython.
                          >
                          It was removed at one point in the dim, dark past (circa Python 1.4) on an
                          experimental basis. Aside from the huge amount of work, it resulted in
                          significantly lower performance for single-threaded apps (that is, the
                          common case). Maybe more effort should have been put in at that time to
                          improve performance, but that didn't happen. Much more water has gone under
                          the bridge at this point, so extracting the GIL from the core would be
                          correspondingly more difficult.
                          >
                          Given the effort that GIL-removal would take, I'm beginning to wonder if
                          PyPy doesn't offer a better way forward than CPython, in terms of
                          execution speed improvements returned per developer-hour.

                          regards
                          Steve
                          --
                          Steve Holden +44 150 684 7255 +1 800 494 3119
                          Holden Web LLC/Ltd http://www.holdenweb.com
                          Skype: holdenweb http://holdenweb.blogspot.com
                          Recent Ramblings http://del.icio.us/steve.holden

                          Comment

                          • skip@pobox.com

                            #28
                            Re: threading support in python


                            SteveGiven the effort that GIL-removal would take, I'm beginning to
                            Stevewonder if PyPy doesn't offer a better way forward than CPython,
                            Stevein terms of execution speed improvements returned per
                            Stevedeveloper-hour.

                            How about execution speed improvements per hour of discussion about removing
                            the GIL? ;-)


                            Skip

                            Comment

                            • skip@pobox.com

                              #29
                              Re: threading support in python


                              RichardIt would probably be easier to find smarter friends than to
                              Richardremove the GIL from Python.

                              And if the friends you find are smart enough, they can remove the GIL for
                              you!

                              Skip

                              Comment

                              • sjdevnull@yahoo.com

                                #30
                                Re: threading support in python

                                Bryan Olson wrote:
                                I think it's even worse. The standard Python library offers
                                shared memory, but not cross-process locks.
                                File locks are supported by the standard library (at least on Unix,
                                I've not tried on Windows). They work cross-process and are a normal
                                method of interprocess locking even in C code.

                                Comment

                                Working...