calling functions across threads

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Steven Bethard

    #1

    calling functions across threads

    I'm playing around with some threading stuff right now, and I'm having a
    little trouble calling a function from one thread that affects another.
    Here's my setup:

    py> import os, threading, time
    py> def write(file_in, input_lines):
    .... for line in input_lines:
    .... time.sleep(0.5)
    .... file_in.write(l ine)
    .... file_in.flush()
    .... file_in.close()
    ....
    py> def read(file_out, output_list):
    .... while True:
    .... line = file_out.readli ne()
    .... if not line:
    .... break
    .... output_list.app end(line)
    ....
    py> def runthreads(lst) :
    .... file_in, file_out, file_err = os.popen3('cat' )
    .... write_thread = threading.Threa d(
    .... target=write, args=(file_in,
    .... ['%s\n' % x for x in range(10)]))
    .... read_thread = threading.Threa d(target=read,
    .... args=(file_out, lst))
    .... write_thread.st art()
    .... read_thread.sta rt()
    .... write_thread.jo in()
    .... read_thread.joi n()
    ....

    Basically, I start one thread to read and one thread to write (from a
    os.pipe). This all works fine for me:

    py> lst = []
    py> runthreads(lst)
    py> lst
    ['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

    I run into a problem though when I try to call an update method every
    time I read a line:

    py> class updatinglist(li st):
    .... def __init__(self, updater):
    .... super(updatingl ist, self).__init__( )
    .... self.updater = updater
    .... def append(self, item):
    .... super(updatingl ist, self).append(it em)
    .... self.updater(le n(self))
    ....
    py> def update(i):
    .... print i
    ....
    py> lst = updatinglist(up date)
    py> runthreads(lst)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    py> lst
    ['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']

    I get the correct output, but if you run this yourself, you'll see that
    the numbers 1 through 10 aren't printed in sync with the writes (i.e.
    every half second); they're all printed at the end. Could someone
    explain to me why this happens, and how (if possible) I can get the
    numbers printed in sync with the appends to the list?

    Thanks,

    Steve
  • Thomas Rast

    #2
    Re: calling functions across threads

    Steven Bethard <steven.bethard @gmail.com> writes:
    [color=blue]
    > I get the correct output, but if you run this yourself, you'll see
    > that the numbers 1 through 10 aren't printed in sync with the writes
    > (i.e. every half second); they're all printed at the end. Could
    > someone explain to me why this happens, and how (if possible) I can
    > get the numbers printed in sync with the appends to the list?[/color]

    I tried your code, and got the expected behaviour, i.e. the numbers
    are printed every half second. Maybe you have a buffering problem?

    $ python2.4 -V
    Python 2.4
    $ uname -a
    Linux thomas 2.6.9 #6 Sun Dec 19 17:45:53 CET 2004 i686 GNU/Linux

    - Thomas

    --
    If you want to reply by mail, substitute my first and last name for
    'foo' and 'bar', respectively, and remove '.invalid'.

    Comment

    • Steven Bethard

      #3
      Re: calling functions across threads

      Thomas Rast wrote:[color=blue]
      > Steven Bethard <steven.bethard @gmail.com> writes:
      >[color=green]
      >>I get the correct output, but if you run this yourself, you'll see
      >>that the numbers 1 through 10 aren't printed in sync with the writes
      >>(i.e. every half second); they're all printed at the end. Could
      >>someone explain to me why this happens, and how (if possible) I can
      >>get the numbers printed in sync with the appends to the list?[/color]
      >
      >
      > I tried your code, and got the expected behaviour, i.e. the numbers
      > are printed every half second. Maybe you have a buffering problem?
      >
      > $ python2.4 -V
      > Python 2.4
      > $ uname -a
      > Linux thomas 2.6.9 #6 Sun Dec 19 17:45:53 CET 2004 i686 GNU/Linux[/color]

      FWIW, I'm using Python 2.4 on a Windows XP box. But it looks like my
      problems were due to using PythonWin. When I tried the same code at the
      command prompt python, it worked just fine.

      Anyone know why PythonWin would do this? I originally ran into this
      problem using simliar code in a module that I used with the Ellogon
      (www.ellogon.org) framework. Maybe if I can solve the problem for
      PythonWin, I can translate that into a solution for Ellogon too...

      Steve

      Comment

      • Fernando Perez

        #4
        Re: calling functions across threads

        Steven Bethard wrote:
        [color=blue]
        > I get the correct output, but if you run this yourself, you'll see that
        > the numbers 1 through 10 aren't printed in sync with the writes (i.e.
        > every half second); they're all printed at the end. Could someone
        > explain to me why this happens, and how (if possible) I can get the
        > numbers printed in sync with the appends to the list?[/color]

        This is just a shot in the dark, as I'm quite ignorant of threading details.
        But what happens if you try adding a sys.stdout.flus h() call after the print
        statement in your custom update() method? It may just be a flushing problem
        what makes the output appear out of sync...

        Cheers,

        f

        Comment

        • Steven Bethard

          #5
          Re: calling functions across threads

          Fernando Perez wrote:[color=blue]
          > Steven Bethard wrote:
          >
          >[color=green]
          >>I get the correct output, but if you run this yourself, you'll see that
          >>the numbers 1 through 10 aren't printed in sync with the writes (i.e.
          >>every half second); they're all printed at the end. Could someone
          >>explain to me why this happens, and how (if possible) I can get the
          >>numbers printed in sync with the appends to the list?[/color]
          >
          >
          > This is just a shot in the dark, as I'm quite ignorant of threading details.
          > But what happens if you try adding a sys.stdout.flus h() call after the print
          > statement in your custom update() method? It may just be a flushing problem
          > what makes the output appear out of sync...[/color]

          Strangely enough, that causes PythonWin to hang... Why that would be
          true, I have no idea...

          Steve

          Comment

          • Fernando Perez

            #6
            Re: calling functions across threads

            Steven Bethard wrote:
            [color=blue]
            > Fernando Perez wrote:[color=green]
            >> Steven Bethard wrote:
            >>
            >>[color=darkred]
            >>>I get the correct output, but if you run this yourself, you'll see that
            >>>the numbers 1 through 10 aren't printed in sync with the writes (i.e.
            >>>every half second); they're all printed at the end. Could someone
            >>>explain to me why this happens, and how (if possible) I can get the
            >>>numbers printed in sync with the appends to the list?[/color]
            >>
            >>
            >> This is just a shot in the dark, as I'm quite ignorant of threading details.
            >> But what happens if you try adding a sys.stdout.flus h() call after the print
            >> statement in your custom update() method? It may just be a flushing problem
            >> what makes the output appear out of sync...[/color]
            >
            > Strangely enough, that causes PythonWin to hang... Why that would be
            > true, I have no idea...[/color]

            Mmh. I wouldn't be surprised if under pythonwin, sys.stdout is not the true
            python sys.stdout. Check the following:

            sys.stdout is sys.__stdout__

            The answer is probably false. In that case, they may have implemented some
            incomplete object whose flush method is broken, or something similar. I can't
            confirm, as I don't have windows access, so this is just a guess.

            Cheers,

            f

            Comment

            • Steven Bethard

              #7
              Re: calling functions across threads

              Fernando Perez wrote:[color=blue]
              > Steven Bethard wrote:
              >
              >[color=green]
              >>Fernando Perez wrote:
              >>[color=darkred]
              >>>Steven Bethard wrote:
              >>>
              >>>
              >>>
              >>>>I get the correct output, but if you run this yourself, you'll see that
              >>>>the numbers 1 through 10 aren't printed in sync with the writes (i.e.
              >>>>every half second); they're all printed at the end. Could someone
              >>>>explain to me why this happens, and how (if possible) I can get the
              >>>>numbers printed in sync with the appends to the list?
              >>>
              >>>
              >>>This is just a shot in the dark, as I'm quite ignorant of threading details.
              >>>But what happens if you try adding a sys.stdout.flus h() call after the print
              >>>statement in your custom update() method? It may just be a flushing problem
              >>>what makes the output appear out of sync...[/color]
              >>
              >>Strangely enough, that causes PythonWin to hang... Why that would be
              >>true, I have no idea...[/color]
              >
              >
              > Mmh. I wouldn't be surprised if under pythonwin, sys.stdout is not the true
              > python sys.stdout. Check the following:
              >
              > sys.stdout is sys.__stdout__
              >
              > The answer is probably false. In that case, they may have implemented some
              > incomplete object whose flush method is broken, or something similar. I can't
              > confirm, as I don't have windows access, so this is just a guess.[/color]

              Just to verify, yes, the answer is False:

              py> import sys
              py> sys.stdout is sys.__stdout__
              False

              Is there a list to ask PythonWin specific questions somewhere, or should
              I just wait for a PythonWin expert around here?

              Steve

              Comment

              • Steve Holden

                #8
                Re: calling functions across threads

                Steven Bethard wrote:
                [color=blue]
                > Fernando Perez wrote:
                >[color=green]
                >> Steven Bethard wrote:
                >>
                >>[color=darkred]
                >>> Fernando Perez wrote:
                >>>
                >>>> Steven Bethard wrote:
                >>>>
                >>>>
                >>>>
                >>>>> I get the correct output, but if you run this yourself, you'll see
                >>>>> that
                >>>>> the numbers 1 through 10 aren't printed in sync with the writes (i.e.
                >>>>> every half second); they're all printed at the end. Could someone
                >>>>> explain to me why this happens, and how (if possible) I can get the
                >>>>> numbers printed in sync with the appends to the list?
                >>>>
                >>>>
                >>>>
                >>>> This is just a shot in the dark, as I'm quite ignorant of threading
                >>>> details.
                >>>> But what happens if you try adding a sys.stdout.flus h() call after
                >>>> the print
                >>>> statement in your custom update() method? It may just be a flushing
                >>>> problem
                >>>> what makes the output appear out of sync...
                >>>
                >>>
                >>> Strangely enough, that causes PythonWin to hang... Why that would be
                >>> true, I have no idea...[/color]
                >>
                >>
                >>
                >> Mmh. I wouldn't be surprised if under pythonwin, sys.stdout is not
                >> the true
                >> python sys.stdout. Check the following:
                >>
                >> sys.stdout is sys.__stdout__
                >>
                >> The answer is probably false. In that case, they may have implemented
                >> some
                >> incomplete object whose flush method is broken, or something similar.
                >> I can't
                >> confirm, as I don't have windows access, so this is just a guess.[/color]
                >
                >
                > Just to verify, yes, the answer is False:
                >
                > py> import sys
                > py> sys.stdout is sys.__stdout__
                > False
                >
                > Is there a list to ask PythonWin specific questions somewhere, or should
                > I just wait for a PythonWin expert around here?
                >
                > Steve[/color]

                There's a mailing list at python-win32@python.or g - it's listed on
                www.python.org should you choose to subscribe, and the volume isn't
                generally high (maybe 5-10 messages per day).

                regards
                Steve
                --
                Steve Holden http://www.holdenweb.com/
                Python Web Programming http://pydish.holdenweb.com/
                Holden Web LLC +1 703 861 4237 +1 800 494 3119

                Comment

                • It's me

                  #9
                  Re: calling functions across threads

                  I haven't play with the thread stuff in Python (yet) but in general terms
                  (from a C mind), one should not expect read/write actions to be sequential
                  across threads. I would assume the Python threads eventually goes back to
                  some system calls for thread handling. If that were the case, you should
                  not be surprised at all that the I/O sequences appear to be quite
                  unpredictable.

                  If you absolutely, positively wants them to come out in a certain way, you
                  need to build in additionally serialization mechanisims in your code (like
                  use semaphores and stuff).



                  "Steven Bethard" <steven.bethard @gmail.com> wrote in message
                  news:NVBAd.2803 09$V41.151261@a ttbi_s52...[color=blue]
                  > I'm playing around with some threading stuff right now, and I'm having a
                  > little trouble calling a function from one thread that affects another.
                  > Here's my setup:
                  >
                  > py> import os, threading, time
                  > py> def write(file_in, input_lines):
                  > ... for line in input_lines:
                  > ... time.sleep(0.5)
                  > ... file_in.write(l ine)
                  > ... file_in.flush()
                  > ... file_in.close()
                  > ...
                  > py> def read(file_out, output_list):
                  > ... while True:
                  > ... line = file_out.readli ne()
                  > ... if not line:
                  > ... break
                  > ... output_list.app end(line)
                  > ...
                  > py> def runthreads(lst) :
                  > ... file_in, file_out, file_err = os.popen3('cat' )
                  > ... write_thread = threading.Threa d(
                  > ... target=write, args=(file_in,
                  > ... ['%s\n' % x for x in range(10)]))
                  > ... read_thread = threading.Threa d(target=read,
                  > ... args=(file_out, lst))
                  > ... write_thread.st art()
                  > ... read_thread.sta rt()
                  > ... write_thread.jo in()
                  > ... read_thread.joi n()
                  > ...
                  >
                  > Basically, I start one thread to read and one thread to write (from a
                  > os.pipe). This all works fine for me:
                  >
                  > py> lst = []
                  > py> runthreads(lst)
                  > py> lst
                  > ['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']
                  >
                  > I run into a problem though when I try to call an update method every
                  > time I read a line:
                  >
                  > py> class updatinglist(li st):
                  > ... def __init__(self, updater):
                  > ... super(updatingl ist, self).__init__( )
                  > ... self.updater = updater
                  > ... def append(self, item):
                  > ... super(updatingl ist, self).append(it em)
                  > ... self.updater(le n(self))
                  > ...
                  > py> def update(i):
                  > ... print i
                  > ...
                  > py> lst = updatinglist(up date)
                  > py> runthreads(lst)
                  > 1
                  > 2
                  > 3
                  > 4
                  > 5
                  > 6
                  > 7
                  > 8
                  > 9
                  > 10
                  > py> lst
                  > ['0\n', '1\n', '2\n', '3\n', '4\n', '5\n', '6\n', '7\n', '8\n', '9\n']
                  >
                  > I get the correct output, but if you run this yourself, you'll see that
                  > the numbers 1 through 10 aren't printed in sync with the writes (i.e.
                  > every half second); they're all printed at the end. Could someone
                  > explain to me why this happens, and how (if possible) I can get the
                  > numbers printed in sync with the appends to the list?
                  >
                  > Thanks,
                  >
                  > Steve[/color]


                  Comment

                  Working...