Interesting Thread Gotcha

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Hendrik van Rooyen

    Interesting Thread Gotcha


    I thought I would share this nasty little gotcha with the group.

    Consider the following code fragment:

    <start>
    print 'starting kbd thread'
    keyboard_thread = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
    print 'starting main loop'
    error = Mainloop(s,port _q,active_q_lis t)
    <end>

    It produces, as output, the following:

    starting kbd thread
    we get here - a

    It does not print 'starting main loop', the Mainloop routine
    is never executed, and no exceptions are raised.

    Here is the offending routine that seems to capture the control:

    <start>
    def kbd_driver(out_ q,in_q):
    """
    thread to look for keyboard input and to put it on the queue out_q
    also looks for replies on in_q and prints them
    """

    kbdname = '/dev/stdin'

    kbd = open(kbdname,'r +',1) # Reading, line buffered

    unblock(kbd) # Call the magic to unblock keyboard
    print 'we get here - a'
    while True:

    try:
    d = kbd.readline() # see if any kbd input
    except:
    IOError
    try:
    msg=in_q.get(bl ock=False)
    except Queue.Empty:
    time.sleep(0.1)
    continue
    print msg
    time.sleep(0.1)
    continue
    d = d.rstrip() # get rid of line feed
    out_q.put([d + '\r',in_q]) # add a carriage return and return q and send
    to port
    <end>


    The unblock is a routine that unblocks a port using fcntl - it
    is not the problem. In case you don't believe me, here it is:

    def unblock(f):
    """Given file 'f', sets its unblock flag to true."""

    fcntl.fcntl(f.f ileno(), fcntl.F_SETFL, os.O_NONBLOCK)

    I will post the solution tomorrow when I read my mail,
    if no one has spotted it by then.

    - Hendrik


  • Dan

    #2
    Re: Interesting Thread Gotcha

    On Jan 15, 10:07 am, "Hendrik van Rooyen" <m...@microcorp .co.za>
    wrote:
    I thought I would share this nasty little gotcha with the group.
    >
    Consider the following code fragment:
    >
    <start>
    print 'starting kbd thread'
    keyboard_thread = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
    print 'starting main loop'
    error = Mainloop(s,port _q,active_q_lis t)
    <end>
    >
    It produces, as output, the following:
    >
    starting kbd thread
    we get here - a
    >
    It does not print 'starting main loop', the Mainloop routine
    is never executed, and no exceptions are raised.
    >
    Here is the offending routine that seems to capture the control:
    >
    <start>
    def kbd_driver(out_ q,in_q):
    """
    thread to look for keyboard input and to put it on the queue out_q
    also looks for replies on in_q and prints them
    """
    >
    kbdname = '/dev/stdin'
    >
    kbd = open(kbdname,'r +',1) # Reading, line buffered
    >
    unblock(kbd) # Call the magic to unblock keyboard
    print 'we get here - a'
    while True:
    >
    try:
    d = kbd.readline() # see if any kbd input
    except:
    IOError
    try:
    msg=in_q.get(bl ock=False)
    except Queue.Empty:
    time.sleep(0.1)
    continue
    print msg
    time.sleep(0.1)
    continue
    d = d.rstrip() # get rid of line feed
    out_q.put([d + '\r',in_q]) # add a carriage return and return q and send
    to port
    <end>
    >
    The unblock is a routine that unblocks a port using fcntl - it
    is not the problem. In case you don't believe me, here it is:
    >
    def unblock(f):
    """Given file 'f', sets its unblock flag to true."""
    >
    fcntl.fcntl(f.f ileno(), fcntl.F_SETFL, os.O_NONBLOCK)
    >
    I will post the solution tomorrow when I read my mail,
    if no one has spotted it by then.
    >
    - Hendrik
    >>keyboard_thre ad = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
    Needs to be
    >>keyboard_thre ad = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
    Commas are important!

    -Dan

    Comment

    • Hendrik van Rooyen

      #3
      Re: Interesting Thread Gotcha

      "Dan" <the,,,ail.comw rote:

      >keyboard_threa d = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
      >
      Needs to be
      >keyboard_threa d = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
      >
      Commas are important!
      >
      -Dan
      Absolutely! - well spotted!

      As the first correct respondent, you win the freedom to spend a week in
      Naboomspruit at your own expense.

      It would have been nice, however, to have gotten something like:

      TypeError - This routine needs a tuple.

      instead of the silent in line calling of the routine in question,
      while failing actually to start a new thread.

      It seems to act no different from plain old:

      kbd_driver (port_q,kbd_q)

      Is it worth the trouble of learning how to submit a bug report?

      - Hendrik



      Comment

      • Duncan Booth

        #4
        Re: Interesting Thread Gotcha

        "Hendrik van Rooyen" <mail@microcorp .co.zawrote:
        It would have been nice, however, to have gotten something like:
        >
        TypeError - This routine needs a tuple.
        >
        instead of the silent in line calling of the routine in question,
        while failing actually to start a new thread.
        Given that the start_new_threa d function never actually got called, what
        code exactly do you expect to complain about the absence of a tuple?
        >
        It seems to act no different from plain old:
        >
        kbd_driver (port_q,kbd_q)
        >
        Is it worth the trouble of learning how to submit a bug report?
        On your own code? There doesn't appear to be a bug in anyone else's code
        here.

        Comment

        • Bjoern Schliessmann

          #5
          Re: Interesting Thread Gotcha

          Hendrik van Rooyen wrote:
          Absolutely! - well spotted!
          This is no threading problem at all; not even a syntax problem. If
          you don't know exactly what start_new_threa d and kbd_driver
          functions do it's impossible to tell if your code does what is
          intended.
          It would have been nice, however, to have gotten something like:
          >
          TypeError - This routine needs a tuple.
          >
          instead of the silent in line calling of the routine in question,
          while failing actually to start a new thread.
          Exactly which part of the code should give you this warning?
          Is it worth the trouble of learning how to submit a bug report?
          For your problem not, IMHO, as a bug report for it will be closed
          quickly.

          Regards,


          Björn

          --
          BOFH excuse #330:

          quantum decoherence

          Comment

          • Diez B. Roggisch

            #6
            Re: Interesting Thread Gotcha

            Hendrik van Rooyen wrote:
            "Dan" <the,,,ail.comw rote:
            >
            >
            >>keyboard_thre ad = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
            >>
            >Needs to be
            >>keyboard_thre ad = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
            >>
            >Commas are important!
            >>
            >-Dan
            >
            Absolutely! - well spotted!
            >
            As the first correct respondent, you win the freedom to spend a week in
            Naboomspruit at your own expense.
            >
            It would have been nice, however, to have gotten something like:
            >
            TypeError - This routine needs a tuple.
            >
            instead of the silent in line calling of the routine in question,
            while failing actually to start a new thread.
            You can't prevent the silent inline-calling - otherwise, how would you do
            this:

            def compute_thread_ target():
            def target():
            pass
            return target

            thread.start_ne w_thread(comput e_thread_target ())


            Of course start_new_threa d could throw an error if it got nothing callable
            as first argument. No idea why it doesn't.

            Diez

            Comment

            • Dan

              #7
              Re: Interesting Thread Gotcha

              On Jan 16, 11:06 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
              Hendrik van Rooyen wrote:
              "Dan" <the,,,ail.comw rote:
              >
              >keyboard_threa d = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
              >
              Needs to be
              >keyboard_threa d = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
              >
              Commas are important!
              >
              -Dan
              >
              Absolutely! - well spotted!
              >
              As the first correct respondent, you win the freedom to spend a week in
              Naboomspruit at your own expense.
              >
              It would have been nice, however, to have gotten something like:
              >
              TypeError - This routine needs a tuple.
              >
              instead of the silent in line calling of the routine in question,
              while failing actually to start a new thread.
              >
              You can't prevent the silent inline-calling - otherwise, how would you do
              this:
              >
              def compute_thread_ target():
              def target():
              pass
              return target
              >
              thread.start_ne w_thread(comput e_thread_target ())
              >
              Of course start_new_threa d could throw an error if it got nothing callable
              as first argument. No idea why it doesn't.
              >
              Diez
              Of course, in his case, having start_new_threa d throw an error
              wouldn't have helped, since he went into an infinite loop while
              evaluating the parameters for start_new_threa d.

              Would it be possible to have pychecker (or some such) warn that there
              is an insufficient parameter count to start_new_threa d? I guess that
              would require knowing the type of thread. . .

              -Dan

              Comment

              • Diez B. Roggisch

                #8
                Re: Interesting Thread Gotcha

                Dan schrieb:
                On Jan 16, 11:06 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
                >Hendrik van Rooyen wrote:
                >>"Dan" <the,,,ail.comw rote:
                >>>>>>keyboard_ thread = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
                >>>Needs to be
                >>>>>>keyboard_ thread = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
                >>>Commas are important!
                >>>-Dan
                >>Absolutely! - well spotted!
                >>As the first correct respondent, you win the freedom to spend a week in
                >>Naboomsprui t at your own expense.
                >>It would have been nice, however, to have gotten something like:
                >>TypeError - This routine needs a tuple.
                >>instead of the silent in line calling of the routine in question,
                >>while failing actually to start a new thread.
                >You can't prevent the silent inline-calling - otherwise, how would you do
                >this:
                >>
                >def compute_thread_ target():
                > def target():
                > pass
                > return target
                >>
                >thread.start_n ew_thread(compu te_thread_targe t())
                >>
                >Of course start_new_threa d could throw an error if it got nothing callable
                >as first argument. No idea why it doesn't.
                >>
                >Diez
                >
                Of course, in his case, having start_new_threa d throw an error
                wouldn't have helped, since he went into an infinite loop while
                evaluating the parameters for start_new_threa d.
                >
                Would it be possible to have pychecker (or some such) warn that there
                is an insufficient parameter count to start_new_threa d? I guess that
                would require knowing the type of thread. . .
                What has this to do with the second argument? It's perfectly legal to
                have a function as thread-target that takes no arguments at all, so
                enforcing a second argument wouldn't be helpful - all it would do is to
                force all developers that don't need an argument tuple to pass the empty
                tuple. So there was no insufficient argument count.

                And none of these would solve the underlying problem that in python
                expressions are evaluated eagerly. Changing that would mean that you end
                up with a totally new language.

                the only thing that could help to a certain extend would be static
                types. Which we don't want here :)

                Diez

                Comment

                • Dan

                  #9
                  Re: Interesting Thread Gotcha

                  On Jan 16, 1:33 pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
                  Dan schrieb:
                  >
                  >
                  >
                  On Jan 16, 11:06 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
                  Hendrik van Rooyen wrote:
                  >"Dan" <the,,,ail.comw rote:
                  >>>>>keyboard_t hread = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
                  >>Needs to be
                  >>>>>keyboard_t hread = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
                  >>Commas are important!
                  >>-Dan
                  >Absolutely! - well spotted!
                  >As the first correct respondent, you win the freedom to spend a week in
                  >Naboomspruit at your own expense.
                  >It would have been nice, however, to have gotten something like:
                  >TypeError - This routine needs a tuple.
                  >instead of the silent in line calling of the routine in question,
                  >while failing actually to start a new thread.
                  You can't prevent the silent inline-calling - otherwise, how would you do
                  this:
                  >
                  def compute_thread_ target():
                  def target():
                  pass
                  return target
                  >
                  thread.start_ne w_thread(comput e_thread_target ())
                  >
                  Of course start_new_threa d could throw an error if it got nothing callable
                  as first argument. No idea why it doesn't.
                  >
                  Diez
                  >
                  Of course, in his case, having start_new_threa d throw an error
                  wouldn't have helped, since he went into an infinite loop while
                  evaluating the parameters for start_new_threa d.
                  >
                  Would it be possible to have pychecker (or some such) warn that there
                  is an insufficient parameter count to start_new_threa d? I guess that
                  would require knowing the type of thread. . .
                  >
                  What has this to do with the second argument? It's perfectly legal to
                  have a function as thread-target that takes no arguments at all, so
                  enforcing a second argument wouldn't be helpful - all it would do is to
                  force all developers that don't need an argument tuple to pass the empty
                  tuple. So there was no insufficient argument count.
                  >
                  And none of these would solve the underlying problem that in python
                  expressions are evaluated eagerly. Changing that would mean that you end
                  up with a totally new language.
                  >
                  the only thing that could help to a certain extend would be static
                  types. Which we don't want here :)
                  >
                  Diez
                  It doesn't seem to be legal in my version of python (or the doc):
                  >>import thread
                  >>def bat():
                  print "hello"

                  >>thread.start_ new_thread(bat)
                  Traceback (most recent call last):
                  File "<pyshell#1 2>", line 1, in <module>
                  thread.start_ne w_thread(bat)
                  TypeError: start_new_threa d expected at least 2 arguments, got 1
                  >>thread.start_ new_thread(bat, ())
                  2256hello

                  >>>
                  -Dan

                  Comment

                  • Diez B. Roggisch

                    #10
                    Re: Interesting Thread Gotcha

                    Dan schrieb:
                    On Jan 16, 1:33 pm, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
                    >Dan schrieb:
                    >>
                    >>
                    >>
                    >>On Jan 16, 11:06 am, "Diez B. Roggisch" <de...@nospam.w eb.dewrote:
                    >>>Hendrik van Rooyen wrote:
                    >>>>"Dan" <the,,,ail.comw rote:
                    >>>>>>>>keyboar d_thread = thread.start_ne w_thread(kbd_dr iver (port_q,kbd_q))
                    >>>>>Needs to be
                    >>>>>>>>keyboar d_thread = thread.start_ne w_thread(kbd_dr iver, (port_q,kbd_q))
                    >>>>>Commas are important!
                    >>>>>-Dan
                    >>>>Absolutel y! - well spotted!
                    >>>>As the first correct respondent, you win the freedom to spend a week in
                    >>>>Naboomsprui t at your own expense.
                    >>>>It would have been nice, however, to have gotten something like:
                    >>>>TypeError - This routine needs a tuple.
                    >>>>instead of the silent in line calling of the routine in question,
                    >>>>while failing actually to start a new thread.
                    >>>You can't prevent the silent inline-calling - otherwise, how would you do
                    >>>this:
                    >>>def compute_thread_ target():
                    >>> def target():
                    >>> pass
                    >>> return target
                    >>>thread.start _new_thread(com pute_thread_tar get())
                    >>>Of course start_new_threa d could throw an error if it got nothing callable
                    >>>as first argument. No idea why it doesn't.
                    >>>Diez
                    >>Of course, in his case, having start_new_threa d throw an error
                    >>wouldn't have helped, since he went into an infinite loop while
                    >>evaluating the parameters for start_new_threa d.
                    >>Would it be possible to have pychecker (or some such) warn that there
                    >>is an insufficient parameter count to start_new_threa d? I guess that
                    >>would require knowing the type of thread. . .
                    >What has this to do with the second argument? It's perfectly legal to
                    >have a function as thread-target that takes no arguments at all, so
                    >enforcing a second argument wouldn't be helpful - all it would do is to
                    >force all developers that don't need an argument tuple to pass the empty
                    >tuple. So there was no insufficient argument count.
                    >>
                    >And none of these would solve the underlying problem that in python
                    >expressions are evaluated eagerly. Changing that would mean that you end
                    >up with a totally new language.
                    >>
                    >the only thing that could help to a certain extend would be static
                    >types. Which we don't want here :)
                    >>
                    >Diez
                    >
                    It doesn't seem to be legal in my version of python (or the doc):
                    >
                    >>>import thread
                    >>>def bat():
                    print "hello"
                    >
                    >
                    >>>thread.start _new_thread(bat )
                    >
                    Traceback (most recent call last):
                    File "<pyshell#1 2>", line 1, in <module>
                    thread.start_ne w_thread(bat)
                    TypeError: start_new_threa d expected at least 2 arguments, got 1
                    >>>thread.start _new_thread(bat , ())
                    2256hello
                    Ah, I thought it was optional, as in the threading.Threa d(target=...,
                    args=....)-version. Sorry for not looking that up.

                    Then you'd might stand a chance that pychecker can find such a situation
                    - but of course not on a general level, as in the above - that would
                    only work with type-annotations.



                    Diez

                    Comment

                    • Sion Arrowsmith

                      #11
                      Re: Interesting Thread Gotcha

                      Diez B. Roggisch <deets@nospam.w eb.dewrote:
                      >Of course start_new_threa d could throw an error if it got nothing callable
                      >as first argument. No idea why it doesn't.
                      It does:
                      >>thread.start_ new_thread(None , None)
                      Traceback (most recent call last):
                      File "<stdin>", line 1, in ?
                      TypeError: first arg must be callable

                      --
                      \S -- siona@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
                      "Frankly I have no feelings towards penguins one way or the other"
                      -- Arthur C. Clarke
                      her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

                      Comment

                      • Hendrik van Rooyen

                        #12
                        Re: Interesting Thread Gotcha

                        "Bjoern Schliessmann" <usenet-ourmet.comwrote :

                        Hendrik van Rooyen wrote:
                        Absolutely! - well spotted!
                        This is no threading problem at all; not even a syntax problem. If
                        you don't know exactly what start_new_threa d and kbd_driver
                        functions do it's impossible to tell if your code does what is
                        intended.
                        It would have been nice, however, to have gotten something like:
                        >
                        TypeError - This routine needs a tuple.
                        >
                        instead of the silent in line calling of the routine in question,
                        while failing actually to start a new thread.
                        Exactly which part of the code should give you this warning?

                        I am obviously missing something.

                        My understanding is that, to start a new thread, one does:

                        NewThreadID = thread.start_ne w_thread(NameOf RoutineToStart,
                        (ArgumentToCall _it_with,second Arg,Etc))

                        This calls start_new_threa d with the name and the arguments to pass.

                        If one omits the comma, then start_new_threa d is surely stilled called,
                        but with an argument that is now a call to the routine in question, which
                        somehow causes the problem.

                        So start_new_threa d is the code that that is executed, with a bad set of
                        arguments - one thing, (a call to a routine) instead of two things -
                        a routine and a tuple of arguments.

                        Everywhere else in Python if you give a routine the incorrect number of
                        arguments, you get an exception. Why not here?

                        - Hendrik



                        Comment

                        • Peter Otten

                          #13
                          Re: Interesting Thread Gotcha

                          Hendrik van Rooyen wrote:
                          "Bjoern Schliessmann" <usenet-ourmet.comwrote :
                          >
                          Hendrik van Rooyen wrote:
                          >Absolutely! - well spotted!
                          >
                          This is no threading problem at all; not even a syntax problem. If
                          you don't know exactly what start_new_threa d and kbd_driver
                          functions do it's impossible to tell if your code does what is
                          intended.
                          >
                          >It would have been nice, however, to have gotten something like:
                          >>
                          >TypeError - This routine needs a tuple.
                          >>
                          >instead of the silent in line calling of the routine in question,
                          >while failing actually to start a new thread.
                          >
                          Exactly which part of the code should give you this warning?
                          >
                          I am obviously missing something.
                          >
                          My understanding is that, to start a new thread, one does:
                          >
                          NewThreadID = thread.start_ne w_thread(NameOf RoutineToStart,
                          (ArgumentToCall _it_with,second Arg,Etc))
                          >
                          This calls start_new_threa d with the name and the arguments to pass.
                          >
                          If one omits the comma, then start_new_threa d is surely stilled called,
                          but with an argument that is now a call to the routine in question, which
                          somehow causes the problem.
                          >
                          So start_new_threa d is the code that that is executed, with a bad set of
                          arguments - one thing, (a call to a routine) instead of two things -
                          a routine and a tuple of arguments.
                          >
                          Everywhere else in Python if you give a routine the incorrect number of
                          arguments, you get an exception. Why not here?
                          Python always evaluates the function's arguments first. The check for the
                          correct number of arguments is part of the call and therefore done
                          afterwards:
                          >>def f(x): print x
                          ....
                          >>f(f(1), f(2), f(3))
                          1
                          2
                          3
                          Traceback (most recent call last):
                          File "<stdin>", line 1, in <module>
                          TypeError: f() takes exactly 1 argument (3 given)

                          So if one of the arguments takes forever to calculate you will never see
                          the TypeError:
                          >>def g(x):
                          .... print x
                          .... import time
                          .... while 1: time.sleep(1)
                          ....
                          >>f(f(1), g(2), f(3))
                          1
                          2
                          Traceback (most recent call last):
                          File "<stdin>", line 1, in <module>
                          File "<stdin>", line 4, in g
                          KeyboardInterru pt # I hit Ctrl-C

                          Peter

                          Comment

                          • Hendrik van Rooyen

                            #14
                            Re: Interesting Thread Gotcha

                            "Duncan Booth" <dunc...d.inval idwrote:
                            Given that the start_new_threa d function never actually got called, what
                            code exactly do you expect to complain about the absence of a tuple?
                            I don't understand this assertion.

                            I thought that start_new_threa d was called with a missing comma in
                            its argument list, which had the effect that I am complaining about.

                            Putting the comma in place solved the problem, without any other
                            changes, so why do you say that start_new_threa d was not called?

                            - Hendrik

                            Comment

                            • Hendrik van Rooyen

                              #15
                              Re: Interesting Thread Gotcha

                              "Diez B. Roggisch" <dee,,eb.dewrot e:
                              Hendrik van Rooyen wrote:
                              It would have been nice, however, to have gotten something like:

                              TypeError - This routine needs a tuple.

                              instead of the silent in line calling of the routine in question,
                              while failing actually to start a new thread.
                              >
                              You can't prevent the silent inline-calling - otherwise, how would you do
                              this:
                              >
                              def compute_thread_ target():
                              def target():
                              pass
                              return target
                              >
                              thread.start_ne w_thread(comput e_thread_target ())
                              >
                              >
                              Of course start_new_threa d could throw an error if it got nothing callable
                              as first argument. No idea why it doesn't.
                              Thanks - got it, I think. Doesn't mean I like it, though:
                              >>a = 42
                              >>b = 24
                              >>def do_something(c, d):
                              print c
                              print d
                              >>do_something( a,b)
                              42
                              24
                              >>def harmless():
                              return a
                              >>def evil():
                              while True:
                              pass
                              >>do_something( a)
                              Traceback (most recent call last):
                              File "<pyshell#1 5>", line 1, in ?
                              do_something(a)
                              TypeError: do_something() takes exactly 2 arguments (1 given)
                              >>do_something( harmless())
                              Traceback (most recent call last):
                              File "<pyshell#1 7>", line 1, in ?
                              do_something(ha rmless())
                              TypeError: do_something() takes exactly 2 arguments (1 given)
                              >>>do_something (evil())

                              This hangs and needs OS intervention to kill it - and there is also just
                              one argument, not two.

                              Looks like the arguments are handled one by one without validation
                              till the end. Lets see:
                              >>do_something( a,b,harmless())
                              Traceback (most recent call last):
                              File "<pyshell#1 8>", line 1, in ?
                              do_something(a, b,harmless())
                              TypeError: do_something() takes exactly 2 arguments (3 given)

                              So far, so good.
                              >>>do_something (a,b,evil())
                              This also hangs - the third, extra argument is actually called!

                              Are you all sure this is not a buglet?

                              - Hendrik


                              Comment

                              Working...