BUG trace realtime

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • malv
    #1

    BUG trace realtime

    How would you approach the following?
    In a multithreaded realtime data acquisition system (all python v2.4),
    after hours of running without a snag, without warning python hangs at
    once without leaving any error message or error traceback at all. After
    the incident, the OS itself (linux) appears to remain fully functional.
    Thank you!
    malv

    Tags: None
    • Peter Hansen
      #2
      Re: BUG trace realtime

      malv wrote:[color=blue]
      > How would you approach the following?
      > In a multithreaded realtime data acquisition system (all python v2.4),
      > after hours of running without a snag, without warning python hangs at
      > once without leaving any error message or error traceback at all. After
      > the incident, the OS itself (linux) appears to remain fully functional.[/color]

      1. Define "hangs" more precisely. It's rare that an app can hang
      without any behaviour that can tell you something more about it. For
      example, can you Ctrl-C out of it? Does it have sockets open still? Is
      it using any CPU time? etc... While you're at it, define "realtime"
      too, since many people use it incorrectly and it might mean something
      different to you than it does to everyone else reading it...

      2. Use logging. "import logging" and proceed from there... without
      logging, you have no idea where the problem occurred, and are going to
      spend lots of time guessing where the cause is.

      3. Make sure you are using Queues exclusively for inter-thread
      communication.

      4. Consider whether your external source of data (assuming there is one)
      is the cause. How are you interfacing to it? Serial port? ctypes?
      Something SWIGged for Python? A true "hang" is much more likely in an
      external package than in Python code.

      5. Tell us more about the platform. Which Linux (including version),
      does it have a GUI, what's the basic architecture of the app as it
      relates to the threading and data acquisition stuff, etc. (Of course,
      the problem might not come from threads or data acquisition, but until
      we learn more who could say?)

      -Peter

        Comment

        • malv
          #3
          Re: BUG trace realtime

          Hi Peter,
          Thank you for your extensive reply.
          In the meantime, I solved the problem. I had started out to trace the
          path of the randomly arriving external data through the different
          threaded processing stages. After one of those 'hangs', with some luck,
          I was able to spot the cause of the trouble: a piece of old code that
          wasn't supposed to be out there! Really dumb.
          FYI, to my surprise, Python really quit. No more CPU time, no message,
          nothing. I can't recall having seen this before. You are quite right
          about queues. I use these extensively and carefully. This kind of
          enabled me to trace the path of the externally triggered events without
          too much of a hassle.
          Thanks again,
          malv

            Comment

            • Peter Hansen
              #4
              Re: BUG trace realtime

              malv wrote:[color=blue]
              > After one of those 'hangs', with some luck,
              > I was able to spot the cause of the trouble: a piece of old code that
              > wasn't supposed to be out there! Really dumb.
              > FYI, to my surprise, Python really quit. No more CPU time, no message,
              > nothing. I can't recall having seen this before.[/color]

              If you post the relevant "old code" it's very likely someone could point
              out exactly why it failed, and how it wasn't *Python* that "quit", but
              merely something in your code that was blocking. (In other words,
              Python probably did exactly what you told it to do.)

              In general, Python does not "quit". It's very robust. When you get a
              failure like that, always suspect your own code structure.

              -Peter

                Comment

                • malv
                  #5
                  Re: BUG trace realtime

                  What happened in fact was that a graphics plotting function for
                  monitoring got called twice frorm a different thread, due to an 'old'
                  instruction inadvertently left. Due to the quasi random nature of the
                  incoming data streams, this happened rather very sparingly. As the
                  plotting itself involves quite a bit of code, there are probably
                  several if not many reasons why problems could arise. It was never my
                  intention to reenter this function and make it thread safe. Problem
                  solved.

                  I was indeed a surprise for me to see Python quit this way.
                  Thanks again for your commentary.
                  malv

                    Comment

                    Previous template Next
                    Working...