Strange Hotshot problem...

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • KefX

    Strange Hotshot problem...

    Hi guys! I'm still a bit of a Python newbie, but regardless I decided to embed
    Python into a plugin for a freeware (closed-source) Windows music program
    called Jeskola Buzz. (Man, I can't believe how many people keep telling me
    "Extend Python instead!" when clearly I can't do that in this case. If somebody
    here tells me this, I will explode in an appropriately gory fashion.) It's a
    great program, and it works mainly on plugins other people write. Well, I had
    been working on a guitar synth plugin in the summer, which started to sound
    good. Then I lost the code (I was trying to rewrite it anyway but I wanted to
    keep it as reference.) So I began to think about ways to try to make it better.

    I eventually hit on the idea of using a scripting language to do this. I
    started designing a language with a simple grammar (I like doing this kind of
    thing in a semi-compulsive manner, heh)...but I tossed that aside, worked on a
    ROM hacking/translating project some...well, my ROM hacking project required me
    to write a small script, so I decided it was finally time to learn Python.

    Geez, I love this language. :)

    Anyway, now it's a couple months later, translation project gets put on hold
    for reasons outside of my control, so I'm trying the guitar synth again. And I
    thought...why don't I use Python? I was told I could never get it to run in
    real time (we're talking about DSP, after all). I was about to give up on
    pursuing the idea when I heard about Psyco, which should help speed up
    everything that isn't rewritten as a C extension. I decided I'll give that a
    shot, no matter how ridiculous the idea seems. My experience with ridiculous
    ideas is that sometimes they actually work. :)

    Well, to make a long story longer, I got it running and it does produce sound
    (not the sound I would like to hear from it, though!) But brokenness of the
    sound suggested that it's running too freakin' slow, and it was: Buzz's CPU
    display said my own plugin was hogging the whole CPU! Well, okay, it's not
    hopeless yet, I can profile it and see what I can do about the bottlenecks.

    Which finally leads me to my problem: I can't get Hotshot to work properly. I
    can't fix a bottleneck if I don't know where it is! ;) Remember, I'm calling
    various Python functions from C++. See, in my module's init code, I'm basically
    I'm creating the hotshot.Profile object with the required path, bound to
    _profiler, then I do _profiler.start () ...at the very end, I do
    _profiler.stop( ) and _profiler.close (), and then I try to load the stats back
    with hotshot.stats.l oad() with the same path. Well, it finds the file just
    fine, but it can't read it. It raises an IndexError from trying to pop an empty
    list or some such. I've tested Hotshot with simple test scripts in pure Python
    and it works just fine. I thought Psyco might be the problem, but I commented
    out all use of Psyco in my script and the problem still exists, so I think the
    use of C++ to call Python code is messing it up.

    Is this a bug in the Hotshot package? Is there a workaround? Oh, yes, I also
    uploaded the offending profiler stats snapshot, in case you want to look at it.



    If you open it in the interactive Python interpreter and try
    hotshot.stats.l oad('hotshot.pr of') you should get the same result I do,
    complete with traceback.

    - Kef

    P.S. Sorry for making this post so long :P
  • Neil Hodgson

    #2
    Re: Strange Hotshot problem...

    KefX:[color=blue]
    > Well, it finds the file just
    > fine, but it can't read it. It raises an IndexError from trying to pop an[/color]
    empty[color=blue]
    > list or some such. I've tested Hotshot with simple test scripts in pure[/color]
    Python[color=blue]
    > and it works just fine.[/color]

    I saw a similar issue that could be 'fixed' by running Python in
    optimized '-O' mode rather than normal mode as -O turns off assertions.

    Neil


    Comment

    • Neil Hodgson

      #3
      Re: Strange Hotshot problem...

      I wrote:
      [color=blue]
      > I saw a similar issue that could be 'fixed' by running Python in
      > optimized '-O' mode rather than normal mode as -O turns off assertions.[/color]

      Disregard that - it doesn't solve your problem.

      Neil


      Comment

      • Kef Li Marcus X

        #4
        Re: Strange Hotshot problem...

        Nope...doesn't fix anything.

        - Kef
        [color=blue]
        > I saw a similar issue that could be 'fixed' by running Python in
        >optimized '-O' mode rather than normal mode as -O turns off assertions.[/color]


        Comment

        • KefX

          #5
          Re: Strange Hotshot problem...

          Well, I found a rather unhelpful non-solution (just posting about it so nobody
          else suggests it). I figured that the Work() function (being called the most
          often given the architecture of the plugin, and the only one called from the
          C++ code while outputting sound) would probably have the biggest bottleneck, so
          I renamed my Work function to Work_Impl, and added a new Work function with
          this:

          def Work(num_sample s):
          if _profiler is not None:
          return _profiler.runca ll(Work_Impl, num_samples)
          else:
          return Work_Impl(num_s amples)

          This technically worked (my code finally produced a readable log in the proper
          place). However, it was a failure in that this seemed to circumvent Psyco
          completely (making my code WAY too slow), and the results weren't too helpful
          in any case. An obvious solution is to make sure to call psyco.bind(Work _Impl),
          and while this did speed it up I get no information on the functions that
          Work_Impl called. Is there any other way to profile code that uses Psyco?
          (Other than the older 'profile' module, which is probably even less likely to
          work well...)

          Comment

          • Skip Montanaro

            #6
            Re: Strange Hotshot problem...


            kefx> (Other than the older 'profile' module, which is probably even
            kefx> less likely to work well...)

            The profile module might help, though since it's written in Python, it will
            definitely run slower than the hotshot profiler (what about profile+psyco?) .
            If nothing else you should be able to get some call counts. Another option
            is to use the trace module to count executed lines. If you submit a bug
            report to SourceForge (http://sourceforge.net/projects/python and assign it
            to Fred Drake (fdrake I believe) you'll at least be alerting the primary
            author of the hotshot code.

            Skip


            Comment

            • James Kew

              #7
              Re: Strange Hotshot problem...

              "KefX" <keflimarcvsx@a ol.comNOSPAM> wrote in message
              news:2003102418 1102.26916.0000 0093@mb-m05.aol.com...[color=blue]
              > See, in my module's init code, I'm basically
              > I'm creating the hotshot.Profile object with the required path, bound to
              > _profiler, then I do _profiler.start () ...at the very end, I do
              > _profiler.stop( ) and _profiler.close (), and then I try to load the stats[/color]
              back[color=blue]
              > with hotshot.stats.l oad() with the same path. Well, it finds the file just
              > fine, but it can't read it. It raises an IndexError from trying to pop an[/color]
              empty[color=blue]
              > list or some such.[/color]

              Not that this is going to help, but: me too. I've seen the exact same
              problem -- loading saved stats throws an exception -- when profiling with
              Hotshot.

              This was in a pure-Python project, and it seemed sensitive to the exact code
              being profiled: it'd work fine on some inputs, but then I'd change one or
              two lines to attempt an optimisation and it'd fail.

              I ended up giving up and going back to profile -- slower, but livable with.

              James


              Comment

              Working...