python coding contest

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Claudio Grondi

    #16
    Re: python coding contest

    Claudio Grondi wrote:[color=blue][color=green]
    >> Please send me comments, suggestions and ideas.[/color]
    >
    >
    > Now, after the contest is over I analysed the outcome of it and have
    > come to the conclusion, that there were two major factors which
    > contributed to squeezing of code:
    >
    > (1). usage of available variants for coding of the same thing
    > (2). sqeezing the size of used numeric and string literals
    >
    > As (1) leads to less readable cryptic code it makes not much sense from
    > my point of view to dig deeper in that direction. As already mentioned
    > in this thread by Tim Peters ( pointing to
    > http://spoj.sphere.pl/problems/KAMIL/ ) it seems, that Pearl is here the
    > proper language of choice for such kind of problems anyway.
    >
    > Trying to improve on (2) belongs in my eyes much more into the area of
    > problems discussed in comp.compressio n than to problems belonging into
    > comp.lang.pytho n .
    >
    > So what is my point? Ok, I will mention it at the end of this post.
    >
    > Before that I want to thank the originators of the contest and
    > especially the participants for providing insight into the techniques
    > they have used. I have learned from the contest what lambda expression
    > is good for and how it works where I failed to grasp it from reading
    > tutorials only.
    >
    > I have detected, that it would be a nice thing to have in Python a
    > function able to convert values from binary string to an integer
    > representation as in my eyes both in case of long integer values are
    > more or less the same thing/object. The only difference is probably in
    > the header not in the representation of the actual value in memory - am
    > I right here? Will it make sense to have a string-integer object which
    > value can be used in both contexts as a binary string and a long integer
    > value?
    > Is there in Python any simple way to do the same as the following two
    > following functions I have put together today:
    >
    > def longIntWithBits OfBinaryString( stringToConvert ):
    > intWithBitsOfBi naryString = 0L
    > for singleChar in stringToConvert :
    > intWithBitsOfBi naryString = (intWithBitsOfB inaryString<<8) +
    > ord(singleChar)
    > #:for
    > return intWithBitsOfBi naryString
    > #:def longIntWithBits OfBinaryString( s)
    >
    > def binaryStringWit hBitsOfLongInt( i):
    > listOfCharsOfSt ringWithThePack edInt = []
    > exponent = 1
    > while i > 256**exponent: exponent+=1
    > for byteNo in range(0,exponen t):
    > noOfBitsToShift = byteNo*8
    >
    > listOfCharsOfSt ringWithThePack edInt.append(ch r(i>>noOfBitsTo Shift&0xFF))
    > #:for
    > # reverse (in place) in order to get the highest bits of the integer
    > as leftmost byte
    > listOfCharsOfSt ringWithThePack edInt.reverse()
    > stringWithThePa ckedInt = ''.join(listOfC harsOfStringWit hThePackedInt)
    > return stringWithThePa ckedInt
    > #:def binaryStringWit hBitsOfLongInt( i)
    >
    > print "longIntWithBit sOfBinaryString ('ABBA') =
    > %i"%longIntWith BitsOfBinaryStr ing('ABBA')
    > print
    > "binaryStringWi thBitsOfLongInt (longIntWithBit sOfBinaryString ('ABBA')) =
    > '%s'"%binaryStr ingWithBitsOfLo ngInt(longIntWi thBitsOfBinaryS tring('ABBA'))
    >
    > which gives:
    >
    > longIntWithBits OfBinaryString( 'ABBA') = 1094861377
    > binaryStringWit hBitsOfLongInt( longIntWithBits OfBinaryString( 'ABBA')) =
    > 'ABBA'
    >
    > ?
    >
    > And now my point I have promised to write about:
    >
    > If squeezing code makes it bad code and compressing literals is more or
    > less compression technique and not Python programming, it is maybe a
    > good idea to try to explore what Python distribution provides as data
    > and modules and rewrite the seven_seg module, but with following
    > limitations:
    >
    > 1. it is not allowed to use any literals in the provided code
    > 2. it is not allowed to misuse the names of the identifiers as a kind of
    > literals providing data
    > 3. it is not allowed to use modules or files which doesn't come with the
    > Python distribution.
    >
    > I have no slightest idea if it is possible to program a seven_seg
    > module under such conditions. It could be a sign, that it would be a
    > very interesting challenge worth to get involved into or a sign I have
    > no slightest idea about Python and programming.
    >
    > What do you all think about it?
    >
    > Claudio[/color]
    After some coding trials, it turned out to be quite easy (almost
    trivial) to overcome the problem of not beeing allowed to use any
    literals in the script code, but I suppose, that I am not alone with not
    seeing directly how to code it, so it is maybe a good exercise for a
    Python beginner (like me) to cope a bit with it.
    Knowing this I am curious if it is also comparable easy in other
    programming languages, e.g. when using only a C/C++ compiler and linker
    executables along with the provided libraries and header files? I
    suppose, that each language comes with built-in literals which can be
    utilized in own code to get the full range of required literals into
    identifiers by using only what the language provides itself.
    Am I right or not?

    Claudio

    Comment

    • Claudio Grondi

      #17
      Re: python coding contest

      Steven D'Aprano wrote:[color=blue]
      > On Sun, 01 Jan 2006 03:34:33 +0100, Claudio Grondi wrote:
      >
      >[color=green][color=darkred]
      >>>Please send me comments, suggestions and ideas.[/color]
      >>
      >>Now, after the contest is over I analysed the outcome of it and have
      >>come to the conclusion, that there were two major factors which
      >>contributed to squeezing of code:
      >>
      >> (1). usage of available variants for coding of the same thing
      >> (2). sqeezing the size of used numeric and string literals[/color]
      >
      >
      > [snip]
      >
      >[color=green]
      >>Is there in Python any simple way to do the same as the following two
      >>following functions I have put together today:[/color]
      >
      >
      > They are already pretty simple. You can make them even more simple by
      > using less complicated names and getting rid of the explicit end block
      > markers. It is sometimes useful to put in explicit end block markers when
      > you have long blocks, but when the block is just a single line, well,
      > I don't see the point.
      >
      > Here is another possibility.
      >
      >[color=green][color=darkred]
      >>>>import array
      >>>>A = array.array('b' )
      >>>>n = 1000000
      >>>>while n:[/color][/color]
      >
      > .... A.append(n&255) ; n = n >> 8
      > ....
      >[color=green][color=darkred]
      >>>>A.reverse ()
      >>>>A[/color][/color]
      >
      > array('b', [15, 66, 64])
      >[color=green][color=darkred]
      >>>>15*256**2 + 66*256 + 64[/color][/color]
      >
      > 1000000
      >[color=green][color=darkred]
      >>>>A.tostring( )[/color][/color]
      >
      > '\x0fB@'
      >
      > The reverse transformation is just as easy:
      >
      >[color=green][color=darkred]
      >>>>A = array.array('b' , "\x0fB@") # initialise from a byte string
      >>>>n = 0L
      >>>>for b in A:[/color][/color]
      >
      > .... n = n << 8 | b
      > ....
      >[color=green][color=darkred]
      >>>>n[/color][/color]
      >
      > 1000000L
      >
      > And of course these can be turned into functions.
      >
      >
      >[/color]
      What I have thought about as a simpler/better solution is a method
      allowing to avoid processing the content of the string or long integer
      object by looping over its content. I suppose, that knowing enough about
      Python internals it must be possible to change only the object type not
      beeing forced to process the content i.e. the value itself, what in case
      of big size of data to convert with methods like this above wastes CPU
      time.

      Claudio

      Comment

      • Steven D'Aprano

        #18
        Re: python coding contest

        On Sun, 01 Jan 2006 15:49:58 +0100, Claudio Grondi wrote:

        [color=blue]
        > What I have thought about as a simpler/better solution is a method
        > allowing to avoid processing the content of the string or long integer
        > object by looping over its content.[/color]

        How can you avoid looping over its content? Whether you do it yourself
        using "for byte in array" or similar, or Python does it for you
        (using array.tostring perhaps), *something* has to walk through the bytes.

        If you don't like walking the string, write a function to do it once, and
        then use the function.
        [color=blue]
        > I suppose, that knowing enough about
        > Python internals it must be possible to change only the object type not
        > beeing forced to process the content i.e. the value itself, what in case
        > of big size of data to convert with methods like this above wastes CPU
        > time.[/color]

        I'm reminded of a time I was going for a drive in the country when I drove
        past an apple orchid. Standing in the orchid was a farmer with a pig. He
        lifted the pig into the air, and the pig then bit an apple and slowly
        chewed it. The farmer then carried him over to another branch, and the pig
        ate another apple.

        I was so surprised I stopped my car and wandered over to ask the farmer
        what he was doing.

        "I'm feeding apples to my pig," he replied.

        "Wouldn't it save time to just pick some apples and feed them to the pig?"

        The farmer looked at me like I was an idiot. "What's time to a pig?"


        The moral of the story is, before spending time working on some scheme to
        save CPU time, you better be absolutely sure that firstly, you are going
        to save CPU time, secondly, that it is enough CPU time to be worth saving,
        and thirdly, that you aren't wasting more of your own time to do it.



        --
        Steven.

        Comment

        • Claudio Grondi

          #19
          Re: python coding contest

          Steven D'Aprano wrote:[color=blue]
          > On Sun, 01 Jan 2006 15:49:58 +0100, Claudio Grondi wrote:
          >
          >
          >[color=green]
          >>What I have thought about as a simpler/better solution is a method
          >>allowing to avoid processing the content of the string or long integer
          >>object by looping over its content.[/color]
          >
          >
          > How can you avoid looping over its content? Whether you do it yourself
          > using "for byte in array" or similar, or Python does it for you
          > (using array.tostring perhaps), *something* has to walk through the bytes.
          >
          > If you don't like walking the string, write a function to do it once, and
          > then use the function.
          >
          >[color=green]
          >>I suppose, that knowing enough about
          >>Python internals it must be possible to change only the object type not
          >>beeing forced to process the content i.e. the value itself, what in case
          >>of big size of data to convert with methods like this above wastes CPU
          >>time.[/color]
          >
          >
          > I'm reminded of a time I was going for a drive in the country when I drove
          > past an apple orchid. Standing in the orchid was a farmer with a pig. He
          > lifted the pig into the air, and the pig then bit an apple and slowly
          > chewed it. The farmer then carried him over to another branch, and the pig
          > ate another apple.
          >
          > I was so surprised I stopped my car and wandered over to ask the farmer
          > what he was doing.
          >
          > "I'm feeding apples to my pig," he replied.
          >
          > "Wouldn't it save time to just pick some apples and feed them to the pig?"
          >
          > The farmer looked at me like I was an idiot. "What's time to a pig?"
          >
          >
          > The moral of the story is, before spending time working on some scheme to
          > save CPU time, you better be absolutely sure that firstly, you are going
          > to save CPU time, secondly, that it is enough CPU time to be worth saving,
          > and thirdly, that you aren't wasting more of your own time to do it.
          >
          >
          >[/color]
          It's a funny story :-))

          , but in my eyes not appropriate in given context, because my prior goal
          is to understand some more about Python internals, i.e. what is and if
          it is at all a differerence between the internal representation of a
          string and a long integer.

          I know, I should probably look into the C source of Python, but I
          suppose it could be too hard for me to find the appropriate piece of
          code, so I welcome any hints.

          If I knew the internal representation of string and long integer objects
          and were able to read/write to memory and point an identifier at a given
          memory address, a conversion between long integer and string types were
          probably nothing else as changing some bytes and repointing an
          identifier (assuming that string and long integer values are in memory
          the same data if they represent the same value). That it would save CPU
          time is secondary here, but with increasing costs of energy making the
          number on my electrical power bill higher each year due to higher power
          consumption with increasing number of programs I run (it makes a
          difference of 50 Watt between an algorithm keeping the CPU 100% busy and
          an algorithm using only 1% of it) it is not necessarily paranoia driving
          one to consider potential savings of CPU time.
          In this context the example of the bigdec class comes to my mind, where
          usage of another algorithm made it possible to cut down power
          consumption and time of printing a decimal form of the largest known
          prime number from 7 hours of a 100% busy CPU down to 7 seconds!

          Claudio

          Comment

          • bonono@gmail.com

            #20
            Re: python coding contest


            Steven D'Aprano wrote:[color=blue]
            > I'm reminded of a time I was going for a drive in the country when I drove
            > past an apple orchid. Standing in the orchid was a farmer with a pig. He
            > lifted the pig into the air, and the pig then bit an apple and slowly
            > chewed it. The farmer then carried him over to another branch, and the pig
            > ate another apple.
            >
            > I was so surprised I stopped my car and wandered over to ask the farmer
            > what he was doing.
            >
            > "I'm feeding apples to my pig," he replied.
            >
            > "Wouldn't it save time to just pick some apples and feed them to the pig?"
            >
            > The farmer looked at me like I was an idiot. "What's time to a pig?"[/color]

            Has anyone studied if farmers like him are in general healthier ?

            Comment

            • Peter Hansen

              #21
              Re: python coding contest

              bonono@gmail.co m wrote:[color=blue]
              > Steven D'Aprano wrote:
              >[color=green]
              >>I'm reminded of a time I was going for a drive in the country when I drove
              >>past an apple orchid. Standing in the orchid was a farmer with a pig. He
              >>lifted the pig into the air, and the pig then bit an apple and slowly
              >>chewed it. The farmer then carried him over to another branch, and the pig
              >>ate another apple.
              >>
              >>I was so surprised I stopped my car and wandered over to ask the farmer
              >>what he was doing.
              >>
              >>"I'm feeding apples to my pig," he replied.
              >>
              >>"Wouldn't it save time to just pick some apples and feed them to the pig?"
              >>
              >>The farmer looked at me like I was an idiot. "What's time to a pig?"[/color]
              >
              >
              > Has anyone studied if farmers like him are in general healthier ?[/color]

              Yes, they have stronger arms, but some greater incidence of back
              problems as well. Other than that they're basically just as healthy as
              the rest of us who feed our pigs the normal way.

              -Peter

              Comment

              • Claudio Grondi

                #22
                Re: python coding contest

                Claudio Grondi wrote:[color=blue]
                > Steven D'Aprano wrote:
                >[color=green]
                >> On Sun, 01 Jan 2006 15:49:58 +0100, Claudio Grondi wrote:
                >>
                >>
                >>[color=darkred]
                >>> What I have thought about as a simpler/better solution is a method
                >>> allowing to avoid processing the content of the string or long
                >>> integer object by looping over its content.[/color]
                >>
                >>
                >>
                >> How can you avoid looping over its content? Whether you do it yourself
                >> using "for byte in array" or similar, or Python does it for you
                >> (using array.tostring perhaps), *something* has to walk through the
                >> bytes.
                >>
                >> If you don't like walking the string, write a function to do it once, and
                >> then use the function.
                >>
                >>[color=darkred]
                >>> I suppose, that knowing enough about Python internals it must be
                >>> possible to change only the object type not beeing forced to process
                >>> the content i.e. the value itself, what in case of big size of data
                >>> to convert with methods like this above wastes CPU time.[/color]
                >>
                >>
                >>
                >> I'm reminded of a time I was going for a drive in the country when I
                >> drove
                >> past an apple orchid. Standing in the orchid was a farmer with a pig. He
                >> lifted the pig into the air, and the pig then bit an apple and slowly
                >> chewed it. The farmer then carried him over to another branch, and the
                >> pig
                >> ate another apple.
                >>
                >> I was so surprised I stopped my car and wandered over to ask the farmer
                >> what he was doing.
                >>
                >> "I'm feeding apples to my pig," he replied.
                >>
                >> "Wouldn't it save time to just pick some apples and feed them to the
                >> pig?"
                >>
                >> The farmer looked at me like I was an idiot. "What's time to a pig?"
                >>
                >>
                >> The moral of the story is, before spending time working on some scheme to
                >> save CPU time, you better be absolutely sure that firstly, you are going
                >> to save CPU time, secondly, that it is enough CPU time to be worth
                >> saving,
                >> and thirdly, that you aren't wasting more of your own time to do it.
                >>
                >>
                >>[/color]
                > It's a funny story :-))
                >
                > , but in my eyes not appropriate in given context, because my prior goal
                > is to understand some more about Python internals, i.e. what is and if
                > it is at all a differerence between the internal representation of a
                > string and a long integer.
                >
                > I know, I should probably look into the C source of Python, but I
                > suppose it could be too hard for me to find the appropriate piece of
                > code, so I welcome any hints.
                >
                > If I knew the internal representation of string and long integer objects
                > and were able to read/write to memory and point an identifier at a given
                > memory address, a conversion between long integer and string types were
                > probably nothing else as changing some bytes and repointing an
                > identifier (assuming that string and long integer values are in memory
                > the same data if they represent the same value). That it would save CPU
                > time is secondary here, but with increasing costs of energy making the
                > number on my electrical power bill higher each year due to higher power
                > consumption with increasing number of programs I run (it makes a
                > difference of 50 Watt between an algorithm keeping the CPU 100% busy and
                > an algorithm using only 1% of it) it is not necessarily paranoia driving
                > one to consider potential savings of CPU time.
                > In this context the example of the bigdec class comes to my mind, where
                > usage of another algorithm made it possible to cut down power
                > consumption and time of printing a decimal form of the largest known
                > prime number from 7 hours of a 100% busy CPU down to 7 seconds!
                >
                > Claudio[/color]

                From stringobject.h :
                typedef struct {
                PyObject_VAR_HE AD
                long ob_shash;
                int ob_sstate;
                char ob_sval[1];
                /* Invariants:
                * ob_sval contains space for 'ob_size+1' elements.
                * ob_sval[ob_size] == 0.
                * ob_shash is the hash of the string or -1 if not computed yet.
                * ob_sstate != 0 iff the string object is in stringobject.c' s
                * 'interned' dictionary; in this case the two references
                * from 'interned' to this object are *not counted* in ob_refcnt.
                */
                } PyStringObject;

                From longobject.h :
                typedef struct _longobject PyLongObject; /* Revealed in longintrepr.h */
                From longintrepr.h :
                typedef unsigned short digit;
                struct _longobject {
                PyObject_VAR_HE AD
                digit ob_digit[1];
                };

                From this I mean to see, that the string data header is longer
                containing an additional long and an int compared to long integer data
                header. The long integer seem to be an array of unsigned short values
                and the string an array of characters. My MSDN help tells me, that:
                "Type short int (or simply short) is an integral type that is larger
                than or equal to the size of type char, and shorter than or equal to the
                size of type int." In Microsoft Visual C++ short is 2 bytes long, and
                because I am on an Intel processor the sequence of bytes will differ
                from what I will get when creating a binary string out of a long integer
                due to swapping of byte order. Am I right here?

                My conclusion is, that beeing on e.g. Motorola 68000 based processors
                the actual data behind the string and the long integer types will be the
                same, but beeing on an Intel 386 series processor based systems the
                actual data stored in memory behind a string and long integer will differ.

                So finally the idea of converting binary strings forth and back to long
                integers without looping over the data content does for sure not work on
                Intel processor based systems, but may work on processors which don't
                need swapped bytes for integer operations ( I found this page
                http://www.cs.princeton.edu/~kazad/r.../cs/endian.htm gives a good
                overview about the problems with byte swapping, is there any better
                reference related to this subject online? ).

                As Python tries to be multiplatform, I don't understand why it stores
                long integers as an array of shorts and not as an array of chars?
                Historical reasons? Speed reasons? Any other reasons?

                Wouldn't it be more intuitive to have in Python a data type beeing an
                object with binary data which can be easily 'casted' to both string and
                long integer depending how it should be processed or used?
                Do you see any advantage to have such data type, or am I quite alone
                with this idea?

                Claudio

                Comment

                • Michael Spencer

                  #23
                  Re: python coding contest

                  Claudio Grondi wrote:[color=blue]
                  > ...I analysed the outcome of it and have
                  > come to the conclusion, that there were two major factors which
                  > contributed to squeezing of code:
                  >
                  > (1). usage of available variants for coding of the same thing
                  > (2). sqeezing the size of used numeric and string literals
                  >[/color]
                  ....not only squeezing the size of the literals, but the combined size of the
                  compressed data and the code to expand it. In this respect it turned out to be
                  a surprisingly rewarding challenge, and a nice reinforcement of the Pythonic
                  mantra of seeking performance gains by optimizing algorithms.

                  Michael



                  Comment

                  • Claudio Grondi

                    #24
                    Re: python coding contest

                    Michael Spencer wrote:[color=blue]
                    > Claudio Grondi wrote:
                    >[color=green]
                    >> ...I analysed the outcome of it and have
                    >> come to the conclusion, that there were two major factors which
                    >> contributed to squeezing of code:
                    >>
                    >> (1). usage of available variants for coding of the same thing
                    >> (2). sqeezing the size of used numeric and string literals
                    >>[/color]
                    > ....not only squeezing the size of the literals, but the combined size
                    > of the compressed data and the code to expand it. In this respect it
                    > turned out to be a surprisingly rewarding challenge, and a nice
                    > reinforcement of the Pythonic mantra of seeking performance gains by
                    > optimizing algorithms.
                    >
                    > Michael
                    >[/color]
                    You are right. I have overseen that aspect which makes the contest a
                    kind of Python scripting language related compression challenge.

                    I haven't done it yet, but I consider it interesting to check out how
                    compression algorithms are doing on the three lines 7 segment LCD
                    mimicking text representations of a decimal number generated by
                    seven_seg() compared to size of input string plus the size of the
                    module. I suppose, as the input string is not of minimal possible size
                    compared to the information it carries, compression schemes should be
                    able to beat the 'compression' by seven_seg().

                    Interesting in this context for me is also, that if I understand it
                    right, according to Kolmogorov complexity law, it will be never possible
                    to state if the achieved solution is the shortest possible as it will be
                    also not possible for any other provided (shorter) solution.

                    Claudio

                    Comment

                    Working...