Shared Memory Modules

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • S Green

    Shared Memory Modules

    Does any one now if a shared memory module exists, written in python
    for a windows platform. I now one exists for unix?


    help most appreciated,

    S Green
  • Peter Hansen

    #2
    Re: Shared Memory Modules

    S Green wrote:[color=blue]
    >
    > Does any one now if a shared memory module exists, written in python
    > for a windows platform. I now one exists for unix?[/color]

    Your first message with this same question did in fact make it to
    Usenet and the mailing list.

    Generally speaking, you should allow several days for answers to questions,
    rather than getting anxious just because ten people don't jump up and answer
    it in the first few hours.

    (And my guess is that one doesn't exist, as Windows shared memory would
    probably be a completely unreliable bitch, but that's just my guess. :-)

    -Peter

    Comment

    • Thomas Heller

      #3
      Re: Shared Memory Modules

      steven.green3@b aesystems.com (S Green) writes:
      [color=blue]
      > Does any one now if a shared memory module exists, written in python
      > for a windows platform. I now one exists for unix?
      >
      >
      > help most appreciated,
      >
      > S Green[/color]

      import mmap

      Comment

      • Peter Hansen

        #4
        Re: Shared Memory Modules

        Thomas Heller wrote:[color=blue]
        >
        > steven.green3@b aesystems.com (S Green) writes:
        >[color=green]
        > > Does any one now if a shared memory module exists, written in python
        > > for a windows platform. I now one exists for unix?
        > >
        > >
        > > help most appreciated,
        > >
        > > S Green[/color]
        >
        > import mmap[/color]

        The docs suggest that mmap.mmap(x, x, mmap.MAP_SHARED ) will work under
        Unix but not Windows.

        Does the Windows version really support shared memory, or is the
        multiple-maps-per-file feature only valid within a single process?

        -Peter

        Comment

        • Thomas Heller

          #5
          Re: Shared Memory Modules

          Peter Hansen <peter@engcorp. com> writes:
          [color=blue]
          > Thomas Heller wrote:[color=green]
          >>
          >> steven.green3@b aesystems.com (S Green) writes:
          >>[color=darkred]
          >> > Does any one now if a shared memory module exists, written in python
          >> > for a windows platform. I now one exists for unix?
          >> >
          >> >
          >> > help most appreciated,
          >> >
          >> > S Green[/color]
          >>
          >> import mmap[/color]
          >
          > The docs suggest that mmap.mmap(x, x, mmap.MAP_SHARED ) will work under
          > Unix but not Windows.
          >
          > Does the Windows version really support shared memory, or is the
          > multiple-maps-per-file feature only valid within a single process?[/color]

          Yes, it does. You have to give this memory block a name, though:

          sharedmem = mmap.mmap(0, 16384, "GlobalSharedMe mory")

          This allows to access 16384 bytes of memory, shared across processes,
          not backed up by a file in the filesystem.

          Thomas

          Comment

          • Peter Hansen

            #6
            Re: Shared Memory Modules

            Thomas Heller wrote:[color=blue]
            >
            > Peter Hansen <peter@engcorp. com> writes:
            >[color=green]
            > > Thomas Heller wrote:[color=darkred]
            > >>
            > >> steven.green3@b aesystems.com (S Green) writes:
            > >>
            > >> > Does any one now if a shared memory module exists, written in python
            > >> > for a windows platform. I now one exists for unix?
            > >> >
            > >> >
            > >> > help most appreciated,
            > >> >
            > >> > S Green
            > >>
            > >> import mmap[/color]
            > >
            > > The docs suggest that mmap.mmap(x, x, mmap.MAP_SHARED ) will work under
            > > Unix but not Windows.
            > >
            > > Does the Windows version really support shared memory, or is the
            > > multiple-maps-per-file feature only valid within a single process?[/color]
            >
            > Yes, it does. You have to give this memory block a name, though:
            >
            > sharedmem = mmap.mmap(0, 16384, "GlobalSharedMe mory")
            >
            > This allows to access 16384 bytes of memory, shared across processes,
            > not backed up by a file in the filesystem.[/color]

            Thanks, Thomas.

            In my opinion the documentation on this is entirely unclear. I attach
            it for reference, but I can't offer any suggestions for improvement (as
            I don't know anything about shared memory) except that even after reading
            it a second time, Thomas' answer above is very much news to me:

            '''tagname, if specified and not None, is a string giving a tag name
            for the mapping. Windows allows you to have many different mappings
            against the same file. If you specify the name of an existing tag,
            that tag is opened, otherwise a new tag of this name is created. If
            this parameter is omitted or None, the mapping is created without a
            name. Avoiding the use of the tag parameter will assist in keeping your
            code portable between Unix and Windows. '''

            (from http://www.python.org/doc/current/lib/module-mmap.html)

            -Peter

            Comment

            • Irmen de Jong

              #7
              Re: Shared Memory Modules

              Thomas Heller wrote:
              [color=blue][color=green]
              >>Does the Windows version really support shared memory, or is the
              >>multiple-maps-per-file feature only valid within a single process?[/color]
              >
              >
              > Yes, it does. You have to give this memory block a name, though:
              >
              > sharedmem = mmap.mmap(0, 16384, "GlobalSharedMe mory")
              >
              > This allows to access 16384 bytes of memory, shared across processes,
              > not backed up by a file in the filesystem.[/color]

              That is très cool, it doesn't tell you this in the docs, does it?
              The first argument is 'the file handle' of the file to be mapped,
              and it doesn't say that 0 is valid and means 'no file at all'...

              However, I've just tried it, and managed to crash Python in mmap.pyd
              with an application exception... twice. But trying to reproduce it
              now fails-- Python keeps running.
              I did this:[color=blue][color=green][color=darkred]
              >>> import mmap
              >>> mem=mmap.mmap(0 ,3000000,'irmen ')
              >>> mem[0]='a'
              >>> mem[2000000]='a'[/color][/color][/color]
              and initially, it crashed........ . Python 2.3.2 on win xp)

              --Irmen

              Comment

              • Thomas Heller

                #8
                Re: Shared Memory Modules

                Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:
                [color=blue]
                > Thomas Heller wrote:
                >[color=green][color=darkred]
                >>> Does the Windows version really support shared memory, or is the
                >>> multiple-maps-per-file feature only valid within a single process?[/color]
                >> Yes, it does. You have to give this memory block a name, though:
                >> sharedmem = mmap.mmap(0, 16384, "GlobalSharedMe mory")
                >> This allows to access 16384 bytes of memory, shared across processes,
                >> not backed up by a file in the filesystem.[/color]
                >
                > That is très cool, it doesn't tell you this in the docs, does it?
                > The first argument is 'the file handle' of the file to be mapped,
                > and it doesn't say that 0 is valid and means 'no file at all'...
                >
                > However, I've just tried it, and managed to crash Python in mmap.pyd
                > with an application exception... twice. But trying to reproduce it
                > now fails-- Python keeps running.
                > I did this:[color=green][color=darkred]
                > >>> import mmap
                > >>> mem=mmap.mmap(0 ,3000000,'irmen ')
                > >>> mem[0]='a'
                > >>> mem[2000000]='a'[/color][/color]
                > and initially, it crashed........ . Python 2.3.2 on win xp)[/color]

                This works for me even from the beginning (with 2.3 CVS version, XP Pro).

                Understading which parameters to pass apparently requires

                - reading the mmapmodule.c sources
                - and reading about CreateFileMappi ng and MapViewOfFile in MSDN.

                It would be great if someine could submit a patch for the docs <wink>.

                I vaguely remember, but am not able to find it anymore: didn't AMK once
                had an article about this somewhere?

                Thomas

                Comment

                • Scott David Daniels

                  #9
                  Re: Shared Memory Modules

                  Irmen de Jong wrote:[color=blue]
                  > However, I've just tried it, and managed to crash Python in mmap.pyd
                  > with an application exception... twice. But trying to reproduce it
                  > now fails-- Python keeps running.
                  > I did this:[color=green][color=darkred]
                  > >>> import mmap
                  > >>> mem=mmap.mmap(0 ,3000000,'irmen ')
                  > >>> mem[0]='a'
                  > >>> mem[2000000]='a'[/color][/color]
                  > and initially, it crashed........ . Python 2.3.2 on win xp)
                  > --Irmen[/color]

                  Probably not enough actual memory hanging around. Some systems
                  (and from this I'd guess XP) allocate virtual memory by reserving
                  address space, not actually allocating the RAM and/or backing
                  store for that memory. Python has no control over this, and you
                  have nothing good to do if the memory is over-allocated. When
                  you create the memory, you could walk across it writing into it
                  (forcing it to exist), but that would just force the failure to
                  happen earlier.


                  -Scott David Daniels
                  Scott.Daniels@A cm.Org

                  Comment

                  • Irmen de Jong

                    #10
                    Re: Shared Memory Modules

                    Scott David Daniels wrote:[color=blue]
                    > Irmen de Jong wrote:
                    >[color=green]
                    >> However, I've just tried it, and managed to crash Python in mmap.pyd
                    >> with an application exception... twice. But trying to reproduce it
                    >> now fails-- Python keeps running.
                    >> I did this:[color=darkred]
                    >> >>> import mmap
                    >> >>> mem=mmap.mmap(0 ,3000000,'irmen ')
                    >> >>> mem[0]='a'
                    >> >>> mem[2000000]='a'[/color]
                    >> and initially, it crashed........ . Python 2.3.2 on win xp)
                    >> --Irmen[/color]
                    >
                    >
                    > Probably not enough actual memory hanging around. Some systems
                    > (and from this I'd guess XP) allocate virtual memory by reserving
                    > address space, not actually allocating the RAM and/or backing
                    > store for that memory.[/color]

                    Over-commit is that called, right?
                    I'm sorry but that certainly wasn't the case here.
                    My machine has 512 Mb RAM and about 250 Mb of them
                    allocated when I tried it. (physical RAM that is)

                    Very weird, I cannot reproduce the initial crash I experienced.

                    --Irmen

                    Comment

                    • Bengt Richter

                      #11
                      Re: Shared Memory Modules

                      On Tue, 02 Dec 2003 00:44:15 +0100, Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> wrote:
                      [color=blue]
                      >Scott David Daniels wrote:[color=green]
                      >> Irmen de Jong wrote:
                      >>[color=darkred]
                      >>> However, I've just tried it, and managed to crash Python in mmap.pyd
                      >>> with an application exception... twice. But trying to reproduce it
                      >>> now fails-- Python keeps running.
                      >>> I did this:
                      >>> >>> import mmap
                      >>> >>> mem=mmap.mmap(0 ,3000000,'irmen ')
                      >>> >>> mem[0]='a'
                      >>> >>> mem[2000000]='a'
                      >>> and initially, it crashed........ . Python 2.3.2 on win xp)
                      >>> --Irmen[/color]
                      >>
                      >>
                      >> Probably not enough actual memory hanging around. Some systems
                      >> (and from this I'd guess XP) allocate virtual memory by reserving
                      >> address space, not actually allocating the RAM and/or backing
                      >> store for that memory.[/color]
                      >
                      >Over-commit is that called, right?
                      >I'm sorry but that certainly wasn't the case here.
                      >My machine has 512 Mb RAM and about 250 Mb of them
                      >allocated when I tried it. (physical RAM that is)
                      >
                      >Very weird, I cannot reproduce the initial crash I experienced.
                      >[/color]
                      What's that first zero argument?

                      Snip from mmap docs:
                      """
                      mmap( fileno, length[, tagname[, access]])

                      (Windows version) Maps length bytes from the file specified by
                      the file handle fileno, and returns a mmap object. If length is 0,
                      the maximum length of the map will be the current size of the file
                      when mmap() is called.

                      tagname, if specified and not None, is a string giving a tag name
                      for the mapping. Windows allows you to have many different mappings
                      against the same file. If you specify the name of an existing tag,
                      that tag is opened, otherwise a new tag of this name is created.
                      If this parameter is omitted or None, the mapping is created without a name.
                      Avoiding the use of the tag parameter will assist in keeping your code portable
                      between Unix and Windows.
                      """

                      Does that mean you are trying to use stdin as the open file handle?

                      Regards,
                      Bengt Richter

                      Comment

                      • Irmen de Jong

                        #12
                        Re: Shared Memory Modules

                        Bengt Richter wrote:[color=blue]
                        > What's that first zero argument?[/color]
                        [...][color=blue]
                        > Does that mean you are trying to use stdin as the open file handle?[/color]

                        No it doesn't, according to Thomas Heller's post in this thread.
                        Thomas said 0 means "not associated with a file" on windows...

                        --Irmen

                        Comment

                        • Thomas Heller

                          #13
                          Re: Shared Memory Modules

                          Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:
                          [color=blue]
                          > Bengt Richter wrote:[color=green]
                          >> What's that first zero argument?[/color]
                          > [...][color=green]
                          >> Does that mean you are trying to use stdin as the open file handle?[/color]
                          >
                          > No it doesn't, according to Thomas Heller's post in this thread.
                          > Thomas said 0 means "not associated with a file" on windows...[/color]

                          mmapmodule.c internally uses INVALID_HANDLE_ VALUE of a file handle of 0
                          is used.

                          As I said, read the source (and MSDN). And then submit a documentation
                          patch <wink>.

                          Thomas

                          Comment

                          • Bengt Richter

                            #14
                            Re: Shared Memory Modules

                            On Tue, 02 Dec 2003 10:54:19 +0100, Thomas Heller <theller@python .net> wrote:
                            [color=blue]
                            >Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:
                            >[color=green]
                            >> Bengt Richter wrote:[color=darkred]
                            >>> What's that first zero argument?[/color]
                            >> [...][color=darkred]
                            >>> Does that mean you are trying to use stdin as the open file handle?[/color]
                            >>
                            >> No it doesn't, according to Thomas Heller's post in this thread.
                            >> Thomas said 0 means "not associated with a file" on windows...[/color]
                            >
                            >mmapmodule.c internally uses INVALID_HANDLE_ VALUE of a file handle of 0
                            >is used.[/color]
                            Wouldn't passing None be more pythonic for a python interface?
                            [color=blue]
                            >
                            >As I said, read the source (and MSDN). And then submit a documentation
                            >patch <wink>.
                            >[/color]
                            How does one do that? (being lazy, I guess I could find out via www.python.org)

                            Regards,
                            Bengt Richter

                            Comment

                            • Thomas Heller

                              #15
                              Re: Shared Memory Modules

                              bokr@oz.net (Bengt Richter) writes:
                              [color=blue]
                              > On Tue, 02 Dec 2003 10:54:19 +0100, Thomas Heller <theller@python .net> wrote:
                              >[color=green]
                              >>Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> writes:
                              >>[color=darkred]
                              >>> Bengt Richter wrote:
                              >>>> What's that first zero argument?
                              >>> [...]
                              >>>> Does that mean you are trying to use stdin as the open file handle?
                              >>>
                              >>> No it doesn't, according to Thomas Heller's post in this thread.
                              >>> Thomas said 0 means "not associated with a file" on windows...[/color]
                              >>
                              >>mmapmodule. c internally uses INVALID_HANDLE_ VALUE of a file handle of 0
                              >>is used.[/color]
                              > Wouldn't passing None be more pythonic for a python interface?
                              >[color=green]
                              >>
                              >>As I said, read the source (and MSDN). And then submit a documentation
                              >>patch <wink>.
                              >>[/color]
                              > How does one do that? (being lazy, I guess I could find out via www.python.org)[/color]

                              Finding out how to do this is the easier part ;-)

                              Ok, to get you started:

                              <http://cvs.sourceforge .net/viewcvs.py/python/python/dist/src/Modules/mmapmodule.c>
                              <http://msdn.microsoft. com/library/en-us/fileio/base/creating_named_ shared_memory.a sp>
                              <http://msdn.microsoft. com/library/en-us/fileio/base/createfilemappi ng.asp>
                              <http://msdn.microsoft. com/library/en-us/fileio/base/mapviewoffile.a sp>

                              Thomas

                              Comment

                              Working...