Save Binary data.

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • GMane Python
    #1

    Save Binary data.

    Hello All.
    I have a program that downloads 'gigabytes' of Axis NetCam photos per day.
    Right now, I set up the process to put the images into a queue, and every 30
    or so seconds, 'pop' them from the queue and save them to disc. I save
    them as individual files.

    I think that I'd like to modify it to save into one file 100-200 images,
    so that I don't have directories with 50,000-90,000 frames before handing
    that off to a DivX Encoder.

    I don't know if I need to use something like cPickle, or maybe just save
    them as a binary data file (which would be a temp file until later in the
    day when I open it to begin the encoding process.)

    Can someone please help me with some direction?


    Thank you!
    Dave



    Tags: None
    • Larry Bates
      #2
      Re: Save Binary data.

      Images are binary data, don't do anything to them just save them
      to files on disk in their binary format. The extra processing of
      pickling them isn't going to help. Directories with large
      numbers of files was a problem in FAT16 and FAT32 filesystems but
      not really a problem in NTFS or Linux (at least that I've found).
      Appending 100-200 images together into a single file will surely
      cut down on the number of files. A lot depends on what the next
      step in the process is expecting (e.g. individual files or a
      a stream of data). If it is individual files, you will have to
      split them back apart anyway so keeping them as individual files
      is a benefit.

      Larry Bates

      GMane Python wrote:[color=blue]
      > Hello All.
      > I have a program that downloads 'gigabytes' of Axis NetCam photos per day.
      > Right now, I set up the process to put the images into a queue, and every 30
      > or so seconds, 'pop' them from the queue and save them to disc. I save
      > them as individual files.
      >
      > I think that I'd like to modify it to save into one file 100-200 images,
      > so that I don't have directories with 50,000-90,000 frames before handing
      > that off to a DivX Encoder.
      >
      > I don't know if I need to use something like cPickle, or maybe just save
      > them as a binary data file (which would be a temp file until later in the
      > day when I open it to begin the encoding process.)
      >
      > Can someone please help me with some direction?
      >
      >
      > Thank you!
      > Dave
      >
      >
      >[/color]

        Comment

        • Benjamin Niemann
          #3
          Re: Save Binary data.

          GMane Python wrote:
          [color=blue]
          > Hello All.
          > I have a program that downloads 'gigabytes' of Axis NetCam photos per
          > day.
          > Right now, I set up the process to put the images into a queue, and every
          > 30
          > or so seconds, 'pop' them from the queue and save them to disc. I save
          > them as individual files.
          >
          > I think that I'd like to modify it to save into one file 100-200 images,
          > so that I don't have directories with 50,000-90,000 frames before handing
          > that off to a DivX Encoder.
          >
          > I don't know if I need to use something like cPickle, or maybe just save
          > them as a binary data file (which would be a temp file until later in the
          > day when I open it to begin the encoding process.)
          >
          > Can someone please help me with some direction?[/color]

          You could use the tarfile module to create a single file holding a arbitrary
          number of files - including compression, if you want, but if your images
          are JPEGs then further compression is mostly a waste of CPU cycles. You can
          extract the images later on with either a python script and tarfile or
          using the standard commandline tool 'tar'.

          If the images are uncompressed anyway, you could have a look at the netpbm
          suite and its fileformat (which is pretty simple, but uncompressed and
          would bloat JPEGs to a multiple of the original filesize) which supports
          'image sequences'. Perhaps a DivX encoder could even support this
          fileformat directly as input.

          --
          Benjamin Niemann
          Email: pink at odahoda dot de
          WWW: http://www.odahoda.de/

            Comment

            • Mike Meyer
              #4
              Re: Save Binary data.

              Larry Bates <lbates@syscono nline.com> writes:
              [color=blue]
              > Directories with large numbers of files was a problem in FAT16 and
              > FAT32 filesystems but not really a problem in NTFS or Linux (at
              > least that I've found).[/color]

              Depends on how you define "large" and what Linux file system you're
              using. Of course, if you open the directory in a GUI directory
              browser, you're probably going to be unhappy no matter what the
              underlying file system.

              The standard Unix solution to this is to break the files out into
              subdirectories. Create a subdirectory with the name being the first
              few letters of the file name, and then store the file in that
              subdirectory. Easy to do programmaticall y, it's still easy to find
              files "by hand", and you can make it as fine-grained as you want.

              <mike
              --
              Mike Meyer <mwm@mired.or g> http://www.mired.org/home/mwm/
              Independent WWW/Perforce/FreeBSD/Unix consultant, email for more information.

                Comment

                Previous template Next
                Working...