mirroring files and data via http

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Steve Potter
    #1

    mirroring files and data via http

    I'm working on a project to create a central administration interface
    for several websites located on different physical servers.

    You can think of the websites as a blog type application. My
    administration application will be used to create new blog posts with
    associated media (images, etc..)

    So I am thinking to setting up a script on each of the sites to "phone
    home" once per day and download any new posts and media files.

    I am thinking of transmitting the post data in an xml format that
    could then be decoded an recorded into the database on the slave
    site. Are there any easy ways to encode a python dictionary to and
    from xml?

    For the media files I am thinking that the administration interface
    would also provide an xml file with a list of all of the media files
    required along with an http path to retrieve them and a checksum of
    some sort to verify that they were downloaded correctly.

    Basically I am trying to figure out if anything already exists to
    perform some of these functions or if I have to make them from
    scratch. I know none of it should be too hard, but I hate to re-
    invent the wheel.

    Thanks,

    Steven Potter
    Tags: None
    • Matt Nordhoff
      #2
      Re: mirroring files and data via http

      Steve Potter wrote:
      I'm working on a project to create a central administration interface
      for several websites located on different physical servers.
      >
      You can think of the websites as a blog type application. My
      administration application will be used to create new blog posts with
      associated media (images, etc..)
      >
      So I am thinking to setting up a script on each of the sites to "phone
      home" once per day and download any new posts and media files.
      >
      I am thinking of transmitting the post data in an xml format that
      could then be decoded an recorded into the database on the slave
      site. Are there any easy ways to encode a python dictionary to and
      from xml?
      >
      For the media files I am thinking that the administration interface
      would also provide an xml file with a list of all of the media files
      required along with an http path to retrieve them and a checksum of
      some sort to verify that they were downloaded correctly.
      >
      Basically I am trying to figure out if anything already exists to
      perform some of these functions or if I have to make them from
      scratch. I know none of it should be too hard, but I hate to re-
      invent the wheel.
      >
      Thanks,
      >
      Steven Potter
      It sounds like JSON would be perfect for this. For working with it in
      Python, the simplejson module is popular:

      <http://pypi.python.org/pypi/simplejson>
      --

        Comment

        • Steve Potter
          #3
          Re: mirroring files and data via http

          On Jul 6, 8:19 pm, Matt Nordhoff <mnordh...@matt nordhoff.comwro te:
          Steve Potter wrote:
          I'm working on a project to create a central administration interface
          for several websites located on different physical servers.
          >
          You can think of the websites as a blog type application. My
          administration application will be used to create new blog posts with
          associated media (images, etc..)
          >
          So I am thinking to setting up a script on each of the sites to "phone
          home" once per day and download any new posts and media files.
          >
          I am thinking of transmitting the post data in an xml format that
          could then be decoded an recorded into the database on the slave
          site. Are there any easy ways to encode a python dictionary to and
          from xml?
          >
          For the media files I am thinking that the administration interface
          would also provide an xml file with a list of all of the media files
          required along with an http path to retrieve them and a checksum of
          some sort to verify that they were downloaded correctly.
          >
          Basically I am trying to figure out if anything already exists to
          perform some of these functions or if I have to make them from
          scratch. I know none of it should be too hard, but I hate to re-
          invent the wheel.
          >
          Thanks,
          >
          Steven Potter
          >
          It sounds like JSON would be perfect for this. For working with it in
          Python, the simplejson module is popular:
          >
          <http://pypi.python.org/pypi/simplejson>
          --
          Matt,

          You are correct JSON would be much easier to deal with than xml. That
          takes care of several of the problems.

          Now it really just comes down to the file transfer from one server to
          the other and a verification of some sort that the file transfer was
          not corrupt. Is there a python module that takes care of file
          downloads and verification?

          Thanks,

          Steve

            Comment

            • Matt Nordhoff
              #4
              Re: mirroring files and data via http

              Steve Potter wrote:
              On Jul 6, 8:19 pm, Matt Nordhoff <mnordh...@matt nordhoff.comwro te:
              >Steve Potter wrote:
              >>I'm working on a project to create a central administration interface
              >>for several websites located on different physical servers.
              >>You can think of the websites as a blog type application. My
              >>administratio n application will be used to create new blog posts with
              >>associated media (images, etc..)
              >>So I am thinking to setting up a script on each of the sites to "phone
              >>home" once per day and download any new posts and media files.
              >>I am thinking of transmitting the post data in an xml format that
              >>could then be decoded an recorded into the database on the slave
              >>site. Are there any easy ways to encode a python dictionary to and
              >>from xml?
              >>For the media files I am thinking that the administration interface
              >>would also provide an xml file with a list of all of the media files
              >>required along with an http path to retrieve them and a checksum of
              >>some sort to verify that they were downloaded correctly.
              >>Basically I am trying to figure out if anything already exists to
              >>perform some of these functions or if I have to make them from
              >>scratch. I know none of it should be too hard, but I hate to re-
              >>invent the wheel.
              >>Thanks,
              >>Steven Potter
              >It sounds like JSON would be perfect for this. For working with it in
              >Python, the simplejson module is popular:
              >>
              ><http://pypi.python.org/pypi/simplejson>
              >--
              >
              Matt,
              >
              You are correct JSON would be much easier to deal with than xml. That
              takes care of several of the problems.
              >
              Now it really just comes down to the file transfer from one server to
              the other and a verification of some sort that the file transfer was
              not corrupt. Is there a python module that takes care of file
              downloads and verification?
              >
              Thanks,
              >
              Steve
              Not automatically, but this should be easy to do yourself. You could
              generate a simple JSON or plain text file listing URLs and hashes, and
              then use urllib or urllib2 to download them, and md5, sha1 or (in Python
              2.5) hashlib to verify them. All of those modules are in the standard
              library.
              --

                Comment

                Previous template Next
                Working...