Reading 5MB xml with python

Collapse
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • realin
    Contributor
    • Feb 2007
    • 254
    #1

    Reading 5MB xml with python

    hiya all,

    However, the title says it all but still i want to explain it further. I am trying to fetch a xml file which is lying at different size. The file size is like >5MB. So as soon as i write my code, the browser keeps on waiting for the response and in the end it freezes or displays time out.
    What can be the work around for this ?
    I am using google's app engine, so not exactly core python. And i would like to mention that i started learning python just half an hour back, i am a programmer of php, ruby on rails and little bit of java .. so its app engine is more like servlet has get/post methods..

    Please help me with it, is it possible using threads ? spliting the file or what ?

    Thanks & Regards
    Realin !
    Tags: None
    • Laharl
      Recognized Expert Contributor
      • Sep 2007
      • 849
      #2
      Which XML API are you using, DOM or SAX? For large files (on the order of what you're dealing with), I believe SAX is the better choice.

      You could use a thread to read through the file in the background, but it's going to be reasonably slow no matter what...

        Comment

        • realin
          Contributor
          • Feb 2007
          • 254
          #3
          Originally posted by Laharl
          Which XML API are you using, DOM or SAX? For large files (on the order of what you're dealing with), I believe SAX is the better choice.

          You could use a thread to read through the file in the background, but it's going to be reasonably slow no matter what...
          hiya laharl,

          thanks for the quick reply.. I am using DOM, i will give a try to SAX. Time is no constraint, so what do you suggest now ?

          I am not a die hard fan of threads though, i want my code to be much more simpler, if you think SAX can do it without having to use thread, then i would be more than happy :)

          waiting for ur reply
          thanks :)

            Comment

            • realin
              Contributor
              • Feb 2007
              • 254
              #4
              hiya all,

              I m unable to get a code to fetch XML from a URL link using SAX.
              Can anyone please help me in that .. its really urgent

              please

                Comment

                • Laharl
                  Recognized Expert Contributor
                  • Sep 2007
                  • 849
                  #5
                  I've never actually used SAX (only xml.dom.minidom ), but were I to guess, you'd use something from url or some other networking library to get the XML file from the server and then SAX to process it.

                    Comment

                    • realin
                      Contributor
                      • Feb 2007
                      • 254
                      #6
                      Originally posted by Laharl
                      I've never actually used SAX (only xml.dom.minidom ), but were I to guess, you'd use something from url or some other networking library to get the XML file from the server and then SAX to process it.
                      hiya

                      sorry for late update, was outta town actually. I had to use urllib for doing the same, but google app engine wont allow me to download some bulk size files, so i had to drop the idea of importing data thru XML.

                      However, i found a nice utility called bulkuploader using CSV files. I converted XMLs into CSV using php and then using bulkuploader utility, i just pushed the data in Datastore.

                      thanks for your help, its always good to see you guys coming up and helping :)

                      cheer!!

                        Comment

                        Previous template Next
                        Working...