Sequential XML parsing with xml.sax

[peter@hardy.dropbear.id.au]

Hi hi.

I'm trying to do sequential decompression of a bzipped XML file and
feed it to a SAX parser with the following code.

remotefh = urllib.urlopen( 'file:///home/peter/catalog.rdf.bz2 ')
decompressor = bz2.BZ2Decompre ssor()
handler = CatalogueDocume ntHandler(sys.s tdout)
chunksize = 2048
data = remotefh.read(c hunksize)
while data != '':
out = decompressor.de compress(data)
if out != '':
xml.sax.parseSt ring(out, handler)
data = remotefh.read(c hunksize)

This fails with the first chunk of decompressed data passed to
xml.sax.parseSt ring. I'm suspecting because it's an incomplete fragment
of XML. I've tried with a number of different chunk sizes, putting the
break in different places, but it always fails on the first call. For
reference, the traceback looks like:

xml.sax.parseSt ring(out, handler)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/__init__.py",
line 47, in parseString
parser.parse(in psrc)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py" ,
line 109, in parse
xmlreader.Incre mentalParser.pa rse(self, source)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/xmlreader.py",
line 125, in parse
self.close()
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py" ,
line 226, in close
self.feed("", isFinal = 1)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/expatreader.py" ,
line 220, in feed
self._err_handl er.fatalError(e xc)
File "/usr/lib/python2.4/site-packages/_xmlplus/sax/handler.py", line
38, in fatalError
raise exception
xml.sax._except ions.SAXParseEx ception: <unknown>:15132 :63: no element
found

(line 15132 is the last, incomplete line feed to parseString. FWIW,
it's:
<pgterms:friend lytitle rdf:parseType=" Literal">Search lights o
)

The API reference isn't clear on whether parseString can only handle
discrete bits of valid XML, or if it's designed to be called in this
way. So I'm not sure if I'm misusing the function, or if I've done
something else wrong.

Any pointers?
Thanks,
--
Pete

[Fredrik Lundh]

Re: Sequential XML parsing with xml.sax

peter@hardy.dro pbear.id.au wrote:
[color=blue]
> The API reference isn't clear on whether parseString can only handle
> discrete bits of valid XML[/color]

the documentation says that "parse" expects an XML document,
and that "parseStrin g" is the same thing, but parses from a buffer.

it's probably easier to pass a BZ2File instance to "parse", but if you
insist on doing incremental SAX parsing, the IncrementalPars er class
might be what you need:

http://www.python.org/doc/current/lib/module-xml.sax.xmlreader.html

http://www.python.org/doc/current/lib/incremental-parser-objects.html

</F>

[peter@hardy.dropbear.id.au]

Re: Sequential XML parsing with xml.sax

Hi.

Fredrik Lundh wrote:[color=blue]
> peter@hardy.dro pbear.id.au wrote:
>[color=green]
> > The API reference isn't clear on whether parseString can only handle
> > discrete bits of valid XML[/color]
>
> the documentation says that "parse" expects an XML document,
> and that "parseStrin g" is the same thing, but parses from a buffer.[/color]

OK, so it sounded a lot more ambiguous at 4am. :-)
[color=blue]
> it's probably easier to pass a BZ2File instance to "parse",[/color]

It is easier to retrieve a remote file, and decompress and parse as
separate steps. But I've been wondering if it would be faster / more
efficient to do it without caching.
[color=blue]
> but if you
> insist on doing incremental SAX parsing, the IncrementalPars er class
> might be what you need:[/color]

That'll do the trick nicely. Thanks.

Cheers,
--
Pete