elementtree: line numbers and iterparse

[Stuart McGraw]

I have a broad (~200K nodes) but shallow xml file
I want to parse with Elementtree. There are too many
nodes to read into memory simultaneously so I use
iterparse() to process each node sequentially.

Now I find i need to get and save the input file line
number of each node. Googling turned up a way
to do it by subclassing FancyTreeBuilde r,
(http://groups.google.com/group/comp....9553b4b?hl=en&)
but that tries to read everything at once.

Is there a way to do something similiar with iterparse()?

[Fredrik Lundh]

Re: elementtree: line numbers and iterparse

Stuart McGraw wrote:

I have a broad (~200K nodes) but shallow xml file
I want to parse with Elementtree. There are too many
nodes to read into memory simultaneously so I use
iterparse() to process each node sequentially.
>
Now I find i need to get and save the input file line
number of each node. Googling turned up a way
to do it by subclassing FancyTreeBuilde r,
(http://groups.google.com/group/comp....9553b4b?hl=en&)
but that tries to read everything at once.
>
Is there a way to do something similiar with iterparse()?

something like this could work:

import elementtree.Ele mentTree as ET
import StringIO

data = """\
<doc>
<tag>
<subtag>text</subtag>
<subtag>text</subtag>
</tag>
</doc>
"""

class FileWrapper:
def __init__(self, source):
self.source = source
self.lineno = 0
def read(self, bytes):
s = self.source.rea dline()
self.lineno += 1
return s

# f = FileWrapper(ope n("source.xml ")
f = FileWrapper(Str ingIO.StringIO( data))

for event, elem in ET.iterparse(f, events=["start", "end"]):
if event == "start":
print f.lineno, event, elem

</F>

[Stuart McGraw]

Re: elementtree: line numbers and iterparse


"Fredrik Lundh" <fredrik@python ware.comwrote in message news:mailman.1. 1158124100.1049 1.python-list@python.org ...

Stuart McGraw wrote:

Now I find i need to get and save the input file line
number of each node. Googling turned up a way
to do it by subclassing FancyTreeBuilde r,
(http://groups.google.com/group/comp....9553b4b?hl=en&)
but that tries to read everything at once.

Is there a way to do something similiar with iterparse()?

>
something like this could work:
...snip...

Indeed it does. Many thanks!