XML parsing per record

**Irmen de Jong** · Jul 19 '05, 12:04 AM

Re: XML parsing per record

Willem Ligtenberg wrote:[color=blue]
> I want to parse a very large (2.4 gig) XML file (bioinformatics ofcourse :))
> But I have no clue how to do that. Most things I see read the entire xml
> file at once. That isn't going to work here ofcourse.
>
> So I would like to parse a XML file one record at a time and then be able
> to store the information in another object.
> How should I do that?
>
> Thanks in advance,
>
> Willem Ligtenberg
> A total newbie to python by the way.[/color]

Read about SAX parsers.
This may be of help:

Dev Trademarks - Furm

http://www.devarticles.com/c/a/XML/Parsing-XML-with-SAX-and-Python/

Out of curiousity, why is the data stored in a XML file?
XML is not known for its efficiency....

--Irmen

**Ivan Voras** · Jul 19 '05, 12:04 AM

Re: XML parsing per record

Irmen de Jong wrote:
[color=blue]
> XML is not known for its efficiency....[/color]

<sarcasm> Surely you are blaspheming, sir! XML's the greatest thing
since peanut butter! </sarcasm>

I'm just *waiting* for the day someone finds its use on the rolls of
toilet paper... oh the glorious day...

**Kent Johnson** · Jul 19 '05, 12:04 AM

Re: XML parsing per record

Willem Ligtenberg wrote:[color=blue]
> I want to parse a very large (2.4 gig) XML file (bioinformatics ofcourse :))
> But I have no clue how to do that. Most things I see read the entire xml
> file at once. That isn't going to work here ofcourse.
>
> So I would like to parse a XML file one record at a time and then be able
> to store the information in another object.
> How should I do that?[/color]

You might be interested in this recipe using ElementTree:

http://online.effbot.org/2004_12_01_archive.htm#element-generator

Kent

**Fredrik Lundh** · Jul 19 '05, 12:04 AM

Re: XML parsing per record

Kent Johnson wrote:
[color=blue][color=green]
>> So I would like to parse a XML file one record at a time and then be able
>> to store the information in another object.[/color]
>
> You might be interested in this recipe using ElementTree:
> http://online.effbot.org/2004_12_01_...ment-generator[/color]

if you have ElementTree 1.2.5 or later, the "iterparse" function provides a
more efficient implementation of that pattern:

http://effbot.org/zone/element-iterparse.htm

the cElementTree implemention of "iterparse" is a lot faster than SAX; see
the second table under

http://effbot.org/zone/celementtree.htm#benchmarks

for some figures.

</F>

**William Park** · Jul 19 '05, 12:05 AM

Re: XML parsing per record

Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:[color=blue]
> I want to parse a very large (2.4 gig) XML file (bioinformatics
> ofcourse :)) But I have no clue how to do that. Most things I see read
> the entire xml file at once. That isn't going to work here ofcourse.
>
> So I would like to parse a XML file one record at a time and then be
> able to store the information in another object. How should I do
> that?
>
> Thanks in advance,
>
> Willem Ligtenberg A total newbie to python by the way.[/color]

You may want to try Expat (www.libexpat.org) or Python wrapper to it.
You can feed small piece at a time, say by lines or whatever. Of
course, it all depends on what kind of parsing you have in mind. :-)

Care to post more details?

--
William Park <opengeometry@y ahoo.ca>, Toronto, Canada
Slackware Linux -- because it works.

**Fredrik Lundh** · Jul 19 '05, 12:06 AM

Re: XML parsing per record

William Park wrote:
[color=blue]
> You may want to try Expat (www.libexpat.org) or Python wrapper to it.[/color]

Python comes with a low-level expat wrapper (pyexpat).

however, if you want performance, cElementTree (which also uses expat) is a
lot faster than pyexpat. (see my other post for links to benchmarks and code).

</F>

**Willem Ligtenberg** · Jul 19 '05, 12:12 AM

Re: XML parsing per record

On Sun, 17 Apr 2005 02:16:04 +0000, William Park wrote:
[color=blue]
> Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:[color=green]
>> I want to parse a very large (2.4 gig) XML file (bioinformatics
>> ofcourse :)) But I have no clue how to do that. Most things I see read
>> the entire xml file at once. That isn't going to work here ofcourse.
>>
>> So I would like to parse a XML file one record at a time and then be
>> able to store the information in another object. How should I do
>> that?
>>
>> Thanks in advance,
>>
>> Willem Ligtenberg A total newbie to python by the way.[/color]
>
> You may want to try Expat (www.libexpat.org) or Python wrapper to it.
> You can feed small piece at a time, say by lines or whatever. Of
> course, it all depends on what kind of parsing you have in mind. :-)
>
> Care to post more details?[/color]

The XML file I need to parse contains information about genes.
So the first element is a gene and then there are a lot sub-elements with
sub-elements. I only need some of the informtion and want to store it in
my an object called gene. Lateron this information will be printed into a
file, which in it's turn will be fed into some other program.
This is an example of the XML
<?xml version="1.0"?>
<!DOCTYPE Entrezgene-Set PUBLIC "-//NCBI//NCBI Entrezgene/EN" "NCBI_Entrezgen e.dtd">
<Entrezgene-Set>
<Entrezgene>
<Entrezgene_tra ck-info>
<Gene-track>
<Gene-track_geneid>99 96</Gene-track_geneid>
<Gene-track_status value="secondar y">1</Gene-track_status>
<Gene-track_current-id>
<Dbtag>
<Dbtag_db>Locus ID</Dbtag_db>
<Dbtag_tag>
<Object-id>
<Object-id_id>320632</Object-id_id>
</Object-id>
</Dbtag_tag>
</Dbtag>
<Dbtag>
<Dbtag_db>GeneI D</Dbtag_db>
<Dbtag_tag>
<Object-id>
<Object-id_id>320632</Object-id_id>
</Object-id>
</Dbtag_tag>
</Dbtag>
</Gene-track_current-id>
<Gene-track_create-date>
<Date>
<Date_std>
<Date-std>
<Date-std_year>2003</Date-std_year>
<Date-std_month>8</Date-std_month>
<Date-std_day>28</Date-std_day>
<Date-std_hour>21</Date-std_hour>
<Date-std_minute>39</Date-std_minute>
<Date-std_second>0</Date-std_second>
</Date-std>
</Date_std>
</Date>
</Gene-track_create-date>
<Gene-track_update-date>
<Date>
<Date_std>
<Date-std>
<Date-std_year>2005</Date-std_year>
<Date-std_month>2</Date-std_month>
<Date-std_day>17</Date-std_day>
<Date-std_hour>12</Date-std_hour>
<Date-std_minute>54</Date-std_minute>
<Date-std_second>0</Date-std_second>
</Date-std>
</Date_std>
</Date>
</Gene-track_update-date>
</Gene-track>
</Entrezgene_trac k-info>
<Entrezgene_typ e value="protein-coding">6</Entrezgene_type >
<Entrezgene_sou rce>
<BioSource>
<BioSource_geno me value="genomic" >1</BioSource_genom e>
<BioSource_orig in value="natural" >1</BioSource_origi n>
<BioSource_or g>
<Org-ref>
<Org-ref_taxname>Mus musculus</Org-ref_taxname>
<Org-ref_common>hous e mouse</Org-ref_common>
<Org-ref_db>
<Dbtag>
<Dbtag_db>taxon </Dbtag_db>
<Dbtag_tag>
<Object-id>
<Object-id_id>10090</Object-id_id>
</Object-id>
</Dbtag_tag>
</Dbtag>
</Org-ref_db>
<Org-ref_syn>
<Org-ref_syn_E>mouse </Org-ref_syn_E>
</Org-ref_syn>
<Org-ref_orgname>
<OrgName>
<OrgName_name >
<OrgName_name_b inomial>
<BinomialOrgNam e>
<BinomialOrgNam e_genus>Mus</BinomialOrgName _genus>
<BinomialOrgNam e_species>muscu lus</BinomialOrgName _species>
</BinomialOrgName >
</OrgName_name_bi nomial>
</OrgName_name>
<OrgName_lineag e>Eukaryota; Metazoa; Chordata; Craniata; Vertebrata; Euteleostomi; Mammalia; Eutheria; Euarchontoglire s; Glires; Rodentia; Sciurognathi; Muridae; Murinae; Mus</OrgName_lineage >
<OrgName_gcode> 1</OrgName_gcode>
<OrgName_mgcode >2</OrgName_mgcode>
<OrgName_div>RO D</OrgName_div>
</OrgName>
</Org-ref_orgname>
</Org-ref>
</BioSource_org>
</BioSource>
</Entrezgene_sour ce>
<Entrezgene_gen e>
<Gene-ref>
</Gene-ref>
</Entrezgene_gene >
<Entrezgene_gen e-source>
<Gene-source>
<Gene-source_src>Locu sLink</Gene-source_src>
<Gene-source_src-int>9996</Gene-source_src-int>
<Gene-source_src-str2>9996</Gene-source_src-str2>
<Gene-source_gene-display value="false"/>
<Gene-source_locus-display value="false"/>
<Gene-source_extra-terms value="false"/>
</Gene-source>
</Entrezgene_gene-source>
<Entrezgene_loc us>
<Gene-commentary>
<Gene-commentary_type value="genomic" >1</Gene-commentary_type >
<Gene-commentary_vers ion>0</Gene-commentary_vers ion>
</Gene-commentary>
</Entrezgene_locu s>
<Entrezgene_uni que-keys>
<Dbtag>
<Dbtag_db>Locus ID</Dbtag_db>
<Dbtag_tag>
<Object-id>
<Object-id_id>9996</Object-id_id>
</Object-id>
</Dbtag_tag>
</Dbtag>
</Entrezgene_uniq ue-keys>
<Entrezgene_xtr a-index-terms>
<Entrezgene_xtr a-index-terms_E>LOC3206 32</Entrezgene_xtra-index-terms_E>
</Entrezgene_xtra-index-terms>
</Entrezgene>
</Entrezgene-Set>

**Kent Johnson** · Jul 19 '05, 12:12 AM

Re: XML parsing per record

Willem Ligtenberg wrote:[color=blue][color=green]
>>Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:
>>[color=darkred]
>>>I want to parse a very large (2.4 gig) XML file (bioinformatics
>>>ofcourse :)) But I have no clue how to do that. Most things I see read
>>>the entire xml file at once. That isn't going to work here ofcourse.
>>>
>>>So I would like to parse a XML file one record at a time and then be
>>>able to store the information in another object. How should I do
>>>that?[/color][/color]
>
> The XML file I need to parse contains information about genes.
> So the first element is a gene and then there are a lot sub-elements with
> sub-elements. I only need some of the informtion and want to store it in
> my an object called gene. Lateron this information will be printed into a
> file, which in it's turn will be fed into some other program.
> This is an example of the XML
> <?xml version="1.0"?>
> <!DOCTYPE Entrezgene-Set PUBLIC "-//NCBI//NCBI Entrezgene/EN" "NCBI_Entrezgen e.dtd">
> <Entrezgene-Set>
> <Entrezgene>
> <snip>
> </Entrezgene>
> </Entrezgene-Set>[/color]

This should get you started with cElementTree:

import cElementTree as ElementTree

source = 'Entrezgene.xml '

for event, elem in ElementTree.ite rparse(source):
if elem.tag == 'Entrezgene':
# Process the Entrezgene element
geneid = elem.findtext(' Entrezgene_trac k-info/Gene-track/Gene-track_geneid')
print 'Gene id', geneid

# Throw away the element, we're done with it
elem.clear()

Kent

**William Park** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:[color=blue]
> On Sun, 17 Apr 2005 02:16:04 +0000, William Park wrote:[color=green]
> > Care to post more details?[/color]
>
> The XML file I need to parse contains information about genes.
> So the first element is a gene and then there are a lot sub-elements with
> sub-elements. I only need some of the informtion and want to store it in
> my an object called gene. Lateron this information will be printed into a
> file, which in it's turn will be fed into some other program.[/color]

You have to help us a little more here. Which info do you want to
extract from below example?
[color=blue]
> <Entrezgene-Set>
> ...
> </Entrezgene-Set>[/color]

--
William Park <opengeometry@y ahoo.ca>, Toronto, Canada
Slackware Linux -- because it works.

**Willem Ligtenberg** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

This is all the info I need from the xml file:
ID --> <Gene-track_geneid>32 0632</Gene-track_geneid>

Name --> <Gene-ref>
<Gene-ref_locus>Pzp</Gene-ref_locus>

Startbase --> <Gene-commentary_seqs >
<Seq-loc>
<Seq-loc_int>
<Seq-interval>
<Seq-interval_from>1 26957426</Seq-interval_from>
<Seq-interval_to>126 989473</Seq-interval_to>
<Seq-interval_strand >
<Na-strand value="plus"/>
</Seq-interval_strand >
<Seq-interval_id>
<Seq-id>
<Seq-id_gi>51860766</Seq-id_gi>
</Seq-id>
</Seq-interval_id>
</Seq-interval>
</Seq-loc_int>
</Seq-loc>
</Gene-commentary_seqs >
Endbase

Function --> <Prot-ref_name>
<Prot-ref_name_E>U5 snRNP-specific protein, 200 kDa</Prot-ref_name_E>
<Prot-ref_name_E>U5 snRNP-specific protein, 200 kDa (DEXH RNA helicase
family)</Prot-ref_name_E>
</Prot-ref_name>

DBLink --> <Gene-ref_locus-tag>MGI:2444401 </Gene-ref_locus-tag>
<Gene-commentary_sour ce>
<Other-source>
<Other-source_src>
<Dbtag>
<Dbtag_db>GO</Dbtag_db>
<Dbtag_tag>
<Object-id>
<Object-id_id>5524</Object-id_id>
</Object-id>
</Dbtag_tag>
</Dbtag>
</Other-source_src>
<Other-source_anchor>A TP binding</Other-source_anchor>
<Other-source_post-text>evidence: ISS</Other-source_post-text>
</Other-source>
</Gene-commentary_sour ce>

Product-type --> <Entrezgene_typ e value="protein-coding">6</Entrezgene_type >

gene-comment --> <Gene-ref_desc>activa ting signal cointegrator 1 complex subunit 3-like
1</Gene-ref_desc>

synonym --> <Gene-ref_syn>
<Gene-ref_syn_E>HELIC 2</Gene-ref_syn_E>
<Gene-ref_syn_E>KIAA0 788</Gene-ref_syn_E>
<Gene-ref_syn_E>U5-200KD</Gene-ref_syn_E>
<Gene-ref_syn_E>U5-200-KD</Gene-ref_syn_E>
<Gene-ref_syn_E>A3300 64G03Rik</Gene-ref_syn_E>
</Gene-ref_syn>

EC --> <Prot-ref_ec>
<Prot-ref_ec_E>1.5.1. 5</Prot-ref_ec_E>
<Prot-ref_ec_E>3.5.4. 9</Prot-ref_ec_E>
</Prot-ref_ec>

Chromosome: <SubSource>
<SubSource_subt ype value="chromoso me">1</SubSource_subty pe>
<SubSource_name >6</SubSource_name>
</SubSource>

Some can happen more than once in a record.

On Fri, 22 Apr 2005 02:41:46 -0400, William Park wrote:
[color=blue]
> Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:[color=green]
>> On Sun, 17 Apr 2005 02:16:04 +0000, William Park wrote:[color=darkred]
>> > Care to post more details?[/color]
>>
>> The XML file I need to parse contains information about genes.
>> So the first element is a gene and then there are a lot sub-elements with
>> sub-elements. I only need some of the informtion and want to store it in
>> my an object called gene. Lateron this information will be printed into a
>> file, which in it's turn will be fed into some other program.[/color]
>
> You have to help us a little more here. Which info do you want to
> extract from below example?
>[color=green]
>> <Entrezgene-Set>
>> ...
>> </Entrezgene-Set>[/color][/color]

**Willem Ligtenberg** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

As I'm trying to write the code using cElementTree.
I stumble across one problem. Sometimes there are multiple values to
retrieve from one record for the same element. Like this:
<Prot-ref_name_E>ATP-binding cassette, subfamily G, member 1</Prot-ref_name_E>
<Prot-ref_name_E>ATP-binding cassette 8</Prot-ref_name_E>

How do you get not only the first, but the rest as well, so that I can
store it in a list.

Thanks in advance,

Willem Ligtenberg

On Fri, 22 Apr 2005 13:48:15 +0200, Willem Ligtenberg wrote:
[color=blue]
> This is all the info I need from the xml file:
> ID --> <Gene-track_geneid>32 0632</Gene-track_geneid>
>
> Name --> <Gene-ref>
> <Gene-ref_locus>Pzp</Gene-ref_locus>
>
> Startbase --> <Gene-commentary_seqs >
> <Seq-loc>
> <Seq-loc_int>
> <Seq-interval>
> <Seq-interval_from>1 26957426</Seq-interval_from>
> <Seq-interval_to>126 989473</Seq-interval_to>
> <Seq-interval_strand >
> <Na-strand value="plus"/>
> </Seq-interval_strand >
> <Seq-interval_id>
> <Seq-id>
> <Seq-id_gi>51860766</Seq-id_gi>
> </Seq-id>
> </Seq-interval_id>
> </Seq-interval>
> </Seq-loc_int>
> </Seq-loc>
> </Gene-commentary_seqs >
> Endbase
>
> Function --> <Prot-ref_name>
> <Prot-ref_name_E>U5 snRNP-specific protein, 200 kDa</Prot-ref_name_E>
> <Prot-ref_name_E>U5 snRNP-specific protein, 200 kDa (DEXH RNA helicase
> family)</Prot-ref_name_E>
> </Prot-ref_name>
>
> DBLink --> <Gene-ref_locus-tag>MGI:2444401 </Gene-ref_locus-tag>
> <Gene-commentary_sour ce>
> <Other-source>
> <Other-source_src>
> <Dbtag>
> <Dbtag_db>GO</Dbtag_db>
> <Dbtag_tag>
> <Object-id>
> <Object-id_id>5524</Object-id_id>
> </Object-id>
> </Dbtag_tag>
> </Dbtag>
> </Other-source_src>
> <Other-source_anchor>A TP binding</Other-source_anchor>
> <Other-source_post-text>evidence: ISS</Other-source_post-text>
> </Other-source>
> </Gene-commentary_sour ce>
>
> Product-type --> <Entrezgene_typ e value="protein-coding">6</Entrezgene_type >
>
> gene-comment --> <Gene-ref_desc>activa ting signal cointegrator 1 complex subunit 3-like
> 1</Gene-ref_desc>
>
> synonym --> <Gene-ref_syn>
> <Gene-ref_syn_E>HELIC 2</Gene-ref_syn_E>
> <Gene-ref_syn_E>KIAA0 788</Gene-ref_syn_E>
> <Gene-ref_syn_E>U5-200KD</Gene-ref_syn_E>
> <Gene-ref_syn_E>U5-200-KD</Gene-ref_syn_E>
> <Gene-ref_syn_E>A3300 64G03Rik</Gene-ref_syn_E>
> </Gene-ref_syn>
>
> EC --> <Prot-ref_ec>
> <Prot-ref_ec_E>1.5.1. 5</Prot-ref_ec_E>
> <Prot-ref_ec_E>3.5.4. 9</Prot-ref_ec_E>
> </Prot-ref_ec>
>
> Chromosome: <SubSource>
> <SubSource_subt ype value="chromoso me">1</SubSource_subty pe>
> <SubSource_name >6</SubSource_name>
> </SubSource>
>
> Some can happen more than once in a record.
>
>
> On Fri, 22 Apr 2005 02:41:46 -0400, William Park wrote:
>[color=green]
>> Willem Ligtenberg <WLigtenberg@gm ail.com> wrote:[color=darkred]
>>> On Sun, 17 Apr 2005 02:16:04 +0000, William Park wrote:
>>> > Care to post more details?
>>>
>>> The XML file I need to parse contains information about genes.
>>> So the first element is a gene and then there are a lot sub-elements with
>>> sub-elements. I only need some of the informtion and want to store it in
>>> my an object called gene. Lateron this information will be printed into a
>>> file, which in it's turn will be fed into some other program.[/color]
>>
>> You have to help us a little more here. Which info do you want to
>> extract from below example?
>>[color=darkred]
>>> <Entrezgene-Set>
>>> ...
>>> </Entrezgene-Set>[/color][/color][/color]

**Willem Ligtenberg** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

By the way, I know about findall, but when I iterate thruogh it like:
for x in function:
print 'function', x

I get:
function <Element 'Prot-ref_name_E' at 0xb7d10cf8>
function <Element 'Prot-ref_name_E' at 0xb7d10d10>

But ofcourse I want the information in there...

On Fri, 22 Apr 2005 15:22:17 +0200, Willem Ligtenberg wrote:
[color=blue]
> As I'm trying to write the code using cElementTree.
> I stumble across one problem. Sometimes there are multiple values to
> retrieve from one record for the same element. Like this:
> <Prot-ref_name_E>ATP-binding cassette, subfamily G, member 1</Prot-ref_name_E>
> <Prot-ref_name_E>ATP-binding cassette 8</Prot-ref_name_E>
>
> How do you get not only the first, but the rest as well, so that I can
> store it in a list.
>
> Thanks in advance,
>
> Willem Ligtenberg
>[/color]

**Fredrik Lundh** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

Willem Ligtenberg wrote:
[color=blue]
> As I'm trying to write the code using cElementTree.
> I stumble across one problem. Sometimes there are multiple values to
> retrieve from one record for the same element. Like this:
> <Prot-ref_name_E>ATP-binding cassette, subfamily G, member 1</Prot-ref_name_E>
> <Prot-ref_name_E>ATP-binding cassette 8</Prot-ref_name_E>
>
> How do you get not only the first, but the rest as well, so that I can
> store it in a list.[/color]

findall returns a list of matching elements. if "elem" is the paretnt element,
this gives you a list of the text inside all Prot-ref_name_E child elements:

[e.text for e in elem.findall("P rot-ref_name_E")]

(you have read the elementtree documentation, I hope?)

</F>

**Willem Ligtenberg** · Jul 19 '05, 12:26 AM

Re: XML parsing per record

As you can read in the other post of mine, my problem was with the
iterating through the list. didn't know that you should do. e.text. I did
only print e, not print e.text
Did read documentation, but must admit not everything.

Anyway, thank you very much!

On Fri, 22 Apr 2005 15:47:08 +0200, Fredrik Lundh wrote:
[color=blue]
> Willem Ligtenberg wrote:
>[color=green]
>> As I'm trying to write the code using cElementTree.
>> I stumble across one problem. Sometimes there are multiple values to
>> retrieve from one record for the same element. Like this:
>> <Prot-ref_name_E>ATP-binding cassette, subfamily G, member 1</Prot-ref_name_E>
>> <Prot-ref_name_E>ATP-binding cassette 8</Prot-ref_name_E>
>>
>> How do you get not only the first, but the rest as well, so that I can
>> store it in a list.[/color]
>
> findall returns a list of matching elements. if "elem" is the paretnt element,
> this gives you a list of the text inside all Prot-ref_name_E child elements:
>
> [e.text for e in elem.findall("P rot-ref_name_E")]
>
> (you have read the elementtree documentation, I hope?)
>
> </F>[/color]

XML parsing per record

XML parsing per record

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment