Reducing memeory overheads using perl

[tarunjindal]

Hi

I am Tarun Jindal a s/w engineer working in Satyam computers

I would like to ask some question from you.

We at Satyam is developing a software which will built xml for out clients.

We had developed software in Perl but we are experiencing performance hit.

We built the xml tags according to some criteria but we did not take care of capitalization & ordering, as a result of which we were unable to load the xml into the database as the parser did not validate the xml files.

Later we included sort element function & did Capitalization into our code which made the code run very slow as a result we are looking for some other alternative.

We thought of XSLT but when we try to built 150MB or larger file we get a message of “Out of Memory”

I would appreciate if you kindly look in this matter.
Is there any soultion to increase the heap size or some fucntion through which we cal decerease the memory use.



Platform -: Linux

Language-: Perl



I would be very grateful if you kindly assist in this manner & enlighten me with your opinion.

Thanks & Regards
Tarun Jindal
Software Engineer
Satyam Computers Services Limited.
Satyam CyberSpace
Survery No: 12p, Madhapur
Hyderabad - 500081
Mobile No.: +91-9849278383


I am including the sort element code for the referece

This part of code takes 15 to 20 hours to sort & create 2-3 gb of file..
-->> Can we put some unix or perl fucntion to reduce the memory usage & increase the productivity ..
####To reorder the xml elements according to DTD
sub sortElements {
  my $input = shift;
  $input =~ s/\n//ig;

  my $finalXml;
  my $tagVal;
  for (@orderArray){

      my $curr = $_;
      $tagVal = '';
      if($input =~ /$curr/i){
        $input =~ s/(\<$curr.*\>.*\<\/$curr\>|\<$curr\/\>)/$tagVal = $&/ige;

        if($curr =~ /(Features|Generefs|Geneextrefs|Organisms|CommentGene)/i){
          $tagVal =~ s/(\<$1\s*\/\>|\<$1\s*\>\s*\<\/$1\s*\>)//ig;
        }

        for (@childArray) {
          my $child = $_;
          if($child =~ /(LocOp|Qualifiers|Comment)/i){
            $tagVal =~ s/(\<$1\s*\/\>|\<$1\s*\>\s*\<\/$1\s*\>)//ig;
            $tagVal =~ s/locop/LocOp/ig;
          }
        }
        
        if($curr =~ /Feature/){
          $tagVal =~ s/end_pos/endPos/ig;
          $tagVal =~ s/start_pos/startPos/ig;
        }
        if($curr =~ /Generefs|Geneextrefs/){
         $tagVal =~ s/(\<Generef\s*|\<Geneextref\s*)(\w*) href/$1 $2 xlink\:href/ig;
         $tagVal =~ s/(\<Generef\s*|\<Geneextref\s*)(\w*) type/$1 $2 xlink\:type/ig;
        }
      }
      if(length $tagVal){
        $finalXml .= $tagVal;
      }
  }


  return $finalXml;

}

[Banfa]

I have moved this thread to the Perl forum since it is a Perl question