out of memory in Perl Builder

[anklos]

Hi, I am writing a word stemming program, the program reads from a *.trec file to get query words in <title></title> tag, then stems the query words and output the result to another *.trec file. But perl builder promts "out of memory":

use stem;
use WordNet::QueryData;

$wn = WordNet::QueryData->new;
$stemmer = stem->new($wn);


my $topic= "D:\\Program Files\\Perl Builder2\\MYPERL\\TrainingTopics.trec";
my $mytopic="D:\\Program Files\\Perl Builder2\\MYPERL\\myTopics.trec";
open FILE,"$topic" or die $!;
open MYTOPIC,">$mytopic" or die $!;

while(my $file=<FILE>)
{
	if($file=~/<title>(.+?)<\/title>/)
	{
		print MYTOPIC &stemming($file);
		
	}
	else
	{
	   print MYTOPIC $file;
	}							  
}



sub stemming
{
 	my $in = shift;
    my $out = '';

    foreach my $token (split /[ ,.:;?!]/, $in)
    {
	  		
	  	my @stems = $stemmer->stemWord($token);

			 $out .= $stems[0] . ' ';
		 
    }
	 return $out;
}
close FILE;
close MYTOPIC;

Thanks a lot!

[numberwhun]

Well, you aren't slurping the file into an array so the size of the file shouldn't make a difference so much.

It makes me wonder how much system memory you have and how many programs are running on your machine. If that it not it, then you will probably need to take this to the PerlBuilder forums as it would be outside of the realms of Perl coding.

Regards,

Jeff

**Note: are you trying to run this from Perl Builder? If so, why not run the script from the command line instead? I personally never trust a program to run my Perl script. I would rather the interpreter do it directly, without a middle man so to speak. As you can see, they always seem to find a way to cause issues.

Regards,

Jeff

[KevinADC]

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryD ata queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.

[numberwhun]

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)

[Ganon11]

Oh, ok. Sorry, wasn't aware. I tend to hand code everything and run from the command line. If I do use an IDE, I use either Eclipse (with EPIC plugin for Perl) or Komodo. Both are awesome.

Well, now I know. :)

I'm with you, Numberwhun. Notepad++ to edit, cmd.exe to run.

[KevinADC]

Writing simple scripts in a text editor is fine, but using an IDE is essential to any complex code and even not so complex. You can run syntax checks, pipe data in/out of a script (command line or CGI), write and test code on the fly, use the perl debugger in the IDE, step over code, step into code, run timers , pause a script, see and jump to any subroutine in a single glance (may vary IDE to IDE) see all the variables used in a script (may vary IDE to IDE), and much more.

With the IDE you can open multiple scripts and select the one to run and check the output too, all on the fly if you wanted to.

Make your life easier and use an IDE.

In Perl Builder all the builtin perl functions display a basic syntax usage next to the function if you want it to. So you see something like:

split

(this displays in a little window next to the function)
split /PATTERN/,EXPR,LIMIT
split /PATTERN/,EXPR
split /PATTERN/

No need to look at the documentation if you foget the simple usage syntax. That right there is such a big time saver. And if you wanted to read the entire documentation you just click on the little window and a bigger window pops up with the full perl manpage for the function.

[numberwhun]

Wow, didn't mean to kick you up onto your soap box (j/k). I never said I didn't use an IDE. Komodo has always been my editor of choice but have been playing more with Epic (Eclipse) due to its interesting regex tester.

I agree with you that an IDE is a very useful part of Perl coding, but its not something you cannot live without. The software I support at work is all supported without an IDE, using GVim alone. Some of the files that we make additions to are several thousand lines or more in length.

Regards,

Jeff

[Icecrack]

i use Crimson Editor for my Perl Coding + other Languages. (FREE WARE)
i did use Perl Builder

[KevinADC]

Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.

[Ganon11]

Did I forget to mention I work for PerlBuilder? J/K..... I just had too much coffee this morning.

*Dare I ask?* What IDE do you suggest? Is PerlBuilder free?

[anklos]

Jeff.... Perl Builder is an IDE (its the one I use), it uses perl to run your perl programs same as any IDE.

That script looks like it would use very little memory so I don't know why its running out of memeory unless that module you are using has some known issues. WordNet::QueryD ata queries a database, that is possibly where the memory problem is happening. Not sure what to suggest though.

It has no problem when I just convert one word to its stemming words, but when I try to stemming a number of words from a file, the "out of memory" occurs. I think its probably because the wordnet's database 2. I dont know much about how the database works, hope anyone can give me more suggestion.

PS: You guys are friendly and funny, thank you!

[anklos]

Since there is no WordNet 3.0 for windows, I download the WordNet 2.1 and change its name to WordNet 3.0... As I said, there was no problem to convert one word to its semming word.

[anklos]

I find a problem in my program.

if($file=~/<title>(.+?)<\/title>/)

I thinks this sentence shoud pick the content between the tag <title></title>,
not containing the tag itself. But I test my program and I find the tag is contained in my first split word. For example, <title>perl bulider</title>, the first word is "<title>per l" not "perl"

[KevinADC]

The regexp is probably OK:

my $file = "test <title>Perl Builder<\/title> test";
if ($file =~ /<title>(.+?)<\/title>/) {
   print $1;
}

I think the problem is here:

foreach my $token (split /[ ,.:;?!]/, $in)

There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Per l" and "Builder<\/title>".

[anklos]

The regexp is probably OK:

my $file = "test <title>Perl Builder<\/title> test";
if ($file =~ /<title>(.+?)<\/title>/) {
   print $1;
}

I think the problem is here:

foreach my $token (split /[ ,.:;?!]/, $in)

There is nothing in the above regexp (split) to split the string into tokens except a space. So it gets split into "<title>Per l" and "Builder<\/title>".

I see your point, you are right! Thank you very much!!