Script not listing all the files in the dir

[jonathan184]

Hi The script seems to be working ok with a list files to read a dir from but i got a dir with thousands of files and it does not search through all files, when i do a print of the read dir it comes up with less than ahindred. Could some one please help me, not sure which part is wrong in the code. Need it read and search the whole dir

#!/usr/bin/perl
#use strict;
use warnings;
use File::Copy;

$| = 1;

# Contains a list of product numbers
my $prodfile = "/home/prod.txt";

#Contains a bunch of flat files where i would take a product number and check line in each file of that dir and if they have a line containing the prod number it will print to the output file.
my $srcdir = "/home/archive/";
my $finalfile = "/home/output.txt";

# Check for command line arguments
if ($#ARGV != 0) {
	die "Usage: $0 You need to enter <search string> \n";
} else {
	#print "Arg : $#ARGV\t @ARGV \n";
	$search = "ARGV[0]";
}

print "##### Reading Line from File ##### \n\n\n\n";
open(PRODFILE, "< $prodfile") or die (" Could not open Product File $!");
while (my $product = <PRODFILE>) {
	chomp($product);

	##### Opens Dir and Lists files in archive dir #####
	opendir(DIR, $srcdir) or die "Can't open $srcdir: $!";
	while (my $files = readdir(DIR)) {
		print " This is file $files \n";

		if ((-f == "$files") && (("$files")=~ /^$search/)) {
			print " This product is : $product \n";

			### Open file and searching for matching products ###
			open(INFILE2,"< $srcdir/$files") || die(" Could not open INFILE2 File!");
			open(OUTFILE2, ">> $finalfile") || die("Could not OUTFILE2 open file $!");

			# while (my $findprod = <INFILE2>) {
			#	chomp($findprod);
			#	if($product =~ /$findprod/) {
			#		print OUTFILE2 "$findprod \n";
			#		print "******* Match Found: $findprod ********* \n\n"
			#	}

			#### Checking in line in the file and comaring with prod.txt to see if there is a match ######
			while (my $findprod = <INFILE2>) {
				chomp($findprod);
				# print "qq~if ($product =~ /$findprod/)\n~";

				if ($findprod =~ /$product\|/) {
					print "******* Match Found: $findprod ********* \n\n";
					sleep 2;
					print OUTFILE2 "$findprod \n";
				}
			}
		}
	}
}

close(DIR);
close OUTFILE2;
close INFILE2;
close PRODFILE;

[KevinADC]

this line jumped out at me:

if ((-f == "$files") && (("$files")=~ /^$search/)) {

besides having excessive parenthesis , maybe this:

(-f == "$files")

should be:

(-f $files)

the wole line is better written as:

if (-f $files  &&  $files =~ /^$search/) {

you never need to put double-quotes around a single scalar variable and the over use of parenthesis can lead to problems if you don't understand the implications of using them correctly/incorrectly.

[jonathan184]

Interesting The reason i did it like that was I thought i had to separate each condition. Great thanks for the help I will try that?

[jonathan184]

Hi Kevin
ok the strange thing is when I coded the correct way you told me.

if (-f $files  &&  $files =~ /^$search/) {

it did not come up with errors but the script stopped at after listing the files with the print statement i used

print " This product is : $product \n";

so it listed some of the files and went back to the command prompt like the script was done. Nothing came up and it did not print any lines or output file.

Now just to experiment i tried

[CODE]if (-f == $files  &&  $files =~ /^$search/) {

[/CODE]

I got these errors repeating untilt he end of the script :

Argument "PMD555_3Com.20 060806" isn't numeric in numeric eq (==) at compare.pl line 41, <PRODFILE> line 1.
Use of uninitialized value in numeric eq (==) at compare.pl line 41, <PRODFILE> line 1.
This is file mig_sbl_custome r_06844035000.t xt

But it looked like it was searching through the lines which it suppose to. Since it is not reading all the files in the dir to search through its not finding everything because some files are showing up in the readdir loop and not all. Its crazy.

[KevinADC]

do this for a sanity check:

if (-f $files) {

remove the $files =~ /^$search/ condition, then run the code and see if it prints. If it does, you know the regexp is causing the problem.

This is basic debugging, removing some or all conditions and testing them one at a time until you are able to determine which if any is the culprit.

[KevinADC]

I'll check back later, I have some work to do right now :(

[jonathan184]

Ok I tried that but it still stopping at the print statement above the if statement.

do this for a sanity check:

if (-f $files) {

remove the $files =~ /^$search/ condition, then run the code and see if it prints. If it does, you know the regexp is causing the problem.

This is basic debugging, removing some or all conditions and testing them one at a time until you are able to determine which if any is the culprit.

[jonathan184]

For experimental purposes I tried

if($files) {

and it did go through to the rest of the code and did the print statements.

So I guess you are right something was messing it up.

NP Catch you laters when you get some time.

[KevinADC]

this is in your original code, see any problem with it?

$search = "ARGV[0]";

[jonathan184]

yes the quotes but itried it without the quotes and got the same issue.

this is in your original code, see any problem with it?

$search = "ARGV[0]";

[KevinADC]

do you really want $search to literally equal the string ARGV[0] or the value of scalar $ARGV[0]?

You should never double-quote single scalars, "strict" would have picked that up and warned of a bareword on that line and alerted you to the fact you omitted the $ . Double-quoting single scalars is totally unnecessary and only slows down your script because it forces perl to make copy of the scalar that you never use. Double-quotes are a string operator, which allows constructs like this to work:

my $foo = 'foo';
my $bar = 'bar';
my $foobar = "$foo$bar";

[jonathan184]

Sorry for the late reply Kevin.

I guess I learned it as not literally use " " and if it using literally use ' '
Would this be correct?

[miller]

Sorry for the late reply Kevin.

I guess I learned it as not literally use " " and if it using literally use ' '
Would this be correct?

Hi Jonathan,

No that is not correct.

What Kevin was hoping that you would recognize was the missing $ in front of your variable. Currently you're simply assigning the constant "ARGV[0]" to $search, instead of the variable $ARGV[0], which is the first parameter sent at the command line:

my $search = $ARGV[0];

This is why it is bad to enclose variables in quotes, as it keeps "use strict;" from being able to detect the error.

- Miller

[jonathan184]

I see I understand now. Thank you

[KevinADC]

jonathan184

as Miller said, the missing $ was the specific problem. The bigger problem is the use of double-quotes when you should not be using them. Quotes and parenthesis are often over used by new perl coders, often because of poor perl tutorials or picking up bad habits by looking at other peoples bad code. Hopefully you learned a valuable lesson. This is one situation where "strict" was not able to save you from yourself because of the improper use of quotes.