Perl FTP issue

[psutoad]

I am somewhat a Perl Noob, but I've done some programming with my animal books at my side. This issue, however doesn't make sense to me. I am grabbing data via FTP corresponding to data I've already downloaded (this is a pre-processing front end to some other fortran code). However, in the part indicated below (the second FTP execution), I get a login failure, even both ftp sites allow anonymous FTP.

Any thoughts or suggestions on how I might run this down would be much appreciated.

Sincerely,
Todd

Code pasted below
----------------------------

#!/usr/bin/perl

use Net::FTP;

$ftp_username = 'anonymous';
$ftp_passwd = 'BLOCKED - this would normally be my email'

my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;

#initialize arrays with strings

@month_str = ('01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12');
@day_str = ('01', '02', '03', '04', '05', '06', '07', '08', '09', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31');

# set local AMSU data directory

$AMSUdir = 'data/2005';

chdir $AMSUdir or die "cannot change dir to $AMSU_dir: $!";

@AMSUfiles = glob "*.eos";

# loop through each file to grab relevant ancillary data

foreach $AMSU (@AMSUfiles) {

# do not grab more data if this swath is in same day as last swath

if ($AMSU ne $hold_name) {

# capture the year and the doy

if ($AMSU =~ /\w+(\d{4})\.(\d +)\w\d+\w\d+\w\ d+\w+\.eos/) {

$year = $1*1;
$doy = $2;

# convert dayofyear to date

if ($year % 4 eq 0) {
@days_by_month = (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366);
} else {
@days_by_month = (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);
}

$month = 1;
$dd = $2*1;

foreach $dm (@days_by_month ) {
if ($2 <= $dm) {
$day = $dd;
} else {
$dd = $2 - $dm;
$month++;
}

}

$mo_string = $month_str[$month-1];
$day_string = $day_str[$day-1];

# get the SSMI data

$ssmi_dir = "/ssmi/f15/bmaps_v06/y$year/m$mo_string/";
$ssmi_filename = "f15_$year$mo_s tring$day_strin gv6.gz";

$ssmi_site = 'ftp.ssmi.com';
$ftp_obj = Net::FTP -> new ($ssmi_site, Timeout => 20) or
die "Cannot access $ssmi_site via FTP\n";

$ftp_obj -> login($ftp_user name, $ftp_passwd) or
die "Invalid user name and/or password SSMI\n";

$ftp_obj -> binary;

$ftp_obj -> cwd ("$ssmi_dir" ) or
die "Cannot access directory $ssmi_dir\n";

$ftp_obj -> get("$ssmi_file name");

$ftp_obj -> quit;

# get the SST data

$sst_dir = "/sst/daily/tmi_amsre/";
$sst_filename = "tmi_amsre.fusi on.$year.$doy.v 02.gz";

$sst_site = 'ftp.discover-earth.org';
$ftp_obj -> new ($sst_site, Timeout=>20) or
die "Cannot access $sst_site via FTP\n";

# this next line throws out an invalid login error

$ftp_obj -> login($ftp_user name, $ftp_passwd) or
die "Invalid user name and/or password SST\n";

$ftp_obj -> binary;

$ftp_obj -> cwd ("$sst_dir") or
die "Cannot access directory $sst_dir\n";

$ftp_obj -> get("$sst_filen ame");

$ftp_obj -> quit;

}
}

# set this string for next time through

$hold_name = $AMSU;

# write out the relevant data

print "$year $doy $AMSU $ssmi_filename $sst_filename\n ";

}

# make a new data for all data

$data_dir = "data_$thour$tm in_$yday_$yyear ";
mkdir $data_dir, 0777;

# move all the data

foreach my $file (glob "*.eos") {

my $newfile = "$data_dir/$file";
rename $file, $newfile;

}

foreach my $file (glob "*.gz") {

my $newfile = "$data_dir/$file";
rename $file, $newfile;

}

[GunnarH]

# this next line throws out an invalid login error

$ftp_obj -> login($ftp_user name, $ftp_passwd) or
die "Invalid user name and/or password SST\n";

Then ask Perl why it failed. See in the Net::FTP docs (the SYNOPSIS section) how it can be done.

[psutoad]

Then ask Perl why it failed. See in the Net::FTP docs (the SYNOPSIS section) how it can be done.

That's not helpful - nothing in the synopsis tells me anything more than I already know - that I am unable to log in during the second ftp event. Adding the $ftp_obj -> message to this step returns "connection closed in line 83".

Like I said, I'm relatively new to perl, but I've been programming for 20 years...


So is this something about the FTP package? Something with the use of the variables? Something else? I've been trying stuff for 4 hours.

-T

[miller]

psutoad. I would like to help you, but I must first pick on one thing with regard to your perl programming practices. It is very important that you include the line "use strict;" at the beginning of any perl script that you write. This will force you to always declare your variables with "my", and in this way encourage you to think about the scope of your variables, instead of treating everything as global. Additionally, it as the benefit of helping you find spelling errors immediately, of which you currently have 4 in your script.

Take the below code that I spent 10 minutes reformatting for you. There are a lot more ways that I can clean up the code, but I woudl like you to discover the spelling errors on your own so that you can fix them. After you've done that, I will continue trying to help you debug your problem.

#!/usr/bin/perl

use Net::FTP;

use strict;

my $ftp_username = 'anonymous';
my $ftp_passwd   = 'BLOCKED - this would normally be my email';

my ( $tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst ) =
  localtime time;
$yyear += 1900;

# set local AMSU data directory
my $AMSUdir = 'data/2005';

chdir $AMSUdir or die "cannot change dir to $AMSU_dir: $!";

my @AMSUfiles = glob "*.eos";

# loop through each file to grab relevant ancillary data
my $hold_name = '';

foreach my $AMSU (@AMSUfiles) {

	my $year = '';
	my $doy = '';
	my $ssmi_filename = '';
	my $sst_filename = '';

	# do not grab more data if this swath is in same day as last swath

	if ( $AMSU ne $hold_name ) {

		# capture the year and the doy

		if ( $AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/ ) {

			my $year = $1 * 1;
			my $doy  = $2;

			# convert dayofyear to date

			my @days_by_month = ( $year % 4 eq 0 )
				? ( 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366 )
				: ( 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365 );

			my $month = 1;
			my $dd	= $2 * 1;

			my $day;
			foreach my $dm (@days_by_month) {
				if ( $2 <= $dm ) {
					$day = $dd;
				}
				else {
					$dd = $2 - $dm;
					$month++;
				}

			}

			my $mo_string  = sprintf "%02s", $month;
			my $day_string = sprintf "%02s", $day;

			# get the SSMI data

			my $ssmi_dir	  = "/ssmi/f15/bmaps_v06/y$year/m$mo_string/";
			my $ssmi_filename = "f15_$year$mo_string$day_stringv6.gz";

			my $ssmi_site = 'ftp.ssmi.com';
			my $ftp_obj = Net::FTP->new( $ssmi_site, Timeout => 20 )
			  or die "Cannot access $ssmi_site via FTP\n";

			$ftp_obj->login( $ftp_username, $ftp_passwd )
			  or die "Invalid user name and/or password SSMI\n";

			$ftp_obj->binary;

			$ftp_obj->cwd("$ssmi_dir")
			  or die "Cannot access directory $ssmi_dir\n";

			$ftp_obj->get("$ssmi_filename");

			$ftp_obj->quit;

			# get the SST data

			my $sst_dir	  = "/sst/daily/tmi_amsre/";
			my $sst_filename = "tmi_amsre.fusion.$year.$doy.v02.gz";

			my $sst_site = 'ftp.discover-earth.org';
			$ftp_obj->new( $sst_site, Timeout => 20 )
			  or die "Cannot access $sst_site via FTP\n";

			# this next line throws out an invalid login error

			$ftp_obj->login( $ftp_username, $ftp_passwd )
			  or die "Invalid user name and/or password SST\n";

			$ftp_obj->binary;

			$ftp_obj->cwd("$sst_dir")
			  or die "Cannot access directory $sst_dir\n";

			$ftp_obj->get("$sst_filename");

			$ftp_obj->quit;

		}
	}

	# set this string for next time through

	$hold_name = $AMSU;

	# write out the relevant data

	print "$year $doy $AMSU $ssmi_filename $sst_filename\n";

}

# make a new data for all data

my $data_dir = "data_$thour$tmin_$yday_$yyear";
mkdir $data_dir, 0777;

# move all the data

foreach my $file ( glob "*.eos" ) {
	my $newfile = "$data_dir/$file";
	rename $file, $newfile;
}

foreach my $file ( glob "*.gz" ) {
	my $newfile = "$data_dir/$file";
	rename $file, $newfile;
}

Good night for now.

Note. If you want to explicitly specify a variable in a interpolated string, then you should include it in brackets. "my ${var}is crowded". That reads the variable $var, where as "my $varis crowded" reads the varaible $varis.

Just repost the fixed code, and I will continue helping you as soon as I wake up.

[psutoad]

I would like to help you, but I must first pick on one thing with regard to your perl programming practices. It is very important that you include the line "use strict;" at the beginning of any perl script that you write. This will force you to always declare your variables with "my", and in this way encourage you to think about the scope of your variables, instead of treating everything as global. Additionally, it as the benefit of helping you find spelling errors immediately, of which you currently have 4 in your script.

Take the below code that I spent 10 minutes reformatting for you. There are a lot more ways that I can clean up the code, but I woudl like you to discover the spelling errors on your own so that you can fix them. After you've done that, I will continue trying to help you debug your problem.

Wow, I really appreciate not only the effort, but this commenton 'use strict'. In my Perl books they talk about this in later chapters, but it didn't seem necessary to me and I'd never used the strict command in any scripts I've made over the past 10 years. I will use it from now on, though I really don't understand what purpose it serves other than to find spelling errors.. I will read more on this...

So this is my reworked code - I incorported most of your code reformatting, but not all. There's also a slightly different reordering of commands and a slightly different FTP block. But, the problem remains. Whenever I try to open a second ftp connection within the same loop, it fails every time. I've tried several different combinations... I've changed the $ftp_obj variable to a different one for each FTP block, I've tried troubleshooting every variable by printing them between every command and they all look ok.

Again many thanks for the effort so far, it was more than I expected. I just can't get my head around what I'm doing wrong with these FTP calls.. any continued help is greatly appreciated.

#!/usr/bin/perl

use Net::FTP;
use strict;

#### USER INPUT PARAMETERS

my $ftp_passwd = 'my@email.address'  # your email for anonymous FTP

my $AMSUdir = '/home/ellis/work/classwork/AT652/DREAM/test/'; # the directory where your AMSU data is located

########### DO NOT CHANGE ANYTHING BELOW THIS LINE ###################

my $ftp_username = 'anonymous'; # for anonymous ftp

# get current time for naming files and directories

my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;

# initialize output file name

my $out_file_name = "DREAMfiles_${thour}${tmin}_${yday}_${yyear}.dat";
open OUTFILE, ">$out_file_name";
$| = 1; # clear buffer after each output
 
# change to AMSU data directory and work here

chdir $AMSUdir or die "cannot change dir to $AMSUdir: $!";
 
# load in names of all existing AMSU data

my @AMSUfiles = glob "*.eos";

# output number of files (for IDL program)
 
print OUTFILE "$#AMSUfiles\n";
 
# make a new data directory for all data files

my $data_dir = "data_${thour}${tmin}_${yday}_${yyear}";
mkdir $data_dir, 0777;

# output data directory name for IDL program

print OUTFILE "$data_dir\n";
 
# initialize holders

my $hold_doy = 0;
my $hold_year = 0;

# loop through each AMSU file

foreach my $AMSU (@AMSUfiles) {

 my $ssmi_filename = '';
 my $sst_filename = '';
 my $ncep_uwind_name = '';
 my $ncep_vwind_name = '';
 my $year = '';
 my $doy = '';

 # match file name to extract year and day of year

 if ($AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/) {

  my $year = $1 * 1; # converts year to number
  my $doy = $2;

  # compare day of year to last time through - only download data if new day

  if ($doy !=  $hold_doy) {

   # convert dayofyear to date

   my @days_by_month = ($year % 4 eq 0)
    ? (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366)
    : (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);
   
   my $month = 1;
   my $dd = $2*1;

   my $day;

   foreach my $dm (@days_by_month) {
    if ($2 <= $dm) {
     $day = $dd;
    } else {
     $dd = $2 - $dm;
     $month++;
    }

   }
    
   # grab strings corresponding to date of file

   my $mo_string = sprintf "%02s", $month;
   my $day_string = sprintf "%02s", $day;
   
   # get the SSMI data

   my $ssmi_dir = "/ssmi/f15/bmaps_v06/y${year}/m${mo_string}/";
   my $ssmi_filename = "f15_${year}${mo_string}${day_string}v6.gz";
   
   my $ssmi_site = 'ftp.ssmi.com';

   my $ftp_obj = Net::FTP -> new ($ssmi_site, Timeout => 20) or
    die "Cannot access $ssmi_site via FTP\n";

   $ftp_obj -> login($ftp_username, $ftp_passwd) or
    die "Invalid user name and/or password SSMI\n";

   $ftp_obj -> binary;

   $ftp_obj -> cwd ("$ssmi_dir") or
    die "Cannot access directory $ssmi_dir\n";

   $ftp_obj -> get("$ssmi_filename") or die "Get failed $ssmi_filename";

   # get the SST data

   my $sst_dir = "/sst/daily/tmi_amsre/";
   my $sst_filename = "tmi_amsre.fusion.${year}.${doy}.v02.gz";

   $ftp_obj -> binary;

   $ftp_obj -> cwd ("$sst_dir") or
    die "Cannot access directory $sst_dir\n";

   $ftp_obj -> get("$sst_filename") or die "Cannot get SST file\n";

   $ftp_obj -> quit;
   
  }

  # if new year, grab NCEP data

  if ($year != $hold_year) {

   my $ncep_dir = '/Datasets/ncep.reanalysis.dailyavgs/surface/';
   my $ncep_uwind_name = "uwnd.sig995.${year}.nc";
   my $ncep_vwind_name = "vwnd.sig995.${year}.nc";

   my $ncep_site = "ftp.cdc.noaa.gov";

   my $ncep_obj = Net::FTP -> new ($ncep_site, Timeout => 20) or
    die "Cannot access $ncep_site via FTP\n";

   print "$ncep_obj\n";
 
   $ncep_obj -> binary;

   print "$ncep_obj\n";
   print "$ncep_dir\n";

# error is currently at this step - yet I can manually use FTP to access the directory...
   $ncep_obj -> cwd ("$ncep_dir") or die "Cannot access directory $ncep_dir\n";

   $ncep_obj -> get("$ncep_uwind_name") or die "Get failed u-wind\n";
   $ncep_obj -> get("$ncep_vwind_name") or die "Get failed v-wind\n";

   $ncep_obj -> quit;

  }
 } 

 # hold the day of year and year for comparison to next iteration of loop

 $hold_doy = $doy;
 $hold_year = $year;

 # write out the relevant data

 print OUTFILE "$year $doy $AMSU $ssmi_filename $sst_filename $ncep_uwind_name $ncep_vwind_name\n"; 
 
}

# move all data to new data directory

foreach my $file (glob "*.eos") {

 my $newfile = "$data_dir/$file";
 rename $file, $newfile;

}

foreach my $file (glob "*.gz") {

 my $newfile = "$data_dir/$file";
 rename $file, $newfile;

}

foreach my $file (glob "*.nc") {

 my $newfile = "$data_dir/$file";
 rename $file, $newfile;

}

# end of file

Many thanks
-T

[miller]

The main thing that is missing from your script now are the error messages as stated in http://perldoc.perl.or g/Net/FTP.html I went ahead and added them, and immediate was returned the following error:

"Cwd failed Please login with USER and PASS"

This is definitely beats the old error message as far as clarity. I therefore added the login statement and everything appears to work just fine. You'll also notice that I changed the formatting of the code some and discovered a couple other minor bugs. There are a lot of other enhancements that I could make to your script or suggestions of ones that you can make. But I think it's just important that you get it running for now.

The main lesson learned is that take advantage of all the debugging information that perl gives you, whether that is from the use of directives, or error messages provided by modules. Both were very useful in getting your script to a working state.

Happy Thanksgiving.

#!/usr/bin/perl

use Net::FTP;
use strict;

#### USER INPUT PARAMETERS

my $ftp_passwd = 'my@email.address';  # your email for anonymous FTP
# Was BUG: missing semicolon

my $AMSUdir = '/home/ellis/work/classwork/AT652/DREAM/test/'; # the directory where your AMSU data is located

########### DO NOT CHANGE ANYTHING BELOW THIS LINE ###################

my $ftp_username = 'anonymous'; # for anonymous ftp

# get current time for naming files and directories

my ($tsec, $tmin, $thour, $mday, $mmon, $yyear, $wday, $yday, $isdst) = localtime time;
$yyear += 1900;
# REDUNDANT CODE - from http://perldoc.perl.org/functions/localtime.html
# "If EXPR is omitted, localtime() uses the current time (localtime(time))."

# initialize output file name

my $out_file_name = "DREAMfiles_${thour}${tmin}_${yday}_${yyear}.dat";
open OUTFILE, ">$out_file_name";
my $old_fh = select(OUTFILE); $| = 1; select($old_fh); # clear buffer after each output
# Was BUG: Cleared only STDOUT (not OUTFILE) - http://perldoc.perl.org/functions/select.html

# change to AMSU data directory and work here

chdir $AMSUdir or die "cannot change dir to $AMSUdir: $!";

# load in names of all existing AMSU data

my @AMSUfiles = glob "*.eos";

# output number of files (for IDL program)

print OUTFILE "$#AMSUfiles\n";

# make a new data directory for all data files

my $data_dir = "data_${thour}${tmin}_${yday}_${yyear}";
mkdir $data_dir, 0777;

# output data directory name for IDL program

print OUTFILE "$data_dir\n";

# initialize holders

my $hold_doy = 0;
my $hold_year = 0;

# loop through each AMSU file

foreach my $AMSU (@AMSUfiles) {

	my $ssmi_filename = '';
	my $sst_filename = '';
	my $ncep_uwind_name = '';
	my $ncep_vwind_name = '';
	my $year = '';
	my $doy = '';

	# match file name to extract year and day of year

	if ($AMSU =~ /\w+(\d{4})\.(\d+)\w\d+\w\d+\w\d+\w+\.eos/) {

		my $year = $1 * 1; # converts year to number
		my $doy = $2;

		# compare day of year to last time through - only download data if new day

		if ($doy !=  $hold_doy) {

			# convert dayofyear to date

			my @days_by_month = ($year % 4 eq 0)
				? (31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335, 366)
				: (31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334, 365);

			my $month = 1;
			my $dd = $2*1;

			my $day;

			foreach my $dm (@days_by_month) {
				if ($2 <= $dm) {
					$day = $dd;
				} else {
					$dd = $2 - $dm;
					$month++;
				}
			}

			# grab strings corresponding to date of file
			my $mo_string = sprintf "%02s", $month;
			my $day_string = sprintf "%02s", $day;

			my $ssmi_site = 'ftp.ssmi.com';

			my $ftp = Net::FTP->new($ssmi_site, Timeout => 20)
				or die "Cannot connect to $ssmi_site: $@";
			$ftp ->login($ftp_username, $ftp_passwd)
				or die "Login failed " , $ftp->message;
			$ftp->binary;

			# get the SSMI data
			$ftp->cwd("/ssmi/f15/bmaps_v06/y${year}/m${mo_string}/")
				or die "Cwd failed ", $ftp->message;
			$ftp->get("f15_${year}${mo_string}${day_string}v6.gz")
				or die "Get failed ", $ftp->message;

			# get the SST data
			$ftp->cwd("/sst/daily/tmi_amsre/")
				or die "Cwd failed", $ftp->message;
			$ftp->get("tmi_amsre.fusion.${year}.${doy}.v02.gz")
				or die "Get failed ", $ftp->message;

			$ftp->quit;
		}

		# if new year, grab NCEP data

		if ($year != $hold_year) {

			my $ncep_site = "ftp.cdc.noaa.gov";

			my $ftp = Net::FTP->new($ncep_site, Timeout => 20)
				or die "Cannot connect to $ncep_site: $@";
			$ftp->login($ftp_username, $ftp_passwd)
				or die "Login failed " , $ftp->message;
			$ftp->binary;
			$ftp->cwd('/Datasets/ncep.reanalysis.dailyavgs/surface/')
				or die "Cwd failed ", $ftp->message;
			$ftp->get("uwnd.sig995.${year}.nc")
				or die "Get failed ", $ftp->message;
			$ftp->get("vwnd.sig995.${year}.nc")
				or die "Get failed ", $ftp->message;
			$ftp->quit;

		}
	}

	# hold the day of year and year for comparison to next iteration of loop

	$hold_doy = $doy;
	$hold_year = $year;

	# write out the relevant data

	print OUTFILE "$year $doy $AMSU $ssmi_filename $sst_filename $ncep_uwind_name $ncep_vwind_name\n";

}

# move all data to new data directory

foreach my $file (glob "*.eos") {

	my $newfile = "$data_dir/$file";
	rename $file, $newfile;

}

foreach my $file (glob "*.gz") {

	my $newfile = "$data_dir/$file";
	rename $file, $newfile;

}

foreach my $file (glob "*.nc") {

	my $newfile = "$data_dir/$file";
	rename $file, $newfile;

}

# end of file

[psutoad]

I cannot thank you enough, both for your assistance is finding the bugs (yes, that error message was much more helpful), but more importantly for helping me take better advantage of all perl has to offer. I feel like I learned a great deal from you, which is significantly more valuable than just having code that works.

I apologize for my late reply, as I was on travel for the holiday. I hope your holidays are meaningful and enjoyable.

Thanks again,
-T