FTP- Check valid upload

[xerc]

Hey all,

So I have a script that runs pretty much every 5 minutes or so to look for updated files locally on a server. New files (ranging in size from 1KB - 2MB) are then FTP'd to another server. Without going into detail, one server is a content server, the other is a 'broadcast' server. Anywho, the problem seems to be with the FTP part. I'd say about 20 percent of the time, regardless of file size, something hangs up, such that the file is never delivered completely from the content server to the broadcast server. What's worse, about 5% the incomplete files get locked up. I've ensured there isn't a time out issue with the script, and I admit the Internet connection between the servers isn't great. (One server is in the US, and the other in S. Africa.) Anyway, I've tried my best, but is there a reliable way to verify a file is completely uploaded and not 'corrupted' or incomplete? When I return '$result' from the fput action, I don't get any errors reported -- even when I can visually see the file is incomplete or locked. The only thing I can think is to download the file after uploading and do a byte-by-byte comparison, but that seems a rather ugly and bandwidth/processor taxing method. Thanks in advance for any help. Alternatively, maybe I shouldn't be using PHP for this -- so other suggestions?

[Motoma]

A small question which might affect how I respond: does the server hang up on the same file each time? That is, are there files that will always produce an error?

Hey all,

So I have a script that runs pretty much every 5 minutes or so to look for updated files locally on a server. New files (ranging in size from 1KB - 2MB) are then FTP'd to another server. Without going into detail, one server is a content server, the other is a 'broadcast' server. Anywho, the problem seems to be with the FTP part. I'd say about 20 percent of the time, regardless of file size, something hangs up, such that the file is never delivered completely from the content server to the broadcast server. What's worse, about 5% the incomplete files get locked up. I've ensured there isn't a time out issue with the script, and I admit the Internet connection between the servers isn't great. (One server is in the US, and the other in S. Africa.) Anyway, I've tried my best, but is there a reliable way to verify a file is completely uploaded and not 'corrupted' or incomplete? When I return '$result' from the fput action, I don't get any errors reported -- even when I can visually see the file is incomplete or locked. The only thing I can think is to download the file after uploading and do a byte-by-byte comparison, but that seems a rather ugly and bandwidth/processor taxing method. Thanks in advance for any help. Alternatively, maybe I shouldn't be using PHP for this -- so other suggestions?

[xerc]

A small question which might affect how I respond: does the server hang up on the same file each time? That is, are there files that will always produce an error?

Nope. Different files. Now, files do repeat, so yes, some files do lock up repeatedly, but it doesn't appear to be specific to certain files -- no pattern. Some of the files that lock up will go without a problem for weeks.

[Motoma]

What method are you using to upload the file to the server? Do you have error reporting or error logging enabled? What messages do you get when it crashes? What about the server; did the FTP server log any upload errors?

Alternatively, it sounds like something you could do by setting up database replication, and storing file directly into the database.

[xerc]

Okay, so turned on the PHP log, and here is error I'm getting:

"PHP Warning: ftp_pasv() expects parameter 1 to be resource, boolean given in"

I've checked that both the sending and receiving servers don't have anything screwy with firewalls or limit on FTP connections. Similarly, the PHP timeout is set crazy high, so that's not the issue. What's weird is that this script use to work just fine a month ago. Said script is:

[PHP]

// set up basic connection
$conn_id_af = ftp_connect($ft p_server_af);

// login with username and password
$login_result_a f = ftp_login($conn _id_af, $ftp_user_name_ af, $ftp_user_pass_ af);

// ensure passive mode off
ftp_pasv($conn_ id_af, false);

// check connection
if ((!$conn_id_af) || (!$login_result _af)) {
print "FTP connection to $ftp_server_af_ af failed!<br>";
print "Attempted to connect to $ftp_server_af for user $ftp_user_name_ af<p>";
exit;
} else {
echo "Connected to $ftp_server_af, for user $ftp_user_name_ af<p>";
}

[/PHP]

I'm not sure it matters, but this is called in several other scripts as:

[PHP]

require "af_ftp_connect .php";

[/PHP]

Would include or something treat the variables differently?

[xerc]

I should add one more thing. I've noticed this does not produce an error, except when left to run as command line. Some setting I might have in the .ini file for how variables are treated differently?

[Motoma]

The ftp_connect() is returning false, meaning that the it did not connect, that is why you are getting that error.

Okay, so turned on the PHP log, and here is error I'm getting:

"PHP Warning: ftp_pasv() expects parameter 1 to be resource, boolean given in"

[xerc]

The ftp_connect() is returning false, meaning that the it did not connect, that is why you are getting that error.

So would it be fair to say that it is connecting some times, but other times the connection is just bad -- meaning my code is okay but the connection sucks? If this is the case, would a timer then be good way to proceed do you think? Meaning it tries to connect, if ftp_connect() is false, then it waits say 15-30secs and tries again. Maybe trying a maximum of 3 times. I'm just asking if this is reasonable in case there is a better way. I like the idea mentioned earlier of replicating a MySQL database, but will have to make that a 6 month goal or something until I can redo all the scripts and have enough access to the remote server. Thanks much Motoma for all the help and advice!!!