Automating FTP file transfers

[Limey Drink]

Hi all,

Firstly :-) , is there any where I can search through archived newsgroup
posts so I am not cluttering up the newsgroup with repeated queries ?

And secondly :-), I know this has probably been discussed before but.

I am wanting to do some scripting to automate a few administration tasks,
one of the first tasks is automating FTP file transfers.

I would like to know if the following could be made more robust using python
and its FTP libraries rather than executing native OS commands in a shell
script.

Basically I need to...
1. Check on the local system for new files in a specific directory
If new files exist then...
2. Connect to a remote FTP server.
3. Transfer local files to a specific directory on the remote FTP server.
4. Then remove/archive local files and end session.

This is critical operation and though while in the past I have written DOS
scripts etc. to do simple tasks I have not needed to check for errors as
they weren't absolutely critical tasks. I am going to need to run this
overnight using AT or CRON then I need to know about file transfer errors
etc. and other problems that f I were actually performing the FTP transfer
manually I would get feedback of the problem and could re-try the operation.

Any help would be very appreciated.

[Rene Pijlman]

Re: Automating FTP file transfers

Limey Drink:[color=blue]
>is there any where I can search through archived newsgroup posts[/color]

http://groups.google.com

[color=blue]
>I would like to know if the following could be made more robust using python
>and its FTP libraries rather than executing native OS commands in a shell
>script.
>
>Basically I need to... [upload a file][/color]

Sure. Use ftplib. See

http://www.python.org/doc/2.3.2/lib/module-ftplib.html

Here's a code snippet from one of my scripts.

import os, ftplib
ftp = ftplib.FTP(Host name,Username,P assword)
ftp.cwd(Working Directory)
ftp.storbinary( "STOR " + RemoteZipFile, file(LocalZipFi le, "rb"))
ftp.quit()
os.remove(Local ZipFile)

--
René Pijlman

[Matt Goodall]

Re: Automating FTP file transfers

Limey Drink wrote:
[color=blue]
>Hi all,
>
>Firstly :-) , is there any where I can search through archived newsgroup
>posts so I am not cluttering up the newsgroup with repeated queries ?
>
>[/color]
Yep, info on all the python lists, including archives, are available here:

http://python.org/community/lists.html

[color=blue]
>And secondly :-), I know this has probably been discussed before but.
>
>I am wanting to do some scripting to automate a few administration tasks,
>one of the first tasks is automating FTP file transfers.
>
>I would like to know if the following could be made more robust using python
>and its FTP libraries rather than executing native OS commands in a shell
>script.
>
>Basically I need to...
>1. Check on the local system for new files in a specific directory
>If new files exist then...
>2. Connect to a remote FTP server.
>3. Transfer local files to a specific directory on the remote FTP server.
>4. Then remove/archive local files and end session.
>
>[/color]
Most of this should be fairly straightforward . (1) and (4) should be
covered using the os and os.path modules; for (2) and (3) you need the
ftplib module.

The only problem you are likely to encounter is getting a remote
directory listing. The FTP protocol neglected to formalise the response
to FTP's LIST command and different servers send different results back
:(. If this is a problem I would recommend using one of the ftpparse
Python APIs:

* http://effbot.org/downloads/#ftpparse
* http://c0re.23.nu/c0de/ftpparsemodule/ftpparse.html
[color=blue]
>This is critical operation and though while in the past I have written DOS
>scripts etc. to do simple tasks I have not needed to check for errors as
>they weren't absolutely critical tasks. I am going to need to run this
>overnight using AT or CRON then I need to know about file transfer errors
>etc. and other problems that f I were actually performing the FTP transfer
>manually I would get feedback of the problem and could re-try the operation.
>
>[/color]
The ftplib module reports errors using the Python exception mechanism so
you can make it as robust as you need and take whatever action is
appropriate on errors.

Hope this helps.

Cheers, Matt

--
Matt Goodall, Pollenation Internet Ltd
w: http://www.pollenation.net
e: matt@pollenatio n.net

[Limey Drink]

Re: Automating FTP file transfers

Thanks very much for the help guys I now know how to connect using python
ftp library and I am happy with this but...

Just wondering about the bigger picture with regards to sys admin scripts,
in your opinion, is creating these type of system administration scripts
preferable/easier than using say bash scripts, or DOS .bat scripts ?

Because I don't have alot of experience of sys admin and shell scripting I'm
just wondering if I am applying Python for one of its intended uses doing
these type of scripts.

IMHO i would rather learn and stick with Python than to learn all the
different scripting languages as it is cleaner and more powerful and also if
the script is written correctly it seems could be cross platform.

Just wondering if you think I should be trying to execute the in built OS
programs such as FTP etc. calling them from python or should I be using the
libraries that come with Python such as FTPlib and say if I wanted to send
email should I be using the SMTP lib rather than say executing the mail
program in unix and passing arguments from a Python script.

It would hear interesting to hear your views on this as it would be nice if
I can gain from others experience and not make the same mistakes.

Thanks again

"Limey Drink" <bhicking@bhick ing.plus.com> wrote in message
news:HQqsb.7375 $lm1.50918@ward s.force9.net...[color=blue]
> Hi all,
>
> Firstly :-) , is there any where I can search through archived newsgroup
> posts so I am not cluttering up the newsgroup with repeated queries ?
>
> And secondly :-), I know this has probably been discussed before but.
>
> I am wanting to do some scripting to automate a few administration tasks,
> one of the first tasks is automating FTP file transfers.
>
> I would like to know if the following could be made more robust using[/color]
python[color=blue]
> and its FTP libraries rather than executing native OS commands in a shell
> script.
>
> Basically I need to...
> 1. Check on the local system for new files in a specific directory
> If new files exist then...
> 2. Connect to a remote FTP server.
> 3. Transfer local files to a specific directory on the remote FTP server.
> 4. Then remove/archive local files and end session.
>
> This is critical operation and though while in the past I have written DOS
> scripts etc. to do simple tasks I have not needed to check for errors as
> they weren't absolutely critical tasks. I am going to need to run this
> overnight using AT or CRON then I need to know about file transfer errors
> etc. and other problems that f I were actually performing the FTP transfer
> manually I would get feedback of the problem and could re-try the[/color]
operation.[color=blue]
>
> Any help would be very appreciated.
>
>
>[/color]

[Cameron Laird]

Re: Automating FTP file transfers

In article <aptsb.7427$lm1 .51312@wards.fo rce9.net>,
Limey Drink <bhicking@bhick ing.plus.com> wrote:[color=blue]
>Thanks very much for the help guys I now know how to connect using python
>ftp library and I am happy with this but...
>
>Just wondering about the bigger picture with regards to sys admin scripts,
>in your opinion, is creating these type of system administration scripts
>preferable/easier than using say bash scripts, or DOS .bat scripts ?
>
>Because I don't have alot of experience of sys admin and shell scripting I'm
>just wondering if I am applying Python for one of its intended uses doing
>these type of scripts.
>
>IMHO i would rather learn and stick with Python than to learn all the
>different scripting languages as it is cleaner and more powerful and also if
>the script is written correctly it seems could be cross platform.
>
>Just wondering if you think I should be trying to execute the in built OS
>programs such as FTP etc. calling them from python or should I be using the
>libraries that come with Python such as FTPlib and say if I wanted to send
>email should I be using the SMTP lib rather than say executing the mail
>program in unix and passing arguments from a Python script.
>
>It would hear interesting to hear your views on this as it would be nice if
>I can gain from others experience and not make the same mistakes.[/color]

[Rene Pijlman]

Re: Automating FTP file transfers

Limey Drink:[color=blue]
>Just wondering about the bigger picture with regards to sys admin scripts,
>in your opinion, is creating these type of system administration scripts
>preferable/easier than using say bash scripts, or DOS .bat scripts ?[/color]

Yes! You get real data structures, classes, exception handling, powerful
libraries and such.

For example, you can write system administration scripts in Python that
use a relational database, parse XML config files, send e-mail,
up/download with FTP ...
[color=blue]
>Just wondering if you think I should be trying to execute the in built OS
>programs such as FTP etc. calling them from python[/color]

No. Why would you want to do that? Only as a last resort.
[color=blue]
>or should I be using the libraries that come with Python such as FTPlib and
>say if I wanted to send email should I be using the SMTP lib[/color]

Yes. Otherwise you'll loose the exception handling, classes, data
structures etc. :-)

--
René Pijlman

[Limey Drink]

Re: Automating FTP file transfers

OK then, case closed, Python it is :-)

Its just nice to have your gut feeling backed up by other peoples
experience.

Thanks for all the help guys.


"Rene Pijlman" <reply.in.the.n ewsgroup@my.add ress.is.invalid > wrote in
message news:m435rv429a n7b68feg6slv9ih gjmahopam@4ax.c om...[color=blue]
> Limey Drink:[color=green]
> >Just wondering about the bigger picture with regards to sys admin[/color][/color]
scripts,[color=blue][color=green]
> >in your opinion, is creating these type of system administration scripts
> >preferable/easier than using say bash scripts, or DOS .bat scripts ?[/color]
>
> Yes! You get real data structures, classes, exception handling, powerful
> libraries and such.
>
> For example, you can write system administration scripts in Python that
> use a relational database, parse XML config files, send e-mail,
> up/download with FTP ...
>[color=green]
> >Just wondering if you think I should be trying to execute the in built OS
> >programs such as FTP etc. calling them from python[/color]
>
> No. Why would you want to do that? Only as a last resort.
>[color=green]
> >or should I be using the libraries that come with Python such as FTPlib[/color][/color]
and[color=blue][color=green]
> >say if I wanted to send email should I be using the SMTP lib[/color]
>
> Yes. Otherwise you'll loose the exception handling, classes, data
> structures etc. :-)
>
> --
> René Pijlman[/color]

[Jamey Cribbs]

Re: Automating FTP file transfers

Limey Drink wrote:[color=blue]
> Just wondering about the bigger picture with regards to sys admin scripts,
> in your opinion, is creating these type of system administration scripts
> preferable/easier than using say bash scripts, or DOS .bat scripts ?
>
> Just wondering if you think I should be trying to execute the in built OS
> programs such as FTP etc. calling them from python or should I be using the
> libraries that come with Python such as FTPlib and say if I wanted to send
> email should I be using the SMTP lib rather than say executing the mail
> program in unix and passing arguments from a Python script.[/color]

Just to give you my $.02 on this. I have literally dozens of Python ftp
scripts that run on a Linux server at my company daily via cron. All of
my scripts use ftplib and smtplib. My basic script structure is:

1). Check local directory to see if files are available for sending.
2). Encrypt local files via PGP.
3). FTP files to external vendor using ftplib.
4). Send "success" email to myself and other interested parties via
smtplib.
5). If the script error's out instead, send error email to me so I know
there is a problem.

IMO, Python is an excellent choice for this kind of stuff.

Hope this helps!

Jamey