Multi-threaded FTP Question

**Jeremy Jones** · Jul 11 '06, 06:55 PM

Re: Multi-threaded FTP Question

dbandler@gmail. com wrote:

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:
>
File "C:\Python24\li b\ftplib.py", line 107, in __init__
self.connect(ho st)
File "C:\Python24\li b\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\li b\ftplib.py", line 208, in getresp
resp = self.getmultili ne()
File "C:\Python24\li b\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\li b\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError
>
Is it possible to have more than five simultaneous ftp connections?
>
Thanks.
>
Derek

I replied to this about 4 hours ago from my gmail email account (not my
google groups account associated with the same email addres), but
haven't seen it show up, so I apologize if this is a dupe.

Would you mind posting your code? Are you trying to pass the same FTP
connection object to all 5 threads?

- Jeremy M. Jones

**dbandler@gmail.com** · Jul 11 '06, 08:05 PM

Re: Multi-threaded FTP Question

There are n instances of ftplib.FTP.

Funny thing is that I tried to run the code twice, concurrently, in two
separate IDLE instances. Each instance had four threads/ftp calls.
Again, only five ftp connections were allowed on my computer. The code
errored out on the sixth attempt.

I wonder if there' s some underlying software being called that has a
limit on the number of ftp connections allowed at any point in time.
I'm tempted to try this on cygwin to see if the behavior is different.

Derek

import ftplib, zipfile, datetime, os
from threading import Thread, Lock

class ftpDownload(Thr ead):
def __init__(self, year, quarter):
Thread.__init__ (self)
self.quarter = str(quarter)
self.year = str(year)
self.filename = ""
self.ftp = ftplib.FTP('xxx xxxx')
self.ftp.login( )
self.lock = Lock()

def run(self):
filehandle = open(self.filen ame,'wb')
self.ftp.retrbi nary('RETR /xxxxx/xxxxxxx/' + self.year + '/QTR'
+ self.quarter + '/xxxxxxx.zip', filehandle.writ e)
filehandle.clos e()
self.ftp.close( )

ftplist = []
for year in range(1990,2000 ):
for quarter in range(1,5):
current = ftpDownload(yea r, quarter)
ftplist.append( current)
current.start()

for job in ftplist:
job.join()

Jeremy Jones wrote:

dbandler@gmail. com wrote:

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:

File "C:\Python24\li b\ftplib.py", line 107, in __init__
self.connect(ho st)
File "C:\Python24\li b\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\li b\ftplib.py", line 208, in getresp
resp = self.getmultili ne()
File "C:\Python24\li b\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\li b\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError

Is it possible to have more than five simultaneous ftp connections?

Thanks.

Derek

>
I replied to this about 4 hours ago from my gmail email account (not my
google groups account associated with the same email addres), but
haven't seen it show up, so I apologize if this is a dupe.
>
Would you mind posting your code? Are you trying to pass the same FTP
connection object to all 5 threads?
>
- Jeremy M. Jones

**Jeremy Jones** · Jul 12 '06, 12:55 PM

Re: Multi-threaded FTP Question

Dennis Lee Bieber wrote:

On 11 Jul 2006 06:45:42 -0700, dbandler@gmail. com declaimed the
following in comp.lang.pytho n:
>
Could it be that the SERVER is limiting things to 5
concurrent/parallel connections from any single IP?
>
I know I've encountered sites that only allowed two FTP downloads at
a time...

This is what I was starting to think as well. The only thing that
looked funky with the OP's code was that it looked like he was writing
everything to a filename of "" (unless he's intentionally modified his
code to not show where he's setting that).

- Jeremy M. Jones

**olsongt@verizon.net** · Jul 12 '06, 01:05 PM

Re: Multi-threaded FTP Question

dbandler@gmail. com wrote:

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:
>
File "C:\Python24\li b\ftplib.py", line 107, in __init__
self.connect(ho st)
File "C:\Python24\li b\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\li b\ftplib.py", line 208, in getresp
resp = self.getmultili ne()
File "C:\Python24\li b\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\li b\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError
>
Is it possible to have more than five simultaneous ftp connections?
>
Thanks.
>
Derek

It might be XP SP2's worm protection as well:

Windows XP SP2 tcpip.sys connection limit patch

http://www.speedguide.net/read_articles.php?id=1497

In addition to the tweaks already covered in Win 2k/XP Registry Tweaks and More Win 2k/XP Tweaks, the Windows XP Service Pack 2 introduces a few new issues...

**dbandler@gmail.com** · Jul 12 '06, 04:15 PM

Re: Multi-threaded FTP Question

Thanks so much for your help on this. The server that I'm connecting
to is the culprit. They only allow five connections at a time.

I assumed that it was a code issue. I think that we're conditioned to
expect that the problem is on the software side of things.

-Derek

olsongt@verizon .net wrote:

dbandler@gmail. com wrote:

I'm trying to use ftp in python in a multi-threaded way on a windows
box - python version 2.4.3. Problem is that it appears that it's only
possible to have five instances/threads at one point in time. Errors
look like:

File "C:\Python24\li b\ftplib.py", line 107, in __init__
self.connect(ho st)
File "C:\Python24\li b\ftplib.py", line 132, in connect
self.welcome = self.getresp()
File "C:\Python24\li b\ftplib.py", line 208, in getresp
resp = self.getmultili ne()
File "C:\Python24\li b\ftplib.py", line 194, in getmultiline
line = self.getline()
File "C:\Python24\li b\ftplib.py", line 184, in getline
if not line: raise EOFError
EOFError

Is it possible to have more than five simultaneous ftp connections?

Thanks.

Derek

>
It might be XP SP2's worm protection as well:
>
http://www.speedguide.net/read_articles.php?id=1497

Multi-threaded FTP Question

Multi-threaded FTP Question

Comment

Comment

Comment

Comment

Comment