urrlib2 multithreading error

[viscanti@gmail.com]

Hi,

I'm using urllib2 to retrieve some data usign http in a multithreaded
application.
Here's a piece of code:
req = urllib2.Request (url, txdata, txheaders)
opener = urllib2.build_o pener()
opener.addheade rs = [('User-agent', user_agent)]
request = opener.open(req )
data = request.read(10 24)

I'm trying to read only the first 1024 bytes to retrieve http headers
(if is html then I will retrieve the entire page).
When I use it on a single thread everything goes ok, when I create
multiple threads the execution halts and the program terminates, just
before the last line (when I execute the request.read(.) ). Obviously I
tried to catch the exception but it doesn't work, the interpreter exits
without any exception or message.
How can I solve this?

lv

[Gabriel Genellina]

Re: urrlib2 multithreading error

At Tuesday 16/1/2007 11:41, viscanti@gmail. com wrote:

>When I use it on a single thread everything goes ok, when I create
>multiple threads the execution halts and the program terminates, just
>before the last line (when I execute the request.read(.) ). Obviously I
>tried to catch the exception but it doesn't work, the interpreter exits
>without any exception or message.

Ouch... Can you reduce your program to the minimum code that fails,
and post it?


--
Gabriel Genellina
Softlab SRL






_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

http://www.yahoo.com.ar/respuestas

[Facundo Batista]

Re: urrlib2 multithreading error

viscanti@gmail. com wrote:

I'm using urllib2 to retrieve some data usign http in a multithreaded
application.
Here's a piece of code:
req = urllib2.Request (url, txdata, txheaders)
opener = urllib2.build_o pener()
opener.addheade rs = [('User-agent', user_agent)]
request = opener.open(req )
data = request.read(10 24)
>
I'm trying to read only the first 1024 bytes to retrieve http headers
(if is html then I will retrieve the entire page).

Why so much bother? You just can create the Request, open it, and ask
for the headers:

>>req = urllib2.Request ("http://www.google.com. ar")
>>u = urllib2.urlopen (req)
>>u.headers["content-type"]

'text/html'

>>>

Take into account that you can add the headers where you put
"txheaders" , it's not necessary to use "addheaders ".

And see that I'm not reading the page at all, urllib2.urlopen just
retrieves the headers...

Regards,

--
.. Facundo
..
Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/