Multiple threads/connections in cURL

[adamalton]

Hi, I am using cURL with php to submit a form to a website and then retrieve the page that comes back. That part is easy. The problem is that the site I am posting the request to displays a 'please_wait' page which then redirects you to the page that you actually want to see. I can deal with this fine if I am only sending one request, but the trouble is that I am sending lots of requests (i.e. I am posting the form several times with different form data in each request). If I do this one after the other then I have to post the first request, then wait a couple of seconds and then request the page with the info on it. Then post the next request, wait a few seconds, get the page. Send the next.......ZZZz zzzz

So I need to be able to do them all at once...which cURL can do. But the trouble is that the 'please wait' page redirects you using javascript, so cURL can't automatically pick this up i.e. the CURLOPT_FOLLOWL OCATION doesn't work. The only way to do it is to post the request, and get the 'please wait' page, and then send a separate request for the page with the info on it.

And here's the snag: Posting the request and getting the info page all has to be done on the same connection, otherwise the website that I'm getting the page from doesn't know which page to send me!!

So I need to run several simultaneous cURL connections, then wait a couple of seconds and then USING THE SAME CONNECTIONS send 'GET' requests for the resulting pages.

How??

[adamalton]

I think it might be possible to write a script that posts the request and then requests the resulting page. And then get cURL to access that script on my server lots of times all at once. But I can't work out how to do this.

[adamalton]

The answer: There's no need to keep the same connection.

HTTP is 'stateless' meaning that a server has no way of tracking which user is which just by keeping the connection. The only way a server can determine which user is which (or which request is which) is by putting a cookie onto the user's computer to track who they are. (Or a user could be tracked using hidden form fields.) I had made a mistake in the way that I was dealing with the cookies which falsely led me to believe that there was something else affecting how the server was determining which request was which.

This explains it:

http://en.wikipedia.org/wiki/HTTP_sessions:_Client_vs_Server_side

For anyone looking for how to do multiple cURL transfers look at the example at the bottom of this page of the php manual:

http://uk2.php.net/manual/en/function.curl-multi-exec.php

[Atli]

I'm glad you solved your problem and thank you for sharing the solution with us.